In 1996, Brian Shneiderman made a plea to designers that “information exploration should be a joyous experience”  [Shneiderman 1996, 336]. Noting the rising demand for information-seeking online services in the days before Google’s ubiquity, Shneiderman argued that as online archives expanded that the “visual presentation” as well as “direct-manipulation” of information would be critical for users to be able to deliberately sort through the slew of search results without experiencing cognitive overload. The “visual presentation” of different types of data would therefore be critical to managing user experience, and Shneiderman outlined several data types in order to aide how designers conceptualized the user’s relationship between data and directly manipulating this data. His taxonomy included data that he considered 1-dimensional (e.g., textual data), 2-dimensional (e.g. maps), 3-dimensional (e.g., “real-world” objects like the human body), temporal (e.g., timelines), multi-dimensional (e.g., combination of multiple data types), treed (e.g., hierarchical tree-structured data), and networked (e.g., network visualizations that show relationships between data)[3] [Shneiderman 1996, 337]. Shneiderman (1996) insisted that accommodating several data types simultaneously would be necessary to design information search interfaces, and further suggested that the goals and interests of users conducting searches would likely affect the use of visualization/manipulation tools such as filtering [Shneiderman 1996, 339].

In the last two decades, much of Shneiderman’s taxonomy is highly visible in various iterations of web-based browsing applications and design literature, particularly dynamic queries which are frequently represented as faceted searching. Fagan (2010) noted that contemporary instantiations of faceted search tools aimed to represent data types and provide tools for users to narrow information search results. A typical example might start with a single search term, and then facet by the type or resource, the date range, or even availability of digitized copies (see Figure 2 for an example of faceted search). After examining numerous studies that empirically investigated faceted searching, Fagan (2010) reported on a bevy of benefits to faceted search, perhaps most notably that faceted search may contribute to users finding a greater amount of relevant results as well as creating mental navigation structures and specifically benefits user’s ability to search within specific time-frames or by specific authors [Fagan 2010, 62–63].

Xiao et al. (2009) demonstrated that successfully navigating and interacting with an online search tool, such as Yahoo.com, on a mobile device stresses the importance of limiting the available visual real estate by stacking blocks of space horizontally, and limiting the amount of text simultaneously visible on-screen. Faceted search interfaces, as shown in Figure 2, represent significant obstacles to such challenges from a visual perspective. While designers and scholars have explored mobile-specific interfaces for faceted search, the challenges outlined by [Xiao et al. 2009] are pervasive. For example, [Schneider, Scherp, and Hunz 2013] finds that certain modifications, such as presenting facets in a list or a grid, might significantly improve usability but not necessarily user enjoyability of the search experience. Incorporating lessons from [Shneiderman 1996] and [Xiao et al. 2009], [Kleinen, Scherp, and Staab 2014] designed a custom mobile application that combined text and visuals to conduct faceted search tasks, so that a user might begin by entering or selecting text based on prior knowledge, and then selecting the next facet through a map with geographically specific “points of interest.” In this way, at each point of faceting the user is presented with a different dimensionality of the interface so that they may go from map, to list, to detailed visual, to detailed text. Such an example of faceted searching that incorporates multidimensional data, Kleinen et al. argue, may yield more enjoyable search experiences.

Regardless of the device, it is apparent that online activities like web-browsing and searching are unique performances that are configured with the apparatus that user and site interface through (as well as how that interface is reconfigured with the apparatus). These usability studies further reveal representations and visualizations of dimensions of faceted searching may inhibit or encourage alternate forms of exploration. In particular, an undercurrent of the studies reported in [Fagan 2010] as well as [Schneider, Scherp, and Hunz 2013] is that faceted search interfaces are intended to guide users to specific information based on a priori knowledge or data type dimensions. Kleinen et al. (2014) caution that as online data types, sources, devices, and interfaces increase in size and complexity faceted search interfaces will need to accommodate an “a-priori unknown number of data categories and data instances”  [Kleinen, Scherp, and Staab 2014, 57, emphasis ours]. Therefore, the concern is not simply a matter of device- and screen-size, but rather, a concern of the relationship between what the user wants — in terms of inquiry and affect — and what the archive wants — in terms of liveness and data dimensionality.

Our study is not the first within the digital humanities to draw upon the idea of usability and to consider user experience in ways similar to the studies described in the previous section. For example, [Gibbs and Owens 2012] incorporated a series of virtual panels, much like a focus group, to discuss online tools with historians, and [Warwick 2012] incorporated a usability study to observe novice and expert users of an online archeological database. While both of these studies used methods familiar to UX studies, an emphasis on users with specific background knowledge, expertise, and scholarly practices (i.e., historians or archeologists) suggests greater consideration for use and users in addition to usability.

Since the DH ethos values both the evolution of traditional research practices from humanities scholars as well as knowledge production in humanities classrooms, it is reasonable that DH scholars seeking better understanding and improved designs of digital media are concerned with a wider context than that of the interface. Burkick & Willis (2011) have argued that 21st-century literacy practices are design practices, which transforms digital scholarship and knowledge production into a process of thinking through designing. This user-oriented approach “takes into account not only how an application is used but also the kinds of subject positions, world-views, and models it affords”  [Burdick and Willis 2011, 550]. User-oriented designs would account for Gibbs & Owens’s (2012) recommendation that DH tools be designed more holistically for the needs of humanities scholars and students, specifically tools that are “more transitory than revolutionary”  [Gibbs and Owens 2012, ¶35].

Warrick (2012) has additionally argued that studying users in context is key to understanding how scholars and students of varying expertise use digital tools and media as part of their broader scholarly practices. This encourages use in addition to usability, and attention to what users (i.e., scholars) want and how they might engage in practice with digital tools — regardless of usability [Kemman and Kleppe 2014]. By focusing on the user in context, Kemman & Kleppe (2014) argue that observations will inform design, which may then result in a feedback loop between design and use. This user-centered design process requires iterative attention to use in context, such as through extended and longitudinal usability studies [Warwick 2012].

While this surely requires an extended discussion on its own, it is worth noting that the tension between a user and use for DH scholars and usability from UX approaches are not mutually exclusive, and, as suggested by [Warwick 2012], the two can inform one another. Software and methods intended for usability studies, which may include screen capture, log-file data, and even front-facing cameras to capture facial expressions, may well yield rich and well-rounded understandings of use as well as usability. In our study of BigDIVA, we therefore sought to further understand how new ways of visualizing and encountering data types may serve as a “transitory” practice for humanities scholars, and our use of a usability study may be seen as an intervention to elicit not only potential understandings of how users initially use BigDIVA, but also how they may perceive its potential use in their own practices. In the following section we provide details of BigDIVA’s configuration.

This study was conducted entirely within a labspace designed specifically for usability studies in the library of a Mid-Atlantic U.S.-based university. This lab was outfitted with a desktop computer running Morae software[4], designed for computer-based usability testing, and a Logitech C920 webcam with stereo microphones. Morae was able to simultaneously record on-screen video (including highlighting of cursor actions), audio-video footage from the webcam, and log-file data from the keyboard, mouse, and onscreen applications. The resulting recording (see Figure 3 below) could then be viewed and analyzed with Morae software.

Participants were guided through the study tasks through a Google Forms file that provided instructions for, as well as logged performance of, each task. First, participants were asked to complete a brief pre-study questionnaire to self-report expertise with generic information search, as well as fields of expertise. Next, participants were guided through three search tasks that required using Google to find a famous work by William Blake, a relief etching by William Blake, and an etching of Blake’s “Milton: a Poem.” These three tasks in particular were intended for participants to find increasingly specialized information and to use Google’s faceting feature for searching via dimensions such as Image, Web, or Video. Following these three tasks, participants were then required to use BigDIVA to identify collections with large results for “William Blake,” an individual result from this search, meta-data from a specific result for Blake’s “Night Thoughts,” and an example search result that contains the text string “chastity.” It should be noted that all of the search tasks are listed in the Appendix, and that the BigDIVA search tasks were intentionally longer simply to observe users interacting with BigDIVA for longer. Following the search tasks, the first author engaged the participants in a brief follow-up interview to solicit self-reported experiences with BigDIVA.

Participants (N = 13) were recruited from networks connected to a Mid-Atlantic university in order to engage participants that were likely familiar with web-based archival search tools beyond Google. All participants were currently enrolled in graduate coursework or had completed graduate coursework, and were from a variety of fields and academic positions including tenured professors, graduate teaching/research assistants, library & information scientists, industry workers, and one individual currently between jobs. A small group of participants (N = 5) piloted the study design during the spring of 2015, which led to subsequent adjustments to the study’s task design (i.e., particular phrasing of task instructions, and streamlining of the Google Forms interface). The remaining participants (N = 8) engaged in the study between late 2015 and early 2016, and will make up the primary participant pool to be discussed throughout the remainder of this paper.

Analysis was conducted using Morae Manager to code and extract data, and visualizations were produced in R Studio [RStudio Team 2015]. Participant usability sessions were analyzed according to three dimensions: Time On Tasks (TOT), Actions Within Tasks (AWT), and Log-File data (primarily Mouse and cursor actions). TOT data was primarily continuous, i.e., based upon duration of time a participant spent on each individual task, whereas AWT and Log File data were discrete, i.e., a specific type of action that occurred at a specific moment of time. Table 1 below outlines each specific dimension along with its corresponding codes.

Participants spent a total of 30.51 minutes on the three Google tasks, and 144.78 minutes on the seven BigDIVA tasks (see Table 2 for breakdown of TOT). When looking at TOT for each individual task, as shown in Figure 4 below, we can see that, while in many cases the TOT for BigDIVA tasks varied within the same range as the Google tasks, the BigDIVA tasks witnessed greater variability in terms of TOT. In particular, the two BigDIVA tasks that required the most specific information (Task 2_3a and Task2_4) generally took the longest. A paired samples t-test comparing participants’ mean TOT for Google and BigDIVA tasks demonstrated that mean TOT for BigDIVA was significantly longer ( t (6) = -2.85, p = .029).

Since all participants self-assessed themselves as “Very Good” or “Excellent” Google-users, it is certainly reasonable that they would need to spend less time on the Google search tasks. Directly comparing aggregate TOT for Google and BigDIVA tasks does not account for time necessary to adapt to an unfamiliar search tool; therefore, understanding the value of aggregating TOT across tasks and search platforms requires more nuance. For example, based on comparison of TOT for BigDIVA tasks only, it becomes apparent that following the first BigDIVA task (Task 2_1a), some participants appeared to become more time-efficient (particularly for Task 2_2b and 2_3b), which suggests that they overcame a learning curve and adjusted to aspects of networked browsing. Further, some participants that exhibited similar difficulties with search tasks (particularly Task 2_3a-b and 2_4) spent more time on each task. For example, participant B06 and B08 were both unable to complete these three tasks for similar reasons (as will be discussed below), but B06 spent much longer (5.66 minutes on all three tasks) than B08 (3.74 minutes on all three tasks). This suggests that successful or unsuccessful adaptation to BigDIVA is not necessarily dependent on how much time a user spends on task.

In order to look deeper than TOT data, the AWT data is able to provide more fine-grained comparisons of what participants did for each task with the time allotted. Figure 5 provides a timeline for each participant colored for any occurrence of an AWT action type. This provides a window into the interplay and sequencing between multiple action types over the course of the entire study, and shows that a typical action sequence for Google tasks involved entering a search query, adjusting for search dimension (i.e., Image, News, Scholarly), opening a result (maybe two), and then eventually entering an answer for the task. For BigDIVA, these sequences were more complex as participants were required to explore and manipulate interface elements they were previously unfamiliar with. Interestingly, at first glance of BigDIVA’s force-directed network, all participants found the search bar and entered a query in a matter of seconds. However, once a search query was entered, participants variably played with the network nodes, switched dimensions (e.g., Genre or Format), enlarged the screens, faceted, etc. In other words, the sequence from search query to facet to search result was broken. Entering or modifying the search query makes up a much smaller portion of AWT actions for BigDIVA tasks (with the exception of Tasks 2_3a and 2_4), while experimenting and playing with the faceting features and the nodes were much more frequent. The greater frequency of these action types suggests that participants’ unfamiliarity with BigDIVA — specifically the aspects of networked browsing — are likely indicative of a sense of fascination and playfulness, or even confusion and frustration, with these features in order to understand their function. The novelty of these features may have encouraged participants to spend time playing with these features in order to better understand them, which may explain how some participants spent less time for later tasks.

Log-File actions additionally present a complex picture. While Mouse and cursor actions dominate input actions during the Google tasks, the first four BigDIVA tasks witnessed a steady decrease in mouse action as participants engaged with the keyboard and various other AWT action types more frequently. While Tasks 2_3a-b appear to revert to Google-like behavior, the frequency of mouse and cursor actions glaringly decreases for the BigDIVA tasks, despite the fact that BigDIVA’s functionality is primarily dependent on the cursor (with the exception of the search bar).

Lastly, since not all participants were able to complete every search task, it is worth looking more closely at those participants and what their actions were. Furthermore, in order to have a comparison of these Non-Finisher participants to those that did, we made a subgroup of the two Non-Finishers (B06 and B08) and two Finishers (B01 and B04). This sub-group was determined by developing a rudimentary scoring system for each task in which participants received 1 point for completing the task as designed, 0.5 points for completing the task not as designed, and 0 points for not completing the task at all. After rating all participants, B06 and B08 scored the lowest, while B01 and B04 scored the highest.

In terms of AWT action types, Figure 8 demonstrates both groups overall were not strikingly different. Besides the action type QUIT, both the Finishers and Non-Finishers engaged in similar distribution of AWTs as the Finishers. The primary distinction is for the action type DIMENSIONSWITCH, as the Finishers switched dimensions in BigDIVA more than half as much as the Non-Finishers. After looking more closely at when and how these participants switched dimensions, it was observed that both groups had developed distinct understandings of this feature’s function: the Finishers would switch dimensions and then enter a search query, whereas the Non-Finishers would enter a search query and then switch dimensions. While a seemingly inconsequential distinction in Google, this sequence of action in BigDIVA is the difference between filtering results based on a search query and resetting the force-directed network visualization of the archive to another dimension. Thus, when the Non-Finishers entered a search query, and then switched dimensions, they effectively negated their query and reset the visualization to view all archive content — albeit in a different dimension. In other words, the Non-Finishers were very likely unable to finish all of the BigDIVA tasks because they had failed to adjust to this specific feature of BigDIVA’s functionality, and instead appeared to transfer their understanding of this functionality directly from Google.

Furthermore, when comparing TOT data for the Finishers and Non-Finishers, the Non-Finishers appear to take more time in general to adjust to BigDIVA search tasks. Indeed, by Task 2_3a, the Non-Finishers demonstrated so much difficulty in using BigDIVA in order to complete each search task that they appeared more inclined to abandon the tasks rather than adjust their understanding of the functionality; whereas the Finishers — who had done a better job of understanding the functionality — were willing to spend more time in order to complete the tasks.

While audio-video data captured from the webcam does coordinate on-screen actions with affective reaction as gauged by facial expression, it is beyond the scope of this current study to adequately or appropriately surmise how using BigDIVA resulted in participants’ affective reactions. After all, since BigDIVA was new to the participants while Google was used daily, some manner of novelty effect would be expected. Further, since BigDIVA did require some manner of learning curve (and at least two participants experienced delays due to server interruptions), such analysis would likely have been biased towards affects such as frustration or confusion [D’Mello and Graesser 2012]. Nevertheless, at first glance of BigDIVA’s network visualization all users did display unique momentary reactions. For example, B04 noticeably smiled, B01 appeared surprised, B02 exclaimed quietly to himself “What,” and B08’s brow furrowed. In other words, the novelty of networked browsing appeared to induce affectations — perhaps even “joyous” in some cases — that in and of itself may offer participants hope of a new manner of searching.

Indeed, based on the self-report questionnaire following the search tasks, participants were largely in agreement about the potentialities of networked browsing. While many elaborated on early frustrations with the new functionalities of the search tools, or the initial learning curve, all participants remarked about BigDIVA’s ability to establish an understanding of the relationship between the collections and their individual archive sources. In particular, three common themes were apparent from the participants’ self-reports: that the novelty of BigDIVA’s networked browsing was either “fun” or “interesting,” that BigDIVA could be most useful for specialized archives or use in scholarly settings, and that the networked visualizations may be pedagogically valuable for demonstrating relationships between scholarly sources (see Table 3 below).

Another unique theme expressed by participants related to the nature of web-based search queries altogether. In particular, while participants noted that after a brief adjustment period they were able to successfully search and find specific information using BigDIVA, that they would still use a more typical visualization such as Google — even if networked browsing were an option with Google. Participants justified this preference with the fact that a text-based list of results still might be simpler and quicker if the user already knows what they are looking for; whereas networked browsing would be preferable if the user was not certain what they were looking for. In other words, networked browsing would be more suitable for exploring the contents of an archive without a predisposition for specific results.

Of course, as implied already, our study was not without its limitations. In addition to a small sample size, we wish to emphasize that the comparative design of our study is one such limitation, because, as participants self-reported, BigDIVA prioritized a different kind of browsing than Google. Finding information in BigDIVA did not require a scavenger hunt amongst a hierarchical text-based list that required prior knowledge encoded in the form of a query. Instead, BigDIVA made information available without a search query, thus begging participants to visually untangle and play with the networked catalog. Our study design admittedly prioritized specific search tasks in order to target use of specific features as well as to allow easy comparison to search tasks in Google. Open-ended tasks or longitudinal observation, per [Warwick 2012], would therefore be well-suited alternatives.

At the same time, comparative methods are still valuable for design and experiential considerations of these interfaces. After all, if networked browsing indeed affords unique performative understandings and meaning-making experiences compared to other operationalizations of faceted search functions, then more detailed comparative studies could reveal the distinctions in those processes, particularly comparing similar scholarly faceted search tools such as a university library database. As suggested above, such a comparative study would do well to not solely rely on input-based behavioral measures that privilege the machine components of the interface, and instead privilege the human components in the interface by observing affect, metacognitive processes, as well as self-report measures to emphasize users’ perceptions of their experiences. In particular, such methods could potentially parse out the distinction between users’ understanding of search functionality from archive architecture, as well as how, in their own terms, they may find joy in their browsing experiences.