Classification is a basic, integral and historically significant human function. Defined by Golder and Huberman as an act through which meaning emerges , the practice of classification represents one of the primary intellectual foundations of library and information sciences. Useful classification systems both accurately reflect the contents of a particular collection and allow for the effective and efficient retrieval of items or information. Given the subjective nature of human classification, examining meaning has no universal procedure. From the birth of the library science field until the late 1800s, during which the first edition of Dewey Decimal Classification was published in 1876, followed quickly by the Cutter Expansive Classification System and the introduction of Library of Congress Subject Headings , all libraries and centers of information each used an independent, unstandardized systems of organization, otherwise known as local classification.

With the rise of such classifications as Dewey, Cutter and LC Classification, local classification was mostly retired in the late 19th century in favor of a united system that would allow for understanding across all libraries. This was, of course, until the rise of Internet tagging-based platforms such as Flickr, Twitter, Tumblr, and Delicious presented a challenge to these standardized, centralized systems. Currently information professionals are facing an unprecedented amount of unstructured classification. Classification systems generated by such tagging-based platforms are referred to as folksonomy, or a type of classification system for online content, created by an individual user who tags information with freely chosen keywords; also, the cooperation of a group of people to create such a classification system .

While folksonomies represented an increased diversity of classification, they were perceived mostly as sources of entertainment and documentations of casual colloquialisms rather than a formal system of documentation. However, in 2006 Golder and Huberman conducted a study of folksonomic data generated by users of the website Delicious demonstrating that user-provided tags not only formed in coherent groups, but also accurately described the basic elements of items that were tagged, such as the Who, What, Where, When, Why, and How of items, proving their ability to supplement formal records.

Golder and Huberman’s analysis also revealed two highly subjective categories that may diminish the potential value of crowdsourced metadata: Self Reference (i.e., any tag including the word my, such as mydocuments) and Task Organizing (i.e., tags denoting actions such as readlater or sendtomom). In examining the overall accuracy and reliability of tags, Golder and Huberman concluded: Because stable patterns emerge in tag proportions, minority opinions can coexist alongside extremely popular ones without disrupting the nearly stable consensus choices made by many users .

This research was expanded upon in 2007, when Noll and Meinel compared tags from websites Delicious and Google against author-created metadata and found that the former provided a more accurate representation of items’ overall content. Bischoff et al. [2009] also examined folksonomic metadata within the context of the music industry and found that tags submitted by users at the website Last.fm were comparable with professional metadata being produced at AllMusic.com. Syn and Spring [2009] examined folksonomic classifications within the domain of academic publishing, and also found authoritative metadata to be lacking when compared with its user-generated counterparts. As stated by Noll and Mienel, tags provide additional information which is not directly contained within a document. We therefore argue that integrating tagging information can help to improve or augment document classification and retrieval techniques and is worth further research. . Together, these studies indicate that folksonomies can easily and usefully be stored alongside classical, controlled vocabularies.

While some may have reservations regarding the mixing of folksonomic and controlled vocabularies, these two classification systems need not be viewed as mutually exclusive. Both systems have inherent advantages as well as potential flaws. Controlled vocabularies are reliable and logically structured, but can be somewhat inaccessible to the casual user . Furthermore, they can be time-consuming and expensive to produce and maintain. For example, if one’s local library employs two catalogers at an average salary of $58,960/year , it sets aside $117,920 for implementing controlled vocabularies on a limited number of new collection items alone. In contrast, folksonomies represent a relatively quicker and more cost-effective alternative , with greater public appeal and accessibility, as evidenced by their overwhelming usage on social media. Mai noted: Folksonomies have come about in part in reaction to the perception that many classificatory structures represent an outdated worldview, in part from the belief that since there is no single purpose, goal or activity that unifies the universe of knowledge . As many users have become accustomed to the level of service and interaction styles offered by current popular search engines, traditional searches are unlikely to be very successful and are becoming less frequent as patrons’ behavior is shaped by keyword search engines . As researcher Heather Hedden points out, if an artifact [in today's culture] if it cannot be found, it may as well not exist . However, their lack of centralization renders folksonomies prone to issues of potential data contamination, such as an unorganized, unstructured plurality of subjects and the likelihood of data duplication between users. Regardless, folksonomies and traditional systems of organizations may be used in tandem to address the shortcomings of their respective features, allowing for a more diverse and organized form of classification.

Our research aimed to provide new empirical evidence supporting the value of folksonomies by: (1) directly testing the benefits of adding user-generated folksonomic metadata to a search index and (2) comparing the range and accuracy of tags produced by library and information science professionals and non-professional users. The three main questions guiding this work were:

The present research employed a hybrid form of usability testing utilizing the Metadata Games platform [http://www.metadatagames.org], an online, free and open-source suite of games supported by the National Endowment for the Humanities and developed by Dartmouth College’s Tiltfactor Laboratory. The Metadata Games Project, launched in January 2014, aims to use games to help augment digital records by collecting metadata on image, audio, and film/video artifacts through gameplay . Current Metadata Games media content partners include Dartmouth College’s Rauner Special Collections Library, the British Library, the Boston Public Library, the Sterling and Francine Clark Art Institute Library, UCLA, and Clemson University’s Open Parks Network. Inspired by other successful crowdsourcing efforts, the designers endeavored to create a diverse suite of games that could enable the public to engage with cultural heritage institutions and their digital collections, invite them to contribute knowledge to those collections, and set the stage for users to create and discover new connections among material within and across collections. The Metadata Games platform currently includes a palette of games that cater to a variety of player interests, including both single-player and multi-player games, competitive and cooperative games, and real-time and turn-based games, available for browsers and/or mobile devices. Despite their variety, all the games in the suite are united by a common purpose: to allow players to access media items from a number of cultural heritage institutions’ collections and provide them with the opportunity to contribute new metadata, in the form of single-word or short-phrase tags, within the context of an immersive, enjoyable game experience. The end result is that institutions benefit from increased engagement from a variety of users and acquire a wider array of data about their media collections.

This research represents a collaborative project between the first author, who chose to use the Metadata Games platform as the focus for an independent study project at the Simmons College of Library and Information Sciences, and the co-authors from Dartmouth College’s Tiltfactor Laboratory. To be clear, the goal of the reported study was not to provide a validation of the Metadata Games platform, but rather to study the value of folksonomic metadata more generally; that is, the focus of this research was on the data itself, and the tool employed was intended to be largely incidental and peripheral to the study’s aims. At the time, the Metadata Games project was one of the few open-source metadata gathering tools available for cultural heritage institutions. Other noteworthy examples include the steve.museum, one of the first projects to crowdsource metadata in the cultural heritage domain, which created an online web interface for registered users to submit tags to a selection of works from participating museums [https://code.google.com/p/steve-museum/]. In addition, in 2008, the Library of Congress (LoC) used the photo-sharing service Flickr to gather comments and tags on 3,000 images . From this project, the LoC teamed with Flickr to form the Flickr Commons, a communal page for other cultural heritage institutions with image collections [https://www.flickr.com/commons]. Thus, while the reported study results are specific to datasets gathered by the Metadata Games platform, the conclusions drawn from this study are intended to be generalizable to any organization currently making use of services such as CrowdAsk, LibraryThing or Scripto or considering a crowdsourced metadata project.

According to Nielsen, the number of participants needed for a usability test to be considered statistically relevant is five ; however, because of the additional collection of quantitative data in our hybrid study, sixteen individuals (eight men and eight women; six of whom were aged 18-24, eight aged 25-44, one aged 45-60, and one over 60 years of age) were recruited to participate individually in 30-40 minute sessions. In order to discern any differential patterns of results between librarians and non-librarians, and to separate out any potential advantage that users in the field of Library and Information Sciences might have with content search and metadata generation, a mixed sample (with nine participants recruited from LIS-related fields and seven from non-LIS-related fields) was used for the study.

The study was divided into two main tasks. In the first task, participants were presented with physical facsimiles of five images from the Leslie Jones Collection of the Boston Public Library and instructed to retrieve these items using the Metadata Games search platform. The images presented to participants were divided into the following categories: Places, People (Recognizable), People (Unidentified or Unrecognizable), Events/Actions, Miscellaneous Formats (Posters, Drawings, Manuscripts etc.), as seen in Figures 1-5 below. Upon being given each physical facsimile, participants were timed from the moment they entered their first search term until the moment they clicked on the correct digital item retrieved from the Metadata Games search platform. This practice was adapted to reflect the digital items that would most commonly exist in a typical humanities-based collection, (i.e., photographs, manuscripts, postcards, glass plate negatives and other miscellanea). This design mirrored the common everyday occurrence of patrons attempting to retrieve a specific media item that they have in mind when using a library search index. According to a 2013 PEW Research Study, 82% of people that used the library in the last 12 months did so looking for a specific book, DVD or other resource .

For this image search component of the study, participants were randomly assigned to one of two search index conditions: one with access to controlled vocabularies and folksonomic metadata (i.e., the folksonomy condition) and the other with restricted access only to controlled vocabularies (i.e., the controlled vocabulary condition) [See Figure 6 for a schematic representation of the study design]. Searches were conducted using two different, customized versions of the search index on the Metadata Games website [play.metadatagames.org/search]. The folksonomic metadata was generated by previous users of the Metadata Games platform, whereas the controlled vocabularies attached to the items were generated by Boston Public Library staff. The process of inputting the controlled vocabularies into both versions of the search index required some reformatting. For example, because the version of the search platform used in the study did not allow for special characters such as the dash - or the comma ,, terms such as Boston Red Sox (baseball team) were imported as two individual tags: Boston Red Sox and baseball team.

Additionally, the system returned exact matches only, which meant that if a participant searched for sailboat and the only term present in the system was sailboats, the search would be unsuccessful. This aspect of the study design was necessitated by the technical specifications and functionality of the version of the Metadata Games search index used in the study, rather than a strategic methodological choice. The frequency of preventable exact match retrieval failures is discussed below in Section 4.1.

To further illustrate the differences between the two search index conditions, consider the case of a participant in each condition attempting to retrieve image 3 (see Figure 3 above). In the controlled vocabulary condition, the only search terms that would yield a successful retrieval were: harbors, sailboats, marblehead harbor and glass negatives. In contrast, in the folksonomy condition, a participant would successfully retrieve this item by entering any of the following search terms: harbor, sailboats, water, woman, sail boats, porch, scenic, view, sail, sun, summer, marblehead harbor, boats, dock, harbors, veranda, balcony, girl looking at boats, marina, glass negatives, sailboats on water and yacht.

Immediately following the image search task, participants were instructed to play three different single-player tagging games from the Metadata Games suite: , , and . In the free-tagging game (Figure 7), users are able to input as many tags as they wish for four separate images. (Figure 8), uses the same game mechanic as , but utilizes a more minimalist user interface and presents a more robust image to players. Finally, in (Figure 9), a novice robot asks users to help it learn new words by tagging images. The game presents one image to users and gives them two minutes to input tags, with the constraint that they may only enter one tag for each given word length (i.e., one four-letter word, one five-letter word, and so on). In playing a single session of each game, participants in the present study tagged the same nine images (four images each in and and one in ). Tags from these three games were compiled and compared against the traditional metadata provided by staff from the Boston Public Library.

As of now, there remains debate about the comparative value of traditional and folksonomic metadata as organizational systems for today’s information needs. Nonetheless, there is growing recognition of the fact that folksonomies offer libraries with an ideal return-on-investment scenario with minimum cost (much of which can be off-set by digital humanities grants), maximum output of data , as well as the chance to increase community engagement with their patrons. As the findings of the present study demonstrate, folksonomic metadata, when used in tandem with traditional metadata, increases findability, corrects preventable search failures, and is by and large accurate. Furthermore, the data suggest that given the same tagging conditions, librarians and non-librarians produce a surprisingly similar distribution of useful metadata. Collectively, these findings point to the potential to change the way we search for and organize our most treasured media.

The research team would like to thank all study participants, Tom Blake, Linda Gallant, the Boston Public Library, The Digital Public Library of America, Mary Wilkins Jordan, Jeremy Guillette, and the National Endowment for the Humanities.