Without being aware of the underlying technologies, our behavior is frequently recorded, assessed, and fed back to us via today's Web interfaces. Information systems are now far more advanced in mirroring preferences, curiosities, and yes, knowledge, than they were, say, five years ago. The example which is familiar to us all is Amazon.com.[1] The world's largest bookstore makes clever use of the newest collaborative filtering technology.[2] From a complex database of recorded consumer behavior persuasive, personalized web pages are generated. The idea behind the process is that statistically combining traces of user behavior will yield knowledge about user interests.[3] If you have ever ordered a book that was suggested to you by Amazon, that forms the best proof of the success of the collaborative filtering approach.

Looking at the basic idea behind collaborative filtering systems, one may wonder whether this concept is applicable to the humanities, where the expression of value judgments (as against stating incontrovertible facts) is of such importance. Building a collaborative filtering tool involves first establishing what are supposed to be relevant data, and then modeling the way these data may be recorded and organized, in order to turn data into information, and perhaps collaborative knowledge.[4]

This paper discusses a field test in collaboratively establishing visual similarities, a topic of concern in art history. Art.Similarities is the experimental application used to sample visual similarity judgments.[5] The experiment could be relevant for other disciplines than the history of art, both inside and outside the humanities.

The bulk of research in distributed cognition is text-based — as we shall see in the next paragraph. For this type of research in information science and cognitive psychology an example in the domain of visual cognition may be advantageous. In particular crossing off the intermediary function of verbal language, may encourage the development of scenarios for combining transactional data (partial observations of visual artifacts) to construct emergent knowledge (holistic representations). This is why I believe that in its consequences this field test could support cross-cultural comparisons in visual cognition.

I also expect that the project may shine a new light on the relation between the cultural interpretation of images and words on one side, and the relation between such cultural interpretations and objective image data, e.g. as produced by content-based image retrieval technologies [Eakins & Graham 1999], on the other side. In other words: it could help tracing the line between objective visual facts (as captured for instance in a color histogram) and more interpretive “truths”.

In the practical sphere, because visual attention becomes partly measurable and comparable, the project may contribute to the development of educational applications, in which students learn to reflect on culture as a determinant of visual cognition. And because the interconnectedness of base distinctions and more evolved cultural knowledge is worked out computationally, it becomes conceivable that by means of technology subjects will be offered the tools to vary the compositional elements of a visual configuration, just as when you add or remove query strings based on intermediate search results, when searching large volumes of texts.

I presume that the latent possibilities of the Art.Similarities experiment will only come to the fore when we do not discuss it in isolation. Therefore consider in advance that interesting alternatives may evolve when varying such dimensions as:

Such dimensions are important in assessing the usefulness of the experiment. Just to get an idea of the kind of problem addressed here, consider these two works of art, dating from the early 1930s:

Both images show a striking overall similarity. Experienced observers seeing these works of art, may draw conclusions about artistic influences. And indeed, artists are observers by profession, adopting typical ways of expressing visual thoughts in new visual configurations. The history of art has documented innumerable instances of both explicit and implicit borrowing of formal traits and configurations.[6] This is partly motivated by the common assumption that visual phenomena in cultural artifacts are indices of material, personal and social conditions during the time of creation.

Establishing visual similarities is part of the groundwork in the field of art history, but making these similarity observations verifiable is precisely where the conventional systems of scholarly communication fail, since neither do we have an effective way of discussing visual similarities, nor do we have the means to trace observed similarities in large collections of images.[7]

In recent years we have seen several attempts at constructing frameworks for the study of collaborative knowledge. The subject was variously denoted with terms such as “distributed cognition”, “social cognition”, “situated cognition”, or “group cognition”. The urge to develop theory in this field has increased dramatically since the waves of hypertext enthusiasm in the late 1980s and early 1990s, the subsequent rise of the world wide web and later semantic web, and the popularization, recently, of social software, of which the collaborative filtering systems mentioned earlier are an omnipresent example. In theories of collaborative knowledge the psychology of Lev Vygotsky, who may have been one of the first to envision a socially constructed mind, is frequently referred to. His Mind in Society dates back to 1930 [Vygotsky 1976]. Gavriel Salomon stressed the importance of the complementary concepts of “shared cognition” (knowledge of the world which is continuously established through live interactions among individuals) and “off-loaded cognition” (recording and processing cognitive facts and functions to realia, such as concept maps) [Salomon 1996]. In 1995 Edwin Hutchins published another influential book, Cognition in the Wild, where the author uses the metaphor of ship navigation to pinpoint the cultural nature of cognition: no single individual from the ship's crew is capable of managing all the complex operations that are necessary to sail [Hutchins 1995].

Important building blocks of frameworks for collaborative knowledge are: thought processes, as distributed amongst a group of individuals, representations of these thought processes, as captured in external realia, and mediating processes (i.e. computer systems) that are capable of coordinating both internal and external representations and lifting the newly generated forms of knowledge above the level of consciousness of the individual.

A recent attempt to theorize in this field is Gerry Stahl’s book Group Cognition: Computer Support for Building Collaborative Knowledge [Stahl 2006]. On the basis of a series of experiments, in which the promise of computer supported knowledge negotiation has been tested, Stahl analyses the mechanisms of collaborative knowledge building and ends by presenting a tentative theory of group cognition. In Stahl’s model the sphere of personal understanding is opposed to and merges with a cycle of social knowledge building. Individuals articulate – in public statements – personal perspectives, which in a process of argumentation and rationale are being assimilated with the perspectives of other individuals into collaborative knowledge. This collaborative knowledge is formalized and objectified in cultural artifacts that can be observed and known by individuals. Et cetera. Stahl’s model may help pointing out the stages of the process of knowledge construction and thus facilitate the design of specific forms of computer support [Stahl 2006, 207].

Typical for Stahl’s approach is his focus on verbal discourse. Although in some of his experiments schematic representations of e.g. floor plans or network architectures are crucial to the tasks of his subjects, in most of the experiments he uses chat-like conversations and threaded discussions to unfold his theory. But the kind of knowledge being researched here is visual knowledge constructed by visual means. As yet I have found no models of distributed cognition covering this particular domain. In my model the visual discernments (“personal focus”) of individuals are expressed (“off-loaded”) in visual images, where the computer records and mediates (“negotiation of perspectives”).

It is taken for granted in this study that the visual appearance of a cultural artifact, and thus its potential similarity to other artifacts, is resolvable into literally innumerable formal qualities. Two artifacts may share any number of formal qualities, and these are thought to somehow account for their similarity. Two (visually) identical (=maximally similar) artifacts share immense quantities of visual characteristics, whereas for two very dissimilar artifacts the number of shared visual traits is minimal. In between the extremes the rule would be that the more two cultural artifacts share a set of singular formal/visual attributes/traits, the more they are experienced as being similar to one another. [8] In other words: the actually perceived similarity is defined as the function of an unspecified number of trait-to-trait similarities.

In order to avoid endless image dissections, another assumption adopted here is that the overall visual appearance of a cultural artifact in fact approximates to the weighted sum of relatively few, but distinctive formal/visual attributes/traits. So I assume that culture is a decisive factor in subjects focusing on specific image characteristics: the threat of an infinite number of physical features is being balanced by cultural determinants.[9]

Observers have a number of options to denote a visual concept [Nauta 2001]. In most of these options abstract symbols (words, numbers, etc.) signify visual similarities. Two options, however, concern iconic references. One focuses on the simultaneous display of multiple instances of similar artifacts along one or several specific attribute(s). For instance, Santini, Gupta and Jain discuss an experimental interface in which the user is allowed “to manipulate not only the individual images, but also the relation between them”  [Santini et al. 2001]. An example of fairly successful content-based image retrieval is the system developed by D.P. Huijsmans et al. for the Leiden 19th-Century Portrait Database (LCPD), a database of Dutch carte-de-visite studio portraits (1860-1914). One of the options for consulting the LCPD is a so-called “relevance feedback” interface. After each retrieval action the user is presented a table of results. Feedback is given by just clicking the photographs that are close to what the user was looking for, whereupon the system — using pre-calculated image characteristics — offers a new set of images, ideally closer matching the user's preferences. Et cetera. [Huijsmans & Smeulders 1999] In the other type of iconic denotation, the one elaborated here, image similarities are indicated by simple images exemplifying a restricted set of formal attributes.[10] These simple images are thus conceived as visual denotats.[11] So where Santini et al. avoid immediate deconstruction of holistic appearances, this paper will explicitly analyze similarities in considering iconic denotations.[12] In this way, problematic verbal denomination issues will be evaded. Present-day computers are perfectly suited to create systems for communicating pronouncements on matters of visual similarities through instant presentation of instances.[13]

Another assumption concerning the similarity of artifacts has to do with the quantity of common visual traits. Where two visual artifacts share a substantial number of different visual traits, we say that these artifacts manifest multi-trait similarity. The notion of “style” may be akin to this concept. In the vocabulary of this article style could be defined as multi-trait similarity occurring across multiple images.

Apart from considering the number of visual traits two artifacts have in common, it is important to recognize that the relative weight of these visual traits is of interest: the more observers indicate they have discerned one particular trait, the more relevant this trait will be in any function defining the visual similarity of the artifact to any other artifact. Of course it should be kept in mind that relevance is dependent on the group these observers belong to. We are not considering facts. We need individual human observers to assess the visual qualities of artifacts. And assessments may either apply to the entire surface of a visual artifact or to only a segment of that surface. Once these assessments are recorded they can be analyzed. And the more individuals are involved in assessing visual qualities, the greater the cultural relevance of these assessments will be. Patterns in individual assessments, calculated for social groups, may reveal “high order” notions (such as — perhaps — stylistic notions). That an individual observer giving individual assessments need not be conscious of these high order notions is a fascinating idea.

The following hypothesis aims at focusing the points made so far: The social formation of opinions concerning the visual similarity of cultural artifacts amongst a group of motivated observers can be modeled and recorded without the application of verbal descriptors, using information technology (web technology, databases, statistics), in such a way as to make real use of archived records for purposes like: getting to know broad classes of image attributes; retrieving related images; and learning the preferences and perceptual biases of user classes and individual users.

In order to test the hypothesis an experimental application — Art.Similarities — had to be developed, offering access to a modest quantity of digital images. The application interface had to be constructed in such a way that it would be relatively easy for observers to link visual icons from a restricted but varied set to the digital objects in the collection. Every transaction involving the database would have to be recorded to be able to quantitatively compare the visual discernments of different observers.

Here is an approximation of the processes the application should ideally support:

Since the experiment considers three object types — observers, artifacts, icons — and since these object types are systematically interrelated — observers assign icons to works of art — the analysis may extend into at least three fields of semantic relevance. The primary concern is with how artifacts will be typified or defined as being more or less similar to one another. But an analysis of patterns in observer behavior and the covariance of icon-image assignments can equally yield interesting results.

Before we turn to an inspection of the collected data, however, a few words must be added on the representativeness of the sample. The following analysis of data is explorative. In comparison with today’s large-scale commercial data warehouses the couple of thousand of records in my transaction database is very small. In the real world data mining techniques will not be used on data sets of this size. Add to this that data were not collected systematically: subjects were not asked to visually tag a fixed number of artifacts; they were free to start or stop tagging any time, and artifacts were presented at random. The only choice the subjects had was to skip an artifact and go on with the next object displayed. As a result the choice of artifacts and the distribution of visual tags across the objects represented in the database cannot be fully representative. Still it was decided not to normalize data routinely. The meaning of differing tagging frequencies could be relevant. Below the results of the Art.Similarities experiment will mostly be explored on the basis of raw data, especially where the similarity of artifacts is under consideration.

The setup of the test application is fairly simple. Nevertheless an analysis of database logs already requires substantial pre-processing and the application of sophisticated algorithms to extract measures of central tendency from the rapidly growing files. Just to compare one artifact and its related icons with all other artifacts, one needs to know both how many icons any pair of artifacts has in common, and the number of times any shared icon has been assigned to both artifacts. Below are some indications of what lies hidden in these database tables. My approach is a combination of searching for patterns in the data and doing empirical checks, in particular in the form of a visual assessment of artifacts and icons. Other methods of relating data and reality could be asking subjects (experts) to assess the results of manipulating transaction data, or even in some cases using content-based image retrieval technology to evaluate matches of artifacts.

As was contended in the Introduction, advocates of humanities computing stressed the fact that computational techniques may have the collateral benefit of indicating where in the domain of scholarly research “there are holes in your sketch, [and] where it breaks down”  [Unsworth 2001]. Add to this that in the humanities the process of fine tuning personal interests and cognitive schemata is paramount. There are few if any final answers as to meaning in the visual arts: “the process is often at least as important as the product, and the process itself produces new knowledge and understanding”  [Jörgensen 1999, 315]. That is exactly why I have good hopes that applications like Art.Similarities will be beneficial in art/art history education. More in general I think that the experiment has demonstrated that Web 2.0 technologies can offer new opportunities in humanities research.

From the analysis section we learned:

• that meaningful perspectives on visual artifacts can be pointed out by visual means;
• that useful similarity measures can be derived from multiple (simple) similarity expressions;
• that the composition of efficient visual vocabularies may benefit from visual tagging.

Stock-taking here was primarily aimed at indicating the essential possibilities of a visual labeling and harvesting paradigm for art historically relevant visual traits. But our field test also indicated some limitations of visual labeling. Color is problematic; the size of our displays units is (still) limiting; the power to differentiate between stylistically similar works of art is still indeterminate; we have no sharp idea of what an ideal set of visual denotats may look like; we don't know in what degree such a set could be universal. Nevertheless Art.Similarities also suggested possible directions for future research.

One drawback of collaborative filtering systems is their tendency not to cover the extreme cases [Chen & McCleod 2005], [Sierra 2007]. This issue also came up with the visual voting system proposed here. How can we direct the system to encode the significance of valuable exceptions to the rule, and how can we make it ignore cheating, trivialities and clichés? Some data mining protocol for judging automatically traced deviant votes (outliers) and blowing up or playing down their relative importance would be a valuable extension to Art.Similarities.[30]

Another extension would be the coordination of different solutions to the retrieval of information about same object types. This paper offered rough sketches of a method of classifying visual artifacts independently of other classifying options. But the interdependency of different classification systems is worth considering. Iconographical descriptions (“aboutness”) and descriptions of visual qualities (“offness”) belong to separate notational spheres [Panofsky 1970], [Bätschmann 1986], [Nauta 1993], [Shatford Layne 2002]. Still there is an apparent covariance of these description options: if two artifacts are depictions of a particular theme, chances are that both artifacts share a considerable number of visual characteristics. Such partial description parallels also exist for other types of metadata. For example: once an object has been classified as an “engraving” (qua technique), it is usually redundant to file the artifact as a “grayscale” picture, or as a picture “having linear qualities”. So we could fine tune the current approach by introducing restrictions: if images are in grayscale, don't offer colored icons, and vice versa. Eventually this may lead to the use of exchangeable icon sets, and the related question of how to select these sets of visual icons.

Conventional scholarly communication in art history amounts to a textual exchange of information in perhaps 95% of the cases. With that in mind the idea might come up that somehow the results of visual voting actions should be translated back into textual arguments. Now how will we ever get back to words? Since today's information technology is truly interactive in nature, conclusions based on coordinated visual discernments can be immediately communicated back to the communities involved in the process. One group of observers could be involved in visual voting, while the results from this “subroutine” may be fed back to another part — say the “Next Level” users — of the population. If the second group uses some sort of verbal labeling interface, like the one developed by the Steve.museum initiative [Wyman et al. 2006], promising cross-fertilizations become possible. With the adoption of appropriate standards, an integration of textual annotations, reviews, or the discussion of remarkable patterns in visual behavior becomes possible. In a way the collaborative visual filtering system enables noticing, labeling and re-labeling phenomena, thus giving shape to an augmented social construction of knowledge [Barrett 1989]. The role of a scholar in designing these filtering systems will not be the role of an author of scholarly articles or monographs, but the role of a broker in art historical knowledge, the director of an intricate process of knowledge mediation [Stubenrauch 1993].

If it is assumed that there is no reason to strive for a one-fits-all collection of visual denotats, extended research might be focused on the matching of specific collections of visual artifacts and appropriate sets of icons. Such research efforts could start from the assumption that when the medium of expression is fixed, observers (experts and/or lay people) will just express themselves in the reference materials at hand, focusing in these denotats on what are their peculiar points for attention. This actually simplifies research, since developers need not be explicit about suitable reference options. A myriad of further extensions to the basic visual voting model is now conceivable. Observers from particular user groups, for instance, might be enabled to work with sets of dedicated icons (“private alphabets”). We predict that as a matter of course these series will be composed according to common sense principles, like maximum diversity, sufficient coverage, etc.

Extensions of the voting system may include additional functionalities. One might be the possibility to define subsets of artifacts, based on singular or multiple corresponding artifact-icon associations; these subsets would thus be true collaborative products. Based on an analysis of patterns in voting behavior (or, alternatively, explicit observer profiles), the system could also gradually evolve into a system displaying only the preferences (both artifacts and/or votes) of a particular user group. Finally, since somehow artifacts co-organize icons, the labels (icons) in the interface may be grouped (i.e. organized) by their co-occurence patterns; perhaps these may be taken as so-called emergent semantics [Aberer et al. 2004].

Hopefully this orientation has made clear that the social formation of opinions concerning the visual similarity of cultural artifacts amongst a group of motivated observers can with the use of information technology be modeled and recorded without the application of verbal descriptors, and that interesting data processing options exist for collaboratively and visually characterized artifacts. This offers many starting points for further research.