DHQ: Digital Humanities Quarterly
2021
Volume 15 Number 1
2021 15.1  |  XMLPDFPrint

Leonardo, Morelli, and the Computational Mirror

Alison Langmead  <adlangmead_at_pitt_dot_edu>, University of Pittsburgh
Christopher J. Nygren  <cnygren_at_Pitt_dot_edu>, University of Pittsburgh
Paul Rodriguez  <prodriguez_at_sdsc_dot_edu>, San Diego Supercomputer Center
Alan Craig  <N/A>, Independent Consultant and Researcher

Abstract

By bringing forward and interpreting the results from a collaborative research project that used contemporary computing techniques to investigate Giovanni Morelli’s nineteenth-century method for making stylistic attributions of old master paintings, this article examines how humanists make claims to knowledge and how this process may or may not be modellable or mechanizable within the context of classical, deterministic, digital computation. It begins with an explanation of the rationale behind choosing the Morellian practice of attribution, continues with a survey of another effort at computationally implementing Morelli’s method, and then presents our own computational techniques and results. The article concludes with what we have come to understand about the roles of responsibility, trust, and expertise in the social practice of art attribution, and the dangers in assuming that such human entailments are native to digital computers.

On 15 November 2017, the art world was rocked by a momentous event: Christie’s sold a painting for the staggering sum of $450.3 million dollars, shattering every record for an art auction and thrusting the painting well beyond every other possible claimant for the mantle of “most expensive painting in the world” [Lewis 2019]. The painting is, compared to other auction house blockbusters, quite unassuming (see Figure 1).
Figure 1. 
Christie's employees pose in front of a painting entitled Salvator Mundi by Italian polymath Leonardo da Vinci at a photocall at Christie's auction house in central London on October 22, 2017 ahead of its sale at Christie's New York on November 15, 2017. AFP Contributor/AFP/AFP/Getty Images.
It depicts Christ in half-length, holding a crystalline orb in his left hand while blessing with his right. Christ seems to lock eyes with the beholder, suggesting that he is in fact blessing us, which gives the picture a striking sense of immediacy. But assuming that the painting was made around the year 1500, as suggested by the auction house, this kind of interpersonal interaction with Christ was nothing new; painters like Antonello da Messina and Giovanni Bellini had at that point been experimenting with the interpersonal dynamic between images of the godhead and the individualized beholder for at least a quarter century. So, if “novelty” is not what drove the auction price to such extremes, what did? The social practice of art attribution. The painting has been attributed by the auction house (with the support of some art historians) to Leonardo da Vinci, one of the unrivaled masters of the Italian Renaissance.
There are somewhere between 10 to 20 extant paintings reasonably attributed to Leonardo, so the emergence of another picture would increase that corpus by 5 to 10 percent, a significant expansion. Quickly after the initial attribution, though, expert opinion began to divide. On the one hand, a number of scholars spearheaded by Martin Kemp, a professor at Oxford who specializes on Leonardo’s artistic production, steadfastly supported the attribution [Kemp 2019, 142–7]. On the other, curators like Carmen Bambach (Metropolitan Museum of Art) insisted that the painting is the work of one of Leonardo’s assistants, Giovanni Antonio Boltraffio [Bambach 2019, 2: 278–83].[1] Somewhere in the middle stood Luke Syson, currently Director of the Fitzwilliam Museum in Cambridge, England. In his previous position at the National Gallery in London, Syson had curated a blockbuster show on Leonardo that included the Salvator Mundi largely as a hypothesis, asking whether or not the painting held up to other autograph works by Leonardo [Syson 2011, 300–3 (cat. 91)] and [Lewis 2019, 73–213]. Syson believed parts of the painting were done by Leonardo, but that much of it was painted by assistants, as was common practice in Renaissance workshops where works done almost entirely by workshop assistants often went out under the imprimatur of “the master” [Golden 2004] [O’Malley 2007] [Williams 2017, 173–258] [Neilson 2019].
These contrasting opinions can seem dizzying. If three major scholars who have dedicated their lives to the study of Leonardo cannot come to a consensus about this one painting, whom do we trust? What does it say about the state of knowledge production in the humanities that such a crucial question fails to find consensus among leading experts and yet the painting still commands an astronomical price? Should there, or should there not, be a relatively straightforward answer to the question, “Did Leonardo paint this picture?” If so, and if human connoisseurs cannot satisfactorily answer it, perhaps computers might be enlisted to clarify the problem and address it more empirically. After all, a well-programmed computer algorithm will produce results that are based only on the inputs it is given; a computer cannot swindle in the way that unscrupulous art dealers often have. So, might it be possible to model, computationally and mathematically, a problem as difficult as the attribution of a painting to an individual? Imagine for a moment a computer algorithm that could produce consistent assertions about the authorship of the Salvator Mundi. How might that change what it means to be an art historian or, even, an artist?
This essay is about the possibilities and impossibilities inherent to embedding computation within a social practice like art attribution. It examines how humanists make claims to knowledge and how this process may or may not be modellable or mechanizable within the context of classical, deterministic, digital computation. The example of the recent attribution and sale of the Salvator Mundi is exceptional in its economic scope, of course, but we argue that the event pinpoints an important question about the stakes of bringing computation into the realm of art attribution: Is it possible for a computer to “make an attribution” in any meaningful sense, or are attributions always and only possible within the realm of human judgment? We argue that the practice of art attribution is not solely about “being correct,” that is, “finding the answer,” but is equally about being able to assign responsibility for the success and/or failure of such influential decisions to an intelligent agent. We wish to underline that such responsibility must be granted by human communities to digital computation; computers cannot take responsibility for their actions on their own.
To begin to shed light on these questions, this essay brings forward results from a collaborative research project that used contemporary computing methods to investigate Giovanni Morelli’s nineteenth-century method for making stylistic attributions of old master paintings [Morelli 1893, 1–63]. As digital art historians, we saw in Morelli’s attributional practice a method that could be easily transposed into the language of the digital computer because it describes a set of quasi-computational operations (more on which below). But we did not begin this project expecting — or even trying — to prove Morelli’s method “correct” or “accurate,” nor were we interested in finding out if Morelli’s system could successfully and accurately attribute art works in the context of digital computing. Indeed, we knew from the start that such efforts would be a logical impossibility because of the way the notion of “style” has been constructed and reconstructed over a century and a half since Morelli began publishing his findings.[2] Instead, we wanted to use the computer to force us to come to grips with the details of this foundational, highly formative art historical practice, and to look at it fundamentally anew through the process of computational formalization. We know computers to be excellent at supporting this practice. Because computers are sticklers for details, they serve as a type of “computational mirror.” They are able to show only what they have been asked to reveal — which is not necessarily the same as what the researcher wants to see. We knew that the use of the computer would require us, at each step, to have an “honesty chat” about Morelli’s (and our own) methods.
It was a revelation to discover the precise ways in which the Morellian techniques unfolded themselves in response to our computational approaches. We quickly came to re-contextualize Morelli’s original procedure as a nineteenth-century form of dimensionality reduction and proxy generation that was inseparable from his native human powers of judgment rather than something that parallels a twenty-first-century computational approach to formalizing the world. This unlocked for us the potential broader significance of his method both for art history and also the art market. In doing this work, we came to recognize that what truly lies at the heart of a successful (or failed) art attribution is not simply endorsing the accuracy of formal comparisons, but the ability to participate in a community of trusted experts and to take full responsibility for one’s own inferences and judgments.
This paper will begin with an explanation of the rationale behind choosing the Morellian practice of attribution rather than any other. After briefly surveying another effort at computationally implementing Morelli’s method, we shall present our own computational techniques and results. We will then discuss what we have come to understand about the roles of responsibility, trust, and expertise in the social practice of art attribution, and how it would be dangerous to assume that such human entailments are native to digital computers.

Morelli and His Method

Giovanni Morelli (1816-1891) was one of the foremost connoisseurs — masters of art attribution — in the history of art, and the method he proposed for identifying and authenticating old master pictures has had enormous impact on the discipline [Agosti et al. 1993] [Hinojosa 2009] [Caglioti et. al. 2018] [Barni 2016]. Morelli’s career trajectory spanned the fields of medicine, politics, and art criticism, and could perhaps have only been fostered in the bubbling cauldron that was the European nineteenth century [Anderson 2020]. Born in Italy to parents of Swiss origin, Morelli moved to Munich at age 17 to study medicine at the Ludwig Maximilian University. Although Morelli never practiced medicine, his training in that field would prove foundational for the method of art attribution that he developed later in his career. After graduation, Morelli traveled throughout Europe fostering his love of art and a network of associations in the European art market. In 1848 he was swept up in the revolutionary fervor that was overtaking the Continent and, as a consequence, began a long and somewhat complicated involvement in the emergence of the Italian Republic. He was a senator in the early Republic and served on the committee that established the system of Italian national museums that is largely still intact, albeit in a modified form.
It was only in 1874, at the age of 58, that Morelli began publishing the rudiments of his method for attributing old master paintings. From that moment, until his death in 1891, Morelli wrote numerous volumes, which quickly gained a wide audience. His interventions were published in German under a double-pseudonym: the writings purported to be the work of a Russian man named Ivan Lermolieff, which were then translated into German by Johannes Schwartze, itself a translation of the Italian name “Giovanni Morelli” into German. These publications took the form of dialogues in which Lermolieff converses with an anonymous Italian aristocrat. What we now think of as the “Morellian Method” for attributing artworks emerges as the Italian helps Lermolieff see paintings in a new way. Importantly, the “Morellian method” was not laid down in normative terms; rather, it emerges as the product of dialogic exchange that allowed Morelli to give voice to the discomforts provoked by his new method. The Lermolieff/Morelli figure plays the role of the rube: he is a foreigner who has come to Italy in search of great art. The dialogue’s other character serves as his educated guide, introducing this revolutionary new mode of connoisseurial analysis based on the revealing detail.
The rudiments of Morelli’s method are familiar to art historians operating in all European language traditions [Davis 2011, 86–96]. Prior to his work, the attribution of paintings was a chaotic enterprise entangled in questions of nationalism, capitalism, prestige, and the decline of European aristocracy. Generally speaking, claims to authorship and authenticity were made on the basis of a painting as an integral whole: every piece of the painting was believed to work together to reveal the identity of the artistic agent responsible for having made the picture. This approach is predicated on what Morelli described rather derisively as the “general impression” of a picture, and he developed his own method consciously in opposition to this approach [Morelli 1893, 22–3].
Morelli’s wager was that individual details were key, not this “general impression.” His method assumed that painters revealed their true identity in the small places where they — and by extension also the beholders — were paying the least attention. Anatomical features like fingernails, earlobes, toes, and noses became the locus of authorial identity. Morelli argued that painters were consistent in the way that they painted these features throughout their careers and that these elements became something akin to a personalized pattern. He came to describe these patterns as Grundformen, or “ground forms,” and he offered line-drawn examples in his publications (see Figure 2).[3]
Figure 2. 
Line drawings showing sketches of ears and hands, from [Morelli 1890, 98–99]. Artwork in the public domain; image courtesy Universitätsbibliothek Heidelberg.
In his 1890 (English trans., 1893) dialogue on the Italian paintings in the Borghese and Pamphili collections, Morelli offers an excellent example of how to use Grundformen to make an argument about the attribution and dating of an artwork. In this passage, Morelli stages a scene in which the anonymous Italian companion outlines his method quite clearly for Lermolieff while they are standing in the presence of a portrait of Bishop Ludovico Beccadelli, signed and dated by Titian in 1552. The solidity of this attribution allows Morelli to use the painting as a tool for recognizing the Grundformen that truly distinguish Titian’s art and might thereby be used to solidify other attributions:

It is my object to make you notice everything in a work of art, and in time you will come to see that even details, in themselves insignificant, may lead us to the truth… Look at the hand in this portrait, particularly at the ball of the thumb, which is too strongly developed and at the round form of the ear. In all his early works, and in most of those in his middle period till between 1540-1550, Titian adheres to the same round form of ear... This peculiarity in the ball of the thumb also frequently occurs in his other paintings and in his drawings.  [Morelli 1893, 46–7]

This discussion is illustrated by a small schematic line drawing of a hand with the caption “Tizian's Daumenballen,” which is rendered in the English translation as “The Ball of the Thumb in Titian’s Works” (see Figures 3a & 3b) [Morelli 1893, 47].[4]
Figure 3. 
  • 3a (left): Line drawing showing, “The Ball of the Thumb in Titian's Works,” from [Morelli 1890, 58]. Artwork in the public domain; image courtesy Universitätsbibliothek Heidelberg.
  • 3b (right): The (proper) right hand in Titian’s Vincenzo Cappello, ca. 1540. Oil on canvas. Washington, DC, National Gallery of Art, Samuel H. Kress Collection, 1957.14.3. Artwork in the public domain; photograph provided by National Gallery of Art, Washington, DC.
That Morelli’s texts illustrate the Grundformen through line drawings and not full illustrations that more faithfully reproduce the actual image-in-the-world under scrutiny was not a technical limitation of the day. By the time that Morelli was publishing his books, it was possible to mass produce art historical volumes with full page illustrations of original works of art using a number of techniques, including lithography, engraving, and etching. His choice emphasizes his use of abstraction as a central technique and in so doing also reveals something disconcerting in Morelli’s method, which is that it is predicated on what we might think of as a kind of aesthetic dissection of the painting itself, a shredding of its holistic integrity and a disregard for the particulars of pictorial representation as instantiated in a given painting.
Indeed, the reduction of the entire work of art to the telling detail is one of the more charged elements of Morelli’s method, and he recognized it as such. Rather than see full-page illustrations of the paintings, the reader, like Lermolieff, is asked to encounter an odd array of noses, earlobes, and fingers organized as a grid to facilitate the process of differentiation between artists. With this approach, Morelli’s method challenged the sensibilities of what was, at that point, traditional art history, and the Lermolieff/Morelli character expressed repulsion at the aesthetic violence to which the painting is subjected by his Italian companion, as it is subdivided into hands, ears, and mouths, bemoaning that the “general impression” of the overall work of art is declassed to an afterthought [Morelli 1893, 19–22].
The Grundformen isolated features of the artwork and was a disorientating approach at the time, but for Morelli, the cost was worth the benefit. These schematizations facilitated attribution by staging comparisons, and this step-wise workflow in part helps explain why Morelli’s method is almost irresistible to those interested in algorithmic logic.[5] Morelli’s method was to reduce paintings to a limited set of discrete patterns, and in so doing, locate proxies for authorship in meaningful details he believed had been unconsciously embedded in the pictures. The effectiveness of the Grundformen could not be revealed by a casual look at the entire picture, but instead became potent only through their isolation, extraction, and comparison.

Morelli’s Method as Algorithm

In the same way that the results of Big Data analyses are at their most thrilling when they arise from patterns that are not obvious during data collection, Morelli also strove to reveal invisible patterns of reality that are actually hidden in plain sight.[6] The connection to Big Data here is not simply coincidental. In the twenty-first century, Morelli’s approach might read to the astute technologist like a description, although imprecise, of a set of functions that, once formalized, could be effectively and successfully performed by a digital computer. For example, a supervised system of digital Morellian classification might begin with the identification of Grundformen, that is, the image-based features that typify a particular artist. It might then computationally select and extract those detail-features from a dataset of digital images. Next, it might identify a procedure that would best group these details, and classify them into artist-based clusters. With this trained system in hand, Grundformen could then be extracted from unattributed paintings. By comparing these unattributed features to the computationally classified set, one could produce a probabilistic attribution of this unattributed work of art to an artist, or distribute that probability over a set of artists, all by directly implementing Morelli’s assumption that similarities in the appearance of the Grundformen are effective proxies for an artist’s stylistic “hand.”[7]
Both this brief sketch of a computational Morellian system — as well as Morelli’s original procedure — are fundamentally designed to answer one critical question, “Who painted this painting?” But, as we aim to show in this paper, the notion that art attribution is a “problem” that can be “solved” or a question that can be “answered once and for all,” misses a critical point. Art attribution is a practice fully embedded in sociality. Indeed, as contemporary art historians, we have found Morelli's method to be most useful not to answer the question, “Who painted this?” but rather as a tool for better understanding all the complex, human-facing components that make up the question itself.

Finding the Answer Means First Defining the Question

Our project is not invested in attempting to operationalize Morelli’s system, in part because previous scholars have already shown that this is indeed already possible, if one proceeds with the assumption that art attribution is a problem to be solved. Adrian J. Ryan, in a 2009 dissertation produced for the Department of Classics at the University of KwaZulu-Natal in South Africa, produced just such a system, and we have found that his research offers a fruitful case study for comparison. Ryan is a computer scientist/technologist by training, who, in returning to school for his degree in Classics, created a formalized, digital system designed to imitate and reproduce the Attic vase-painting attributions of Sir John Beazley. We have found his work especially useful for revealing a number of differences between a digital-art-historical understanding of the utility of Morellian method and a computer-scientific one. Most critically, Ryan’s work highlights what it looks like when Morelli’s methods are used to “look for answers,” rather than to situate the work of attribution within a social, art-historical context.
The guiding star for Ryan’s computational system of art attribution is not Giovanni Morelli himself, but instead one of the most important scholars in the history of the humanities to have implemented a Morellian Method: Sir John Beazley (1885-1970). Beazley has long served as a disciplinary figurehead for Morelli’s style of artistic attribution, not only in his home field of Attic vase-painting, but also beyond [Davis 2011, 86–96] [Driscoll 2019] [Neer 2005] [Smith 2005]. Ryan states that his overall goal in implementing a computerized system of art attribution is to discover whether or not, “a computer may be taught to attribute in the same way that an art historian can” [Ryan 2009, 56]. He is clear, both in this passage and throughout his study, that his aim is not to design a classifier that can attribute paintings in some natively computational way, or to justify “the practice of connoisseurship by means of statistics” [Ryan 2009, 14], but instead, “to prove or disprove...that machines may be taught to attribute in the same way [as a human expert]” [Ryan 2009, 2]. The system Ryan would come to produce is, for all intents and purposes, a “Beazley Machine,” that is, a supervised machine learning system designed to mimic and/or agree with any and all of Beazley’s attributions.
Ryan’s data-hungry algorithms function, almost exclusively, by winkling out patterns from examples, and the more examples, the better. For this reason, Beazley seems a very sensible choice for this research. The amount of data and documentation remaining from Beazley’s attributions is impressive by the standards of the humanities. Beazley used in his practice, just as Morelli had, hand-drawn, schematic renderings of the details/features that he deemed emblematic of each Attic vase-painter’s style, drawings that he published in his articles and books (see Figure 4).
Figure 4. 
Beazley's sketches of an amphora attributed to the Kleophrades Painter (early 5th century BCE), from Notebook 3, 1909-1910. Pen and ink on paper. Photograph from Beazley Archive, courtesy of the Classical Art Research Centre, University of Oxford.
Over the course of half a century Beazley came to identify hundreds of vase-painters based on detail-features like knees, hands, and costume. The existence of the large photo archive that Beazley amassed of these ceramics, plus the five published, monumental, volumes of Beazley’s attributions, as well as the archival record of Beazley’s dimensionally-reduced line drawings, convinced Ryan that there might be sufficient data to computationally model Beazley’s method [Beazley 1942] [Beazley 1956] [Beazley 1971].[8] Nevertheless, while the collection of data Beazley created on Attic vases is truly excellent and extensive by art historical standards, Ryan found that they proved insufficient to support the types of machine learning approaches that he had hoped to implement.
The method that Ryan used to solve this problem of data scarcity interests us greatly for what it reveals about the necessary role of human beings in the process of art attribution even implemented computationally. Ryan elected to address it by producing “artificial data sets” that increase the amount of data available for training. That is, Ryan himself sketched a number of prototypical mouths, knees, and ears in the manner of a number of Antique painters, digitized them, and included his own line drawings as part of the Beazleian training data. This technique served very specific computer-scientific ends for Ryan, namely to help discover if training the classification engine on a combination of both original and artificial data reduces the risk of overfitting to a small design set significantly enough to justify the risk of adding non-essential information [Ryan 2009, 100ff]. But, critically, he went so far as to argue that the inclusion of his own drawings — insofar as they gesturally re-enact the work of a trained, human connoisseur — made his computational experiment more accurate.[9] This is, interestingly, in line with Beazley’s own practice. Drawing inspiration from Morelli’s method, Beazley produced schematic line drawings of the forms and figures he observed on Greek vases in the belief that the identity of the artist responsible for painting a given vase might become clear when comparing these schematic surrogates against one another. But occasionally, Beazley begged for the reader’s indulgence as he produced line drawings that were more reflective of his own hand than that of the Greek artist he was attempting to ventriloquize [Beazley 1922, 97]. By inserting his own humanity into his datasets, we find that Ryan is not only following Beazley’s example, but also asserting that trained human actions and decisions play an unassailable role in the process of art attribution. It is Ryan’s judgment that makes this all work, not Beazley’s and not the computer’s.
And yet, when Ryan offers a number of justifications for his work, we feel that most of them are more consonant with the traditions of computer science than the humanities. Crucially, he argues that one of the most fundamentally positive contributions of such a system would be to make the practice of art attribution more efficient, that is, to save human time. Rather than focusing on the central importance of human expertise, Ryan suggests that his system could save connoisseurs from wasting precious time performing “painstaking, difficult” operations, and could free them up to focus on any larger issues at hand [Ryan 2009, 36]. This move to assert that the value of computation in any field is to save time and increase efficiency may function well in domains such as engineering or business, but we find it largely unconvincing in the context of the humanities.[10]
For Ryan, the opposite of an efficient system is a slow, painstaking one. However, in the context of the humanities, this analogy simply does not hold. For a humanist, slow is not necessarily bad and it is certainly not the opposite of “efficient.”[11] The value of the humanities does not directly reside in how quickly it can be done, and indeed the work can be considered un-finishable: full of difficult, fundamentally unanswerable questions that each human generation reassesses anew. We believe that the value of the intellectual work of the humanities partially resides in the fact that it takes time both for human beings to gain trust in one another and also to be trained to make scholarly judgments. Will it, or even can it, “save connoisseurial time” to delegate the practice of, say, visual comparison, to a computer program? Ryan brings no evidence to bear to demonstrate that there are other ways to produce this kind of domain-specific knowledge other than the very practice he is automating, that is, having a human being perform these “painstaking, difficult” tasks, especially those that require “very narrow expertise,” as he himself notes [Ryan 2009, 36].
We do not believe that such a demonstration is possible, and it would be irresponsible to find out simply by trying. The question of whether or not a certain painter painted a given part of a picture is, ultimately, unverifiable. While there is a reality to it, we cannot replicate it as an experiment: we do not have a time machine. What an act of hubris it would be to think that we could at any one point solve the problem of art attribution once and for all! Disagreement and a lack of complete consensus in an interpretive community is not a sign of its failure, or lack of “efficiency;” they are signs of its correct functioning as this community goes about its work of continually producing meaningful, cogent knowledge.
In our digital work with the Morellian Method, we focused not on issues of efficiency, novelty, or even the extent to which we could fully mechanize art attribution, but instead on using the computer to help slow our thoughts down, pick apart what we were seeing, show us some of our assumptions in pain-staking detail, and to dive deeper into the system we were formalizing. Rather than hoping that the computer system would partially mechanize our trained judgement, we hoped the process would allow us to use it more judiciously. We wished to engage with what it would mean to implement the Morellian method within the context of digital computing while maintaining our focus on the intellectual priorities of the humanities, hoping to take advantage of the “computational mirror” to come to learn things about Morelli’s methods and our own. We therefore proposed a technical collaboration with the Extreme Science and Engineering Discovery Environment (XSEDE) group, an NSF-funded, computational and collaborative infrastructure built on top of the United States’ national network of supercomputers, for the development of a proof-of-concept “Morelli Machine” that would take advantage of the team’s interdisciplinary composition to use digital techniques to apply Morelli's method to a large set of images of artworks.

Our Engagement with the “Morelli Machine”

We were keenly aware that the reward structure that would surround the technologist assigned to us by the XSEDE infrastructure would differ greatly from our own as academics, and we wanted to design the details of the project so that it could accommodate both our desire to create recognizable humanistic research as well as fit the requirements of grant-funded technologists who would need to show computational innovation and novel results.[12] Therefore, after our initial proposal was accepted, we designed a two-phase project plan with the input of the XSEDE technologist-consultants Paul Rodriguez and Alan Craig. First, we would apply Morelli’s method in a somewhat strict historical manner. We would ask the computer to extract the eyes, mouths and hands from a dataset of images and then cluster them into formal categories, that is, we would ask the machine to follow the same workflow detailed in Morelli’s writings.[13] We would then look to see what groupings the computer might find, in part to compare the results to the known clusters of artistic similarity, but also to see if the computer was able to group these features into any other sorts of meaningful groups that may not (yet) hold any resemblance to art-historical opinion. We chose to apply a standard computer vision technique known as Histogram of Oriented Gradients for Phase One, rather than either a supervised or unsupervised machine learning system, for two reasons: first, as we will note below, Phase Two was earmarked for such approaches, and second, Histogram of Oriented Gradients, as a feature descriptor that operates similarly to edge detectors, works by isolating dramatic shifts within digital images, an approach that seemed to map well to the process that Morelli proposed.
It was recognized from the very beginning that this work, like Ryan’s, would require data and a great deal of it. We had initially hoped that we could amass between two and five thousand images drawn from a handful institutions, such as the Kress Foundation, The Metropolitan Museum of Art, and the National Gallery. Our plan was to gather this relatively large number of images, or perhaps even more, from ArtSTOR’s collection of digitized and digital images of artworks. While our initial conversations with ArtSTOR seemed promising, in the end, the red tape surrounding the large-scale use of multi-institutional images proved insurmountable to solve in the time we had to address it.
However, Nygren and Langmead knew, because of prior digital humanities work their students Andrea Maxwell and Sarah Reiff Conell had done with the images of the Samuel H. Kress Collection, that the Kress Foundation is very generous in sharing their collection of digital images for computational use, and moreover, offered dependable metadata. After our initial conversations with ArtSTOR faltered, we approached the leadership of the Kress Foundation with a request to use their images for the purposes of this project, and they happily agreed, while noting that perhaps the easiest way to acquire the files would be, ironically, through ArtSTOR. With the help of the Foundation, ArtSTOR was able to provide us with the digitized images of the Kress Collection, a dataset that amounted to 1,866 files, not “Big Data” by computer-scientific standards, but too large and at too-high a resolution for a laptop to process, and was sufficient to kickstart our work.
For the second phase of the project — one we will not have space to fully detail in this essay — we recognized that Morelli’s approach of decomposing the images into distinct human-recognizable features would not be the way that contemporary technologists might choose to approach this work if the goal was clustering and classifying images by artist. It is currently quite difficult and time-consuming to ask a computer to do the work of extracting human-defined, visual details from images, and it only has marginal computer-scientific novelty to approach the problem in this manner. And so, Langmead, Nygren, Rodriguez and Craig collectively designed a plan for Phase Two that would compare and cluster the images in this same dataset using machine learning approaches that let the computer select the important features, rather than forcing the use of human-defined ones.[14] Even though we will not detail the process or the results of this second phase in this paper, we mention it here because we feel that it is important to present the entire scope of the collaboration and the balance we tried to strike between serving the working needs of the humanists and the technologists on this project. Our results from Phase Two will be the focus of a separate study at a later date, and has already been presented by Rodriguez to the XSEDE community and received with interest.[15]
Phase One of the “Morelli Machine” project began in August 2018 and concluded in January 2019. Given that the off-the-shelf digital technologies for facial recognition were, and are, far more advanced than any other type of human body-part-detection, we focused on the process of extracting facial features such as eyes and mouths from the data set, rather than hands or feet. Step one on this path was, therefore, to extract faces from the images. Rodriguez began by trying the Dlib Face Detector and OpenCV Face Recognition libraries, but found in the end that the Google Vision API was the most successful at identifying the faces within our dataset.[16] Google’s technology was then applied to the entire collection, and succeeded in extracting 3,205 faces. This API also returned facial landmarks — that is, polygons that outlined its best guesses as to where the eyes, nose and mouth were located on each face it identified — and Rodriguez used this information to post-process the results and extract our desired facial features as individual, cropped images (see Figure 5).
Figure 5. 
At left, an example face extracted from a digital image of a painting in the Kress Collection showing the red box around the part of the image that the Google Vision API has identified as a face, along with the green, red and purple polygons marking the locations of eyes, nose, and mouth. At center, the face extracted using these guideposts. At right, the extracted mouth alone. Image courtesy Paul Rodriguez.
To fine-tune this process of feature extraction, Rodriguez applied pose (i.e., head turn) estimates, again using the Google Vision API, to adjust the size and shape of these boundary polygons. We wanted to be as careful as possible to extract only our desired facial features, knowing that applying Morelli’s method requires accurate feature extraction. Overall, to the human eye, the results appeared quite good. Rodriguez estimated that around 71% of the faces found within our image files were successfully identified, and within that result set, we were able to extract all of the eyes and mouths [Rodriguez 2020b].
Sufficiently satisfied with the accuracy of the feature extraction, the team then moved to the task of clustering. The team began by focusing on the mouths and Rodriguez turned to the Histogram of Oriented Gradients technique to begin work. Histogram of Oriented Gradients (HOG) is a feature descriptor that operates similarly to edge detectors, that is, rather than treating images as color values, it works by identifying the direction and magnitudes of relative difference between pixels. These differences can then, in turn, be used to detect the edges of and/or identify the human-recognizable objects represented in the images.[17] At each pixel, the algorithm calculates a vector for the “gradient,” that is the amount of numeric change between that pixel and the other pixels adjacent to it horizontally and vertically. Those vectors are then added together to create a main direction and magnitude for each pixel, that is, its oriented gradient (see Figure 6).
Figure 6. 
At left, an image that is zoomed in on the edge of a figure’s mouth, with a black box centered on a single pixel. At right, that same area of the image marked by the black box magnified further, with the red arrows denoting the change in pixel value in the horizontal and vertical dimensions, and the purple arrow being their sum. Image courtesy Paul Rodriguez.
When implementing HOG, the meaning of “change in pixel value” can be tweaked by the technologist to best identify those features of an image that may be more important to the study at hand. This could be color detection, saturation detection, or any other fundamental, numerically represented, feature of a digital image. To work effectively, therefore, a bit of data pre-processing on the images is required to highlight those characteristics that it has been decided are important to the assessment at hand, and to deemphasize those felt to be extraneous. Deciding what was and was not visually important to implementing the Morellian method computationally became something that our team needed to identify at the outset. As art historians focused on implementing Morelli’s method as closely as we could, Nygren and Langmead were very concerned with maintaining as much color information in our analysis. Rodriguez took this request into deep consideration and opted to normalize the eye-images so that the total illumination was constant but relative color values were still present.[18]
As a next step in the HOG process, all pixels in the image are clustered into blocks of four and the vectorized gradients for each of them are then used to calculate nine summary gradients as measured from the block center (see Figure 7). These nine summary vectors are then counted and visualized as a histogram showing the distribution of the sum of their magnitudes, categorized by their orientation. The greater the count in any given bin, the “stronger” the gradient is said to be.
Figure 7. 
At left, a visualization of pixels being assembled into groups of four, and then from each corner, nine new vectors appear, summarizing those pixels’ individual gradient magnitudes. In center, the vectors of that pixel group are visualized in isolation. At right, a visualization of the histogram (count) of the various magnitudes represented by the vectors at center, organized by their degrees (bins are 20 degrees each). Image courtesy Paul Rodriguez.
The penultimate result of the Histogram of Oriented Gradients feature descriptor, then, is that the images have been decomposed into a set of clustered, oriented, summarizing, nine-dimensional vectors that describe the relative numeric change in pixel values across the image, one that also retains information about the relative direction in which that change occurs (see Figure 8).
Figure 8. 
Visualization of the grid of vectors that represent the distribution of oriented gradients across a digital image. Image courtesy Paul Rodriguez.
These vectors can then be visualized as a histogram that can be taken to assert the relative “strength” and “weakness” of the gradients across the image. To conclude the process of applying the Histogram of Oriented Gradients technique, a summarization method is applied: one final feature vector is calculated by taking all of these nine-dimensional vectors and summarizing them into a single, large, ten-dimensional vector that represents, in some ways, the image in its entirety [Rodriguez 2020a].
Once this process had been run on our extracted features, it was time to arrange them into clusters of similarity. The ten-dimensional HOG feature vectors that summarized each image were used as input into a K-means clustering algorithm to gather the images into (hopefully meaningful) groupings. To visualize this information, the original ten-dimensional feature vectors were projected into two-dimensional space using the t-SNE technique, and each point on the resulting graph was then color-coded according to their K-means clustering assignments (Figure 9).[19] We began with mouths.
Figure 9. 
To the upper left, a visualization of the clustering of similarities between mouths, with the particular mouth shown to the upper right (“32”) highlighted. To the upper right, the images drawn from the Kress Collection of the example face “32” used in the workflow. At bottom, the histogram representing the gradients from this particular example all as produced by Paul Rodriguez in the project workflow. Image courtesy Paul Rodriguez.
The initial clustering results for mouths did not inspire anyone on the team to assert that our computational method was successfully grouping the images by any clear, meaningful art-historical metric whatsoever (see Figure 10). Above and beyond the fact that the team had clearly produced an excellent Old-Master-Painting mouth extractor, the clustering results produced by the combination of the t-SNE and K-means procedures was not anywhere near to clustering these features by artist. None of us were even particularly struck by any amount of above-average visual similarity within the clusters. The results for eyes were no better.
Figure 10. 
Three computer-produced similarity clusters of the mouths extracted from digital images of the Kress Collection, as produced by the analysis led by Paul Rodriguez in the project workflow. Image courtesy Paul Rodriguez.
It was becoming clear that HOG was not sorting these images into groups that held visual interest for the team. But even so, because our own research questions asked us to make some form of art-historical, or even visual, sense of the clusters of disembodied mouths, Langmead and Nygren were afforded the time and the opportunity to acknowledge and respond to their own immediate and visceral recognition of the aesthetic violence of Morelli’s method, one perhaps magnified by the scale offered by the digital computer. This moment offered us an experiential taste of what it means to extract recognizably visual features from art objects and array them like so many specimens in a row. Moreover, it suggested how hard it can be to visually interpret these isolated images at this scale, one orders of magnitude more massive than Morelli could have possibly imagined. Morelli himself suggested that it was overwhelming to confront dozens of sketches of fragmented body parts, and seeing thousands of them proved no less disconcerting (see Figure 11).
Figure 11. 
Left and right: Line drawings showing sketches of ears and hands, from [Morelli 1890, 98–99]. Artwork in the public domain; image courtesy Universitätsbibliothek Heidelberg. Center: Similarity clusters 1 (bottom) and 3 (top) from Figure 10. Image courtesy Paul Rodriguez.
When comparing Morelli’s published line drawings of disembodied hands and ears to our data, Langmead and Nygren had been considering these visually comprehensible images as the computational analogs to Morelli’s Grundformen. However, during a team meeting, as Rodriguez took a moment to narrate his process in greater detail, he also shared the set of pre-processed images created for the HOG feature descriptor. It was at that moment, with the grids of mouths in mind, that Nygren and Langmead were suddenly struck with the realization that these pre-processed images — not the disembodied, recognizable mouths that had been extracted by the Google Vision API — were the objects analogous to the Grundformen, the dimensionally-reduced line drawings used by Morelli (see Figure 12). These (to human eyes) yellow and white, almost wholly abstracted forms, were the more fundamental visual evidence of the extreme computational abstraction necessary to make claims about sameness and difference within a computer, not the grid of human-identifiable mouths.
Figure 12. 
Left and right: Line drawings showing sketches of ears and hands, from [Morelli 1890, 98–99]. Image courtesy Universitätsbibliothek Heidelberg. Center: Visualization in four rows, each showing a different example mouth extracted from digital images of the Kress Collection. The columns represent three, dimensionally-reduced forms, at left, the mouths as directly extracted from the original digital images, at center, those same images with the digital information abstracted down to a form more useful to the HOG process, but that are difficult to recognize as mouths to the human eye, and at right, the images at center shown with more of their surrounding image context. Image courtesy Paul Rodriguez.
To explain what we mean let us return to the Beccadelli portrait we introduced above, because it is in the illustration accompanying his discussion of that painting that the ultimate stakes of the Grundformen become clear. Through working with Morelli’s method within the context of digital computing, we came to understand that it is predicated on a misdirection that leads us away from the painting-in-the-world toward an abstract, schematic rendering conjured into existence by the connoisseur. Closely comparing Morelli’s line drawing to the Beccadelli portrait, we noticed that his sketchy rendering is quite deceptive and does not appear to duplicate either hand found in Titian’s painting. To a trained art historian, the hand seen in Morelli’s line drawing actually seems much closer to the hands found in numerous other TItian paintings executed around this same period in Titian’s career (1540-50), such as the Crowning of Thorns, The Allocution of Alfonso d’Avalos, or the Portrait of Vincenzo Cappello (see Figure 13). For a man focused on details, Morelli appears entirely uninterested in offering direct visual parallels between his line drawings and the particulars of this portrait [Uglow 2014] [Brewer 2018].
Figure 13. 
  • 13a. Upper left: The (proper) left hand of the figure in the foreground in red in The Allocution of the Marchese del Vasto to His Troops, 1540-41. Oil on canvas. Madrid, Museo del Prado. Artwork in the public domain; photograph by Francesco Mariani.
  • 13b. Upper right: The (proper) right hand in Titian’s Vincenzo Cappello, ca. 1540. Oil on canvas. Washington, DC, National Gallery of Art, Samuel H. Kress Collection, 1957.14.3. Artwork in the public domain; photograph provided by National Gallery of Art, Washington, DC.
  • 13c. Bottom right: The (proper) right hand of Ludovico Beccadelli in Titian’s Portrait of Ludovico Beccadelli, 1552. Oil on canvas. Florence, Uffizi Galleries. Artwork in the public domain; photograph by Laura Fenelli.
  • 13d. Bottom left: The (proper) right hand of soldier in blue at rear of Titian’s Crowning with Thorns, 1543. Oil on canvas. Paris, Louvre. Artwork in the public domain; photograph ©RMN-Grand Palais / Art Resource, NY.
  • 13e. Center: Line drawing showing, “The Ball of the Thumb in Titian's Works,” from [Morelli 1890, 58]. Artwork in the public domain; image courtesy Universitätsbibliothek Heidelberg.
Elsewhere in his volume, Morelli suggested that the Grundformen should resolve into a “physiological treatise” [Morelli 1893, 10]. Having trained as a physician, Morelli’s Grundformen “identify” the author of a painting in the same way that a diagnostic manual “identifies” a disease: if diagnosis relies on inference based on symptoms, so too does attribution mobilize evidence in the service of inferential reasoning that is ultimately metonymic.[20] It is important to understand that despite Morelli’s insistence that the Grundformen constitute a form of ground truth on top of which he constructed his attributions, his practice made apparent that they are significant personal abstractions intended for use by trained professionals who have a great deal of pre-existing visual expertise. The Grundformen do not exist in any real sense, that is, the Grundform of a Titian hand is not the same thing as a photographic detail of Beccadelli’s hand as painted by Titian. Rather, the Grundformen were, in Morelli’s account, a wholly schematic rendering of the type of hands formulaically produced across Titian’s entire pictorial corpus, which existed firstly in the mind of the scholar and only secondarily as line drawings.
This realization forced us to consider Morelli’s texts in a new light. On the basis of what we learned by attempting to computationally recreate Morelli’s method, it was no longer clear to us that we were making correct assumptions about how he expected his books to be read and what role he intended the Grundformen to play in the process of attribution. On the one hand, Morelli could have expected his readers to take copies of his book to the unattributed painting and compare his line drawings to the image in vivo. On the other, his books could have been taken as an invitation to the reader to produce abstracted line drawings of their own, only comparable to Morelli’s published drawings, to use in the service of their own attributions.
At this point we feel it is worthwhile to underline what we consider one of the major findings of this project, at the level of both art-historical and computational analysis: Morelli’s Grundformen are not simply dimensional reductions of visual data. The Grundformen are embodied renderings of human judgment, the result of hundreds (if not thousands) of hours of deliberatively looking at early modern paintings with the larger intention of developing a method for describing artistic style. Simply looking at an array of extracted images would never result in the production of a Grundform; to do this work, a human must exercise trained judgment to identify precisely what is and what is not important to their own informed understanding of the notion of artistic style and then use those concepts to extract only what is necessary to them. There is no promise that the Grundformen of one scholar would match those of another. Similarly, asking a computer to replicate this workflow cannot be achieved by training the computer to recognize, say, every human hand painted by Titian; rather, it would require the computer to produce its own abstracted rendering of Titian’s “hand,” which encompasses both the hands he painted and his stylistic “hand,” and then to deploy successfully that abstraction identifying artistic style. While current computer techniques are becoming capable of achieving the first step in this process, as represented by the yellow-and-white abstractions made for the Histogram of Oriented Gradients workflow, the higher order function of articulating how these abstractions make meaning in the context of the scholarly community remains beyond its reach without human intervention. For the machines to be able to attribute artwork on their own, we would need to allow them to join our intentional community and delegate some of the responsibility for our judgment to their mechanical processes.

The Art Market

We set about this collaboration because we were curious to see what it would require to implement an art historical method of connoisseurship computationally. We have determined that, for the moment at least, computational techniques cannot attribute artworks without the support of human judgement and expertise.[21] While this situation may change as our relationship to technology transforms over time, our core belief remains that looking to a computer for the “answer” to the question “who painted this picture?” is currently not only misplaced but can also have noxious implications.
Although we did not initiate this collaboration with the art market in mind, it quickly became clear to us that our project would not be perceived as agnostic by that market itself and the stakes of this work partake of this larger ecosystem [Elgammal et al. 2018].[22] This perhaps should not have come as a surprise, since Morelli’s method has had a marked impact on the art market [Agosti et al. 1993]. Because the price of an object on the market is interlinked with other concepts like historic import and artistic “value” (however defined), any mode of verifying the authenticity or attribution of a work of art that is deemed to be trustworthy will, almost as if through capillary motion, flow into the surplus value of the work of art. Over the course of the twentieth century, increasing preference has been given to the evidence produced through technological processes to determine authorship [Carrier 2003] [Wollheim 1973] [Grabar 1988]. In 2016, Sotheby’s purchased the firm Orion Analytical, making them the first auction house to create an in-house scientific lab dedicated to authenticating works of art [Subramanian 2018].[23] The auction houses have sought refuge in the seemingly dispassionate results of scientists, a form of evidence that takes on the air of objectivity and thereby is granted authority, especially by those outside of the scholarly community.
Carrying this tradition forward, computer science researchers have recently been promoting their technical skills with computer vision to hold out the promise of “art verified by AI.”[24] Such interventions draw on the same desire for “objective” data as earlier attempts to incorporate scientific results of pigment analysis and carbon-14 dating, but this time wrapped in the rhetoric of artificial intelligence. The assertion that an attribution has been “computer verified” is likely to increase the value and prestige of old master paintings. Would collectors be more likely to wager the GDP of a small nation, as in the example of the Salvator Mundi above, on the conflicting opinions of embodied art historians or on the declarative output of a computer algorithm? We cannot claim to know with certainty, but it seems as likely as not that a buyer will trust the definitive answer produced by the black box of a computer algorithm over a cacophonous exchange of interpretations between the competing voices of art historians. The art market demands answers, not a series of questions.[25]
As humanists, however, we believe that such questions are truly important. What does the insertion of a computer change about, or add to, the traditional humanistic process of looking at objects and providing well-founded, inferential statements about the past? Is it the immense speed of calculation? Is it the size of the data sets? Could it be that people might trust a computer in a way that they no longer trust intellectual forebears such as Morelli? We have engaged in academic research, and to the extent possible we have attempted to sever our inquiry from the “umbilical cord of gold” that links art, art history, and capitalism.[26] However, our community’s efforts in this domain must remain ongoing.
Research in this field should not only resist instrumentalization by the art market, but also offer perspectives on the serious stakes of what it would mean for human beings to vest computers with the sole responsibility for making attributions. Let us return once again to the thought experiment proposed in the introduction in order to make a specific point. To whom would the purchaser of the work purportedly by Leonardo make recourse in the event that the algorithm that produced a computational attribution of the painting to the master was proven to be faulty? The culpable might include: the computer programmer who developed the algorithm; the art historians (if any) who contributed to the construction of the algorithm; the lines of code in the algorithm; the piece of hardware that ran the algorithm; the auction house that countenanced the results; the buyer who believed the results; or the insurance agency that indemnified the results. In this day and age, it is not possible to imagine that, from such a list, anything other than a human being can be held accountable for such an error, and yet the decision would have been “made” by a machine.[27]
The difficulties faced in attempting to respond to this scenario point, we believe, to the unavoidable truth that computers have not earned their place in the conversation about artistic attribution because the linkage between “style” and authorship is not some immutable law of physics that is easily computable, but rather a humanistic commitment that requires human judgment. To make the practice of art attribution anything else besides a necessary collaboration between the physical object and trained human beings would require a fundamental reassessment of the practice, not simply its mechanization. We would like to argue that all such systems will currently fail to produce convincing results, not because they are incapable of mimicking the formal methods of art attribution, but because they have not (yet) been accepted as members of the community that make up attribution’s social practice, and cannot take responsibility for their conclusions. The computer can present the art historian with a series of statistical probabilities related to authorship, but it is important to note that simply choosing between a menu of options presented by the computer is not sufficient for producing a genuine attribution: art attribution cannot be reduced to decisionism; it requires judgment. We affirm that it is necessary for the community of experts called upon to make these judgments be made up of a broad and representative population of human experience that goes beyond the demographics that have traditionally dominated the space of attribution. It is precisely because human judgment is necessary to the process of attribution that, to make effective meaning, the community must reflect the breadth of the human experience.
While the results of our Phase One workflow were not designed to “give answers” to the problem of attribution, and were certainly not successfully responding to that imperative in any way, they have illuminated the ways that the mechanical processes of abstraction that, at first, seemed similar between our computational approaches and Morelli’s, were in fact utterly different, once contextualized within a social system. When they are presented as abstract entities on the page, Morelli’s Grundformen appear reified; through rhetorical force, they seem to take on an existence that was independent of the paintings in which they may be found. This presents an occasion for “verification” and cross-reference that can be described as quasi-computational and therefore formalizable. But the process of art attribution does not begin and end with the physical materiality of the painting.
Attributing a work of art has always been the work of narrative explication rather than mere reckoning about stylistic markers. It currently requires human judgment, expertise, and trust to become an effective truth in the world. Morelli, too, gestured towards this state of affairs by presenting his method in his text as a product of dialogue, as a conversation embedded in expertise and interpersonal persuasion. Morelli’s Grundformen existed to help his interlocutors see the paintings as he saw them, as a way to convince others of his judgments. Even granting that computational methods eventually rise to the level of being able to make believable assessments of stylistic markers, for their outputs to become truly effective, they would need to earn their place as full participants in the social process of art attribution, and human beings would need to delegate some power of judgment to them.

Coda

When we began this project, we took a leap of faith that we would uncover interesting areas of art-historical inquiry by computationally mechanizing Giovanni Morelli’s method of art attribution. While it was to be expected that our system could match Morelli’s own at the level of the workflow — extract a set of details and compare them — we did not anticipate how our work would show us the complexity of Morelli’s historical process at a more fundamental level. Where Morelli used his native human powers of abstraction and judgement to reduce old master paintings to a series of published line drawings, we found ourselves using the computer’s power of abstraction to create seemingly analogous, dimensionally-reduced representations. In so doing, we recognized that Morelli’s abstractions were effective because they worked in parallel with his human judgment in a way that mechanical processes could never replicate alone.
Our investigation has also suggested to us that there is a fundamental question that the field of art history has not yet addressed and that computers might actually help clarify: to what extent are personalized, identifiable artistic traces, however they are defined (forms, brushstroke, pigment, craquelure, etc...), registered at the level of pixels? Because, insofar as these traces are so registered, they will be legible computationally and therefore computers can identify them, perhaps even more effectively than human beings. We nevertheless argue that pixel-level information, on its own, would be insufficient to produce a credible attribution. That work is done as a social practice; it is a matter of an interaction between the physical traces of the past on the canvas and the trained judgment of human beings. It is an ecology that includes the physical history of the work of art, the archive of historical knowledge external to the work itself, and the particular training of each expert working in a community that has the power to make their findings real and effective in the world.
Classical, deterministic, electronic, digital computers are designed to produce certain results based on the input given and their own physical and logical programming. They only show you what you have set them up to show you — not necessarily what you wanted, needed, or even expected to see. Humanists natively operate in a different space than this, which might suggest that the world of computation cannot speak to the world of the interpretive expert. However, we argue here that this is not the case. Working with the computer, and with trained technologists, has provided us with the opportunity to gain important perspectives by allowing us to produce and manipulate an art-historical model of the world. A truism that has become particularly self-evident during the COVID-19 pandemic is that all computer models are, in some senses, wrong, but some models are useful.[28] Recognizing the ways that our computational model did and did not implement the older art-historical approach was, for us, like both looking through a microscope and a mirror onto the history of our discipline. This exercise forced us to slow down and look at Morelli’s method in minute detail, but in the end, it also reflected our own assumptions back to us, providing us with clear critical distance. Using digital techniques to think through even historically distant modes of art history indicates one path by which our field can do the ongoing work of reshaping its own practices from within while also reaching out and collaborating with other expert communities that see the world differently. Bridging those gaps, we believe, is crucial to developing art history for the digital age.

Notes

[1]  On the possible legal repercussions of Bambach’s opinion, see [Alberge 2019].
[2] Art history has moved on from the foundational ways that Morelli envisioned the historical operation of artistic production and the value of “style” within the field. There are numerous studies offering analyses of this historical shift. Among them, we find the following most informative: [Driscoll 2019] [Freedberg 1989] [Freedberg 2006] [Melius 2011] [Neer 2005] [Opperman 1990] [Scallen 2004] [Smith 2005] [Summers 1989] [Wollheim 1973] [Wollheim 1979] [Wollheim 1995] [Zerner 2014].
[3]  It should be noted that Morelli did not always use his own method. This has been noted by numerous scholars. While this is an important historical truth, our paper takes the computational implementation as a hypothesis and is not ultimately concerned with the question of whether or not Morelli obeyed his own method. On this question, see [Uglow 2014], [Seybold 2016], [Brewer 2018].
[4] For the original German text, see [Morelli 1890, 58].
[5] William Vaughan, the pioneering digital art historian, also linked his work to Morelli’s name in the late 1980s and early 1990s. His research on early digital imagery and their possible role in art historical study, while very valuable and notable in our field’s history, took up a fundamentally different research direction from ours, indeed as he notes, “...there is really no connection between [my] computer system and the method devised by the nineteenth century art historian” [Vaughan 1992, 8]. For more on this work, see [Vaughan 1987].
[6] This now-long-standing cultural attraction to the wonder and surprise of revealing the unseen using information in plain sight has been investigated in the context of digital computing, and even likened to a fetish [Chun 2011, 15–9]. Revealing the invisibly visible is also related to the concept of “clues” that Carlo Ginzburg discusses — using Morelli himself as evidence — in the context of the humanities [Ginzburg 1980].
[7] There are at least three layers to the meaning of “hand.” The first and most basic is what we might think of as the hand as a piece of semantic content; under discussion here is a human hand as rendered in paint and then reproduced photographically. Second, the claim to “hand” also refers to the fact (or inference) that Titian spread the paint on the picture surface using his own hand. Third, we have the hand as the historical construct of Titian’s style as constituted by the field of art history. It is important to disentangle these different layers of meaning in order to understand which aspects of Morelli’s method might be rendered computational. For more on the use of effective proxies in the digital humanities, see [Langmead and Newbury 2020].
[8] Portions of the Beazley archive are available online thanks to the University of Oxford’s Classical Art Research Centre, https://www.beazley.ox.ac.uk/, accessed 2 July 2020.
[9] “However, since the method proposed in this chapter is meant to be, to put it colloquially, an automation of the instinct of a skilled art historian rather than an objective method for identifying painters, artificial form sets by a skilled art historian are arguably more valuable [than other forms of data], since they better encode the intuition of the art historian” [Ryan 2009, 123].
[10] For example, efficiency and effectiveness are crucial to the work at the intersection of computer science and medicine, see [Register et al. 2019]. Computer science also prizes efficiency and effectiveness as an independent field, see [Shin 2019]. There are also works at the intersection of the humanities and computer science that prize these characteristics, however we do not believe our project is subject to those same demands, see [Zlabinger 2019].
[11]  In recent years, the concept of “slowness” has undergone something of a revival among mindful practitioners in art history and the academy more broadly, see [Berg and Seeber 2016], [Roberts 2013], [Tishman 2017].
[12] In the computing and information sciences, “novel results” are the coin of the realm, even serving as a major criterion for tenure and promotion. On this, see [Computing Research Association n.d.] and [Luzón Marco 2000]. On the alignment of priorities and reward structures in interdisciplinary work, see [Berg-Fulton et al. 2018].
[13] As detailed in the workplan of Paul Rodriguez, the lead technologist/consultant on this project, we planned to, “apply existing image processing techniques and procedures for segmentation and object border detection, current state-of-the art methods like convolution neural networks for object detection and classification, and open source tools for face detection and facial feature identification in order to divide [a dataset of] old-master paintings into a variety of segments” [Rodriguez 2018]
[14] It did not pass by us unnoticed that this second approach returns, in a sense, to looking at the painting as a whole which holds an interesting parallel to taking in the “general impression” of a painting to classify/attribute it — an approach directly derided by Morelli.
[15] Partial results from both Phase One and Phase Two can be found in [Rodriguez et al. 2020].
[16] For more on the dlib face detector, see [Ponnusamy 2018]. For more on OpenCV, see [Rosebrock 2018]. For more on the Google Vision Detect Faces API, see [Google 2020].
[17] As a technical side note, in this project, summarization was used to ensure that Histogram of Oriented Gradients was implemented in a manner that was rotational invariant [Rodriguez 2020c].
[18] “Relative” here means that, for a three-channel RGB image, the sum of all pixels in each channel will consistently add to 1 and the sum of all pixels at any given image location will also add to 1 across channels [Rodriguez 2019].
[19] For more on t-SNE, please see [van der Maaten n.d.]. For a useful, accessible talk on the subject, see [van der Maaten 2013]. For more on K-means clustering and machine learning, see [Trevino 2016].
[20] Carlo Ginzburg also famously addresses the “medical semiotics” of Morelli’s work, alongside that of Sir Arthur Conan Doyle and Sigmund Freud, in [Ginzburg 1980, esp 12ff].
[21] It may be more successful in the realm of graphic works, as suggested by [Elgammal et al. 2018].
[22] [Elgammal et al. 2018], [Ellis and Johnson Jr. 2019], and [Floridi 2018] all demonstrate possible roles that computers might take up in this ecosystem, and what may be at stake. Ahmed Elgammal is also a co-founder of Artrendex, LLC, a company that offers “Art Trend Analytics: Innovative AI technology for the Art Market,” which includes a product called “Art Verified by AI: AI for Authentication,” http://www.artrendex.com/.
[23]  For more on Orion Analytical, LLC, see http://orionanalytical.com/.
[24] Two of the three authors of a recent study that proposed to use automated analysis of drawings to determine their authenticity listed their primary affiliation as Artrendex, LLC, a startup that promises to use computers to verify works available on the market [Elgammal et al. 2018].
[25] For a truly cutting assessment of the anti-ethical disposition of the market for old master paintings, see [Burke 2020].
[26] We are indebted to Paul Jaskot for reminding us of this phrase originally used by the art critic Clement Greenberg [Jaskot 2019, 22]. For the original, see [Greenberg 1939].
[27] “The conclusion generalizes. We should not delegate to reckoning systems, nor trust them with, tasks that require full fledged judgment - should not inadvertently use or rely on systems that, on the one hand, would need to have judgment in order to function properly or reliably, but that, on the other hand, utterly lack any such intellectual capacity” [Cantwell Smith 2019, 126–7].
[28] The dictum, “All models are wrong, but some are useful,” is associated with the statistician George Box [Box 1979]. On the utility of the (wrong but useful) models used in the Covid-19 pandemic, see [Holmdahl and Buckee 2020]. For more on the utilities of computational models within the humanities, see [Langmead and Newbury 2020].

Works Cited

Agosti et al. 1993 Agosti, Giacomo, Maria Elisabetta Manca, Matteo Panzeri, and Marisa Dalai Emiliani (eds.). (1993). Giovanni Morelli e la cultura dei conoscitori: atti del convegno internazionale, Bergamo, 4-7 giugno 1987. Bergamo: P. Lubrina.
Alberge 2019 Alberge, Dalya. (2019). “Leonardo da Vinci expert declines to back Salvator Mundi as his painting.” The Guardian, 2 June. Accessed 30 June 2020: https://www.theguardian.com/artanddesign/2019/jun/02/leonardo-da-vinci-expert-carmen-bambach-says-she-wont-back-salvator-mundi-as-his-painting
Anderson 2020 Anderson, Jaynie. (2020). The Life of Giovanni Morelli in Risorgimento Italy (Milan: Officina Libraria).
Bambach 2019 Bambach, Carmen (2019). Leonardo da Vinci Rediscovered (New Haven and London: Yale University Press).
Barni 2016 Barni, Daniele. (2016). Lo sguardo della critica: i conoscitori d'arte in Italia tra XIX e XX secolo (Turin: Cartman).
Beazley 1922 Beazley, John Davidson (1922). “Citharoedus,” The Journal of Hellenic Studies 42, 70-98.
Beazley 1942 Beazley, John Davidson (1942). Attic Red-Figure Vase-Painters, 3 volumes (Oxford, Clarendon Press).
Beazley 1956 Beazley, John Davidson (1956). Attic Black-Figure Vase-Painters (Oxford, Clarendon Press).
Beazley 1971 Beazley, John Davidson (1971). Paralipomena: Additions to Attic Black-Figure Vase-Painters and to Attic Red-figure Vase-painters (Oxford: Oxford University Press).
Berg and Seeber 2016 Berg, Maggie and Barbara K. Seeber. (2016).The Slow Professor: Challenging the Culture of Speed in the Academy (Toronto, University of Toronto Press).
Berg-Fulton et al. 2018 Berg-Fulton, Tracey, Alison Langmead, Thomas Lombardi, David Newbury, and Christopher Nygren. (2018). “A Role-Based Model for Successful Collaboration in Digital Art History.” International Journal for Digital Art History, 3, 152-80. DOI: https://doi.org/10.11588/dah.2018.3.34297.
Box 1979 Box, George. (1979). “Robustness in the Strategy of Scientific Model Building: Technical Report #1954.” Madison, Wisconsin: Mathematics Research Center, University of Wisconsin-Madison. https://apps.dtic.mil/dtic/tr/fulltext/u2/a070213.pdf, accessed June 29, 2020.
Brewer 2018 Brewer, John. (2018). “Giovanni Morelli,” https://brewersblog.org/2018/01/28/giovanni-morelli/, accessed 11 May 2020.
Burke 2020 Burke, Jill. (2020). In “Decolonizing Art History,” edited by Catherine Grant and Dorothy Price. Art History, 43, 17-18. DOI: https://doi.org/10.1111/1467-8365.12490.
Caglioti et. al. 2018 Caglioti, Francesco, Andrea De Marchi, and Alessandro Nova (eds.). (2018). I Conoscitori Tedeschi Tra Otto E Novecento, edited by, (Milan: Officina Libraria)
Cantwell Smith 2019 Cantwell Smith, Brian. (2019). The Promise of Artificial Intelligence: Reckoning and Judgment. Cambridge and London: MIT Press.
Carrier 2003 Carrier, David. (2003). “In Praise of Connoisseurship.” The Journal of Aesthetics and Art Criticism, 61, 159-69.
Chun 2011 Chun, Wendy Hui Kyong. (2011). Programmed Visions: Software and Memory (Cambridge: MIT Press).
Computing Research Association n.d. Computing Research Association. (n.d.). “Evaluating Computer Scientists and Engineers for Promotion and Tenure.” https://cra.org/resources/best-practice-memos/evaluating-computer-scientists-and-engineers-for-promotion-and-tenure/, accessed June 29, 2020.
Davis 2011 Davis, Whitney. (2011). A General Theory of Visual Culture (Princeton: Princeton University Press).
Driscoll 2019 Driscoll, Eric. (2019). “Beazley’s Connoisseurship: Aesthetics, Natural History, and Artistic Development,” in Dossier. Corps antiques: morceaux choisis, edited by Florence Gherchanoc and Stéphanie Wyler (Paris: Éditions de l’École des hautes études en sciences sociales): 101-120, http://books.openedition.org/editionsehess/13689, accessed July 1, 2020.
Elgammal et al. 2018 Elgammal, Ahmed, Yan Kang, and Milko Den Leeuw. (April 2018). “Picasso, Matisse, or a Fake? Automated Analysis of Drawings at the Stroke Level for Attribution and Authentication.” Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 42-50. https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17356/15669, accessed June 29, 2020.
Ellis and Johnson Jr. 2019 Ellis, Margaret Holben and C. Richard Johnson Jr. (2019). “Computational Connoisseurship: Enhanced Examination Using Automated Image Analysis.” Visual Resources, 35, 125-140. DOI: https://doi.org/10.1080/01973762.2019.1556886.
Floridi 2018 Floridi, Luciano. (2018). “Artificial Intelligence, Deepfakes and a Future of Ectypes.” Philosophy & Technology, 31, 317-321.
Freedberg 1989 Freedberg, Sydney J. (1989). “Some Thoughts on Berenson, Connoisseurship, and the History of Art.” I Tatti Studies in the Italian Renaissance, 3, 11-26.
Freedberg 2006 Freedberg, David. (2006). “Why Connoisseurship Matters,” in Munuscula amicorum Contributions on Rubens and his Colleagues in Honour of Hans Vlieghe, edited by K. Van der Stighelen (Turnhout: Brepols), 29-43.
Ginzburg 1980 Ginzburg, Carlo. (Spring 1980). “Morelli, Freud and Sherlock Holmes: Clues and Scientific Method,” translated by Anna Davin. History Workshop, 9, 5-36.
Golden 2004 Golden, Andrea. (2004). “Creating and Re-Creating: The Practice of Replication in the Workshop of Giovanni Bellini,” in Giovanni Bellini and the Art of Devotion, edited by Kasl, Ronda. (Indianapolis: Indianapolis Museum of Art), 91-127.
Google 2020 Google, “Face Detection,” Firebase, last updated April 8, 2020. https://firebase.google.com/docs/ml-kit/detect-faces, accessed June 29, 2020.
Grabar 1988 Grabar, Oleg. (1988). “Between Connoisseurship and Technology: A Review.” Muqarnas, 5, 1-8.
Greenberg 1939 Greenberg, Clement. (Fall 1939). “Avant-Garde and Kitsch.” Partisan Review, 6, 34-49.
Hinojosa 2009 Hinojosa, Lynne Walhout. (2009). “The Connoisseur and the Spiritual History of Art: Morelli and Berenson,” in idem, The Renaissance, English Cultural Nationalism, and Modernism, 1860–1920 (New York: Palgrave Macmillan), 89-111.
Holmdahl and Buckee 2020 Holmdahl, Inga and Caroline Buckee. (May 15, 2020). “Wrong but Useful — What Covid-19 Epidemiologic Models Can and Cannot Tell Us.” The New England Journal of Medicine. DOI: https://www.nejm.org/doi/full/10.1056/NEJMp2016822.
Jaskot 2019 Jaskot, Paul. (2019). “Digital Art History as the Social History of Art: Towards the Disciplinary Relevance of Digital Methods.” Visual Resources, 35, 21-33.
Kemp 2019 Kemp, Martin. (2019). Leonardo by Leonardo (New York: Callaway).
Langmead and Newbury 2020 Langmead, Alison and David Newbury. (2020). “Pointers and Proxies: Thoughts on the Computational Modeling of the Phenomenal World.” In The Routledge Companion to Digital Humanities and Art History, edited by Kathryn Brown. London: Routledge, 358-373.
Lewis 2019 Lewis, Ben (2019). The Last Leonardo: The Secret Lives of the World’s Most Expensive Painting. London: Ballantine.
Luzón Marco 2000 Luzón Marco, José. (2000). “The Construction of Novelty in Computer Science Papers.” Revista Alicantina de Estudios Ingleses, 13, 123-140. https://core.ac.uk/download/pdf/16358965.pdf, accessed June 29, 2020.
Melius 2011 Melius, Jeremy. (2011). “Connoisseurship, Painting, and Personhood,” Art History, 34, 288-309
Morelli 1890 Morelli, Giovanni. (1890). Kunstkritische Studien über italienische Malerei; die Galerien Borghese und Doria Panfili in Rom. (Leipzig F.A. Brockhaus).
Morelli 1893 Morelli, Giovanni. (1893). Italian Painters: Critical Studies of Their Works. By Giovanni Morelli (Ivan Lermolieff). The Borghese and Doria-Pamphili Galleries in Rome, translated by Constance Jocelyn Ffoulkes (London: John Murray).
Neer 2005 Neer, Richard. (2005). “Connoisseurship and the Stakes of Style,” Critical Inquiry, 32, 1-26.
Neilson 2019 Neilson, Christina. (2019). Practice & Theory in the Italian Renaissance Workshop: Verrocchio and the Epistemology of Making Art (Cambridge: Cambridge University Press).
Opperman 1990 Opperman, Hal. (1990). “The Thinking Eye, the Mind That Sees: The Art Historian as Connoisseur.” Artibus Et Historiae, 11, 9-13.
O’Malley 2007 Michelle O’Malley. (2007). “Quality, Demand, and the Pressures of Reputation: Rethinking Perugino.” Art Bulletin 89, 674-693.
Ponnusamy 2018 Ponnusamy, Arun. (April 17, 2018). “CNN-Based Face Detector from dlib.” Towards Data Science. https://towardsdatascience.com/cnn-based-face-detector-from-dlib-c3696195e01c, accessed June 29, 2020.
Register et al. 2019 Register, Shilpa, Michelle Brown, and Marjorie Lee White. (2019). “Using Healthcare Simulation in Space Planning to Improve Efficiency and Effectiveness within the Healthcare System,” Health Systems, 8, no. 3 (2019): 184-189, https://doi.org/10.1080/20476965.2019.1569482.
Roberts 2013 Roberts, Jennifer. (2013). “The Power of Patience,” Harvard Magazine (November-December, 2013), 40-43. https://harvardmagazine.com/2013/11/the-power-of-patience, accessed June 29, 2020.
Rodriguez 2018 Rodriguez, Paul. (March 8, 2018). “ECSS Workplan: The ‘Morelli Machine:’ A Proposal Testing a Critical, Algorithmic Approach to Art History,” n.p.
Rodriguez 2019 Rodriguez, Paul. (December 18, 2019). Internal Team Report.
Rodriguez 2020a Rodriguez, Paul. (January 12, 2020). Internal Team Report.
Rodriguez 2020b Rodriguez, Paul. (January 17, 2020). Email Correspondence with the Team.
Rodriguez 2020c Rodriguez, Paul. (March 9, 2020). Email Correspondence with the Team.
Rodriguez et al. 2020 Rodriguez, Paul, Alan Craig, Alison Langmead, and Christopher J. Nygren. (2020). “Extracting and Analyzing Deep Learning Features for Discriminating Historical Art.” In Proceedings of the Practice & Experience in Advanced Research Computing Conference [PEARC] 2020 (Portland, OR) [in publication].
Rosebrock 2018 Rosebrock, Adrian. (September 24, 2018). “OpenCV Face Recognition.” Pyimagesearch. https://www.pyimagesearch.com/2018/09/24/opencv-face-recognition/, accessed June 29, 2020.
Ryan 2009 Ryan, Adrian John. (January 2009). “Computer Aided Techniques for the Attribution of Attic Black-Figure Vase-Paintings Using the Princeton Painter as a Model.” PhD dissertation, University of Kwazulu-Natal, South Africa.
Scallen 2004 Scallen, Catherine B. (2004). Rembrandt, Reputation, and the Practice of Connoisseurship (Amsterdam: Amsterdam University Press).
Seybold 2016 Seybold, Dietrich. (2016). The Giovanni Morelli Monograph, 2016, http://www.seybold.ch/Dietrich/TheGiovanniMorelliMonograph, accessed 11 May 2020.
Shin 2019 Shin, Shin-Shing. (2019). “Empirical Study on the Effectiveness and Efficiency of Model-Driven Architecture Techniques.” Software & Systems Modeling, 18, 3083-3096. DOI: https://doi.org/10.1007/s10270-018-00711-y.
Smith 2005 Smith, Tyler Jo. (2005). “The Beazley Archive: Inside and Out,” Art Documentation: Journal of the Art Libraries Society of North America, 24, 22-25.
Subramanian 2018 Subramanian, Samanth. (June 15, 2018). “How to Spot a Perfect Fake: The World’s Top Art Forgery Detective.” The Guardian. https://www.theguardian.com/news/2018/jun/15/how-to-spot-a-perfect-fake-the-worlds-top-art-forgery-detective, accessed June 29, 2020.
Summers 1989 Summers, David. (1989). “‘Form,’ Nineteenth-Century Metaphysics, and the Problem of Art Historical Description,” Critical Inquiry, 15, 372-406.
Syson 2011 Syson, Luke, ed. (2011). Leonardo da Vinci: Painter at the Court of Milan (London: National Gallery of Art).
Tishman 2017 Tishman, Shari. (2017). Slow Looking: The Art and Practice of Learning Through Observation. Milton: Routledge.
Trevino 2016 Trevino, Andrea. (December 6, 2016). “Introduction to K-means Clustering.” Oracle AI and Data Science Blog. https://blogs.oracle.com/datascience/introduction-to-k-means-clustering, accessed June 29, 2020.
Uglow 2014 Uglow, Luke. (2014). “Giovanni Morelli and his Friend Giorgione: Connoisseurship, Science and Irony.” Journal of Art Historiography 11 (December 2014): https://arthistoriography.files.wordpress.com/2014/11/uglow.pdf, accessed 2 July 2020.
Vaughan 1987 Vaughan, William. (1987). “The Automated Connoisseur: Image Analysis and Art History,” in History and Computing, edited by Peter Denley and Deian Hopkin (Manchester: Manchester University Press), 215-221.
Vaughan 1992 Vaughan, William. (1992). “Automated Picture Referencing: A Further Look at ‘Morelli,’” Computers and the History of Art 2, no. 2, 7-18.
Williams 2017 Williams, Robert. (2017). Raphael and the Redefinition of Art in Renaissance Italy. Cambridge: Cambridge University Press, 2017.
Wollheim 1973 Wollheim, Richard. (1973). “Giovanni Morelli and the Origins of Scientific Connoisseurship.” In On Art and the Mind: Essays and Lectures, by Richard Wollheim. London: Allen Lane, 177-201.
Wollheim 1979 Wollheim, Richard. (1979). “Pictorial Style: Two Views,” in The Concept of Style, edited by Berel Lang (Philadelphia: University of Pennsylvania Press), 129-145.
Wollheim 1995 Wollheim, Richard. (1995). “Style in Painting,” in The Question of Style in Philosophy and the Arts, edited by Caroline Van Eck, James McAllister and Renée van de Vall (Cambridge: Cambridge University Press, 1995), 37-49.
Zerner 2014 Henri Zerner. (2014). “What Gave Connoisseurship Its Bad Name? (1987),” reprinted in Historical Perspectives in the Conversation of Works of Art on Paper, edited by Margaret Hoben Ellis (Los Angeles: The Getty Conservation Institute), 59-61.
Zlabinger 2019 Zlabinger, Markus. (2019). “Improving the Annotation Efficiency and Effectiveness in the Text Domain.” In Advances in Information Retrieval, ECIR 2019, edited by L. Azzopardi, et al. Cham: Springer, 343-347. DOI: https://doi.org/10.1007/978-3-030-15719-7_46.
van der Maaten 2013 van der Maaten, Laurens. (June 24, 2013). “Visualizing Data Using t-SNE.” Google Tech Talks, YouTube. https://youtu.be/RJVL80Gg3lA, accessed June 29, 2020.
van der Maaten n.d. van der Maaten, Laurens. (n.d.). “t-SNE.” Laurens van der Maaten. https://lvdmaaten.github.io/tsne/, accessed June 29, 2020.
2021 15.1  |  XMLPDFPrint