Case Study 1: Material Form in Film and Media Studies (Burges)

Questions of material form are central in the film and media studies classes I teach for the College of Arts, Sciences, and Engineering at the University of Rochester, including the two I discuss here: "The Poetics of Television" and "Introduction to Media Studies". In "The Poetics of Television", for example, we spend significant time studying the different ways in which the episode is a form of inscription that organizes the audiovisual experience of television in narratively open and closed ways. In "Introduction to Media Studies," we discuss television not only from this narrative perspective, but also from the perspective of television as a historically variable technology for transmitting sounds and images onto a screen that was, for many decades, primarily part of the TV set. These are both material to the form of television, with the latter especially providing the specific audiovisual means by which television mediates and materializes narrative, information, and advertising for its viewers. In my classes, the question of material form — of how a medium is a matter of form — is not reducible to solely inquiring into these means in order to secure that which is, to recall Clement Greenberg , irreducibly exclusive to it. It instead involves pursuing lines of inquiry with students in which we explore the specificity of a range of media — from television and film to poetry and song — through the shifting constellations of qualities constrained and enabled by diverse audiovisual means in the first place .

Digital annotation in Mediate indelibly contributes to this pursuit, especially through the schemas that provide the basis of the highly collaborative — as we will show — marking that students do over the course of a semester in classes such as "The Poetics of Television" and "Introduction to Media Studies." The schemas we have designed so far try to capture the constellations of qualities that make up any medium one might annotate in Mediate. In "The Poetics of Television," the schemas were designed around aural, visual, and narrative qualities in order to show how sound, image, and story are respectively constructed on TV.

Visual Schema.

A sample of markers generated using the Visual Schema on a scene from Buffy the Vampire Slayer, Season Four.

Narrative Schema.

A sample of markers generated using the Narrative Schema on a scene from Game of Thrones, Season Three.

The schemas for "Introduction to Media Studies," were designed with comparison in mind, so one focused on a set of markers for annotating poems read aloud by their authors, the other on a set of markers for annotating pop songs by individual singers and bands.

Poetry Schema.

A sample of markers and observations generated using the Poetry Schema on a video of Tracy K. Smith reading

Narrative Schema.

A sample of markers and observations generated using the Narrative Schema on Beyoncé's

Over the course of a semester, students in these two classes worked collaboratively in groups of four to six people to annotate on the basis of the schema or schemas that group was assigned, building toward long papers in which they explored a wide range of topics through distant and close readings performed in writing and through visualizations. Regardless of the topic, these papers almost universally exhibited a deep knowledge of the material form of the audiovisual medium under study; the specific interplay of sight and sound embodied by TV, for instance, became "second nature" to one student, "so much so that when I watch TV now I automatically mark the episode in the back of my mind" . This is the result of what the students characterize as the "immersive" dimension of Mediate in which time spent marking, however "irritating" and "monotonous" and "tedious" in its slowness over the semester, generates a profound concept of the qualities that give material form to their audiovisual experience . This is most palpable when one looks more closely at final projects students completed, in which their collaborative efforts yielded a remarkable level of quantitative data and qualitative observation worth understanding in greater nuance.We believe it is important not only to include these specifics, but also to attribute the papers to the students themselves. Moreover, as needed, we obtained permission from various students in the groups cited from "The Poetics of Television" and "Introduction to Media Studies" to share their work; in the latter, we conducted focus groups where students also signed off on us sharing their feedback and ideas. The comments from students in "Experiments at the Edges of 20th Century Music" and "Linguistics and Advertising," however, have remained anonymous since we had less of this infrastructure in place.

In an essay for "The Poetics of Television" entitled "The Formal Nucleus of Television, and Its Subservience to Narrative" , the students argued that dialogue is a key element of the "formal nucleus" of TV by exploring the nexus of sound and story in four historically and generically variable series defined by open narration (Game of Thrones, Dark Shadows, Guiding Light, and Robotech). On the basis of hundreds of markers the students in this group annotated, they argue that dialogue is an elementally generative feature of the "aural design" of television that "advance[s] the narrative progression of an episode." This is due to dialogue allowing "all details pertinent to the comprehension of the narrative, in terms of both plot and character, to be enumerated in explicit, unequivocal, and economical terms." This group further contends that the television camera often obeys the human voice, suggesting that visual design flows from aural design, for instance, on the basis of the 54 on-offscreen and off-onscreen shifts of sound vis-à-vis the image and diegesis that the group marked in the infamous Red Wedding scene of Game of Thrones while annotating in the Aural Schema. While both the visible and the audible are subservient to narrative in the argument this group's paper makes, it nonetheless richly charts the interplay of sight and sound within storytelling on TV, revealing how that mediated interplay — that audiovisuality — lets narrative take material form on screen.

"Introduction to Media Studies" similarly attuned students to material form. But rather than focusing on one audiovisual medium to achieve this end, as in "The Poetics of Television," I used a comparative approach in which I asked students to work in groups to annotate poetry and pop songs to which they listened closely and repeatedly in Mediate. Students were not allowed to pull print versions of the poems they were annotating on the basis of oral renditions by their authors, forcing them to use their ears to grasp the differences between poems and songs marked using schemas developed for each of these genres. What one group observed about those differences will sound obvious: while rhythmically structured and lineated language is the medium of poetry, pop songs are much more musical in their means, depending on instrumentation, chord progression, beat groupings and so on .

But what is less obvious is how the students in this group came to experience this difference because Mediate introduced annotation into audition. In annotating, they heard what was material to the form of poetry and pop songs as a fixed and motivated structure of aural notation. The songs went from being an internal experience of sound and music to appearing as an externalized — because now annotated — form of audiovisual inscription. Thus the long paper this group produced focused less on material form in favor of trying to pinpoint what constellation of audible qualities inscribe what they call "intensity." For this group, intensity refers to how "affective response" and "aesthetic emotion" in a mediated genre of sound turns upon medium-specific features, as in their LP record player-inspired visualizations of two songs, Troye Sivan's "Wild" and Taylor Swift's "This Love." Audiovisualities in their own right, these experimental data visualizations show how distinct kinds of vocal stress, chord changes, back-up singing, and instrumentation not only mark the form of these songs, but also form the possibility of having a musically "intense" reaction to them as well.

Experimental Visualization of Formal and Intensity Markings for Troye Sivan’s "Wild."

Experimental Visualization of Formal and Intensity Markings for Taylor Swift’s "This Love."

In classes such as "The Poetics of Television" and "Introduction to Media Studies," digital annotation in Mediate enables students to work together to see and hear the material form of a range of audiovisual media. It is important that they are working together, collaborating to annotate such that both within and across their groups they are able to explore audiovisuality in a collective way that has both quantitative and qualitative effects. On the one hand, the quantitative dimension of their collaborative efforts is visible in the plenitude of markers that each group generates as a working collective, and in how they draw on the audiovisual data produced by other groups marking in the same class to understand the annotation that has occurred in a given group. On the other hand, the repeated marking required to yield thousands of data points they can share with one another engenders a quality of description and interpretation that shows how close they understand the material forms of film, television, poetry, and pop songs from multiple audiovisual angles of seeing and hearing. As we have more fully argued elsewhere , however, that "truth" depends upon the discussions that often emerge over how to collectively define and collaboratively mark a unit of analysis within and across groups as they annotate features of material form; these discussions over how to see and hear, though, only further shore up both that no medium is a natural given, and that every audiovisual experience is mediated.

Case Study 2: The Cultural Contexts of Music History (Mueller)

Although connected institutionally, Eastman School of Music (ESM) is quite different from the rest of the university in terms of its student population, educational goals, and curriculum. All of ESM's approximately 500 undergraduate students are music majors, with a primary focus on Western classical music.At the undergraduate level, ESM also offers robust degree programs in jazz performance and jazz composition. Graduate degree tracks include music leadership, conducting, early music, film composition, opera, music theory, ethnomusicology, and musicology. Our students are some of the best young musicians in the world, meaning that most view their classroom activities through the lens of their future careers in performance, composition, and pedagogy. To even gain admittance, they need to have remarkable expertise and years of specialized training in a tradition built around individual composers and master performers. So, while these students bring a strong passion for music into the classroom, doing academic work often forces them to confront viewpoints and approaches that are frequently taken as natural rather than culturally constructed.

Most of the traditions represented at ESM are heavily reliant on musical notation, a highly advanced system of written symbols that has, over many centuries, enabled the development and circulation of music that originated in Western Europe. In many respects, the very presence of notation constitutes the tradition . Musical notation can also be understood as a form of audiovisual inscription that communicates specific information about both how to perform music and also how an individual piece functions melodically, harmonically, rhythmically, and formally . Reading music, as we would say, is a presumed skill that students rely upon as they move through the robust series of classes in music theory and history, both of which differently emphasize score-based analysis of internal (within a piece) and external (within a tradition) musical features. As a result, students are very comfortable working with written music, as well as the specialized language used to describe it. Still, the culture surrounding classical music has calcified certain perspectives, especially on what it means to do analysis. For many students, music analysis is too often conceived of as an action solely in visual terms, rather than an act that takes place in a far more expanded audiovisual realm.

In Fall 2019, I introduced Mediate into my "Experiments at the Edges of 20th Century Music," a required course within the undergraduate music history core. My goal was to foreground listening, rather than reading, as the primary means of analysis. As exceptional performers already, my students listen with an expertly attuned understanding of musical performance. But while they continually analyze music while listening, they do not always listen historically — that is, attend to how specific musical moments express historically contingent beliefs about culture, society, or the many processes of music making. There is no inherent problem with their modality of listening. However, it is not always congruous with my major pedagogical goal: to examine and interpret how music and its written tradition are both heavily mediated creations, dependent on historically situated actors with different investments and values. By asking students to pay attention differently, Mediate not only foregrounds listening but also asks students to translate that listening into specific observations represented visually. In effect, this reverses the standard audiovisual direction of music making. Rather than move from visual inscription (notation) to aural expression (performance), Mediate renders what is heard into specific visual markings of that performance. Unsettling the assumed relationships between the visual and the aural — by putting listening first — encourages alternative viewpoints to come into the forefront of the analytical process.

Inspired by previous uses of Mediate in the classroom, I had students complete a semester-long collective analysis project. Our work began before I introduced the Mediate platform by continually asking students during class discussions and daily responses to listen through a series of five interrelated questions oriented towards cultural contexts:

Who or what shaped this particular music or performance?

What would it be like to perform this music?

What are the musical materials used in the creation of this piece?

How does this musical material transform?

What do you think the creators were trying to say or accomplish with this music?I adapted these questions from composer David Kirkland Garner (University of South Carolina).

For their Mediate project, students organized into groups of two to four. Each student within those groups picked one or more of these questions and marked up their audio in Mediate from those perspectives. I developed specific markers for each question to aid in this process, but also encouraged students to develop others to meet their specific needs.

Categories of Analysis Schema.

As they listened, students would mark everything from the seemingly obvious — what they might otherwise notice without thinking — to those details obscured by the rapid unfolding of any time-based art. After creating several hundred markers, each group began to decipher their efforts and develop a thesis for their written analysis with the same five questions again providing a road map. The class went through this process in two different iterations. First, all groups analyzed the first movement of William Grant Still's Symphony No. 1. Then, each group picked a piece of music or specific performance from a given list of artists and composers.

A sample of markers and observations generated using the Categories of Analysis Schema on Ida Handel performing Part One of the Carmen Fantasy.

Two general observations emerged out of the individual reflections written at the end of the semester. First, the slow, sometimes-tedious markup process requires active listening. Many students discussed how they came to notice subtle details and complexities precisely because the marking up process made "passive" or "casual" listening impossible. The intensity involved with repeated listening did not always change their initial opinions, but rather increased the precision and specificity of their observations. One student remarked with surprise about "how much could happen within one tiny second." Second, the processes of collective listening encouraged individuals to consider multiple vantage points. Collaboration through group work is not always smooth or easy, and it is sometimes unpopular. But by learning from or being challenged by their colleagues — perhaps even by becoming a "cultural context" in miniature — many students reported that the dialogic experience enabled them to make connections that were perhaps not obvious to them before.

The written work of each group also proved how valuable doing analysis away from the score could be. Many groups wrote about meaningful moments that would have otherwise remained hidden by only looking at the written notation — the particular use of vibrato, the background noise in a recording, a reoccurring timbre, or the use of space or a particular texture in the orchestration. Through the analysis of seemingly discrete details in relation to the background of the composer or other cultural influences, students then found ways to relate what happens musically to what that performance might mean more broadly, which is to say historically. As one student commented, the analysis provided a way to understand how music functioned as an interconnected web of historical events, musical influences, and experiences of "real life people." In comparison with previous semesters, the written work of the students in this class was at once more precise and bolder in their conclusions.

The slow and collective process of listening through Mediate allows students to re-situate their otherwise expert ears towards music as a form of audiovisual inscription. Music is a time-based art that exists in performance, yet it nevertheless remains heavily dependent on the visual realm. The specialized language used in traditional forms of analysis — a Neapolitan sixth chord, for one example — describes both what music sounds and looks like. As a digital platform that creates a method for analysis, Mediate makes the audiovisualities of music clear. Musical culture is and has always been an audiovisual culture as well, and new possibilities surface for students about this fact when they experience music through Mediate.

Case Study 3: Social Function in Linguistics (Armoskaite)

The Department of Linguistics in the College of Arts, Sciences, and Engineering at the University of Rochester, while a part of the social sciences, is a hub of interdisciplinary research with ties to other departments, including Music (with a focus on perception and production of sounds), Brain and Cognitive Science (with a focus on meaning and language processing), Anthropology (with focus on culture and language intersections), and Psychology (with a focus of child language acquisition). As a field, linguistics covers a vast number of topics and methodologies, hence it is impossible to provide a general description that would fit the range. For the purposes of this case study, it will suffice to state (i) that linguistics focuses on of the makeup of grammar, which is a set of sub-systems of sound, form and meaning; (ii) and that these subsystems are used for communication, a function that interacts with social conventions and societal values, a.o. .

My course "Language and Advertising," requires students to consider language use in the context of audiovisual marketing against the backdrop of current social trends. While Linguistics does not have a Business or Marketing track, the course routinely is taken by business majors and consistently is a popular elective among other non-Linguistics majors. I face a diverse group of students with different backgrounds, skills, and assumptions, though united by the three common denominators. First, the ubiquity of advertising in their lives gives them a false sense of familiarity and assumed knowledge of the medium; second, they want to learn the nuts and bolts of the advertising machine; and third, they possess limited knowledge of linguistics. Over the course of the semester, they learn that the social function of language in advertising is to manipulate, with commercial media, working to apply psychological pressure through a mode of audiovisuality that depends on influencing our emotions and circumventing our rational mind, a.o. .

Situating a familiar medium — advertising — within a likely unfamiliar field — linguistics — necessarily slows down the students. They learn to analyze the linguistic components that, to call back to the film and media studies case above, "make" the medium. The material form that interests a linguist, however, includes not only sounds and images of the kind that interest Burges and Mueller in their courses, but also the very structure of language as humans speak, read, and hear it. For example, each of the following posters can be deciphered in the linguistic terms of sound, form, and meaning, which shows that even print advertising functions as an audiovisual medium.

The American Red Cross "Missing Types" campaign (2018) presents a sound-based puzzle for the viewer to acoustically fill in, whether in silence or out loud: all the missing elements are vowels. The viewer goes in search of those vowels, enjoying a language game that depends on visual absence engendering audio presence. And because the vowels are associated with types of blood, this audiovisual play becomes a linguistic mechanism for soliciting blood donations.

The Snickers "Satisfectellent" (TBWA 2007) advertisement plays upon another element of grammar, namely, derivation of words. In this case, a word that is possible, but does not exist, is created. The novelty of the coined word is the striking — and strikingly audiovisual — feature of the advertisement: the joy of recognition of the brand of the snack is fused with the unexpectedness of the word.

Finally, the Greenpeace "Straws Suck: Gull" (Rethink 2018) advertisement exploits the shades of meaning of the verb "to suck." The painful visual is certainly not the first association we have about sucking through straws, which we may think about in sonic and/or tactile terms primarily. But the unexpected visual connotation is meant to shock us into changing our habits of consumption.

However, language in print advertising engenders audiovisual experience in a far more static way than the advertising that flows across our many screens as moving images and dynamic sounds. The latter contains hundreds of speech patterns in addition to innumerable cues for our ears and eyes. Superimposed on moving images, these patterns and cues come at a viewer at a speed that barely allows them to register the component parts, let alone perform a thorough analysis. Mediate creates a space for such analysis external to the ephemerally immediate modes in which we normally consume commercial media. In so doing, it makes legible that language plays a constitutive part in the interplay of sight and hearing — that all three of these human abilities contribute to the social function of manipulation that is the raison d'etre of often brilliant acts of commodified audiovisuality that want us to buy or buy into something. Mediate gets students to treat the audiovisuality of advertising as a constellation of elements variously linguistic, optical, and aural — as an object of analysis.

This takes time over the course of the semester. In "Language and Advertising" (Fall 2018 and Fall 2019), I spent three weeks teaching students the fundamentals of linguistics using similar examples to the print advertisements discussed above. Building on this print unit, we then turned to the challenge of analyzing commercials using Mediate. The students began by analyzing an ad without the use of Mediate. Then I modeled the slow and detailed analysis conducted through Mediate by providing them with samples of my own marked-up commercial. We discussed their observations, along with my own, which gently lets them understand how many details they've missed in their own observations. The students were then trained on the platform and introduced to the schema over two class sessions. After that initial introduction, the students took charge of their own learning with only a light editorial supervision.

Circumventing the top-down didacticism of the traditional lecture, Mediate allowed the students to immerse themselves into the material on their own. Working in groups, they were tasked with selecting and analyzing at least two video ads, splitting up the work of marking (what we call "coding" in the field of linguistics) amongst themselves. Rather than a traditional linguistic analysis of a video ad that might include a transcription of the text divorced from the audio and visual cues, Mediate facilitated a holistic approach to the interplay and interdependencies of the audiovisualities commercial media employ.

A sample of markers and observations generated using the Linguistics Schema on Cadillac's "The Future is Here" 60 second spot.

In slowing down their observations, they are forced to think about each element in the totality of the advertisement such that it emerges as a linguistic object of audiovisual analysis, with the manipulative properties of this object becoming ever clearer as its social function over the time they spend in Mediate. As one student noted, "there is no escape but to analyze." At the end of the semester, the groups presented their analyses of commercial media, with the entire class responding. This collective response included debates over how each group defined certain linguistic, audio, and visual units of analysis, and about the conclusions about manipulation in commercial media that each group reached. These discussions — aided by the carefully coded examples in Mediate — were paramount in helping students build an applied understanding of the social function of advertising from an audiovisual angle grounded in linguistics as an interdisciplinary field.

Despite their engagement and increased capacity with audiovisual analysis through Mediate, there still is room for greater interdisciplinary collaboration in "Language and Advertising", especially in light of the cross-disciplinary sense of audiovisuality we are advancing in this article. In my course, I welcome film experts as occasional invited speakers. But through my discussions with Burges and Mueller, I have realized that more sophisticated ways of tuning into the linguistic, visual, and sonic patterns would offer further opportunities to explore the range and means of consumer manipulation. For example, thus far, I have left out musical aspects completely as I lack relevant training. In the future iterations of this course, I think about the potential for more nuanced analysis if I could harness Mueller's expertise in helping students define the auditory components, if Burges could work with students to delve further into the visual form — that is, if students benefitted not only from their collective defining through marking, but also the collective expertise of a more intentionally cross-disciplinary approach to teaching and research.