Computational art Explorations of Linguistic Possibility Spaces: comparative translingual close readings of Daniel C. Howe’s Automatype and Radical of the Vertical Heart 忄

This essay discusses two distinct but related works by Daniel C. Howe. Automatype, 2012-16, is, in the words of the artist’s description, “a networked installation (composed of analog TV monitors and RaspberryPIs running custom software) that explores the creation of aesthetic, linguistic meaning via anticipation, juxtaposition, and association. The algorithm at the heart of the work continually computes the minimum number of substitutions required to transform each valid English word into the next, deriving a near infinite number of combinations of words and phrases, letter-by-letter, substitution-by-substitution.”[Howe 2012-16a] The algorithm running on each of the (usually nine) displays in an Automatype installation is the same, and close reading here effectively focuses on this constituent of the work. Radical of the Vertical Heart 忄, 2019-21, the most recent work in what Howe considers a series,

searches the Chinese lexicon by repeatedly making minimal changes (stroke-by-stroke) to the sub-components of characters, in order to arrive at a new word. Rather than evaluating letters, as in alphabetic machines, this logographic [sic] reader analyses the radicals, components and strokes of characters. When the machine lands on sensitive words, such as those disallowed by China’s Great Firewall (or those now illegal in Hong Kong), a break occurs and the machine jumps from traditional [Chinese script forms] to simplified.

[Howe 2019-21a] As a critical feature of RotVH, this is aligned with other work by Howe, often engaged with important problems and structures of our predominant digital culture – surveillance, security, power asymmetries, freedom of expression, etc. This essay touches on these matters, particularly insofar as they are addressed by algorithm, data curation, and data structuring, and also comparatively, insofar as they are drawn out by and critiqued in this work’s remarkable translinguistic and transcultural contexts. But for more extensive discussion of the digital politics of Howe’s work, and its aestheticization, the reader is referred to this other work itself.

Both these works have been presented to their readers and audiences as examples of electronic literature, often sited in necessarily sculptural gallery installation.[1] They are simultaneously examples of computational art and, as such, they are afforded the potential for re-presentation in other forms, particularly distinct graphic and audiovisual manifestations on computer monitors. The code or software of these works is at the core of what they are, artifactually.[2] What the code produces as “display” may be sited or made manifest in any number of 2D or 3D forms and spaces. This discussion, however, concerns itself, chiefly, with the code itself and what is presented by the code as, basically, typographic form on a 2D display, with some accompanying audio: letters and words in the case of Automatype; strokes, characters, and two-character compounds in the case of RotVH. Audio is used to signal particular relationships between generated forms and items in lexicons that have been derived from the languages of the respective works.

The typographic forms generated by these works are of course referred, by those who engage with them, to language. At any one moment, they can often – not always – be read as such, as language; or rather as words in, at least, one of two natural languages, English on the one hand, Chinese on the other. Except that the situation immediately becomes more complicated. Automatype does have an English lexicon at its disposal but, as it “explores” a mathematically abstracted space of Roman-letteral “spelling,” quite apart from “spellings” that are non-lexical in any language, it will encounter and display word-like forms that may be shared by natural languages other than English (any of those whose typography also uses Roman letters), or forms which are not orthographically English although they may be orthographically correct and readable in another language.

In the case of RotVH, the situation is similarly complicated but radically different. Arbitrary or regularly abstracted rearrangements or substitutions of any particular character’s strokes or sub-elements will not – or only in extremely rare cases – produce an orthographically readable character in Chinese or a language other than Chinese, but any of the orthographically readable characters which are produced may be read, entirely differently, in any number of Chinese dialects or, indeed, in any number of other dialects of certain natural languages such as Japanese or Korean. Since Daniel Howe currently lives, works, and exhibits in Hong Kong, readings of RotVH, for example, have often been anticipated in Mandarin (Putonghua) and Cantonese, which are considered mutually unintelligible dialects of Chinese.

Raising such complications at the outset of this close reading does at least two things. It highlights the way in which similarly motivated algorithmic processes may generate language-driven computational artworks with entirely different readings, particularly since these “readings,” quite apart from being metaphoric – that is, critical or aesthetic – are also readings of distinct “written” linguistic materials which co-constitute the works. And it also reminds us of deep problems concerning the relationship between, on the one hand, what we call “text” or “writing” or “typography,” or now also – in the context of computation – “strings of characters,” and, on the other, language as such.

For text is not language unless and until is either actually read, or unless and until it is considered to be, potentially, readable. This is a statement from my own philosophy and, indeed, ontology of language. [3] In the context of my theory I am happy to refer this statement to a species-unique human faculty of language;[4] in the present context I ask only that those reading this essay agree that the words of Automatype and the characters of RotVH are subject to the possibility, at least, of human reading and interpretation, and that this does have a bearing on their appreciation as art, in particular any art of language that they propose. Their code, however, executes and generates typographic forms without regard to any human reading that may or may not be taking place. From a linguistic perspective, it is the various lexicons and associated data structures embedded in these projects’ software –pre-determined and adopted by the artist prior to any execution of the code – which establish relationships with human reading. The processes which generate their displayed linguistic forms do not.

They do not, that is, unless, as readers and scholars we believe and assert, amongst other things, that reading and interpretation are reducible to regular, formal, combinatorial processes. One of the important tasks of critical code studies is to articulate this relationship between the code-composed programs (“programs” both literal and figurative) insofar as they are generative of linguistic form, and the practices of reading that we bring to them in order to appreciate what they express as language or, indeed, language art. To be clear, I am not in the camp of those who take language to be computable (reducible to computation) in any sense of this hypothesis that is abstracted or divorced from evolution or, indeed, history. In close reading these two works, I highlight the disjunctures between coded processes and human reading, and will even argue that these disjunctures, articulated, are themselves amongst the works’ most significant and affective readings. The disjunctures are brought into relief by the similarity of algorithmic process across these works, in contrast to their radically different integrations with the languages they engage. Both works are, I believe, easy enough to understand in terms of what their code is doing, while their readings are also clearly indicative of quite complex differences of language and culture; and also, in the case of RotVH, sociopolitics.

Concluding these introductory and anticipatory remarks, it bears mentioning that this relation between code-generated text and human reading has the same structure, fundamentally, as that which obtains for the increasing predominant practices of Natural Language Processing that are driven by Deep Learning. Except that, when Deep Learning is operative, human readers’ ability to articulate and understand the coded processes are – is it right to say “literally”? – redacted. Deep Learning language models’ “encoding” of “representations” is often presented as a “black box,” received as such even by willing experts. And while, in principle, exhaustive formal analysis may be held to be possible, what is actually concealed from us, as human readers and consumers (sic) of the generated pseudo-language, is a number of unsubstantiated assumptions concerning things that we do not (and perhaps cannot) know, scientifically or otherwise. [5] I would summarize this by saying that, although we speak and read, we still do not know what language is, nor how or why some of it may be aesthetically significant for us. As to this how and why, when we consider Automatype and RotVH, at least we have a chance.

The artist documentation pages for these works offer links to “Project Home Pages,” Online Versions, and various public manifestations. The Project Home Page for Automatype is minimal, and features an evaluative description extracted from a review by Brian Kim Stefans [Stefans 2011]. The Home for RotVH however takes us to a page which links the two works explicitly, categorizing them both as “Atomic Language Machines” (ALMs), defined as “discrete computational entities with the potential to change the direction, intent, or magnitude of a literary vector. In general, ALMs can be defined as members of the simplest class of mechanisms able to realize linguistic advantage.” [Howe 2021] “Linguistic advantage” is one of those terms deployed in the context of computational linguistics to provide or explain motivation in Natural Language Processing, something used to test against a generated linguistic form in order to decide if it has, in some sense, advantageously “succeeded” or achieved a “goal” which is usually interpretable by (and “advantageous” to) humans. For both Automatype and RotVH it is advantageous to spell out an item from the lexicons of English and Chinese, respectively.

The “atomism” of ALMs is referred by Howe to a concept of “simplest class of mechanism” and is linked with other such classes of mechanism which are cited as deployed, for example, in The Readers Project [Cayley and Howe 2009].[6] Here, I read “atomic” as indicative of, as it were, an elemental similarity between the class of mechanism driving Automatype and that which animates RotVH. The ALMs page quotes the following code snippet abstracted from the actual code driving RotVH:

“Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.” [Wikipedia contributors 2021] This metric is the measure of “linguistic advantage” that underlies both Automatype and RotVH. When we consider Automatype – working with English words spelt out in Roman letters – this distance is a relatively straightforward concept to grasp. Typographic words are, indeed, sequences or “strings” of characters. For each word in a lexicon (or its derivatives) we can calculate a Levenshtein distance to any other word. Having done so, we can start with any word we like and, having found its “nearest” Levenshteinian lexical neighbor, we may animate a typographic display which performs the minimum number of edits on our original string of characters in order to transform it into its neighbor. This is precisely what Automatype does, and by disallowing any “turning back” (to a previously displayed lexical item) it effectively represents an ALM that would travel least-distance paths from item to item until it had visited and exhausted all the items of its English lexicon.[7]

Another connotation of “atomic,” perhaps only latent in Howe’s ALMs webpage, comes into play at this point. The strings and sequences over which Levenshtein distances are calculated must be composed of integral, indivisible – at least for the purposes and processing of the algorithm – tokens, the “atoms” of the symbolic system of which they are expressions. The atoms of Automatype’s sequences are letters taken from a familiar, quite clearly delineated set, one that is widely shared over a number of linguistically integrated domains. In linguistic reality, the Roman letters used for English orthography are problematically related to what they purport to represent, the constituents of spoken English “sound-images” (as Saussure, for one, would have it). But leaving these problems of reference, representation, and transcription to one side, “English” letters can be read straightforwardly by humans as from a finite set of integral elements. And although minimally semantic, they are included as lexical items in most dictionaries. Letters are, generally speaking, typographic atoms of reading and are amenable to contemporary computation as such.

But what are the correspondent “atoms” of Chinese typography, of the Chinese writing system? Lexicons in the Chinese cultural sphere typically have characters as the “head words” of their articles although modern dictionaries may also use “compounds” of characters, usually consisting of two characters. In modern Chinese these compounds correspond, linguistically, to the words of English dictionaries. Nonetheless, the Chinese character is, culturally speaking, the “atom” of reading for the languages which use this logographic or, more properly, morphographic system of inscription. RotVH works with two-character compounds. If characters were RotVH’s atoms then Levenshtein distance would be calculated on the basis of character insertions, deletions, and substitutions. Since single characters are always morphemes and may often correspond with English words in modern Chinese (while typically doing so in pre-modern Chinese) they are clearly of a different linguistic order as compared to Roman letters. Morpheme insertion, deletion, and substitution would read – if it did motivate the algorithms of RotVH – as incommensurate with the corresponding operations of Automatype, although, in this speculative condition, RotVH would still exhaustively traverse a lexicon consisting of two-character compounds and would still sometimes display non-lexical items. (And it would still also be able to signal and respond to any of its encounters with politically charged two-character terms). But it would never be non-semantic. Its operations would always yield readable (and perhaps occasionally poetic) meanings.

The artist recognized these circumstances and went deeper into the analysis of Chinese characters, coming up with a remarkable and effective compromise for his aesthetic purposes along, implicitly, with novel proposals for conceptualizing and calculating Levenshtein distances across the domain of Chinese characters. This is Howe’s comment on his “logographic distance” calculation:

/*
 * logographic distance
 * - number of different full characters less 1 (via basic Levenshtein)
 * - plus Levenshtein distance between two decompositions for each char
 * from [ ⿰ ⿱ ⿻ ⿳ ⿺ ⿸ ⿲ ⿹ ⿴ ⿵ ⿶ ⿷ ]
 * - this gives an integer distance >= 0 (with 0 for identical strings)
 * - the floating point component (0 <= f < 1) is added by comparing the number
 of strokes in differing sub-parts normalized against a max-stroke count
 (not used in production)
 */

The distance calculation is a sum of staged Levenshtein distances. A first distance [1] is calculated between strings of integral characters (the length of these strings being always equal to two in the lexical domain that is addressed by RotVH). Then [2] each of the characters in the strings being compared is assigned to one of twelve patterns of (de)composition which are typical of Chinese characters. There is no generally accepted, rigorous analysis of this feature of characters, but sub-elements of characters are traditionally recognized and read into character composition and patterns of disposition for these elements fall into one of the twelve such that determinations are made and may even be assigned in some character dictionaries. Interestingly the patterns quoted above are represented by Unicode glyphs and thus – although these glyphs’ reference is much further divorced from linguistic significance, in the sense of any sound-image denotation, than that of alphabetic letters – their implicit deployment in Howe’s calculation resonates with the letters-as-tokens approach that is assumed in standard Levenshtein calculations over alphabetic orthographic typography. Finally, [3] calligraphic (or sinotypographic) strokes are also recognized as finest-grained elements of characters. Ordered strokes are what compose the higher-level sub-elements of the (de)composition patterns. Howe documents the possibility of using stroke-token sequences for each of the decomposed sub-elements and adding these to calculate a correspondingly finer-grained Levenshtein distance between, for example characters sharing the same (de)composition pattern. He proposes to do this in an implicitly weighted manner by adding these distances as a “floating point” (fractional) component. In practice, however, in the actual “production” version of RotVH, Howe decided to ignore this component.[8]

RotVH is art not science. It may well, however, engage more practically, empirically, and experimentally with scientific analyses or formulations of editing distance over the Chinese lexicon than other efforts which have been put forward in the context of science explicitly. The choice of next operation – here of insertion, deletion, or substitution – is on display to artistically motivated readers and viewers and will thus bring in other aesthetically implicated considerations apart from Levenshtein distance, the most obvious being based on keeping a “history” of all words or character compounds so far displayed, and disallowing repetitions. This principle applies to both Automatype and RotVH and is what enables these ALMs to take their shortest possible – conceptual-art – walk through an entire lexicon. For RotVH, Howe needs to establish additional pragmatic-aesthetic criteria, beyond the shared concept.

[O]ther criteria also affect selection (from the larger pool of candidates) including whether they differ on the same “side” (left or right character in the word) as the last few changes, so as to avoid the same character remaining constant for long periods, and whether any are trigger words. In the current version I ended up ignoring the floating point part of the distance metric… in order to get a set of words essentially tied in distance, so that I could then choose between them according to these other metrics.

[9] The “larger pool of candidates” refers to all those compounds for potential display which are “tied” in terms of Levenshtein distance after stages [1] and [2] above.

This is the point at which – without abandoning its inevitably computational and code-driven conceptual aesthetics, both also addressing the domain of Natural Language Processing, and both shared with Automatype – RotVH shifts its engines of motivation in order to adopt a critical art aesthetic, one that is designed to inform its readers, creatively, concerning the sociopolitical valences of certain items in the Chinese lexicon. RotVH contains a data file of “trigger” words, those compounds which are flagged by the “Great Firewall,” by Chinese state surveillance of linguistic internet traffic, or which have been signaled as politically taboo in what is now China’s Hong Kong Special Administrative Region. Here is a snapshot from the JSON file (triggers.json) to which RotVH refers [Howe 2019-21b]:

This is a snapshot which demonstrates the wide-ranging scope of Chinese lexical surveillance, flagging terms with regard to: sexual practices; “reforming” politicians; gender- and class-based affiliations; the Tian’anmen Square massacre and its “student movement” (likely including student movements in themselves); and even a controversial television series. The “lang” property of each two-character item is indicative of one of two main Chinese systems of inscription. Those words with the same triggering “def” property may occur in “trad[itional]” (more strokes, greater complexity) or “simp[lified]” spellings. Traditional characters are still widely and officially used in regions of the Chinese culture sphere – notably and with political significance Taiwan and Hong Kong – which are, to whatever extent, still “outside” the People’s Republic of China. The PRC, on the other hand, has instituted and adopted its own “reformed” and “simplified” character orthography. Some spellings – untouched by reform – belong to “both” systems. There is an underlying ideality to characters (or compounds) in either spelling. Essentially, they refer to the same “form-as-read” in the Chinese of their speaker-readers (I would call this a “gram” of the implicit archi-writing) and thus they are equally “triggering” for state surveillance.

Whereas the audiovisual behaviors of Automatype – apart from those which represent edit operations – are confined to signaling the ALM’s “arrival” at a lexical word, those of RotVH signal not only lexical arrival but also whether or not the newly displayed word is a “trigger,” a surveilled word. Then also, after a distinct sound and a flash of red, RotVH is also triggered to switch to orthographic explorations in whatever is the other system of Chinese spelling, either “trad[itional]” or “simp[lified],” depending on which of the two it was exploring when disrupted by a “trigger.” This shift of orthographic systems might perhaps be interpreted as a futile attempt to “misspell” and thus elude surveillance, but it can never be more than a jolt to either system since the underlying “trigger” is, as we have seen, the same word in any Chinese that matters to the surveillance operations of its PRC state overseers. Regardless of the overseers’ indifference (or rather their fixation on transgression) or the computational indifference of RotVH’s execution, the change of orthographic systems will, nonetheless, resonate with, and may mildly traumatize, Chinese readers since each system has ideological and political alignments and associations.[10]

The typical installation version of Automatype consisted of a number of networked, otherwise independently operating instances of the ALM in, for example, a 3x3 grid, each exploring the same lexicon separately but on its own path from a different starting point, making minimal displayed edits, and arriving intermittently at actual lexical items, English words. This grid arrangement overlays a form suggestive of visual or pattern poetry and thus also an aesthetic, a poetic that is not programmatically related (not integrated by code) with the conceptual Natural Language Processing aesthetic of the ALM itself. Automatype’s poetic overlay-in-installation has not been a significant focus of attention for this essay although it is what allowed Brian Kim Stefans, in his remarks on Automatype to say that “You will spend either 10 seconds or 5 minutes staring at this thing [the grid]; you will also see either a bunch of random words, or occasionally, if not always, engaging samples of minimalist poetry” [Stefans 2011].

Not only are Automatype and RotVH closely related in that they share essentially the same coded, programmatic procedure for “walking through” a lexicon and thus essentially the same computational and conceptual NLP aesthetic; they are also similarly structured in that both have been given, by their maker, an additional, overlaid aesthetic. And although the code of RotVH refers and reacts to its trigger word data, this is an additional and distinct coded operation of RotVH, dependent on an additional human-compiled data file, only of significance to or affective of this ALM’s readers for reasons that are sociopolitical rather than merely linguistic. The immediately following concluding remark should be part of a much larger discussion, but we might begin to take away something beyond the code-critical from this comparative reading by reflecting on how the overlaid aesthetic for a project within the domain of global English tends to engage with formal arrangement and poetics, whereas the closely related, subsequent project, addressing what is now perhaps the planet’s “other” global language, engages sociopolitics and critical art practices.

There is at least one more important general point to make that is based on the descriptions and analysis that we have just undertaken, and it has transcultural critical resonance. From both a code critical perspective and from one attuned to careful, responsible humanistic readings of computational art, any writing on this art must remain or become more critically aware of the culture of computation and its history. This is an imperative within the sphere of what is the globally predominant regime of computation, and it has hardly been addressed, as it must be, within an overarching context that is transcultural at the level of distinct “civilizations.”[11]

We have seen that, addressing the Chinese system of inscription, an operational analysis of its elements must be done by way of bespoke or imported data structures and bespoke or imported functions – even when this is for the purposes of animating the same Natural Language Processing concept or operation – here, a Levenshtein distance-based “shortest walk” through a lexicon. By contrast, for global English in particular, and for “western” languages having integrated alphabetic systems of inscription, the data structures and functions are already more or less built into actually existing computational infrastructure. The historical reasons for this integration of computation and the alphabet are quite well known. But this is no reason for critical complacency, particularly when we recall that text-as-orthography is in no way consistent with even a linguistic-scientific analysis of language as such.[12] We call orthographic spellings “words,” but this is both pragmatic, living-culture convention and scientific misdirection. If historical and contingent computational infrastructures reinforce our misdirected conventions, this has implications far beyond the misapplication or bespoke adaptation of these infrastructures to systems of inscription for which they are ill-adapted. Close reading of Automatype and RotVH allows us to encounter and explore these contemporary transcultural critical aporias. These two ALMs both read “words” but read them differently, because even the spellings of these words are culturally situated and involve radically different relations to linguistic structure. The ALMs nonetheless deploy algorithms for lexical traversal which are essentially the same. The code is radically different in each case, not only due to the cultural situation of the “words,” but also because the code itself – and contemporary computation as whole – is culturally situated and adapted to particular global locations.