Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies

The use of large transformer-based machine learning models to predict the three-dimensional structure of proteins from amino acid sequences (“protein folding”) has been widely perceived as one of the most important achievements in contemporary artificial intelligence research. \autocitesmoussad_2023,metz_2023,offert_2021,raghu_2020 AlphaFold, released by DeepMind in 2020, was the first model to suggest this view. “It will change everything,” a 2020 Nature article quotes evolutionary biologist Andrei Lupas. [7] “This will change medicine. It will change research. It will change bioengineering.” [7, 1] And while the unfettered optimism of four years ago has since transformed into a more cautious assessment of AlphaFold’s capabilities, [53] protein folding – and one of its main projected “downstream” applications, drug discovery – is still cited as one of the most “exceptionally useful” [53] use cases for large transformer-based machine learning models. Most recently, the utilization of a large transformer-based machine learning model to design new CRISPR candidate proteins [44] – “blueprints for microscopic mechanisms that can edit your DNA”, as a New York Times [31] report on the discovery reads – has promised even more immediate returns. It is perhaps not surprising, then, that between the first and final submissions of this article, John Jumper and Demis Hassabis at Google received the 2024 Nobel Prize in chemistry for AlphaFold and its potential impact on protein science.

At the same time, a different transformer-based model, the large language model ChatGPT, has dominated headlines ever since its release in 2022, eclipsing the comparably extensive coverage that AlphaFold, as a purely scientific model, and subsequent protein folding models had received by orders of magnitude (apart, maybe, from the reporting in the immediate aftermath of the Nobel Prize). ChatGPT’s remarkable capability to both hold a casual conversation and produce working computer programs suggests an “understanding” of both natural and formal languages that goes far beyond that of previous generations of models, and points to a future in which computation itself becomes “neural” – a future where even essential operations and calculations will be facilitated by a transformer layer. \autocites[See][]offert_2023a,offert_2023b

AlphaFold and ChatGPT, obviously, represent two very different use cases of transformers. But beyond their “two cultures” lies a strange commonality. Both systems’ claim to fame is their ability to reveal the “meaning”¹¹1Meaning is set in quotes here in reference to Hannes Bajohr’s argument that, while what is learned by large language models appears meaning-like, it is missing many of the characteristics of what humans would designate as meaning. Bajohr suggest the term “dumb meaning” to describe this not-here-nor-there character of computational meaning-making. See [3]. hidden in long sequences of tokens. ChatGPT is seen as a success because it can infer what is “meant” by a sequence of words; AlphaFold is seen as a success because it can infer what is “meant” – i.e., which three-dimensional protein structure is described – by a sequence of amino acids. This ability to infer “meaning” derives, technically, from the transformer’s ability to model “long-range relationships in the input sequence beyond their sequential neighborhoods.” [33] Both systems, in other words, due to their exposure to very large sets of training data, can “see” higher-order structure where humans cannot.

To the humanist, this development of course sounds like structuralism 2.0 [16], both due to its intuitive appeal – we like to think that we are thinking in abstractions, so abstraction is thought of as thinking – and due to the large number of potential applications. Structuralism’s appeal was that of a meta-science – much like that of cybernetics, to which it is historically and systematically related. [17] The same meta-science appeal is reflected in statements like the following made by Ali Madani, whose company develops the protein folding model targeting CRISPR mentioned above: “These A.I. models learn from sequences — whether those are sequences of characters or words or computer code or amino acids.” [31]

Its claim to universality, then, is the transformer’s selling point. In the following, we would like to argue that this claim to universality, particularly when viewed historically, is misleading. Concretely, it masks the fact that the direction of the simile, overwhelmingly, points from language to proteins: “Proteins, as a sequence of amino acids,” – as one recent paper reads – “can be viewed precisely as a language and therefore modeled using neural architectures developed for natural language.” [57, 1] Another recent paper proposes that the ability of pre-trained language models “to represent, manipulate, and extrapolate more abstract, nonlinguistic patterns may allow them to serve as basic versions of general pattern machines.” [32] In accounts like this, it is the language-ness of proteins (and of language), in other words, that renders the transformer architecture universal, and enables it to model two very different domains.

This expansion of the domain of language into structural biology seems ripe with implications for the emerging field [41] of critical AI studies which, so far, has focused almost exclusively on “cultural” models like ChatGPT. What is more, the critique of critical AI studies is almost always cultural critique in the cultural studies sense: a critique of culture, with culture. “I know it when I see it,” “it” being “wrong” culture, aesthetically, ethically, and ideologically. The model has produced a “wrong” picture, a “wrong” text, which points to the model’s limitations. Such idiosyncratic methodologies [38] necessarily have to ignore scientific models which simply cannot be evaluated in this way.

In this paper, we suggest that, by all but ignoring the vast landscape of scientific models within and beyond structural biology, critical AI studies is missing an entire angle of critique that draws from the history of science in general, and the history of knowledge in particular. We ask: what kind of knowledge-making do transformers afford, independent of their application? And what can we learn, specifically, about the “cultural” model ChatGPT, from the “scientific” model AlphaFold and its many siblings and successors?

To answer these questions, in a first step, we trace the emergence and historical development of the language paradigm in biology which has guided epistemic assumptions about structural biological processes since the mid 20th century. Biological processes, at least since the discovery of the genetic code, have been understood as language-like, if not as “linguistic,” i.e. as sharing structural properties (e.g. a “grammar”) with human language. The use of large transformer-based machine learning models can be understood as a direct continuation of this “traditional” language paradigm in computational biology in the 21st century.

In a second step, we argue that the transformer architecture works so well exactly because it is, counterintuitively, “non-linguistic.” While both natural language and protein folding models deal with sequences, this is where the commonality ends. Amino acids are not letters and amino acid sequences are not sentences.²²2They are, for instance, much, much longer. Seemingly peripheral (as in, not directly connected to the attention mechanism) techniques like word embedding, positional encoding, and subword tokenization “infuse” tokens with a continuity that could not be more language-unlike, and actually is much closer, analogically, to a physical, not a symbolic understanding of the world. Word embedding, for instance, produces continuous concepts which are manifested as points in an embedding space and that are at the same time more and less “ambiguous” than words: they are precise as they are enumerable but at the same time they can not be reduced to any existing word, as they reside in the “real” (as in real numbers, not as in Lacan) gaps that emerge from modeling a set of discrete tokens as a continuous space. Positional encoding gives an additional “spin” to tokens that encodes their position in a sentence, a concept equally far removed from the discrete world of language. Subword tokenization, finally, dissolves the very concept of the word, and gives equal importance to partials and composites. Before any data can be attended to, in other words, it has already lost its linguistic character.

We finally posit, in a third step, that transformers have to be understood as approaches to knowledge-making distinct from domain-specific affordances. Not only are transformers not “language models” (an understanding that is already questioned by the mere existence of transformers in other cultural domains, for instance in the form of image models), they are also not “linguistic” in the sense that they share structural properties with human language. The intuitive (humanist, and implicitly structuralist) interpretation that the many capabilities of contemporary artificial intelligence systems are a result of the fact that “all the world is language” – itself historically determined by the early Cartesian emphasis on linguistic compositionality [9] as a defining indicator of human-ness – is thus a fallacy. Instead, we suggest that transformers allow for the operationalization of sequential intuition, a concept we model after Peter Galison’s distinction between the “image” and “logic” modalities of scientific reasoning. Protein folding can be understood through sequential intuition. But as we are dealing with unknown (or at least under-formalized) bio-physical processes, their transformation into sequential operations is not straightforward and thus left to the machine. That machine, however, requires continuity (in the computational sense) to model continuity (in the physical sense) – which means that any linguistic qualities introduced must immediately be dissolved again, through word embedding, positional encoding, and subword tokenization.

Given this historical prevalence of the language paradigm in biology, today’s “intuitive” use of “language models” for protein folding connects the early nexus of thermodynamics and information theory, as well as the entanglement of structural biology and structural linguistics, to the transformer architecture.⁶⁶6It should be noted that we are intentionally skipping over multiple steps in the technical history of structural biology here, including earlier forms of machine learning models developed during the 1980s and 1990s. While a further investigation of these intermediate forms would contribute to the historical study of protein folding, and would likely give additional evidence to the prevalence of the language paradigm in structural biology, our aim here is to show instead how the particular technical innovations introduced by the transformer architecture restructure the language paradigm itself.

The transformer architecture is a deep learning architecture especially attuned to handling sequential input data. While not generative by design⁷⁷7Specifying the not-necessarily generative nature of transformers is relevant here, as we will argue that we need to understand “transformer” as an epistemic structure separate from, but related to, the epistemic structure of “natural” language, and protein “language”, independent of sequence-completion tasks., it rose to prominence as the most innovative aspect of generative architectures like GPT, and, subsequently, pre-trained language models like OpenAI’s ChatGPT 4. The literature (both within computer science and critical artificial intelligence studies) is thus ripe with more or less complex descriptions of the transformer architecture’s design principles.⁸⁸8We would like to direct the reader’s attention, in particular to the video tutorial by 3Blue1Brown (https://www.3blue1brown.com/lessons/gpt), the code-based tutorial “The Annotated Transformer” (https://nlp.seas.harvard.edu/annotated-transformer/), and Stephen Wolfram’s blog post “What Is ChatGPT Doing … and Why Does It Work?” (https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/). Thus, we will limit our own summary here to its eponymous core mechanism, the attention mechanism [55], which sharply distinguishes the transformer architecture from earlier approaches.

Attention, in simple terms, is a mechanism that directs a model’s focus to multiple parts of an input sequence at once. Earlier model architectures based on Markov chains ⁹⁹9Subsequent development of more advanced models, such as RNNs and LSTMs, aimed to address the long-term dependency limitations inherent in Markov chains. However, these models still struggled with long-term dependencies due to issues like the vanishing gradient problem. From a statistical perspective, neither RNNs nor LSTMs exhibit long memory capabilities. See [26]. in which only directly preceding tokens would be considered when trying to predict the next token in a sequence ignored, for instance, the impact of tokens near the beginning of a sentence on tokens at the end. That different parts of the input sequence are considered in parallel (mathematically modeled as a matrix multiplication operation) thus not only means that the sequence is processed more effectively, but also that long-range dependencies are taken into account – which in turn means that more complex input-output mappings are possible. In the case of proteins, if – following Anfinsen’s Dogma – all information needed to model the folding process is present – but not immediately visible – in the amino acid sequence, a sequential input (an amino acid chain) is used to predict a more complex three-dimensional structure (a protein), similarly to how one could use a sentence to predict a forthcoming word or sentence.

However, in this seemingly logical translation from language to biomolecule, a critical question is raised around how exactly computers represent both words and protein structures. Below, we address this issue of the apparent biological-linguistic universality of the transformer architecture by pointing out where the affordances of the model leave the realm of language and enter a separate, autonomous epistemic space which can be adapted to both biological and linguistic phenomena. We pay special attention – as it were – to essential but often disregarded pre-processing steps which, counterintuitively, represent the least linguistic aspects of the transformer: subword tokenization, word embedding, and positional encoding.

How exactly are the pre-processing techniques described above put to work? What role do they play in the larger ecosystem of “scientific” and “cultural” models? Here, we need to turn to the actual object of this paper, transformer-based machine learning models of the protein folding process. We focus on one case study in particular – the ESM2 model produced by Meta [27] – as an example of a protein folding model fully determined by the language paradigm.

One of the most important components of any machine learning system is data¹⁴¹⁴14Multiple large-scale datasets are available for computational protein folding. The oldest one is the Protein Data Bank (PDB), a worldwide repository providing a comprehensive collection of experimentally determined three-dimensional structures of proteins, nucleic acids, and complex assemblies. It is widely used by researchers in the field of structural biology to analyze protein-protein interactions and perform various other kinds of computational analyses of protein structures in addition to protein folding. In addition to the PDB, there are other databases specifically designed to support the study of protein folding, including Protein Folding Database (PFD) (with an emphasis on protein folding kinetics and thermodynamics data) and ProteinNet (designed specifically for deep learning approaches, including diverse protein families and structures). – in our case, data that encode those long-range dependencies the transformer architecture aims to decode. The ESM2 model models the MGnify90 dataset [42], which comprises curated and annotated protein sequences from a wide range of organisms found “in the wild”, including those found in microbes in the soil, deep in the ocean, and even inside our bodies – vastly outnumbering those that make up animal and plant life.

Compared to other protein prediction models such as AlphaFold, AlphaFold 2 and RoseTTAFold, the ESM family of models circumvents the need to go through the time-consuming step of multiple sequence alignment.¹⁵¹⁵15The multiple sequence alignment (MSA)-based approach to protein folding, exemplified by models like AlphaFold, AlphaFold 2 and RoseTTAFold, incorporates two transformer blocks to iteratively refine representation generated by the previous step of coevolutionary analysis, where input sequences are compared with protein sequences whose 3D structures were experimentally determined, to search for matches. These components are crucially important when it comes to constructing full-length 3D structures. Thus far, this approach has outperformed the “language model”-based method, demonstrated by AlphaFold 2’s significant lead at CASP 14 meeting. However, the MSA-based method’s reliance on comparing and matching requires a high computational cost which at the same time limits its effectiveness in predicting those sequences that cannot be identified and matched with homologs. These orphan sequences count for 20% of all metagenomic sequences. This limitation has paved the way for the “language model”-based approach, which does not require comparing and matching. Instead, the ESM architecture requires just the amino acid sequence of the protein as input. The model output is composed of 3-D coordinates that describe the position of each atom in the protein molecule. In that sense, ESM2 is a “pure” “language model” – a fact emphasized throughout the research paper and blog post accompanying its development.

Among other methods from natural language processing, ESM2 – a transformer-based model – employs “masked language modelling.” During its training phase, approximately 15 per cent of the amino acid positions in a sequence are randomly obscured – either masked, shuffled to a different amino acid, or left unchanged. The model is then trained to predict these obscured residues based on the bi-directional context of sequences, i.e. considering both the preceding and subsequent residues around any given position in a sequence. From this process, the model then learns a “translation” – to speak with Robson and Shannon – from sequence to structure.

ESM2, importantly, has been employed to predict three-dimensional protein structures on a large scale, contributing to the so-called ESM Atlas, which incorporates predictions from ESM2 and other related models. The ESM Atlas provides the “first large-scale view of the structures of metagenomic proteins encompassing hundreds of millions of proteins.” [12] As such, it is first and foremost a demonstration of the projected efficiency of transformer-based protein folding. As the project blog post states: “Advancements in gene sequencing have made it possible to catalog billions of metagenomic protein sequences. Although we know that these proteins exist, because we have discovered their sequences, understanding their biology is a staggering challenge. Determining the three-dimensional structures for hundreds of millions of proteins experimentally is far beyond the reach of time-intensive laboratory techniques such as X-ray crystallography, which can take weeks to years for a single protein. Computational approaches can give us insight into metagenomics proteins that isn’t possible with experimental techniques.” [12]

What, then, enables this large-scale success, if we are to believe the authors of the project? A paper investigating the potential of ESM2 to generalize de novo proteins beyond existing ones, attributes its success – perhaps unsurprisingly – exactly to the model’s linguistic nature: “We hypothesize that there exists a deep underlying grammar in protein sequences that makes it possible for the language model to generalize. […] Classically this form generalization has been enabled by an energy function grounded in physics that captures the native folded state. […] [N]ew deep learning approaches may capture something similar to the physical energy. The success of language models on this problem suggests that deep patterns in sequences may offer an alternative path to generalization, independent of an explicit model of the underlying physics.” [56, 3]

It is this paradoxical mapping of language and physics – a mapping that we find already in Robson – that structures the project’s epistemic perspective. Language and physics are different but the same, models of nature but also models of each other. “Like the text of an essay or letter,” the blog post reads, “proteins can be written as sequences of characters. […] Learning to read this language of biology poses extraordinary challenges. While a protein sequence and a passage of text can both be written down as characters, there are deep and fundamental differences between them. A protein sequence describes the chemical structure of a molecule, which folds into a complex three-dimensional shape according to the laws of physics.”

In large language models, we witness emergent behaviors¹⁶¹⁶16The degree to which these behaviors are truly emergent is contested, see for instance [58] and [45]. as the model scales up. The rules of grammar, for instance, are never explicitly “taught” to any large language model, they are only latent in the training data. The model “learns” these rules simply because grammatically-correct language has an almost infinitely higher likelihood to appear in the data than nonsense. At the same time, as the popular image of the Library of Babel, most famously described by Borges, suggests, the number of meaningful (i.e. syntactically and semantically valid) token sequences is vanishingly small for natural languages. Token sequences structured by rules thus appear in sharp contrast to random ones, which makes learning easier. A parallel form of learning is assumed to take place in transformer-based models trained on protein sequences. The strength of ESM2 is thus premised on the assumption that the generative forces behind meaning-making in human language may have analogues in the biological “language” of proteins. What the authors suggest is, almost word for word, albeit in terms of the current generation of “language technology,” Robson’s leap from thermodynamics to information theory almost half a century prior. Transformers learn the rules of folding akin to a physical model, but “differently.” The “deep grammar” of symbolized bases, in other words, can represent the epistemic structure of physical reality just as well as its actual epistemic structure (after all, it is physical laws that govern the molecular processes of protein folding.).

Here, then, lies the epistemic dilemma at the core of contemporary transformer models.

The operations performed by subword tokenization, word embedding, and positional encoding suggest that whatever transformers operate on and learn from has lost its “linguistic” character. We might collect a large datasets of “natural” language to train a transformer model but what it actually operates on is anything but “natural”. Pragmatically, we would not recognize the results of the three preprocessing steps described above as language-like anymore.

At the same time, before preprocessing even takes place we have arguably already entered a “non-linguistic” space, even if we bracket the bottom parts of the stack.¹⁷¹⁷17For the sake of our argument there is only software, to paraphrase (and invert) Kittler’s famous ontological perspective. As many others have pointed out [50], basic text encoding and its many idiosyncrasies already dilutes the linguistic character of language before it is fed to any software in the proper sense: language is not a medium of computation.¹⁸¹⁸18Yet. See [34]. We thus have to be more precise about what exactly is being lost here, and about what exactly we mean by “linguistic” or “language-(un)like” in the first place.

Crucially, it is exactly the fact that proteins are treated like language that powers protein folding models. Although it is true that amino acids are not letters and amino acid sequences are not sentences it is equally true that they must become letters and sentences in order for them to be inputted as data into a transformer.¹⁹¹⁹19We are quoting reviewer two of the paper here, and would like to thank them for their invaluable input on this argument. The language paradigm, in other words, is, and always has been, an operational paradigm: not only a way to read reality through the lens of language, but a way to understand.

More precisely: while the language paradigm breaks down if we understand it as doing the representational work of “encoding”, it becomes productive if we understand it as providing a generalized framework for the discrete modeling of continuous systems, exactly in the sense of information theory. It is no coincidence, then, that Hans Blumenberg, in the final chapter of The Readability of the World, quotes quantum physicist Erwin Schrödinger stating: “The great revelation of quantum theory was that features of discreteness were discovered in the Book of Nature, in a context in which anything but continuity seemed to be absurd according to the views held until then.”²⁰²⁰20Op. cit. [6, p. 312].

But why not move directly to information theory then, why the detour through language, arguably only the case study of information theory. If language, in transformers, is a special case, if transformers are “general pattern machines” [32], if the extraction of patterns from linguistic data is just one application, why, then, do we stick to the metaphors derived from the language paradigm even at the fringes of their usefulness – again, amino acids are not letters and amino acid sequences are not sentences.

The answer, we would like to suggest, is operationalized intuition. As Peter Galison famously argues in Image and Logic [14], and perhaps even more explicitly in “Images Scatter into Data, Data Gather into Images”: neither “the ‘pictorial-representative” nor the “analytical-logical” exist as fixed positions in scientific imaging [15, p. 322]. And both are needed, desperately, to learn, to know: “We must have scientific images because only images can teach us. We are human, and as such, we depend on specificity and materiality to learn. Only pictures can develop within us the intuition needed to proceed further towards abstraction. […] And yet, we cannot have images because images deceive. Pictures create artifactual expectations, they incline us to reason on false premises. We are human, and as such are easily led astray by the siren call of material specificity. Logic, not imagery, is the acid test of truth that strips away the shoddy inferences that accompany the mis-seeing eye” [15, p. 300]. Language, we might state then, provides the medium for another, particular kind of intuition that is not afforded by physics – that is not afforded, in fact, by either “image” or “logic”: an intuition about sequences. There lie the roots of the language paradigm as metaphor – and, in the last instance, also as technique. It is only in the medium of language that we can reason about sequences, and operations on sequences. Shannon [48] develops his information theory from statistical patterns in the English language, but perhaps more importantly in explicit reference to instances of low and high entropy language in literature, namely Ogden and Joyce [22]. Indeed, the very idea of statistical patterns in language can be traced back to literature, namely to Markov’s [28] analysis of Eugene Onegin from which the notion of the Markov model of dependent probabilities emerges.

We are facing, thus, a double transformation, a kind of epistemic detour, in protein folding models. To make physical reality amenable to sequential intuition it has to be made linguistic. We just know that the secret to protein folding must be expressible as a complex pipeline of sequential operations (and the persistence and success of the language paradigm shows that this intuition has merit). But as we are dealing with unknown (or at least under-formalized) bio-physical processes in protein folding, their transformation into sequential operations is not straightforward – it is, in fact, precisely non-linear – and thus left to the machine. That machine – the transformer – however, on the one hand cannot process language in the first place, and on the other hand needs continuity (in the computational sense) to model continuity (in the physical sense), which means that our language-reality must immediately be dissolved again, through the preprocessing steps described above. It is only then that enough “space” is created, through the introduction of artificial continuity, and quite literally in the gaps between embedded data points, that new knowledge can be produced.²¹²¹21For a longer and more general version of this argument, see [39]. In other words: in protein folding, we take an epistemic detour through language because we desire the sequence-based reasoning afforded by language without rendering physical reality entirely metaphorical. The transformer, then, does exactly more and less than language. It removes almost all non-operationalizable sense-making dimensions (think, for instance of inter- or paratextuality, of performativity and contingency, or of anything metaphorical that is more complex than a simple analogy) – but it also adds new sense-making dimensions through subword tokenization, word embedding, and positional encoding.

If we began by suggesting that by all but ignoring the vast landscape of scientific models within and beyond structural biology, critical artificial intelligence studies is missing an entire angle of critique, we can now specify: it is exactly in the epistemic structure of the transformer architecture that “cultural” and “scientific” concepts become entangled. Transformers are “cultural” because they render physical reality “linguistic” to afford sequential reasoning, and they are “scientific” because this “linguistic” version of physical reality is only created to be immediately resolved back into numbers (or, more precisely, vectors). The persistence of the language paradigm in biology has always suggested a trading zone. In the transformer age, this trading zone has become material. At the same time, no recurrent neural network, no deep convolutional neural network, and no diffusion network affords the kind of knowledge afforded by the transformer. Architecture- and model-specificity, in other words, matters – and this suggests much work ahead for critical artificial intelligence studies.

The authors would like to thank Elena Aronova, Anna Schewelew, and Rita Raley, as well as the two anonymous reviewers, for their time and helpful suggestions.