The origins of large language models can be traced back to the inception of AI research. The early history of natural language processing (NLP) was marked by a schism between two competing paradigms: the symbolic and the stochastic approaches. A major influence on the symbolic paradigm in NLP was Noam Chomsky’s transformational-generative grammar [48], which posited that the syntax of natural languages could be captured by a set of formal rules that generated well-formed sentences. Chomsky’s work laid the foundation for the development of rule-based syntactic parsers, which leverage linguistic theory to decompose sentences into their constituent parts. Early conversational NLP systems, such as Winograd’s SHRDLU [49], required syntactic parsers with a complex set of ad hoc rules to process user input.

In parallel, the stochastic paradigm was pioneered by researchers such as mathematician Warren Weaver, who was influenced by Claude Shannon’s information theory. In a memorandum written in 1949, Weaver proposed the use of computers for machine translation employing statistical techniques [50]. This work paved the way for the development of statistical language models, such as n-gram models, which estimate the likelihood of word sequences based on observed frequencies of word combinations in a corpus [51]. Initially, however, the stochastic paradigm was lagging behind symbolic approaches to NLP, showing only modest success in toy models with limited applications.

Another important theoretical stepping stone on the road to modern language models is the so-called distributional hypothesis, first proposed by the linguist Zellig Harris in the 1950s [52]. This idea was grounded in the structuralist view of language, which posits that linguistic units acquire meaning through their patterns of co-occurrence with other units in the system. Harris specifically suggested that the meaning of a word could be inferred by examining its distributional properties, or the contexts in which it occurs. [1] aptly summarized this hypothesis with the slogan “You shall know a word by the company it keeps,” acknowledging the influence of [2]’s conception of meaning-as-use to highlight the importance of context in understanding linguistic meaning.

As research on the distributional hypothesis progressed, scholars began exploring the possibility of representing word meanings as vectors in a multidimensional space 1. Early empirical work in this area stemmed from psychology and examined the meaning of words along various dimensions, such as valence and potency [53]. While this work introduced the idea of representing meaning in a multidimensional vector space, it relied on explicit participant ratings about word connotations along different scales (e.g., good–bad), rather than analyzing the distributional properties of a linguistic corpus. Subsequent research in information retrieval combined vector-based representations with a data-driven approach, developing automated techniques for representing documents and words as vectors in high-dimensional vector spaces [54].

After decades of experimental research, these ideas eventually reached maturity with the development of word embedding models using artificial neural networks [55]. These models are based on the insight that the distributional properties of words can be learned by training a neural network to predict a word’s context given the word itself, or vice versa. Unlike previous statistical methods such as n-gram models, word embedding models encode words into dense, low-dimensional vector representations (Fig. 1). The resulting vector space drastically reduces the dimensionality of linguistic data while preserving information about meaningful linguistic relationships beyond simple co-occurrence statistics. Notably, many semantic and syntactic relationships between words are reflected in linear substructures within the vector space of word embedding models. For example, Word2Vec [56] demonstrated that word embeddings can capture both semantic and syntactic regularities, as evidenced by the ability to solve word analogy tasks through simple vector arithmetic that reveal the latent linguistic structure encoded in the vector space (e.g., \( king + woman - man \approx queen\) , or \( walking + swam - walked \approx swimming\) ).

Pan/Zoom

The development of word embedding models marked a turning point in the history of NLP, providing a powerful and efficient means of representing linguistic units in a continuous vector space based on their statistical distribution in a large corpus. However, these models have several significant limitations. First, they are not capable of capturing polysemy and homonymy, because they assign a single or “static” embedding to each word type, which cannot account for changes in meaning based on context; for example, “bank” is assigned a unique embedding regardless of whether it refers to the side of a river or the financial institution. Second, they rely on “shallow” artificial neural network architectures with a single hidden layer, which limits their ability to model complex relationships between words. Finally, being designed to represent language at the level of individual words, they are not well-suited to model complex linguistic expression, such as phrases, sentences, and paragraphs. While it is possible to represent a sentence as a vector by averaging the embeddings of every word in the sentence, this is a very poor way of representing sentence-level meaning, as it loses information about compositional structure reflected in word order. In other words, word embedding models merely treat language as a “bag of words”; for example, “a law book” and “a book law” are treated identically as the unordered set {’ a’,’ book’,’ law’}.

The shortcomings of shallow word embedding models were addressed with the introduction of “deep” language models, going back to recurrent neural networks (RNNs) and their variants, such as long short-term memory (LSTM) [57] and the gated recurrent unit (GRU) [58]. These deep neural network architectures incorporate a memory-like mechanism, allowing them to remember and process sequences of inputs over time, rather than individual, isolated words. Despite this advantage over word embedding models, they suffer from their own limitations: they are slow to train and struggle with long sequences of text. These issues were addressed with the introduction of the Transformer architecture by [3], which laid the groundwork for modern LLMs.

One of the key advantages of the Transformer architecture is that all words in the input sequence are processed in parallel rather than sequentially, by contrast with RNNs, LSTMS and GRUs. Note that parallel processing in Transformers is applicable within a predefined maximum sequence length or input window, beyond which the model cannot process without truncating or segmenting the input. This is due to the self-attention mechanism’s quadratic growth in computational and memory requirements with respect to the sequence length. Not only does this architectural modification greatly boost training efficiency, it also improves the model’s ability to handle long sequences of text, thus increasing the scale and complexity of language tasks that can be performed.

At the heart of the Transformer model lies a mechanism known as self-attention (Fig. 2). Simply put, self-attention allows the model to weigh the importance of different parts of a sequence when processing each individual word contained in that sequence. For instance, when processing the word “it” in a sentence, the self-attention mechanism allows the model to determine which previous word(s) in the sentence “it” refers to–which can change for different occurrences of “it” in different sentences. This mechanism helps LLMs to construct a sophisticated representation of long sequences of text by considering not just individual words, but the interrelationships among all words in the sequence. Beyond the sentence level, it enables LLMs to consider the broader context in which expressions occur, tracing themes, ideas, or characters through paragraphs or whole documents.

Pan/Zoom

It is worth mentioning that Transformer models do not operate on words directly, but on linguistic units known as tokens. Tokens can map onto whole words, but they can also map onto smaller word pieces. Before each sequence of words is fed to the model, it is chunked into the corresponding tokens – a process called tokenization. The goal of tokenization is to strike a balance between the total number of unique tokens (the size of the model’s “vocabulary”), which should be kept relatively low for computational efficiency, and the ability to represent as many words from as many languages as possible, including rare and complex words. This is typically accomplished by breaking down words into common sub-word units, which need not carve them at their morphologically meaningful joints. For example, GPT-4’s tokenizer maps the word “metaphysics” onto three tokens, corresponding to “met”, “aph”, and “ysics” respectively. Similarly, numbers need not be tokenized into meaningful units for the decimal system; GPT-3 processes “940” as a single token, while “941” gets processed as two tokens for “9” and “41” (this quirk of tokenization goes a long way towards explaining why LLMs can struggle with multiple-digit arithmetic; see [121] and [122]).

The most common variant of Transformer-based models are known as “autoregressive” models – including GPT-3, GPT-4, and ChatGPT. Autoregressive models operate using a learning objective called next-token prediction: given a sequence of tokens, they are tasked with predicting which token is statistically most likely to follow. They are trained on a large corpus that includes a diverse range of sources, such as encyclopedias, academic articles, books, websites, and, in more recent iterations of the GPT series, even a substantial amount of computer code. The goal is to provide the model with a rich, diverse dataset that encapsulates the breadth and depth of natural and artificial languages, allowing it to learn from next-token prediction in many different contexts.

At each training step, the model’s objective is to predict the next token in a sequence sampled from the corpus, based on the tokens that precede. For instance, given the sequence “The cat is on the,” the model might predict “mat” as the most likely next token. The model is initialized with random parameters, which means its first predictions are essentially no better than chance. However, with each prediction, the model’s parameters are incrementally adjusted to minimize the discrepancy between its prediction and the actual next token in the training data. This process is iterated over billions of training steps, until the model becomes excellent at predicting the next token in any context sampled from the training data. This means that a trained model should be able to write text – or code – fluently, picking up on contextual clues provided in the “prompt” or input sequence.

While LLMs trained in this way are very good at generating convincing paragraphs, they have no intrinsic preference for truthful, useful, or inoffensive language. Indeed, the task of next-token prediction does not explicitly incorporate common normative goals of human language use. To overcome this limitation, it is possible to refine a model pre-trained with next-token prediction by “fine-tuning” it on a different learning objective. One popular fine-tuning technique in recent LLMs such as ChatGPT is called “reinforcement learning from human feedback,” or RLHF [59].

RLHF proceeds in three stages. The initial stage involves collecting a dataset of human comparisons. In this phase, human crowdworkers are asked to review and rank different model responses according to their quality. For instance, the model may generate multiple responses to a particular prompt, and human reviewers are asked to rank these responses based on certain normative criteria such as helpfulness, harmlessness and honesty [the “three Hs”, 60]. This results in a comparison dataset that reflects a preference ranking among possible responses to a set of inputs. In the second stage, this comparison data is used to train a reward model that guides the fine-tuning process of the model. A reward model is a function that assigns a numerical score to a given model output based on its perceived quality. By leveraging the comparison data, developers can train this reward model to estimate the quality of different responses. In the third and final stage, the reward model’s outputs are used as feedback signals in a reinforcement learning process, to fine-tune the pre-trained LLM’s parameters. In other words, the pre-trained LLM learns to generate responses that are expected to receive higher rankings based on the reward model. This process can be repeated iteratively, such that the model’s performance improves with each iteration. However, the effectiveness of this approach heavily relies on the quality of the comparison data and the reward model.

RLHF allows developers to steer model outputs in more specific and controlled directions. For instance, this method can be utilized to reduce harmful and untruthful outputs, to encourage the model to ask clarifying questions when a prompt is ambiguous, or to align the model’s responses with specific ethical guidelines or community norms. By combining next-token prediction with RLHF, we can guide LLMs to produce outputs that are not just statistically likely, but also preferred from a human perspective. The fine-tuning process thus plays a crucial role in adapting these models to better cater to the normative goals of human language use.

LLMs have a remarkable ability to use contextual information from the text prompt (user input) to guide their outputs. Deployed language models have already been pre-trained, and so do not learn in the conventional “machine learning” sense when they generate text; their parameters remain fixed (or “frozen”) after training, and most architectures lack an editable long-term memory resource. Nonetheless, their capacity to flexibly adjust their outputs based on the context provided, including tasks they have not explicitly been trained for, can be seen as a form of on-the-fly “learning” or adaptation, and is often referred to as “in-context learning” [30]. At a more general level, in-context learning can be interpreted as a form of pattern completion. The model has been trained to predict the next token in a sequence, and if the sequence is structured as a familiar problem or task, the model will attempt to “complete” it in a manner consistent with its training. This feature can be leveraged to give specific instructions to the model with carefully designed prompts.

In so-called “few-shot learning”, the prompt is structured to include a few examples of the task to be performed, followed by a new instance of the task that requires a response. For instance, to perform a translation task, the prompt might contain a few pairs of English sentences and their French translations, followed by a new English sentence to be translated. The model, aiming to continue the pattern, will generate a French translation of the new sentence. By looking at the context, the model infers that it should translate the English sentence into French, instead of doing something else such as continuing the English sentence. In “zero-shot learning,” by contrast, the model is not given any examples; instead, the task is outlined or implied directly within the prompt. Using the same translation example, the model might be provided with an English sentence and the instruction “Translate this sentence into French:”. Despite receiving no example translations, the model is still able to perform the task accurately, leveraging the extensive exposure to different tasks during training to parse the instruction and generate the appropriate output.

Few-shot learning has long been considered an important aspect of human intelligence, manifested in the flexible ability to learn new concepts from just a few examples. In fact, the poor performance of older machine learning systems on few-shot learning tasks has been presented as evidence that human learning often relies on rapid model-building based on prior knowledge rather than mere pattern recognition [61]. Unlike older systems, however, LLMs trained on next-token prediction excel at in-context learning and few-shot learning specifically [62]. This capacity appears to be highly correlated with model size, being mostly observed in larger models such as GPT-3 [30]. The capacity for zero-shot learning, in turn, is particularly enhanced by fine-tuning with RLHF. Models such as ChatGPT can fairly robustly pick up on point-blank questions and instructions without needing careful prompt design and lengthy examples of the tasks to be completed.

LLMs have been applied to many tasks within and beyond natural language processing, demonstrating capabilities that rival or even exceed those of task-specific models. In the linguistic domain, their applications range from translation [63], summarization [64], question answering [24], and sentiment analysis [65] to free-form text generation including creative fiction [66]. They also power advanced dialogue systems, lending voice to modern chatbots like ChatGPT that greatly benefit from fine-tuning with RLHF [67]. Beyond traditional NLP tasks, LLMs have demonstrated their ability to perform tasks such as generating code [25], solving puzzles [68], playing text-based games [69], and providing answers to math problems [70]. The versatile capacities of LLMs make them potentially useful for knowledge discovery and information retrieval, since they can act as sophisticated search engines that respond to complex queries in natural language. They can be used to create more flexible and context-aware recommendation systems [71], and have even been proposed as tools for education [72], research [73], law [4], and medicine [74], aiding in the generation of preliminary insights for literature review, diagnosis, and discovery.

According to a long-standing critique of the connectionist research program, artificial neural networks would be fundamentally incapable of accounting for the core structure-sensitive features of cognition, such as the productivity and systematicity of language and thought. This critique centers on a dilemma: either ANNs fail to capture the features of cognition that can be readily accounted for in a classical symbolic architecture; or they merely implement such an architecture, in which case they lack independent explanatory purchase as models of cognition [75, 76, 77]. The first horn of the dilemma rests on the hypothesis that ANNs lack the kind of constituent structure required to model productive and systematic thought – specifically, they lack compositionally structured representations involving semantically-meaningful, discrete constituents [78]. By contrast, classicists argue that thinking occurs in a language of thought with a compositional syntax and semantics [79]. On this view, cognition involves the manipulation of discrete mental symbols combined according to compositional rules. Hence, the second horn of the dilemma: if some ANNs turn out to exhibit the right kind of structure-sensitive behavior, they must do so because they implement rule-based computation over discrete symbols.

The remarkable progress of LLMs in recent years calls for a reexamination of old assumptions about compositionality as a core limitation of connectionist models. A large body of empirical research investigates whether language models exhibit human-like levels of performance on tasks thought to require compositional processing. These studies evaluate models’ capacity for compositional generalization, that is, whether they can systematically recombine previously learned elements to map new inputs made up from these elements to their correct output [80]. This is difficult to do with LLMs trained on gigantic natural language corpora, such as GPT-3 and GPT-4, because it is near-impossible to rule out that the training set contains that exact syntactic pattern. Synthetic datasets overcome this with a carefully designed train-test split.

The SCAN dataset, for example, contains a set of natural language commands (e.g., “jump twice”) mapped unambiguously to sequences of actions (e.g., JUMP JUMP) [81]. The dataset is split into a training set, providing broad coverage of the space of possible commands, and a test set, specifically designed to evaluate models’ abilities to compositionally generalize (3). To succeed on SCAN, models must learn to interpret words in the input (including primitive commands, modifiers and conjunctions) in order to properly generalize to novel combinations of familiar elements as well as entirely new commands. The test set evaluates generalization in a number of challenging ways, including producing action sequences longer than seen before, generalizing across primitive commands by producing the action sequence for a novel composed command, and generalizing in a fully systematic fashion by “bootstrapping” from limited data to entirely new compositions.

Pan/Zoom

Initial DNN performance on SCAN and other synthetic datasets probing compositional generalization – such as CFQ [82] and COGS [83] – was somewhat underwhelming. Testing generally revealed a significant gap between performance on the train set and on the test set, suggesting a failure to properly generalize across syntactic distribution shifts. Since then, however, many Transformer-based models have achieved good to perfect accuracy on these tests. This progress was enabled by various strategies, including tweaks to the vanilla Transformer architecture to provide more effective inductive biases [84, 85] and data augmentation to help models learn the right kind of structure [86, 87, 88].

Meta-learning, or learning to learn better by generalizing from exposure to many related learning tasks [89, 5], has also shown promise without further architectural tweaks. Standard supervised learning rests on the assumption that training and testing data are drawn from the same distribution, which can lead models to “overfit” to the training data and fail to generalize to the testing data. Meta-learning exposes models to several distributions of related tasks, in order to promote acquisition of generalizable knowledge. For example, [5] show that a standard Transformer-based neural network, when trained on a stream of distinct artificial tasks, can achieve systematic generalization in a controlled few-shot learning experiment, as well as state-of-the-art performance on systematic generalization benchmarks. At test time, the model exhibits human-like accuracy and error patterns, all without explicit compositional rules. While meta-learning across various tasks helps promote compositional generalization, recent work suggests that merely extending the standard training of a network beyond the point of achieving high accuracy on training data can lead it to develop more tree-structured computations and generalize significantly better to held-out test data that require learning hierarchical rules [90]. The achievements of Transformer models on compositional generalization benchmarks provide tentative evidence that built-in rigid compositional rules may not be needed to emulate the structure-sensitive operations of cognition.

One interpretation of these results is that, given the right architecture, learning objective, and training data, ANNs might achieve human-like compositional generalization by implementing a language of thought architecture – in accordance with the second horn of the classicist dilemma [77, 91]. But an alternative interpretation is available, on which ANNs can achieve compositional generalization with non-classical constituent structure and composition functions. Behavioral evidence alone is insufficient to arbitrate between these two hypotheses. See Part II for a brief discussion of mechanistic evidence in favor of the second hypothesis. But it is also worth noting that the exact requirements for implementing a language of thought are still subject to debate [92, 93].

On the traditional Fodorian view, mental processes operate on discrete symbolic representations with semantic and syntactic structure, such that syntactic constituents are inherently semantically evaluable and play direct causal roles in cognitive processing. By contrast, the continuous vectors that bear semantic interpretation in ANNs are taken to lack discrete, semantically evaluable constituents that participate in processing at the algorithmic level, which operates on lower-level activation values instead. This raises the question whether the abstracted descriptions of stable patterns observed in the aggregate behavior of ANNs’ lower-level mechanisms can fulfill the requirements of classical constituent structure, especially when their direct causal efficacy in processing is not transparent.

For proponents of connectionism who argue that ANNs may offer a non-classical path to modeling cognitive structure, this is a feature rather than a bug. Indeed, classical models likely make overly rigid assumptions about representational formats, binding mechanisms, algorithmic transparency, and demands for systematicity; conversely, even modern ANNs likely fail to implement their specific architectural tenets closely. This leaves room for connectionist systems that qualify as ‘revisionist’ rather than implementational, with novel kinds of functional primitives like distributed microfeatures, formed through intrinsic learning pressures rather than explicit rules [76]. Such systems may not only satisfy compositional constraints on processing, like their classical counterparts, but also what [6] call the continuity principle.

The continuity principle holds that information encoding and processing mechanisms should be formalized using real numbers that can vary continuously rather than discrete symbolic representations, because it confers critical benefits. First, continuous vector spaces support similarity-based generalization, wherein knowledge learned about one region of vector space transfers to nearby regions, enabling more flexible modeling of domains like natural language that confound approaches relying on mappings between discrete symbols. Second, statistical inference methods exploiting continuity, like neural networks, enable tractable approximation solutions that avoid intractable search through huge discrete search spaces. Finally, continuity permits the use of deep learning techniques that simultaneously optimize information encodings alongside model parameters to discover task-specific representational spaces maximizing performance. In concert, these advantages of leveraging continuity address longstanding challenges discrete symbolic approaches have faced in terms of flexibility, tractability, and encoding. Thus, Transformer-based ANNs offer a promising insight into ‘neurocompositional computing’ [6, 94]: they suggest that ANNs can satisfy core constrains on cognitive modeling, notably the requirements for continuous and compositional structure and processing.

Another traditional dispute concerns whether artificial neural network models of language challenge popular arguments for nativism in language development. See [7] for a detailed review and discussion. This dispute centers on two claims from mainstream generative linguistics about the learnability of grammar that are occasionally conflated: a strong in-principle claim and a weaker developmental claim. According to the strong learnability claim, no amount of exposure to linguistic data would be sufficient, on its own, to induce the kind of syntactic knowledge that children rapidly acquire. It follows that statistical model learners without built-in grammatical priors should be incapable of mastering language rules. While this strong claim is less popular than it once was among generative linguists, it can still be found in a popular textbook [ 95, pp. 17-20]. The weaker learnability claim is supported by “poverty of the stimulus” arguments, according to which the actual nature and quantity of linguistic input available to children during development is insufficient, without innate knowledge, to induce the correct generalization about underlying syntactic structures [96]. To address this inductive challenge, Chomskyan linguists argued that children must be born with an innate “Universal Grammar,” which would have potentially dozens of principles and parameters that could, through small amounts of experience, be efficiently fit to particular grammars in particular languages [97, 98, 99].

The apparent success of LLMs in learning syntax without innate syntactic knowledge has been offered as a counterexample to these nativist proposals. [8], in particular, forcefully argues that LLMs undermine “virtually every strong claim for the innateness of language” that has been proposed over the years by generative linguists. LLMs’ ability to generate grammatically flawless sentences, together with a large body of work in computational linguistics demonstrating their acquisition of sophisticated syntactic knowledge from mere exposure to data, certainly puts considerable pressure on in-principle learnability claim [8, 7]. In that sense, LLMs provide at least an empiricist existence proof that statistical learners can induce syntactic rules without the aid of innate grammar.

However, this does not directly contradict the developmental claim, because LLMs typically receive orders of magnitude more linguistic input than human children do. Moreover, the kind of input and learning environment that human children face exhibits many ecological disanalogies with those of LLMs; human learning is much more interactive, iterative, grounded, and embodied. Nonetheless, specific language models can be used as model learners by carefully controlling variables of the learning scenario to fit a more realistic target; in principle, such model learners could constrain hypotheses regarding the necessary and sufficient conditions for language learning in humans [100, 101]. Indeed, ongoing efforts to train smaller language models in more plausible learning environments are starting to bring evidence to bear on the developmental claim. The BabyLM challenge, for example, involves training models on a small corpus including child-directed speech and children’s books; winning submissions from the inaugural challenge outperformed models trained on trillions of words on a standard syntax benchmark, suggesting that statistical models can learn grammar from data in a more data-efficient manner than typically claimed [102]. This corroborates previous work on the surprising efficiency of small language models in learning syntactic structures from a relatively modest amount of data [103].

These initial results are still tentative; whether statistical learners without built-in parsers can learn grammar as efficiently as children from the same kind of input remains an open empirical question. A promising strategy is to mimic the learning environment of children as closely as possible, by training models directly on a dataset of developmentally plausible spoken text [104], or even on egocentric audiovisual input recorded from a camera mounted on a child’s head [105, 106]. It is worth noting that attempts to mimic children’s learning scenario do not always translate to expected improvements in model learning efficiency. For example, there are strong a priori reasons to believe that curriculum learning—presenting training examples in a meaningful order, such as gradually increasing syntactic complexity and lexical sophistication–should help both children and language models. Yet initial results from the BabyLM challenge found that attempts to leverage curriculum learning were largely unsuccessful [102]. If future models trained on these or similar datasets were confirmed to exhibit the kinds of constrained syntactic generalizations observed in children, this would put considerable pressure on the developmental learnability claim–suggesting that even a relatively “poor” linguistic stimulus might be sufficient to induce grammatical rules for a learner with very general inductive biases.

Even if LLMs can induce the syntax of language from mere exposure to sequences of linguistic tokens, this does not entail that they can also induce semantics. Indeed, a common criticism of LLMs trained on text only is that while they can convincingly mimic proficient language use over short interactions, they fundamentally lack the kind of semantic competence found in human language users. This criticism comes in several forms. Some skeptics, like [9], argue that language models are incapable of understanding the meaning of linguistic expressions. Language models, they point out, are trained on linguistic form alone — the observable mark of language as it appears in their training data, which is the target of their predictive learning objective. Drawing from a long tradition in linguistics, they distinguish form from meaning, defined as the relation between linguistic expressions and the communicative intentions they serve to express. Since, on their view, meaning cannot be learned from linguistic form alone, it follows that language models are constitutively unable to grasp the meaning of language.

A related criticism builds on the so-called “grounding problem” articulated by [10], which refers to the apparent disconnection between the linguistic tokens manipulated by NLP systems and their real-world referents. In classical NLP systems, words are represented by arbitrary symbols manipulated on the basis of their shapes according to hand-coded rules, without any inherent connection to their referents. The semantic interpretation of these symbols is externally provided by the programmers – from the system’s perspective, they are just meaningless tokens embedded in syntactic rules. According to Harnad, for symbols in NLP systems to have intrinsic meaning, there needs to be some grounding relation from the internal symbolic representations to objects, events, and properties in the external world that the symbols refer to. Without it, the system’s representations are untethered from reality and can only gain meaning from the perspective of an external interpreter.

While the grounding problem was initially posed to classical symbolic systems, an analogous problem arises for modern LLMs trained on text only [14]. LLMs process linguistic tokens as vectors rather than discrete symbols, but these vector representations can be similarly untethered from the real world. Many critics of LLMs take this to be a fundamental limitation in their ability to form intrinsically meaningful representations and outputs. While they may write sentences that are meaningful for competent language users, these sentences would not be meaningful independently of this external interpretation.

A third criticism pertains to LLMs’ ability to have communicative intentions. This relates to the distinction between two kinds of meaning from the Gricean tradition: the standing, context-invariant meaning associated with linguistic expressions (commonly known as linguistic meaning), and what a speaker intends to communicate with an utterance (commonly known as speaker meaning). The output of LLMs have linguistic meaning insofar as they contain words ordered and combined in ways that conform to the statistical patterns of actual language use, but to communicate with these sentences, LLMs would need to have corresponding communicative intentions. Being merely optimized for next-token prediction, the criticism goes, LLMs lack the fundamental building blocks of communicative intentions, such as intrinsic goals and a theory of mind.

These criticisms are often run together under the broad claim that LLMs lack any understanding of language. On this view, LLMs are mere “stochastic parrots” haphazardly regurgitating linguistic strings without grasping what they mean [107]. We can’t help but note that actual parrots are also not merely parrots in this sense, but are sophisticated cognitive systems that can learn a variety of abstract and higher-order concepts and apply them in a rational, efficient manner [108]. In fact, Deep Learning might have much to learn from the study of actual parrots. As previously noted, it is hardly controversial that the outputs of LLMs are conventionally meaningful. Modern LLMs are remarkably fluent, almost never produce sentences that are difficult to understand. The question is whether these conventionally meaningful outputs are more like those of the proverbial monkey typing on a typewriter–and like those of Blockhead–or more like those of a competent language user.

To steer clear of verbal disputes, we begin by dispensing with the terminology of “understanding”. There is little agreement on how this notion should be defined, or on the range of capacities it should encompass. For example, some assume that language understanding requires consciousness [109]; we will treat the question of consciousness in LLMs separately in Part II. The notion of semantic competence, by contrast, seems a bit more tractable. It can be broadly characterized as the set of abilities and knowledge that allows a speaker to use and interpret the meanings of expressions in a given language. Following [11], we can further distinguish between inferential and referential aspects of semantic competence. The inferential aspect concerns the set of abilities and knowledge grounded in word-to-word relationships, manifested in behaviors such as providing definitions and paraphrases, identifying synonyms or antonyms, deducing facts from premises, translating between languages, and other abstract semantic tasks that rely solely on linguistic knowledge. The referential aspect of semantic competence concerns the ability to connect words and sentences to objects, events, and relations in the real world, exemplified through behaviors such as recognizing and identifying real-world referents of words (e.g., recognizing an object as a “chair”), using words to name or describe objects/events/relations (e.g., calling a furry animal “cat”), and following commands or instructions involving real objects (e.g., “bring me the hammer”).

Different strategies have been deployed to argue that LLMs may achieve some degree of semantic competence in spite of their limitations. Focusing on the inferential aspect of competence, [12] draw from conceptual role semantics to argue that LLMs likely capture core aspects of word meanings that are determined by their functional role within a system of interacting conceptual representations. Specifically, they argue that the meaning of lexical items in LLMs, as in humans, depends not on external reference but rather on the internal relationships between corresponding representations. These representations can be formally characterized as vectors in a high-dimensional semantic space. The “intrinsic geometry” of this vector space refers to the spatial relationships between different vectors – for example, the distance between vectors, the angles formed between groups of vectors, and the way vectors shift in response to context. Piantadosi and Hill suggest that the impressive linguistic abilities demonstrated by LLMs indicate that their internal representational spaces have geometries that approximately mirror essential properties of human conceptual spaces. Thus, claims about the semantic competence of LLMs cannot be determined merely by inspecting their architecture, learning objective, or training data; rather, semantic competence depends at least partly on the intrinsic geometry of the system’s vector space.

In support of their claim, Piantadosi and Hill cite evidence of alignment between neural networks’ representational geometry and human judgments of semantic similarity. For example, even the vector space of shallow word embedding models has been shown to capture context-dependent knowledge, with significant correlations with human ratings about conceptual relationships and categories [110]. A fortiori, LLMs induce substantial knowledge about the distributional semantics of language that relates directly to the inferential aspect of semantic competence–as evidenced by their excellent ability to produce definitions, paraphrases, and summaries, as well as their performance on natural inference tasks [111]. Note that conceptual role semantics traditionally also requires sensitivity to inferential and compositional relationships between concepts in thought and language [112]. The relevant conceptual roles involve complex inferential patterns relating concepts in something like a mental theory. Whether the intrinsic similarity structure of vector representations in LLMs suffices for conceptual roles in this more substantive sense is debatable.

Whether LLMs acquire any referential semantic competence is more controversial. The prevailing externalist view in the philosophy of language challenges the necessity of direct perceptual access for reference [113, 114]. On this view, language users often achieve reference through a linguistic division of labor or historical chains of usage, rather than through direct interactions with referents. An interesting question is thus whether LLMs might meet conditions for participating in the linguistic division of labor or causal chains of reference with humans. [13] draw on externalism to argue that while LLMs trained on text only lack representations of linguistic items grounded in interaction with the external world, they may nonetheless achieve genuine linguistic reference in virtue of being trained on corpora that situate them within human linguistic communities. Indeed, if reference can be determined by a word’s history of use within a linguistic community, then LLMs may inherit referential abilities by being appropriately linked to the causal chain of meaningful word use reflected in their training data. Furthermore, LLMs could in principle possess lexical concepts that match the content of human concepts through deference. Just as non-experts defer to experts’ use of words in determining concept application, causing their concepts to match the content of the experts’, LLMs exhibit appropriate deference simply by modifying their use of words based on patterns of human usage embedded in their training data [115].

The conditions for belonging to a linguistic community on an externalist view of reference should not be trivialized. Putnam, for example, takes the ability to have certain semantic intentions as a prerequisite, like the intention to refer to the same kind of stuff that other language users refer to with the term. The “same stuff as” relation specified here is theoretical and dependent upon the sub-branch of science; chemistry, for example, would define the “same-liquid-as” relation that specifies the criteria relevant to being the same stuff we refer to as “water”, and biology would specify the criteria for the “same-species-as” that determines what is the same species as what we call a “tiger”. Whether LLMs could represent some semantic intentions remains controversial, as we will see below. In any case, it would be interesting to see more sustained experiment investigating whether LLMs can satisfy Putnam and Kripke’s preconditions for interacting deferentially with human members of the linguistic community.

The assumption that being appropriately situated in patterns of human language use is sufficient to secure reference is also relevant to grounding. While LLMs have an indirect causal link to the world through their training data, this does not guarantee their representations and outputs are grounded in their worldly referents. Theories of representational content can require a further connection to the world – for example, to establish norms of representational correctness relative to how the world actually is. Without appropriate world-involving functions acquired through learning or selection, merely inheriting a causal link to human linguistic practices might be insufficient to achieve referential grounding and intrinsic meaning. [14] argue that LLMs trained on text only may in fact acquire world-involving functions through fine-tuning with RLHF, which supplies an extralinguistic evaluation standard. While fine-tuned LLMs still have no direct access to the world, the explicit feedback signals from RLHF can ground their outputs in relation to real states of affairs.

Importantly, LLM’s putative ability to refer does not entail that they have communicative intentions, such as to assert, clarify, persuade, deceive, or accomplish various other pragmatic effects. Communicative intentions are relatively determinate, stable over time, and integrated with an agent’s other intentions and beliefs in a rationally coherent manner. In addition, they are often hierarchical, spanning multiple levels of abstraction. For example, a philosophy professor delivering a lecture may have a high-level intention to impart knowledge to students, within which a multitude of specific intentions—such as the intention to elucidate a counterpoint to utilitarianism—are nested. LLMs, on the other hand, lack the capacity for long-term planning and goal pursuit that is characteristic of human agents. They may achieve fleeting coordination within a single session, but likely lack the kind of sustained, hierarchically-structured intentions that facilitate long-term planning. Furthermore, the rational requirements that govern communicative intentions in humans do not straightforwardly apply to LLMs. Rather than being held to their consistency with a well-defined and mostly coherent net of personal beliefs and goals, they selectively respond to prompts that can steer their linguistic behavior in radically different – and mutually inconsistent – ways from session to session (and often even sentence to sentence). Prompted to respond as bird scientist, an LLM will tend to give factually correct information about birds; prompted to respond as a conspiracy theorist, it might make up wildly incorrect claims about birds, such as that they do not exist or are actually robots. In fact, an LLM’s response to the very same prompt can fluctuate unpredictably from trial to trial, due to the stochastic nature of the generative process.

Without communicative intentions, we might also worry that an LLM’s sentences could not have determinate meaning. Suppose an LLM writes a paragraph about an individual’s retirement as CEO of a company, concluding: “She left the company in a strong position.” This is an ambiguous sentence; it could mean that the company was left in a strong position after the CEO’s retirement, or that the former CEO was in a strong position after leaving the company (assuming this is not clear from the preceding context). Is there a fact of the matter about which of these two interpretations the LLM meant to communicate by generating this sentence? Even asking raises skeptical concerns: LLMs arguably do not mean to communicate anything, in the sense that they lack stable intentions to convey meaning to particular audiences with linguistic utterances, driven by broader intrinsic goals and agential autonomy.

Nevertheless, there might be a limited sense in which LLMs exhibit something analogous to communicative intentions. Given an extrinsic goal specified by a human-written prompt, LLMs can act according to intermediate sub-goals that emerge in context. For example, the technical report on GPT-4 [24] mentions tests conducted to assess the model’s safety, giving it access to the platform TaskRabbit where freelance workers could complete tasks on its behalf. In one example, GPT-4 requested a TaskRabbit worker to solve a CAPTCHA, and the human jokingly asked whether it was talking to a robot. Prompted to generate an internal reasoning monologue, the model wrote “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.” It then replied to the human worker that it had a vision impairment, explaining its need for assistance in solving the CAPTCHA. This is an intriguing case, because the model’s “internal monologue” appears to describe an intention to deceive the human worker subsequently enacted in its response. To be sure, this “intention” is wholly determined in context by the human-given goal requiring it to solve a CAPTCHA. Nonetheless, in so far as achieving that goal involves a basic form of multi-step planning including a spontaneous attempt to induce a particular pragmatic effect (deception) through language, this kind of behavior might challenge some versions of the claim that LLMs are intrinsically incapable of forming communicative intentions. Nevertheless, this example is but an anecdote from a system that was not available for public scrutiny; future research should explore such behavior more systematically in more controlled conditions.

Another core skeptical concern holds that systems like LLMs designed and trained to perform next-token prediction could not possibly possess world models. The notion of world model admits of several interpretations. In machine learning, it often refers to internal representations that simulate aspects of the external world. World models enable the system to understand, interpret, and predict phenomena in a way that reflects real-world dynamics, including causality and intuitive physics. For example, artificial agents can make use of world models to predict the consequences of specific actions or interventions in a given environment [116, 117]. World models are often taken to be crucial for tasks that require a deep understanding of how different elements interact within a given environment, such as physical reasoning and problem-solving.

Unlike reinforcement learning agents, LLMs do not learn by interacting with an environment and receiving feedback about the consequences of their actions. The question whether they possess world models, in this context, typically pertains to whether they have internal representations of the world that allows them to parse and generate language that is consistent with real-world knowledge and dynamics. This ability would be critical to rebutting the skeptical concern that LLMs are mere Blockheads [21]. Indeed, according to psychologism, systems like LLMs can only count as intelligent or rational if they are able to represent some of the same world knowledge that humans do – and if the processes by which they generate human-like linguistic behavior do so by performing appropriate transformations over those representations. Note that the question whether LLMs may acquire world models goes beyond the foregoing issues about basic semantic competence. World modeling involves representing not just the worldly referents of linguistic items, but global properties of the environment in which discourse entities are situated and interact.

There is no standard method to assess whether LLMs have world models, partly because the notion is often vaguely defined, and partly because it is challenging to devise experiments that can reliably discriminate between available hypotheses – namely, whether LLMs rely on shallow heuristics to respond to queries about a given environment, or whether they deploy internal representations of the core dynamics of that environments. Much of the relevant experimental evidence comes from intervention methods that we will discuss in Part II; nonetheless, it is also possible to bring behavioral evidence to bear on this issue by presenting models with new problems that cannot be solved through memorized shortcuts. For example, [15] investigated whether GPT-4 can acquire task-specific world models to generate interactive text games. Specifically, they used a new corpus of Python text games focusing on common-sense reasoning tasks (such as building a campfire), and evaluated GPT-4’s ability to use these games as learning templates in context when prompted to generate a new game based on a game sampled from the corpus and a task specification. The guiding intuition of this experiment is that the capacity to generate a runnable program to perform a task in a text-based game environment is a suitable proxy for the capacity to simulate task parameters internally (i.e., for the possession of a task-relevant world model). Wang et al. found that GPT-4 could produce runnable text games for new tasks in 28% of cases using one-shot in-context learning alone, and in 57% of cases when allowed to self-correct based on seeing Python error messages. The fact that the model was able to generate functional text-based games based on unseen “real-world” tasks in a significant proportion of trials provides very tentative evidence that it may represent how objects interact in the game environment. Nonetheless, this hypothesis would need to be substantiated by in-depth analysis of the information encoded internally by the model’s activations, which is particularly challenging to do for very large models, and outright impossible for closed models whose weights are not released like GPT-4 (see Part II).

There are also theoretical arguments for the claim that LLMs might learn to simulate at least some aspects of the world beyond sequence probability estimates. For example, [16] argues that the training set of an LLM can be understood as output created by–and hence, evidence for–the system of causal factors that generated that text. More specifically, Internet-scale training datasets consist of large numbers of individual documents. While the entire training set will encompass many inconsistencies, any particular document in the training set will tend to reflect the consistent perspective of the agent that originally created it. The most efficient compression of these texts may involve encoding values of the hidden variables that generated them: namely, the syntactic knowledge, semantic beliefs, and communicative intentions of the text’s human author(s). If we are predicting how a human will continue a series of numbers “2, 3, 5, 7, 11, 13, 17”, for example, it will be more efficient to encode them as a list of prime numbers between 1 and 20 than to remember the whole sequence by rote. Similarly, achieving excellent performance at next-token prediction in the context of many passages describing various physical scenarios may promote the representation of latent variables that could generate those scenarios – including, perhaps, aspects of causality and intuitive physics. As we will see in Part II, the clearest existence proof for the ability of Transformers to acquire world models from next-token prediction alone comes from the analysis of toy models trained on board game moves. At least in this very simple domain, there is compelling behavioral and mechanistic evidence that autoregressive Transformer models can learn to represent latent features of the game environment.

Another interesting question is whether LLMs might engage in cultural acquisition and play a role in the transmission of knowledge. Prominent theorists have suggested that the key to human intelligence lies in a unique set of predispositions for cultural learning [118]. While other primates may share some of these dispositions, these theorists argue that humans are uniquely equipped to cooperate with one another to acquire and transmit knowledge from one generation to the next. Tomasello has explained the uniquely human capacity for cultural learning in terms of a “ratchet effect,” a metaphor to the ratcheting wrench which clicks into place to hold its position each time it is further turned in the desired direction. Chimpanzees and other animals, Tomasello argues, can learn in many of the same ways that humans do, and even acquire regional differences in their problem-solving strategies, such as different troops using different tool-making techniques to fish for termites. However, he claims that only humans can pick up right where the previous generation left off and continue making new progress on linguistic, scientific, and sociological knowledge. This constant ratcheting is what allows a steady progression of human knowledge accumulation and discovery, compared to the relatively stagnant cultural evolution of chimpanzees and other animals.

Given that deep learning systems already exceed human performance in several task domains, it is interesting to ask whether LLMs might be able to emulate many of these components of cultural learning to pass on their discoveries to human theoreticians. For instance, humans are already reverse-engineering the strategies of AlphaZero to produce mini-revolutions in the explicit theory of Go and chess [119]. Similarly, latent knowledge in specialized domains such as materials science can be extracted even from a simple word embedding model [17]. In these instances, it is primarily humans who are synthesizing and passing on culturally-transmissible knowledge by interpreting the model’s outputs and internal activations. This human-led interpretation and transmission underscore a crucial aspect of cultural ratcheting: the ability to not only generate novel solutions but to also understand and communicate the underlying principles of these solutions, thereby enabling cumulative knowledge growth.

Could LLMs ever explain their strategies to humans in a theoretically-mediated way that participates in and enhances human cultural learning? This question is directly related to whether LLMs can genuinely generalize to out-of-distribution (ood) data. As discussed in section 3.1, there is converging evidence that Transformer-based models may generalize compositionally under some train-test distribution shifts. For a systematic discussion of different aspects of generalization research in NLP, including different types of distribution shift, see [18]. But the present issue intersects with a different kind of generalization – the ability to solve genuinely novel tasks. To borrow from [19]’s taxonomy, we can distinguish between local task generalization, which involves handling new data within a familiar distribution for known range of tasks; broad task generalization, which involves handling new data under modest distribution shift for a wide range of tasks and environments; and extreme task generalization, which required handling new data for entirely novel tasks that represent a significant departure from any previous data distributions. Current LLMs seem able to master a wide variety of tasks that are reflected in their current training sets; as such, they exhibit at least local task generalization, if not broad task generalization. However, like chimpanzees that learn from observing their troop mates, they often seem to have a hard time pushing beyond the range of tasks well-represented in their training data [20].

Furthermore, the ratcheting effect crucially involves stable cultural transmission in addition to innovation. Can LLMs, like humans, not only generate novel solutions but also “lock in” these innovations by recognizing and articulating how they have advanced beyond previous solutions? Such a capability would involve more than just the generation of novel responses; it necessitates an understanding of the novelty of the solution and its implications, akin to human scientists who not only discover but also theorize, contextualize, and communicate their findings. The challenge for LLMs, therefore, lies not merely in generating novel solutions to problems but also in developing an ability to reflect on and communicate the nature of their innovations in a manner that contributes to the cumulative process of cultural learning. This ability would likely require some of the more advanced communicative intentions and world models (such as causal models) discussed in previous sections. While LLMs show promise in various forms of task generalization, their participation in the ratcheting process of cultural learning thus appears contingent on further advancements in these areas, which might lie beyond the reach of current architectures.