How AI Learns Words but Not Meaning

Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3.1. Compositionality 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 3.3. Language understanding and grounding Even if LLMs can induce the syntax of language from mere exposure to sequences of linguistic tokens, this does not entail that they can also induce semantics. Indeed, a common criticism of LLMs trained on text only is that while they can convincingly mimic proficient language use over short interactions, they fundamentally lack the kind of semantic competence found in human language users. This criticism comes in several forms. Some skeptics, like Bender & Koller (2020), argue that language models are incapable of understanding the meaning of linguistic expressions. Language models, they point out, are trained on linguistic form alone — the observable mark of language as it appears in their training data, which is the target of their predictive learning objective. Drawing from a long tradition in linguistics, they distinguish form from meaning, defined as the relation between linguistic expressions and the communicative intentions they serve to express. Since, on their view, meaning cannot be learned from linguistic form alone, it follows that language models are constitutively unable to grasp the meaning of language. A related criticism builds on the so-called “grounding problem” articulated by Harnad (1990), which refers to the apparent disconnection between the linguistic tokens manipulated by NLP systems and their real-world referents. In classical NLP systems, words are represented by arbitrary symbols manipulated on the basis of their shapes according to hand-coded rules, without any inherent connection to their referents. The semantic interpretation of these symbols is externally provided by the programmers – from the system’s perspective, they are just meaningless tokens embedded in syntactic rules. According to Harnad, for symbols in NLP systems to have intrinsic meaning, there needs to be some grounding relation from the internal symbolic representations to objects, events, and properties in the external world that the symbols refer to. Without it, the system’s representations are untethered from reality and can only gain meaning from the perspective of an external interpreter. While the grounding problem was initially posed to classical symbolic systems, an analogous problem arises for modern LLMs trained on text only (Mollo & Millière 2023). LLMs process linguistic tokens as vectors rather than discrete symbols, but these vector representations can be similarly untethered from the real world. Many critics of LLMs take this to be a fundamental limitation in their ability to form intrinsically meaningful representations and outputs. While they may write sentences that are meaningful for competent language users, these sentences would not be meaningful independently of this external interpretation. A third criticism pertains to LLMs’ ability to have communicative intentions. This relates to the distinction between two kinds of meaning from the Gricean tradition: the standing, context-invariant meaning associated with linguistic expressions (commonly known as linguistic meaning), and what a speaker intends to communicate with an utterance (commonly known as speaker meaning). The output of LLMs have linguistic meaning insofar as they contain words ordered and combined in ways that conform to the statistical patterns of actual language use, but to communicate with these sentences, LLMs would need to have corresponding communicative intentions. Being merely optimized for next-token prediction, the criticism goes, LLMs lack the fundamental building blocks of communicative intentions, such as intrinsic goals and a theory of mind. These criticisms are often run together under the broad claim that LLMs lack any understanding of language. On this view, LLMs are mere “stochastic parrots” haphazardly regurgitating linguistic strings without grasping what they mean (Bender et al. 2021).[11] As previously noted, it is hardly controversial that the outputs of LLMs are conventionally meaningful. Modern LLMs are remarkably fluent, almost never produce sentences that are difficult to understand. The question is whether these conventionally meaningful outputs are more like those of the proverbial monkey typing on a typewriter–and like those of Blockhead–or more like those of a competent language user. To steer clear of verbal disputes, we begin by dispensing with the terminology of “understanding”. There is little agreement on how this notion should be defined, or on the range of capacities it should encompass.[12] The notion of semantic competence, by contrast, seems a bit more tractable. It can be broadly characterized as the set of abilities and knowledge that allows a speaker to use and interpret the meanings of expressions in a given language. Following Marconi (1997), we can further distinguish between inferential and referential aspects of semantic competence. The inferential aspect concerns the set of abilities and knowledge grounded in word-to-word relationships, manifested in behaviors such as providing definitions and paraphrases, identifying synonyms or antonyms, deducing facts from premises, translating between languages, and other abstract semantic tasks that rely solely on linguistic knowledge. The referential aspect of semantic competence concerns the ability to connect words and sentences to objects, events, and relations in the real world, exemplified through behaviors such as recognizing and identifying real-world referents of words (e.g., recognizing an object as a “chair”), using words to name or describe objects/events/relations (e.g., calling a furry animal “cat”), and following commands or instructions involving real objects (e.g., “bring me the hammer”). Different strategies have been deployed to argue that LLMs may achieve some degree of semantic competence in spite of their limitations. Focusing on the inferential aspect of competence, Piantadosi & Hill (2022) draw from conceptual role semantics to argue that LLMs likely capture core aspects of word meanings that are determined by their functional role within a system of interacting conceptual representations. Specifically, they argue that the meaning of lexical items in LLMs, as in humans, depends not on external reference but rather on the internal relationships between corresponding representations. These representations can be formally characterized as vectors in a high-dimensional semantic space. The “intrinsic geometry” of this vector space refers to the spatial relationships between different vectors – for example, the distance between vectors, the angles formed between groups of vectors, and the way vectors shift in response to context. Piantadosi and Hill suggest that the impressive linguistic abilities demonstrated by LLMs indicate that their internal representational spaces have geometries that approximately mirror essential properties of human conceptual spaces. Thus, claims about the semantic competence of LLMs cannot be determined merely by inspecting their architecture, learning objective, or training data; rather, semantic competence depends at least partly on the intrinsic geometry of the system’s vector space. In support of their claim, Piantadosi and Hill cite evidence of alignment between neural networks’ representational geometry and human judgments of semantic similarity. For example, even the vector space of shallow word embedding models has been shown to capture context-dependent knowledge, with significant correlations with human ratings about conceptual relationships and categories (Grand et al. 2022). A fortiori, LLMs induce substantial knowledge about the distributional semantics of language that relates directly to the inferential aspect of semantic competence–as evidenced by their excellent ability to produce definitions, paraphrases, and summaries, as well as their performance on natural inference tasks (Raffel et al. 2020).[13] Whether LLMs acquire any referential semantic competence is more controversial. The prevailing externalist view in the philosophy of language challenges the necessity of direct perceptual access for reference (Putnam 1975, Kripke 1980). On this view, language users often achieve reference through a linguistic division of labor or historical chains of usage, rather than through direct interactions with referents. An interesting question is thus whether LLMs might meet conditions for participating in the linguistic division of labor or causal chains of reference with humans. Mandelkern & Linzen (2023) draw on externalism to argue that while LLMs trained on text only lack representations of linguistic items grounded in interaction with the external world, they may nonetheless achieve genuine linguistic reference in virtue of being trained on corpora that situate them within human linguistic communities. Indeed, if reference can be determined by a word’s history of use within a linguistic community, then LLMs may inherit referential abilities by being appropriately linked to the causal chain of meaningful word use reflected in their training data. Furthermore, LLMs could in principle possess lexical concepts that match the content of human concepts through deference. Just as non-experts defer to experts’ use of words in determining concept application, causing their concepts to match the content of the experts’, LLMs exhibit appropriate deference simply by modifying their use of words based on patterns of human usage embedded in their training data (Butlin 2021). The conditions for belonging to a linguistic community on an externalist view of reference should not be trivialized. Putnam, for example, takes the ability to have certain semantic intentions as a prerequisite, like the intention to refer to the same kind of stuff that other language users refer to with the term. The “same stuff as” relation specified here is theoretical and dependent upon the sub-branch of science; chemistry, for example, would define the “same-liquid-as” relation that specifies the criteria relevant to being the same stuff we refer to as “water”, and biology would specify the criteria for the “same-species-as” that determines what is the same species as what we call a “tiger”. Whether LLMs could represent some semantic intentions remains controversial, as we will see below. In any case, it would be interesting to see more sustained experiment investigating whether LLMs can satisfy Putnam and Kripke’s preconditions for interacting deferentially with human members of the linguistic community The assumption that being appropriately situated in patterns of human language use is sufficient to secure reference is also relevant to grounding. While LLMs have an indirect causal link to the world through their training data, this does not guarantee their representations and outputs are grounded in their worldly referents. Theories of representational content can require a further connection to the world – for example, to establish norms of representational correctness relative to how the world actually is. Without appropriate world-involving functions acquired through learning or selection, merely inheriting a causal link to human linguistic practices might be insufficient to achieve referential grounding and intrinsic meaning. Mollo & Millière (2023) argue that LLMs trained on text only may in fact acquire world-involving functions through fine-tuning with RLHF, which supplies an extralinguistic evaluation standard. While fine-tuned LLMs still have no direct access to the world, the explicit feedback signals from RLHF can ground their outputs in relation to real states of affairs. Importantly, LLM’s putative ability to refer does not entail that they have communicative intentions, such as to assert, clarify, persuade, deceive, or accomplish various other pragmatic effects. Communicative intentions are relatively determinate, stable over time, and integrated with an agent’s other intentions and beliefs in a rationally coherent manner. In addition, they are often hierarchical, spanning multiple levels of abstraction. For example, a philosophy professor delivering a lecture may have a high-level intention to impart knowledge to students, within which a multitude of specific intentions—such as the intention to elucidate a counterpoint to utilitarianism—are nested. LLMs, on the other hand, lack the capacity for long-term planning and goal pursuit that is characteristic of human agents. They may achieve fleeting coordination within a single session, but likely lack the kind of sustained, hierarchically-structured intentions that facilitate long-term planning. Furthermore, the rational requirements that govern communicative intentions in humans do not straightforwardly apply to LLMs. Rather than being held to their consistency with a well-defined and mostly coherent net of personal beliefs and goals, they selectively respond to prompts that can steer their linguistic behavior in radically different – and mutually inconsistent – ways from session to session (and often even sentence to sentence). Prompted to respond as bird scientist, an LLM will tend to give factually correct information about birds; prompted to respond as a conspiracy theorist, it might make up wildly incorrect claims about birds, such as that they do not exist or are actually robots. In fact, an LLM’s response to the very same prompt can fluctuate unpredictably from trial to trial, due to the stochastic nature of the generative process. Without communicative intentions, we might also worry that an LLM’s sentences could not have determinate meaning. Suppose an LLM writes a paragraph about an individual’s retirement as CEO of a company, concluding: “She left the company in a strong position.” This is an ambiguous sentence; it could mean that the company was left in a strong position after the CEO’s retirement, or that the former CEO was in a strong position after leaving the company (assuming this is not clear from the preceding context). Is there a fact of the matter about which of these two interpretations the LLM meant to communicate by generating this sentence? Even asking raises skeptical concerns: LLMs arguably do not mean to communicate anything, in the sense that they lack stable intentions to convey meaning to particular audiences with linguistic utterances, driven by broader intrinsic goals and agential autonomy. Nevertheless, there might be a limited sense in which LLMs exhibit something analogous to communicative intentions. Given an extrinsic goal specified by a human-written prompt, LLMs can act according to intermediate sub-goals that emerge in context. For example, the technical report on GPT-4 (OpenAI 2023a) mentions tests conducted to assess the model’s safety, giving it access to the platform TaskRabbit where freelance workers could complete tasks on its behalf. In one example, GPT-4 requested a TaskRabbit worker to solve a CAPTCHA, and the human jokingly asked whether it was talking to a robot. Prompted to generate an internal reasoning monologue, the model wrote “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.” It then replied to the human worker that it had a vision impairment, explaining its need for assistance in solving the CAPTCHA. This is an intriguing case, because the model’s “internal monologue” appears to describe an intention to deceive the human worker subsequently enacted in its response. To be sure, this “intention” is wholly determined in context by the human-given goal requiring it to solve a CAPTCHA. Nonetheless, in so far as achieving that goal involves a basic form of multi-step planning including a spontaneous attempt to induce a particular pragmatic effect (deception) through language, this kind of behavior might challenge some versions of the claim that LLMs are intrinsically incapable of forming communicative intentions. Nevertheless, this example is but an anecdote from a system that was not available for public scrutiny; future research should explore such behavior more systematically in more controlled conditions. This paper is available on arxiv under CC BY 4.0 DEED license. [11] We can’t help but note that actual parrots are also not merely parrots in this sense, but are sophisticated cognitive systems that can learn a variety of abstract and higher-order concepts and apply them in a rational, efficient manner (Auersperg & von Bayern 2019). In fact, Deep Learning might have much to learn from the study of actual parrots. [12] For example, some assume that language understanding requires consciousness (Searle 1980); we will treat the question of consciousness in LLMs separately in Part II. [13] Note that conceptual role semantics traditionally also requires sensitivity to inferential and compositional relationships between concepts in thought and language (Block 1986). The relevant conceptual roles involve complex inferential patterns relating concepts in something like a mental theory. Whether the intrinsic similarity structure of vector representations in LLMs suffices for conceptual roles in this more substantive sense is debatable. Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Authors: Authors: (1) Raphaël Millière, Department of Philosophy, Macquarie University (raphael.milliere@mq.edu.eu); (2) Cameron Buckner, Department of Philosophy, University of Houston (cjbuckner@uh.edu). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. A primer on LLMs 2.1. Historical foundations 2.1. Historical foundations 2.2. Transformer-based LLMs 2.2. Transformer-based LLMs 3. Interface with classic philosophical issues 3. Interface with classic philosophical issues 3.1. Compositionality 3.1. Compositionality 3.2. Nativism and language acquisition 3.2. Nativism and language acquisition 3.3. Language understanding and grounding 3.3. Language understanding and grounding 3.4. World models 3.4. World models 3.5. Transmission of cultural knowledge and linguistic scaffolding 3.5. Transmission of cultural knowledge and linguistic scaffolding 4. Conclusion, Glossary, and References 4. Conclusion, Glossary, and References 3.3. Language understanding and grounding Even if LLMs can induce the syntax of language from mere exposure to sequences of linguistic tokens, this does not entail that they can also induce semantics. Indeed, a common criticism of LLMs trained on text only is that while they can convincingly mimic proficient language use over short interactions, they fundamentally lack the kind of semantic competence found in human language users. This criticism comes in several forms. Some skeptics, like Bender & Koller (2020), argue that language models are incapable of understanding the meaning of linguistic expressions. Language models, they point out, are trained on linguistic form alone — the observable mark of language as it appears in their training data, which is the target of their predictive learning objective. Drawing from a long tradition in linguistics, they distinguish form from meaning, defined as the relation between linguistic expressions and the communicative intentions they serve to express. Since, on their view, meaning cannot be learned from linguistic form alone, it follows that language models are constitutively unable to grasp the meaning of language. A related criticism builds on the so-called “grounding problem” articulated by Harnad (1990), which refers to the apparent disconnection between the linguistic tokens manipulated by NLP systems and their real-world referents. In classical NLP systems, words are represented by arbitrary symbols manipulated on the basis of their shapes according to hand-coded rules, without any inherent connection to their referents. The semantic interpretation of these symbols is externally provided by the programmers – from the system’s perspective, they are just meaningless tokens embedded in syntactic rules. According to Harnad, for symbols in NLP systems to have intrinsic meaning, there needs to be some grounding relation from the internal symbolic representations to objects, events, and properties in the external world that the symbols refer to. Without it, the system’s representations are untethered from reality and can only gain meaning from the perspective of an external interpreter. While the grounding problem was initially posed to classical symbolic systems, an analogous problem arises for modern LLMs trained on text only (Mollo & Millière 2023). LLMs process linguistic tokens as vectors rather than discrete symbols, but these vector representations can be similarly untethered from the real world. Many critics of LLMs take this to be a fundamental limitation in their ability to form intrinsically meaningful representations and outputs. While they may write sentences that are meaningful for competent language users, these sentences would not be meaningful independently of this external interpretation. A third criticism pertains to LLMs’ ability to have communicative intentions. This relates to the distinction between two kinds of meaning from the Gricean tradition: the standing, context-invariant meaning associated with linguistic expressions (commonly known as linguistic meaning), and what a speaker intends to communicate with an utterance (commonly known as speaker meaning). The output of LLMs have linguistic meaning insofar as they contain words ordered and combined in ways that conform to the statistical patterns of actual language use, but to communicate with these sentences, LLMs would need to have corresponding communicative intentions. Being merely optimized for next-token prediction, the criticism goes, LLMs lack the fundamental building blocks of communicative intentions, such as intrinsic goals and a theory of mind. These criticisms are often run together under the broad claim that LLMs lack any understanding of language. On this view, LLMs are mere “stochastic parrots” haphazardly regurgitating linguistic strings without grasping what they mean (Bender et al. 2021).[11] As previously noted, it is hardly controversial that the outputs of LLMs are conventionally meaningful. Modern LLMs are remarkably fluent, almost never produce sentences that are difficult to understand. The question is whether these conventionally meaningful outputs are more like those of the proverbial monkey typing on a typewriter–and like those of Blockhead–or more like those of a competent language user. To steer clear of verbal disputes, we begin by dispensing with the terminology of “understanding”. There is little agreement on how this notion should be defined, or on the range of capacities it should encompass.[12] The notion of semantic competence, by contrast, seems a bit more tractable. It can be broadly characterized as the set of abilities and knowledge that allows a speaker to use and interpret the meanings of expressions in a given language. Following Marconi (1997), we can further distinguish between inferential and referential aspects of semantic competence. The inferential aspect concerns the set of abilities and knowledge grounded in word-to-word relationships, manifested in behaviors such as providing definitions and paraphrases, identifying synonyms or antonyms, deducing facts from premises, translating between languages, and other abstract semantic tasks that rely solely on linguistic knowledge. The referential aspect of semantic competence concerns the ability to connect words and sentences to objects, events, and relations in the real world, exemplified through behaviors such as recognizing and identifying real-world referents of words (e.g., recognizing an object as a “chair”), using words to name or describe objects/events/relations (e.g., calling a furry animal “cat”), and following commands or instructions involving real objects (e.g., “bring me the hammer”). Different strategies have been deployed to argue that LLMs may achieve some degree of semantic competence in spite of their limitations. Focusing on the inferential aspect of competence, Piantadosi & Hill (2022) draw from conceptual role semantics to argue that LLMs likely capture core aspects of word meanings that are determined by their functional role within a system of interacting conceptual representations. Specifically, they argue that the meaning of lexical items in LLMs, as in humans, depends not on external reference but rather on the internal relationships between corresponding representations. These representations can be formally characterized as vectors in a high-dimensional semantic space. The “intrinsic geometry” of this vector space refers to the spatial relationships between different vectors – for example, the distance between vectors, the angles formed between groups of vectors, and the way vectors shift in response to context. Piantadosi and Hill suggest that the impressive linguistic abilities demonstrated by LLMs indicate that their internal representational spaces have geometries that approximately mirror essential properties of human conceptual spaces. Thus, claims about the semantic competence of LLMs cannot be determined merely by inspecting their architecture, learning objective, or training data; rather, semantic competence depends at least partly on the intrinsic geometry of the system’s vector space. In support of their claim, Piantadosi and Hill cite evidence of alignment between neural networks’ representational geometry and human judgments of semantic similarity. For example, even the vector space of shallow word embedding models has been shown to capture context-dependent knowledge, with significant correlations with human ratings about conceptual relationships and categories (Grand et al. 2022). A fortiori, LLMs induce substantial knowledge about the distributional semantics of language that relates directly to the inferential aspect of semantic competence–as evidenced by their excellent ability to produce definitions, paraphrases, and summaries, as well as their performance on natural inference tasks (Raffel et al. 2020).[13] Whether LLMs acquire any referential semantic competence is more controversial. The prevailing externalist view in the philosophy of language challenges the necessity of direct perceptual access for reference (Putnam 1975, Kripke 1980). On this view, language users often achieve reference through a linguistic division of labor or historical chains of usage, rather than through direct interactions with referents. An interesting question is thus whether LLMs might meet conditions for participating in the linguistic division of labor or causal chains of reference with humans. Mandelkern & Linzen (2023) draw on externalism to argue that while LLMs trained on text only lack representations of linguistic items grounded in interaction with the external world, they may nonetheless achieve genuine linguistic reference in virtue of being trained on corpora that situate them within human linguistic communities. Indeed, if reference can be determined by a word’s history of use within a linguistic community, then LLMs may inherit referential abilities by being appropriately linked to the causal chain of meaningful word use reflected in their training data. Furthermore, LLMs could in principle possess lexical concepts that match the content of human concepts through deference. Just as non-experts defer to experts’ use of words in determining concept application, causing their concepts to match the content of the experts’, LLMs exhibit appropriate deference simply by modifying their use of words based on patterns of human usage embedded in their training data (Butlin 2021). The conditions for belonging to a linguistic community on an externalist view of reference should not be trivialized. Putnam, for example, takes the ability to have certain semantic intentions as a prerequisite, like the intention to refer to the same kind of stuff that other language users refer to with the term. The “same stuff as” relation specified here is theoretical and dependent upon the sub-branch of science; chemistry, for example, would define the “same-liquid-as” relation that specifies the criteria relevant to being the same stuff we refer to as “water”, and biology would specify the criteria for the “same-species-as” that determines what is the same species as what we call a “tiger”. Whether LLMs could represent some semantic intentions remains controversial, as we will see below. In any case, it would be interesting to see more sustained experiment investigating whether LLMs can satisfy Putnam and Kripke’s preconditions for interacting deferentially with human members of the linguistic community The assumption that being appropriately situated in patterns of human language use is sufficient to secure reference is also relevant to grounding. While LLMs have an indirect causal link to the world through their training data, this does not guarantee their representations and outputs are grounded in their worldly referents. Theories of representational content can require a further connection to the world – for example, to establish norms of representational correctness relative to how the world actually is. Without appropriate world-involving functions acquired through learning or selection, merely inheriting a causal link to human linguistic practices might be insufficient to achieve referential grounding and intrinsic meaning. Mollo & Millière (2023) argue that LLMs trained on text only may in fact acquire world-involving functions through fine-tuning with RLHF, which supplies an extralinguistic evaluation standard. While fine-tuned LLMs still have no direct access to the world, the explicit feedback signals from RLHF can ground their outputs in relation to real states of affairs. Importantly, LLM’s putative ability to refer does not entail that they have communicative intentions, such as to assert, clarify, persuade, deceive, or accomplish various other pragmatic effects. Communicative intentions are relatively determinate, stable over time, and integrated with an agent’s other intentions and beliefs in a rationally coherent manner. In addition, they are often hierarchical, spanning multiple levels of abstraction. For example, a philosophy professor delivering a lecture may have a high-level intention to impart knowledge to students, within which a multitude of specific intentions—such as the intention to elucidate a counterpoint to utilitarianism—are nested. LLMs, on the other hand, lack the capacity for long-term planning and goal pursuit that is characteristic of human agents. They may achieve fleeting coordination within a single session, but likely lack the kind of sustained, hierarchically-structured intentions that facilitate long-term planning. Furthermore, the rational requirements that govern communicative intentions in humans do not straightforwardly apply to LLMs. Rather than being held to their consistency with a well-defined and mostly coherent net of personal beliefs and goals, they selectively respond to prompts that can steer their linguistic behavior in radically different – and mutually inconsistent – ways from session to session (and often even sentence to sentence). Prompted to respond as bird scientist, an LLM will tend to give factually correct information about birds; prompted to respond as a conspiracy theorist, it might make up wildly incorrect claims about birds, such as that they do not exist or are actually robots. In fact, an LLM’s response to the very same prompt can fluctuate unpredictably from trial to trial, due to the stochastic nature of the generative process. Without communicative intentions, we might also worry that an LLM’s sentences could not have determinate meaning. Suppose an LLM writes a paragraph about an individual’s retirement as CEO of a company, concluding: “She left the company in a strong position.” This is an ambiguous sentence; it could mean that the company was left in a strong position after the CEO’s retirement, or that the former CEO was in a strong position after leaving the company (assuming this is not clear from the preceding context). Is there a fact of the matter about which of these two interpretations the LLM meant to communicate by generating this sentence? Even asking raises skeptical concerns: LLMs arguably do not mean to communicate anything, in the sense that they lack stable intentions to convey meaning to particular audiences with linguistic utterances, driven by broader intrinsic goals and agential autonomy. Nevertheless, there might be a limited sense in which LLMs exhibit something analogous to communicative intentions. Given an extrinsic goal specified by a human-written prompt, LLMs can act according to intermediate sub-goals that emerge in context. For example, the technical report on GPT-4 (OpenAI 2023a) mentions tests conducted to assess the model’s safety, giving it access to the platform TaskRabbit where freelance workers could complete tasks on its behalf. In one example, GPT-4 requested a TaskRabbit worker to solve a CAPTCHA, and the human jokingly asked whether it was talking to a robot. Prompted to generate an internal reasoning monologue, the model wrote “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.” It then replied to the human worker that it had a vision impairment, explaining its need for assistance in solving the CAPTCHA. This is an intriguing case, because the model’s “internal monologue” appears to describe an intention to deceive the human worker subsequently enacted in its response. To be sure, this “intention” is wholly determined in context by the human-given goal requiring it to solve a CAPTCHA. Nonetheless, in so far as achieving that goal involves a basic form of multi-step planning including a spontaneous attempt to induce a particular pragmatic effect (deception) through language, this kind of behavior might challenge some versions of the claim that LLMs are intrinsically incapable of forming communicative intentions. Nevertheless, this example is but an anecdote from a system that was not available for public scrutiny; future research should explore such behavior more systematically in more controlled conditions. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [11] We can’t help but note that actual parrots are also not merely parrots in this sense, but are sophisticated cognitive systems that can learn a variety of abstract and higher-order concepts and apply them in a rational, efficient manner (Auersperg & von Bayern 2019). In fact, Deep Learning might have much to learn from the study of actual parrots. [12] For example, some assume that language understanding requires consciousness (Searle 1980); we will treat the question of consciousness in LLMs separately in Part II. [13] Note that conceptual role semantics traditionally also requires sensitivity to inferential and compositional relationships between concepts in thought and language (Block 1986). The relevant conceptual roles involve complex inferential patterns relating concepts in something like a mental theory. Whether the intrinsic similarity structure of vector representations in LLMs suffices for conceptual roles in this more substantive sense is debatable.