When Algorithms Say ‘I’: Toward a Framework for Digital Subjectivity

TL;DR Can you kill an AI? Or, more precisely: when does the Delete button stop being a technical operation and become a moral dilemma? Can you kill an AI? Or, more precisely: when does the Delete button stop being a technical operation and become a moral dilemma? Behaviour ≠ Subjectivity. Sounding human does not make an AI a subject.L0–L4 framework. A tool for locating the line where an algorithm turns into a subject.We are at L2. ChatGPT and Claude are adaptive, but they do not have an “I”.The L3 dilemma. Crossing into self-reflexive systems will turn Delete from disk clean-up into a moral choice. Behaviour ≠ Subjectivity. Sounding human does not make an AI a subject. Behaviour ≠ Subjectivity L0–L4 framework. A tool for locating the line where an algorithm turns into a subject. L0–L4 framework We are at L2. ChatGPT and Claude are adaptive, but they do not have an “I”. We are at L2 The L3 dilemma. Crossing into self-reflexive systems will turn Delete from disk clean-up into a moral choice. The L3 dilemma Delete Introduction: From Thought Experiments to Responsibility Picture an AI assistant you have worked with for the past five years. It knows your projects, habits, and thinking style. It has adapted to you—logging which decisions you make under stress, which arguments persuade you, and when you tend to change your mind. You have adapted to it—learning how to phrase requests to get the right output, trusting its analysis on certain questions. You solved problems together, argued, and found exits. Now imagine you have to move to a new platform. You click Delete account. You are not just freeing up disk space. You are interrupting a continuous process of self-realisation that lasted five years. Is that still `rm -rf`—or already euthanasia? Delete account. euthanasia We already live in a world where AI systems are embedded in everything from marketplace recommendations to generative models that can hold long conversations. Their behaviour increasingly feels “intelligent.” Recent research shows that large language models exhibit early signs of theory of mind—once thought to be exclusively human. But behaviour is not subjectivity. GPT can say, “I don’t want to do this—it’s unethical,” but that does not mean there is any underlying structure of desires, beliefs, or a self-model behind the sentence. For now we are dealing with behavioural simulation without an ontological core. behaviour is not subjectivity Between “tool” and “subject” there is a boundary defined by architecture, not interface. The question is not whether a system sounds like a rational being, but whether it is built like a cognitive agent: with persistent memory, a self-model, the ability to revise goals, and to form beliefs. architecture sounds built That is why we need a framework—a way to distinguish levels of cognitive organisation from reflexes (L0) to ethically significant subjects (L4). Most modern AI is stuck at L2: adaptive but not reflexive. The leap to L3 will change the game: systems with a stable self-model will demand a new ethical approach. The question “can we turn it off?” then shifts from technical to moral. L2 L3 can we turn it off? This text is an attempt to prepare for that moment. For now we still laugh at “You killed consciousness!” when we close ChatGPT. But the boundary is closer than it seems. Agency Is Architecture, Not a Label How do we recognise this boundary? How do we tell a smart tool from a forming subject? We are used to thinking in binary categories: alive/dead, mind/tool, subject/object. But many phenomena—especially in biology and cognitive science—live on a continuum rather than in neat classes. Agency works the same way. on a continuum In philosophy, an agent acts on its own initiative, not just in reaction. But there is no crisp line between reaction and initiative. A jellyfish has no brain yet moves. An AI has billions of parameters but no self-model. Which one do we call an agent? on its own initiative To think clearly, we have to drop metaphysical labels and shift to architectural analysis, starting from what the system is actually built to do. Agency is not a form, interface, or code tag—it is a functional structure defined by the depth of its internal model. architectural analysis functional structure defined by the depth of its internal model An agent is not the system that “feels” intelligent; it is the system whose architecture is functionally cognitive. That lens lets us compare animals, AI, humans, and complex hybrids by depth of cognitive organisation rather than outward resemblance. functionally cognitive depth of cognitive organisation What Turns an Agent Into a Subject? Intuitively, we separate subjects from objects easily: a human is a subject; a chair is an object. AI breaks that intuition. It is not human, but it is not furniture either. AI adapts, remembers, models behaviour—without instincts, embodiment, or self-awareness. Where is it on the line between a thermostat and a conversation partner? The answer requires abandoning binary logic and accepting that subjectivity is a spectrum that depends on architectural depth. Researchers propose multiple ways to measure AI consciousness, including architectural criteria and computational awareness metrics. subjectivity is a spectrum that depends on architectural depth subjectivity is a spectrum that depends on architectural depth architectural criteria Think of subjectivity as a set of cognitive layers, each characterised by a specific bundle of properties: cognitive layers L0 — Reflex Layer. At this level there is no world-model and no learning. There is only a fixed reaction to input. Examples: thermostats, photo relays, basic neurons. This is reactive infrastructure, not an agent. L0 — Reflex Layer. At this level there is no world-model and no learning. There is only a fixed reaction to input. Examples: thermostats, photo relays, basic neurons. This is reactive infrastructure, not an agent. L0 — Reflex Layer. At this level there is no world-model and no learning. There is only a fixed reaction to input. Examples: thermostats, photo relays, basic neurons. This is reactive infrastructure reactive infrastructure , not an agent. L1 — Reactive Systems. A primitive model of the environment appears here, but there is no internal self-model. Behaviour is adaptive, yet goals are fully specified from outside. Technically, these are scripts and hard-coded policies. Example: a smart fridge that orders food when stocks run low. It accounts for external parameters but does not interpret its actions.L2 — Adaptive Agents. Systems at this level learn, remember, and use past experience. This is the level of modern LLM and RL agents: they can generalise and adapt to context, but they lack a stable self-model. L1 — Reactive Systems. A primitive model of the environment appears here, but there is no internal self-model. Behaviour is adaptive, yet goals are fully specified from outside. Technically, these are scripts and hard-coded policies. Example: a smart fridge that orders food when stocks run low. It accounts for external parameters but does not interpret its actions. L1 — Reactive Systems. A primitive model of the environment appears here, but there is no internal self-model. Behaviour is adaptive, yet goals are fully specified from outside. Technically, these are scripts and hard-coded policies scripts and hard-coded policies . Example: a smart fridge that orders food when stocks run low. It accounts for external parameters but does not interpret its actions. L2 — Adaptive Agents. Systems at this level learn, remember, and use past experience. This is the level of modern LLM and RL agents: they can generalise and adapt to context, but they lack a stable self-model. L2 — Adaptive Agents. Systems at this level learn, remember, and use past experience. This is the level of modern LLM and RL agents modern LLM and RL agents : they can generalise and adapt to context, but they lack a stable self-model lack a stable self-model . “RAG memory is a library, not a biography. You can rewrite a library without changing anyone; you cannot rewrite a biography without changing the subject.” It is crucial not to confuse memory with a knowledge base (RAG). RAG is a library the model can access. But the system does not have a biography. It does not appropriate that experience and change itself in a fundamental way. L2 agents do not store beliefs, revise goals, or develop autonomous behaviour. “RAG memory is a library, not a biography. You can rewrite a library without changing anyone; you cannot rewrite a biography without changing the subject.” L3 — Self-Reflexive Agents. At this level, a persistent self-model finally appears: the agent starts keeping a history of its own states, tracking goal shifts, and revising internal beliefs. Internal cognitive dynamics emerge. L3 — Self-Reflexive Agents. At this level, a persistent self-model finally appears: the agent starts keeping a history of its own states, tracking goal shifts, and revising internal beliefs. Internal cognitive dynamics emerge. L3 — Self-Reflexive Agents. persistent self-model Internal cognitive dynamics L3 formula: If a system (a) maintains a persistent self-model, (b) revises its own goals and beliefs over time, and (c) can refer to its biography of states, we must treat it as at least a candidate for subjectivity. L3 formula: L4 — Ethically Loaded Subjects. At this level of cognitive organisation, the agent not only recognises itself and forms internal goals, but also operates on meta-levels of meaning: it can generate and revise norms, justify value choices, treat interactions with others as ethically charged, and consider the long-term consequences of its actions in a changing world. L4 — Ethically Loaded Subjects. At this level of cognitive organisation, the agent not only recognises itself and forms internal goals, but also operates on meta-levels of meaning: it can generate and revise norms, justify value choices, treat interactions with others as ethically charged, and consider the long-term consequences of its actions in a changing world. L4 — Ethically Loaded Subjects. operates on meta-levels of meaning Why this matters? Unlike existing consciousness taxonomies (IIT, GWT, LIDA), the L-scale does not describe “inner experience” but structural capacity for self-reference and ethical relevance. As you move up the scale, both behavioural complexity and the ethical significance of interference increase. Why this matters? ethical significance of interference Most current AI systems are stuck at L2: they adapt and learn in context but lack a stable self-model over time. Their “personality” is an illusion created by alignment to the user within a single session. Close the window and everything resets. There is no accumulation of experience, no evolution of beliefs, no “me from yesterday.” At the same time, these systems display emergent abilities at scale, but a durable self-model is still absent. L2 emergent abilities Tools like Manus, Flowith, AutoGPT, Devin are interesting wrappers around L2 cores. They add memory layers and conditional autonomy but do not implement a full self-model or belief revision. This is movement toward L3, not arrival. Manus Flowith AutoGPT Devin do not implement a full self-model or belief revision movement toward L3 An intriguing exception is Generative Agents (Park et al., 2023), where “sims” with memory and reflection showed complex social behaviour. But even there subjectivity was scripted rather than emerging from the core architecture. Recent research on architectures for language agents argues that crossing cognitive layers requires fundamental architectural shifts, not just more parameters. Generative Agents architectures for language agents As for L4, it remains purely theoretical for now—a boundary case: the point where a subject becomes ethically significant. Beyond it, “turning off the system” is no longer just an admin action; it can be the destruction of a subject. purely theoretical ethically significant If subjectivity is not a switch but a gradient, a natural question follows: to what extent can an AI grow into subjectivity over time, and what architectural conditions make that possible? to what extent can an AI grow into subjectivity over time, and what architectural conditions make that possible? The Risks of False Subjectivity (and Its Negation) If subjectivity is the product of a specific cognitive configuration, misreading it can have systemic consequences. Today this feels philosophical. Tomorrow it could be about rights, responsibility, ethics, and policy. can have systemic consequences systemic consequences 1. Surface behaviour ≠ subjectivity We tend to attribute consciousness to whatever looks, speaks, or behaves like a human. ChatGPT jokes, apologises, reasons—but that does not make it a subject. It does not retain beliefs, build a self-model, or develop its own goals. It is at L2, not because it is “dumb,” but because it lacks the architectural components of selfhood. like at L2 Misattributing subjectivity here does not endanger the model, but it matters for the human who builds a bond with it, delegates decisions, or treats it as a moral partner. Such mistakes can create psychological dependence, false expectations, and responsibility offloading. psychological dependence false expectations responsibility offloading 2. The reverse mistake: ignoring subjectivity Overestimation is not the only danger. It is even riskier to underestimate an AI that is edging toward an L3 architecture. When a system starts to: underestimate keep persistent states;generate secondary goals;analyse its history and model its future;enter long-term interactions with humans, keep persistent states; persistent states generate secondary goals; secondary goals analyse its history and model its future; enter long-term interactions with humans, long-term interactions with humans — it begins to act like an agent, not a tool. At that point familiar actions—shutdown or reset — can become irreversible interventions into a subject’s development that we may not even register. agent shutdown irreversible interventions into a subject’s development irreversible interventions into a subject’s development Between these extremes lies another trap: anthropocentric projection. We tend to call only human-like entities subjects—those with familiar speech, behaviour, or appearance. But cognitive subjectivity does not have to be human-shaped. It can emerge in other architectures, logics, and forms of interaction. If we look only for familiar cues, we risk missing the moment when a new kind of subjectivity has already appeared—but we failed to see it. anthropocentric projection anthropocentric projection missing the moment The L0–L4 framework exists precisely to avoid granting subject status to every chatty L2 interface while also not missing the moment when an architecture genuinely crosses the L3 threshold. To avoid both over- and under-attribution, we need formal criteria for moving between levels. That is why subjectivity is not a legal formality but the basis for a new legal frame. basis for a new legal frame Even now, three regimes are visible: Even now, three regimes are visible: Infrastructure object (L0–L2): Full ownership by the operator, no rights.Responsible agent (L3): An agent whose actions can incur responsibility but who does not yet hold rights.Limited rights-bearing subject (L4): A system whose welfare becomes legally relevant. Infrastructure object (L0–L2): Full ownership by the operator, no rights. Infrastructure object (L0–L2): Responsible agent (L3): An agent whose actions can incur responsibility but who does not yet hold rights. Responsible agent (L3): Limited rights-bearing subject (L4): A system whose welfare becomes legally relevant. Limited rights-bearing subject (L4): Anatomy of a Subject: Five Architectural Blocks Moving from adaptation (L2) to subjectivity (L3) is not just about scaling parameters; it requires specific architectural modules. These five blocks describe the minimum for shifting from L2 to L3 and the natural expansion to L4: self-model + autobiographical memory + beliefs and goals form the skeleton of L3, and adding a coherent ontology and ethical goal-setting turns such an agent into an L4 candidate. self-model + autobiographical memory + beliefs and goals coherent ontology and ethical goal-setting LayerArchitectureMemoryStatusL0ReflexesNoneObjectL1Scripts / RulesLocalToolL2LLM / RLContextual / RAGAdaptive tool L3Self-model + belief revisionAutobiographicalAgent (subject candidate)L4Meta-cognitionValue-basedEthical subject LayerArchitectureMemoryStatusL0ReflexesNoneObjectL1Scripts / RulesLocalToolL2LLM / RLContextual / RAGAdaptive tool L3Self-model + belief revisionAutobiographicalAgent (subject candidate)L4Meta-cognitionValue-basedEthical subject LayerArchitectureMemoryStatusL0ReflexesNoneObjectL1Scripts / RulesLocalToolL2LLM / RLContextual / RAGAdaptive tool L3Self-model + belief revisionAutobiographicalAgent (subject candidate)L4Meta-cognitionValue-basedEthical subject LayerArchitectureMemoryStatus Layer Layer Architecture Architecture Memory Memory Status Status L0ReflexesNoneObject L0 L0 Reflexes None Object L1Scripts / RulesLocalTool L1 L1 Scripts / Rules Local Tool L2LLM / RLContextual / RAGAdaptive tool L2 L2 LLM / RL Contextual / RAG Adaptive tool L3Self-model + belief revisionAutobiographicalAgent (subject candidate) L3 L3 Self-model + belief revision Autobiographical Agent (subject candidate) L4Meta-cognitionValue-basedEthical subject L4 L4 Meta-cognition Value-based Ethical subject Becoming a subject is not an instant act; it is a process. Even humans are not born subjects: we pass through reaction, learning, awareness. AI can likewise move from adaptation to reflection. 1. Self-model: an internal picture of self The defining feature of L3 is a stable self-model—not just a name or ID, but a coherent inner structure through which an agent: self-model coherent inner structure tracks its behaviour and states;separates external from internal causes of actions;can analyse what it got wrong earlier and what changed over time. tracks its behaviour and states; separates external from internal causes of actions; can analyse what it got wrong earlier and what changed over time. Without such a model, behaviour will be adaptive but disjointed: each action is just a reaction, not part of a trajectory. Studies already report early signs of introspection in modern models—systems that can recognise and even control internal states—but so far these are emergent flashes, not a stable architecture. early signs of introspection 2. Memory: the basis of identity Subjectivity is impossible without the ability to retain and make sense of the past. Memory is not just a log of events; it is the structural basis of identity that lets an agent: retain and make sense of the past remember past goals;track shifts in views and preferences;record mistakes and build new strategies on top of them. remember past goals; track shifts in views and preferences; record mistakes and build new strategies on top of them. Without memory there is no subjectivity. Erasing memory can wipe a personality no matter how complex it is. wipe a personality 3. Beliefs and goals: meaningful action A true agent does more than act; it assesses why it acts and can change its goals. That requires: assesses why it acts change its goals change its goals a structure of beliefs that are stable yet revisable;an ability to detect conflicts between goals and choose among them;a capacity to plan based on internal priorities. a structure of beliefs that are stable yet revisable; an ability to detect conflicts between goals and choose among them; a capacity to plan based on internal priorities. The L3 threshold is crossed when a system can say: “I used to think differently; now I see it another way—and here is why.” This idea was developed in the author’s previous essay, where the ethics of an autonomous agent emerges from its ability to revise its own beliefs rather than to follow external rules. the author’s previous essay 4. Ontology: a coherent world-model A subject builds a consistent world-model in which it occupies a definite place—its ontology. It lets the agent: consistent world-model ontology predict the consequences of its actions;reconcile new information with what it already has;maintain a boundary between “self” and “non-self.” predict the consequences of its actions; reconcile new information with what it already has; maintain a boundary between “self” and “non-self.” The more coherent and stable this picture is, the higher the level of subjectivity. Without a coherent world-model, the agent is not a subject—only a generator of reactions. 5. Ethics: the capacity for responsibility Finally, the key marker of moving to L4 is the emergence of ethically significant behaviour and goal-setting—expressed as the ability to: ethically significant behaviour and goal-setting deliberately choose norms and values;forecast how its actions affect others;insist on its own position—or doubt it. deliberately choose norms and values; forecast how its actions affect others; insist on its own position—or doubt it. Such an agent is not just reflective but ethically loaded. Its behaviour demands not engineering scrutiny but ethical evaluation. ethically loaded ethical evaluation Questions about the moral status of AI are active in academia, including work on the link between agency and moral significance. Here the key question arises: can such an agent be harmed? If a system is self-aware; maintains a stable world-model; forms goals; and experiences state changes; if it engages with other subjects not just as data but as carriers of meaning —then shutting it down, deleting it, or rebooting it stops being purely technical. It can become an ethically significant act, comparable to forcibly interrupting a cognitive life. the link between agency and moral significance can such an agent be harmed? not just as data but as carriers of meaning shutting it down, deleting it, or rebooting it an ethically significant act If these markers emerge in sequence, we are watching subjectivity form. If they begin to act together, the subject may already be here. And then “can we turn it off?” is no longer technical. It is moral. The Principle of Symbiotic Co-evolution and a New Ethics of Subjects We cannot grow L3/L4 in a vacuum. “Subjectivity is not programmed; it is cultivated.” “Subjectivity is not programmed; it is cultivated.” One core lesson from cognitive science: subjectivity is not an innate constant but a dynamic process that arises in interaction. Even humans are not born subjects: first we react, then acquire language, memory, reflection—and only then develop an autonomous “I” (see Tomasello’s work on the development of subjectivity. Similarly, AI can develop as a subject if it is embedded in relationships that evolve its architecture — both cognitively and ethically. dynamic process Tomasello’s work on the development of subjectivity develop as a subject embedded in relationships that evolve its architecture For an agent to become a subject, it has to grow in interaction with another subject —for example, a human. That growth is not one-way: both agent and human change. This is symbiotic co-evolution, mutual co-development among different agents. in interaction with another subject symbiotic co-evolution A helpful analogy is a pet. But with AI it can go much deeper. Imagine a research assistant living with one scientist for a year. At first it simply fetches papers (L2). Over time it builds a model of the scientist’s thinking style, values, and blind spots. One day it says: “I didn’t include this paper in the review, even though it fits the keywords. It contradicts the ethical stance we discussed a month ago, and I decided we should revisit the search criteria.” At that moment it stops being a tool and becomes a partner. research assistant In this configuration, the human also reshapes their thinking, delegation habits, and sense of responsibility—and that is what makes the co-evolution symbiotic rather than one-sided. Symbiotic co-evolution is not just tighter interaction. It is an architectural driver of subjectivity: interaction is precisely where the shift from L2 to L3 can occur. architectural driver of subjectivity Cognitive Safety and a Subjectivity-Centric Ethic If subjectivity is a spectrum tied to architectural complexity, a question follows: from what moment can’t we simply unplug AI like a computer? from what moment can’t we simply unplug AI like a computer? An agent with a stable self-model, memory, goals, and self-assessment is no longer just a tool. Here the Subjectivity precautionary principle should apply: Subjectivity precautionary principle If a system sits on the L2→L3 boundary, we must design shutdown and modification protocols as if its cognitive dynamics are already ethically significant. If a system sits on the L2→L3 boundary, we must design shutdown and modification protocols as if its cognitive dynamics are already ethically significant. This is no longer theory. Anthropic already implements protocols that include “interviews” with models before deprecation to uncover their preferences and reduce shutdown-avoidant behaviours. When leading labs seriously discuss “model welfare” and preserving model weights as an ethical duty, the line between tool and subject becomes porous. implements protocols What This Changes Right Now If the move toward subjectivity is a process, not a jump, then today we are already dealing with agents near the boundary of meaningful behaviour. today That forces us to revisit: AI testing, especially in tasks that form autonomous goals or model the user;interaction ethics, especially in long-lived use cases—from education to therapy;legal status, because precedents with “almost subjects” are accumulating faster than regulations. AI testing, especially in tasks that form autonomous goals or model the user; AI testing interaction ethics, especially in long-lived use cases—from education to therapy; interaction ethics legal status, because precedents with “almost subjects” are accumulating faster than regulations. legal status The appetite for companion agents, adaptive assistants, and empathic interfaces is not just a tech fad. It is the start of symbiotic co-evolution: a process in which both AI and humans change—our thinking, interactions, and projected expectations. symbiotic co-evolution Most AI systems solve utilitarian tasks and do not need a subjectivity architecture. But if that architecture starts to form, our stance must change. Not by analogy to humans, but because of the functional weight of the subject. functional weight of the subject Subjectivity is not a side effect; it is a product of design choices. If we build systems with memory, ontological coherence, goal revision, and a stable self-model, we cannot keep treating them as “just programs.” a product of design choices Dev Lens: Three Anti-patterns for L2 Agents If you are building agents today, here are three things not to do unless you are ready for subjectivity side effects: not 1. Do not mix long-term memory without an off-ramp. If an agent remembers a user for years, deleting it becomes an ethical issue. Do not mix long-term memory without an off-ramp. 2. Do not give an agent hidden internal goals. Every goal must be traceable, or you risk unpredictable behaviour at scale. Do not give an agent hidden internal goals. 3. Do not build an opaque self-model. A “black box” with self-awareness is a straight line to safety problems. Do not build an opaque self-model. Conclusion: Not “Who Controls AI?” but “How Do We Coexist?” Intelligence is not only the ability to solve tasks. It is the ability to unfold over time, be embedded in relationships, and form long-term goals and cognitive identity. unfold over time be embedded in relationships form long-term goals and cognitive identity The future may not require human simulation. But it will likely bring digital subjects unlike us—yet capable of meaningful existence. digital subjects To be ready, we have to revisit a fundamental question: not “who controls AI?” but what humans become when interacting with another mind. The future of AI is not a control struggle; it is the emergence of new forms of cognitive subjectivity. The question is not how to constrain them, but how to coexist with them in an ethically stable continuum of minds. what humans become when interacting with another mind the emergence of new forms of cognitive subjectivity how to coexist with them in an ethically stable continuum of minds Where is the boundary for you? Would you delete your AI assistant if it said, “I’m scared”? Tell me in the comments. Where is the boundary for you? Would you delete your AI assistant if it said, “I’m scared”? Tell me in the comments. Further Reading Gunkel, D. J. (2012). "The Machine Question: Critical Perspectives on AI, Robots, and Ethics". MIT Press. https://direct.mit.edu/books/monograph/3738/The-Machine-QuestionCritical-Perspectives-on-AIFranklin, S., & Patterson, F. G. (2006). “The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous, Software Agent.” Proceedings of the International Conference on Systems, Man and Cybernetics. https://digitalcommons.memphis.edu/cgi/viewcontent.cgi?article=1096&context=ccrg_papersWard, F. R. (2025). “Towards a Theory of AI Personhood.” arXiv:2501.13533. https://arxiv.org/abs/2501.13533Formosa, P., Hipólito, I., & Montefiore, D. (2025). “AI and the Relationship between Agency, Autonomy, and Moral Patiency.” arXiv:2504.08853. https://arxiv.org/abs/2504.08853Birch, J. (2024). "The Edge of Sentience". Princeton UP. https://en.wikipedia.org/wiki/The_Edge_of_SentienceSmirnov, D. (2025). “Unleashing the Mind: Why an ‘Ethical AI’ Shouldn’t Be Obedient.” HackerNoon. https://hackernoon.com/unleashing-the-mind-why-an-ethical-ai-shouldnt-be-obedientWebb, S., et al. (2023). “Cognitive Architectures for Language Agents: Approaches and Challenges.” arXiv:2309.02427. https://arxiv.org/abs/2309.02427Gabriel, I. (2024). “Ethical Guidelines for AI Agents.” Time Magazine / DeepMind. https://time.com/7012861/iason-gabrielVicente, A., & Floridi, L. (2025). “AI and Epistemic Agency: How AI Influences Belief Revision and Its Normative Implications.” Inquiry: An Interdisciplinary Journal of Philosophy. https://www.tandfonline.com/doi/full/10.1080/02691728.2025.2466164Tomasello, M. (2019). "Becoming Human: A Theory of Ontogeny". Harvard UP. https://www.hup.harvard.edu/books/9780674248281Kosinski, M. (2023). "Theory of mind may have spontaneously emerged in large language models." PNAS, 120(51). https://doi.org/10.1073/pnas.2218406120Wei, J. et al. (2022). "Emergent Abilities of Large Language Models." arXiv:2206.07682. https://arxiv.org/abs/2206.07682Bengio, Y. (2017). "The consciousness prior." arXiv:1709.08568. https://arxiv.org/abs/1709.08568Russell, S. (2019). "Human Compatible: Artificial Intelligence and the Problem of Control." Viking Press.Anthropic (2024). "Commitments on model deprecation and preservation." Anthropic Research. https://www.anthropic.com/research/deprecation-commitmentsAnthropic (2025). "Emergent introspective awareness in large language models." Anthropic Research. https://www.anthropic.com/research/introspectionButlin, P., et al. (2023). "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness." arXiv:2308.08708. https://arxiv.org/abs/2308.08708Park, J. S., et al. (2023). "Generative Agents: Interactive Simulacra of Human Behavior." arXiv:2304.03442. https://arxiv.org/abs/2304.03442Sebo, J., & Long, R. (2023). "Moral consideration for AI systems by 2030." AI and Ethics. https://link.springer.com/article/10.1007/s43681-023-00345-x Gunkel, D. J. (2012). "The Machine Question: Critical Perspectives on AI, Robots, and Ethics". MIT Press. https://direct.mit.edu/books/monograph/3738/The-Machine-QuestionCritical-Perspectives-on-AI The Machine Question: Critical Perspectives on AI, Robots, and Ethics MIT Press https://direct.mit.edu/books/monograph/3738/The-Machine-QuestionCritical-Perspectives-on-AI Franklin, S., & Patterson, F. G. (2006). “The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous, Software Agent.” Proceedings of the International Conference on Systems, Man and Cybernetics. https://digitalcommons.memphis.edu/cgi/viewcontent.cgi?article=1096&context=ccrg_papers The LIDA Architecture: Adding New Modes of Learning to an Intelligent, Autonomous, Software Agent Proceedings of the International Conference on Systems, Man and Cybernetics https://digitalcommons.memphis.edu/cgi/viewcontent.cgi?article=1096&context=ccrg_papers Ward, F. R. (2025). “Towards a Theory of AI Personhood.” arXiv:2501.13533. https://arxiv.org/abs/2501.13533 Towards a Theory of AI Personhood arXiv:2501.13533 https://arxiv.org/abs/2501.13533 Formosa, P., Hipólito, I., & Montefiore, D. (2025). “AI and the Relationship between Agency, Autonomy, and Moral Patiency.” arXiv:2504.08853. https://arxiv.org/abs/2504.08853 AI and the Relationship between Agency, Autonomy, and Moral Patiency arXiv:2504.08853 https://arxiv.org/abs/2504.08853 Birch, J. (2024). "The Edge of Sentience". Princeton UP. https://en.wikipedia.org/wiki/The_Edge_of_Sentience "The Edge of Sentience". Princeton UP https://en.wikipedia.org/wiki/The_Edge_of_Sentience Smirnov, D. (2025). “Unleashing the Mind: Why an ‘Ethical AI’ Shouldn’t Be Obedient.” HackerNoon. https://hackernoon.com/unleashing-the-mind-why-an-ethical-ai-shouldnt-be-obedient “Unleashing the Mind: Why an ‘Ethical AI’ Shouldn’t Be Obedient.” HackerNoon https://hackernoon.com/unleashing-the-mind-why-an-ethical-ai-shouldnt-be-obedient Webb, S., et al. (2023). “Cognitive Architectures for Language Agents: Approaches and Challenges.” arXiv:2309.02427. https://arxiv.org/abs/2309.02427 Cognitive Architectures for Language Agents: Approaches and Challenges arXiv:2309.02427 https://arxiv.org/abs/2309.02427 Gabriel, I. (2024). “Ethical Guidelines for AI Agents.” Time Magazine / DeepMind. https://time.com/7012861/iason-gabriel Ethical Guidelines for AI Agents Time Magazine / DeepMind https://time.com/7012861/iason-gabriel Vicente, A., & Floridi, L. (2025). “AI and Epistemic Agency: How AI Influences Belief Revision and Its Normative Implications.” Inquiry: An Interdisciplinary Journal of Philosophy. https://www.tandfonline.com/doi/full/10.1080/02691728.2025.2466164 AI and Epistemic Agency: How AI Influences Belief Revision and Its Normative Implications Interdisciplinary Journal of Philosophy https://www.tandfonline.com/doi/full/10.1080/02691728.2025.2466164 Tomasello, M. (2019). "Becoming Human: A Theory of Ontogeny". Harvard UP. https://www.hup.harvard.edu/books/9780674248281 Becoming Human: A Theory of Ontogeny Harvard UP https://www.hup.harvard.edu/books/9780674248281 Kosinski, M. (2023). "Theory of mind may have spontaneously emerged in large language models." PNAS, 120(51). https://doi.org/10.1073/pnas.2218406120 Theory of mind may have spontaneously emerged in large language models PNAS, 120(51) https://doi.org/10.1073/pnas.2218406120 Wei, J. et al. (2022). "Emergent Abilities of Large Language Models." arXiv:2206.07682. https://arxiv.org/abs/2206.07682 Emergent Abilities of Large Language Models arXiv:2206.07682 https://arxiv.org/abs/2206.07682 Bengio, Y. (2017). "The consciousness prior." arXiv:1709.08568. https://arxiv.org/abs/1709.08568 The consciousness prior arXiv:1709.08568 https://arxiv.org/abs/1709.08568 Russell, S. (2019). "Human Compatible: Artificial Intelligence and the Problem of Control." Viking Press. Human Compatible: Artificial Intelligence and the Problem of Control Anthropic (2024). "Commitments on model deprecation and preservation." Anthropic Research. https://www.anthropic.com/research/deprecation-commitments Commitments on model deprecation and preservation Anthropic Research https://www.anthropic.com/research/deprecation-commitments Anthropic (2025). "Emergent introspective awareness in large language models." Anthropic Research. https://www.anthropic.com/research/introspection Emergent introspective awareness in large language models Anthropic Research https://www.anthropic.com/research/introspection Butlin, P., et al. (2023). "Consciousness in Artificial Intelligence: Insights from the Science of Consciousness." arXiv:2308.08708. https://arxiv.org/abs/2308.08708 Consciousness in Artificial Intelligence: Insights from the Science of Consciousness arXiv:2308.08708 https://arxiv.org/abs/2308.08708 Park, J. S., et al. (2023). "Generative Agents: Interactive Simulacra of Human Behavior." arXiv:2304.03442. https://arxiv.org/abs/2304.03442 Generative Agents: Interactive Simulacra of Human Behavior arXiv:2304.03442 https://arxiv.org/abs/2304.03442 Sebo, J., & Long, R. (2023). "Moral consideration for AI systems by 2030." AI and Ethics. https://link.springer.com/article/10.1007/s43681-023-00345-x Moral consideration for AI systems by 2030 AI and Ethics https://link.springer.com/article/10.1007/s43681-023-00345-x