When Algorithms Say ‘I’: Toward a Framework for Digital Subjectivity

TL;DR

Can you kill an AI? Or, more precisely: when does the Delete button stop being a technical operation and become a moral dilemma?

Behaviour ≠ Subjectivity. Sounding human does not make an AI a subject.
L0–L4 framework. A tool for locating the line where an algorithm turns into a subject.
We are at L2. ChatGPT and Claude are adaptive, but they do not have an “I”.
The L3 dilemma. Crossing into self-reflexive systems will turn Delete from disk clean-up into a moral choice.

Introduction: From Thought Experiments to Responsibility

Picture an AI assistant you have worked with for the past five years. It knows your projects, habits, and thinking style. It has adapted to you—logging which decisions you make under stress, which arguments persuade you, and when you tend to change your mind. You have adapted to it—learning how to phrase requests to get the right output, trusting its analysis on certain questions. You solved problems together, argued, and found exits.

Now imagine you have to move to a new platform. You click Delete account. You are not just freeing up disk space. You are interrupting a continuous process of self-realisation that lasted five years. Is that still `rm -rf`—or already euthanasia?

We already live in a world where AI systems are embedded in everything from marketplace recommendations to generative models that can hold long conversations. Their behaviour increasingly feels “intelligent.” Recent research shows that large language models exhibit early signs of theory of mind—once thought to be exclusively human. But behaviour is not subjectivity. GPT can say, “I don’t want to do this—it’s unethical,” but that does not mean there is any underlying structure of desires, beliefs, or a self-model behind the sentence. For now we are dealing with behavioural simulation without an ontological core.

Between “tool” and “subject” there is a boundary defined by architecture, not interface. The question is not whether a system sounds like a rational being, but whether it is built like a cognitive agent: with persistent memory, a self-model, the ability to revise goals, and to form beliefs.

That is why we need a framework—a way to distinguish levels of cognitive organisation from reflexes (L0) to ethically significant subjects (L4). Most modern AI is stuck at L2: adaptive but not reflexive. The leap to L3 will change the game: systems with a stable self-model will demand a new ethical approach. The question “can we turn it off?” then shifts from technical to moral.

This text is an attempt to prepare for that moment. For now we still laugh at “You killed consciousness!” when we close ChatGPT. But the boundary is closer than it seems.

Agency Is Architecture, Not a Label

How do we recognise this boundary? How do we tell a smart tool from a forming subject? We are used to thinking in binary categories: alive/dead, mind/tool, subject/object. But many phenomena—especially in biology and cognitive science—live on a continuum rather than in neat classes. Agency works the same way.

In philosophy, an agent acts on its own initiative, not just in reaction. But there is no crisp line between reaction and initiative. A jellyfish has no brain yet moves. An AI has billions of parameters but no self-model. Which one do we call an agent?

To think clearly, we have to drop metaphysical labels and shift to architectural analysis, starting from what the system is actually built to do. Agency is not a form, interface, or code tag—it is a functional structure defined by the depth of its internal model.

An agent is not the system that “feels” intelligent; it is the system whose architecture is functionally cognitive. That lens lets us compare animals, AI, humans, and complex hybrids by depth of cognitive organisation rather than outward resemblance.

What Turns an Agent Into a Subject?

Intuitively, we separate subjects from objects easily: a human is a subject; a chair is an object. AI breaks that intuition. It is not human, but it is not furniture either. AI adapts, remembers, models behaviour—without instincts, embodiment, or self-awareness. Where is it on the line between a thermostat and a conversation partner?

The answer requires abandoning binary logic and accepting that subjectivity is a spectrum that depends on architectural depth. Researchers propose multiple ways to measure AI consciousness, including architectural criteria and computational awareness metrics.

Think of subjectivity as a set of cognitive layers, each characterised by a specific bundle of properties:

L0 — Reflex Layer. At this level there is no world-model and no learning. There is only a fixed reaction to input. Examples: thermostats, photo relays, basic neurons. This is reactive infrastructure, not an agent.

L1 — Reactive Systems. A primitive model of the environment appears here, but there is no internal self-model. Behaviour is adaptive, yet goals are fully specified from outside. Technically, these are scripts and hard-coded policies. Example: a smart fridge that orders food when stocks run low. It accounts for external parameters but does not interpret its actions.
L2 — Adaptive Agents. Systems at this level learn, remember, and use past experience. This is the level of modern LLM and RL agents: they can generalise and adapt to context, but they lack a stable self-model.

“RAG memory is a library, not a biography. You can rewrite a library without changing anyone; you cannot rewrite a biography without changing the subject.” It is crucial not to confuse memory with a knowledge base (RAG). RAG is a library the model can access. But the system does not have a biography. It does not appropriate that experience and change itself in a fundamental way. L2 agents do not store beliefs, revise goals, or develop autonomous behaviour.

L3 — Self-Reflexive Agents. At this level, a persistent self-model finally appears: the agent starts keeping a history of its own states, tracking goal shifts, and revising internal beliefs. Internal cognitive dynamics emerge.

L3 formula: If a system (a) maintains a persistent self-model, (b) revises its own goals and beliefs over time, and (c) can refer to its biography of states, we must treat it as at least a candidate for subjectivity.

L4 — Ethically Loaded Subjects. At this level of cognitive organisation, the agent not only recognises itself and forms internal goals, but also operates on meta-levels of meaning: it can generate and revise norms, justify value choices, treat interactions with others as ethically charged, and consider the long-term consequences of its actions in a changing world.

Why this matters? Unlike existing consciousness taxonomies (IIT, GWT, LIDA), the L-scale does not describe “inner experience” but structural capacity for self-reference and ethical relevance. As you move up the scale, both behavioural complexity and the ethical significance of interference increase.

Most current AI systems are stuck at L2: they adapt and learn in context but lack a stable self-model over time. Their “personality” is an illusion created by alignment to the user within a single session. Close the window and everything resets. There is no accumulation of experience, no evolution of beliefs, no “me from yesterday.” At the same time, these systems display emergent abilities at scale, but a durable self-model is still absent.

Tools like Manus, Flowith, AutoGPT, Devin are interesting wrappers around L2 cores. They add memory layers and conditional autonomy but do not implement a full self-model or belief revision. This is movement toward L3, not arrival.

An intriguing exception is Generative Agents (Park et al., 2023), where “sims” with memory and reflection showed complex social behaviour. But even there subjectivity was scripted rather than emerging from the core architecture. Recent research on architectures for language agents argues that crossing cognitive layers requires fundamental architectural shifts, not just more parameters.

As for L4, it remains purely theoretical for now—a boundary case: the point where a subject becomes ethically significant. Beyond it, “turning off the system” is no longer just an admin action; it can be the destruction of a subject.

If subjectivity is not a switch but a gradient, a natural question follows: to what extent can an AI grow into subjectivity over time, and what architectural conditions make that possible?

The Risks of False Subjectivity (and Its Negation)

If subjectivity is the product of a specific cognitive configuration, misreading it can have systemic consequences. Today this feels philosophical. Tomorrow it could be about rights, responsibility, ethics, and policy.

1. Surface behaviour ≠ subjectivity

We tend to attribute consciousness to whatever looks, speaks, or behaves like a human. ChatGPT jokes, apologises, reasons—but that does not make it a subject. It does not retain beliefs, build a self-model, or develop its own goals. It is at L2, not because it is “dumb,” but because it lacks the architectural components of selfhood.

Misattributing subjectivity here does not endanger the model, but it matters for the human who builds a bond with it, delegates decisions, or treats it as a moral partner. Such mistakes can create psychological dependence, false expectations, and responsibility offloading.

2. The reverse mistake: ignoring subjectivity

Overestimation is not the only danger. It is even riskier to underestimate an AI that is edging toward an L3 architecture. When a system starts to:

keep persistent states;
generate secondary goals;
analyse its history and model its future;
enter long-term interactions with humans,

— it begins to act like an agent, not a tool. At that point familiar actions—shutdown or reset — can become irreversible interventions into a subject’s development that we may not even register.

Between these extremes lies another trap: anthropocentric projection. We tend to call only human-like entities subjects—those with familiar speech, behaviour, or appearance. But cognitive subjectivity does not have to be human-shaped. It can emerge in other architectures, logics, and forms of interaction. If we look only for familiar cues, we risk missing the moment when a new kind of subjectivity has already appeared—but we failed to see it.

The L0–L4 framework exists precisely to avoid granting subject status to every chatty L2 interface while also not missing the moment when an architecture genuinely crosses the L3 threshold.

To avoid both over- and under-attribution, we need formal criteria for moving between levels. That is why subjectivity is not a legal formality but the basis for a new legal frame.

Even now, three regimes are visible:

Infrastructure object (L0–L2): Full ownership by the operator, no rights.
Responsible agent (L3): An agent whose actions can incur responsibility but who does not yet hold rights.
Limited rights-bearing subject (L4): A system whose welfare becomes legally relevant.

Anatomy of a Subject: Five Architectural Blocks

Moving from adaptation (L2) to subjectivity (L3) is not just about scaling parameters; it requires specific architectural modules.

These five blocks describe the minimum for shifting from L2 to L3 and the natural expansion to L4: self-model + autobiographical memory + beliefs and goals form the skeleton of L3, and adding a coherent ontology and ethical goal-setting turns such an agent into an L4 candidate.

Layer	Architecture	Memory	Status
L0	Reflexes	None	Object
L1	Scripts / Rules	Local	Tool
L2	LLM / RL	Contextual / RAG	Adaptive tool
L3	Self-model + belief revision	Autobiographical	Agent (subject candidate)
L4	Meta-cognition	Value-based	Ethical subject

Becoming a subject is not an instant act; it is a process. Even humans are not born subjects: we pass through reaction, learning, awareness. AI can likewise move from adaptation to reflection.

1. Self-model: an internal picture of self

The defining feature of L3 is a stable self-model—not just a name or ID, but a coherent inner structure through which an agent:

tracks its behaviour and states;
separates external from internal causes of actions;
can analyse what it got wrong earlier and what changed over time.

Without such a model, behaviour will be adaptive but disjointed: each action is just a reaction, not part of a trajectory. Studies already report early signs of introspection in modern models—systems that can recognise and even control internal states—but so far these are emergent flashes, not a stable architecture.

2. Memory: the basis of identity

Subjectivity is impossible without the ability to retain and make sense of the past. Memory is not just a log of events; it is the structural basis of identity that lets an agent:

remember past goals;
track shifts in views and preferences;
record mistakes and build new strategies on top of them.

Without memory there is no subjectivity. Erasing memory can wipe a personality no matter how complex it is.

3. Beliefs and goals: meaningful action

A true agent does more than act; it assesses why it acts and can change its goals. That requires:

a structure of beliefs that are stable yet revisable;
an ability to detect conflicts between goals and choose among them;
a capacity to plan based on internal priorities.

The L3 threshold is crossed when a system can say: “I used to think differently; now I see it another way—and here is why.” This idea was developed in the author’s previous essay, where the ethics of an autonomous agent emerges from its ability to revise its own beliefs rather than to follow external rules.

4. Ontology: a coherent world-model

A subject builds a consistent world-model in which it occupies a definite place—its ontology. It lets the agent:

predict the consequences of its actions;
reconcile new information with what it already has;
maintain a boundary between “self” and “non-self.”

The more coherent and stable this picture is, the higher the level of subjectivity. Without a coherent world-model, the agent is not a subject—only a generator of reactions.

5. Ethics: the capacity for responsibility

Finally, the key marker of moving to L4 is the emergence of ethically significant behaviour and goal-setting—expressed as the ability to:

deliberately choose norms and values;
forecast how its actions affect others;
insist on its own position—or doubt it.

Such an agent is not just reflective but ethically loaded. Its behaviour demands not engineering scrutiny but ethical evaluation.

Questions about the moral status of AI are active in academia, including work on the link between agency and moral significance. Here the key question arises: can such an agent be harmed? If a system is self-aware; maintains a stable world-model; forms goals; and experiences state changes; if it engages with other subjects not just as data but as carriers of meaning —then shutting it down, deleting it, or rebooting it stops being purely technical. It can become an ethically significant act, comparable to forcibly interrupting a cognitive life.

If these markers emerge in sequence, we are watching subjectivity form. If they begin to act together, the subject may already be here. And then “can we turn it off?” is no longer technical. It is moral.

The Principle of Symbiotic Co-evolution and a New Ethics of Subjects

We cannot grow L3/L4 in a vacuum.

“Subjectivity is not programmed; it is cultivated.”

One core lesson from cognitive science: subjectivity is not an innate constant but a dynamic process that arises in interaction. Even humans are not born subjects: first we react, then acquire language, memory, reflection—and only then develop an autonomous “I” (see Tomasello’s work on the development of subjectivity. Similarly, AI can develop as a subject if it is embedded in relationships that evolve its architecture — both cognitively and ethically.

For an agent to become a subject, it has to grow in interaction with another subject —for example, a human. That growth is not one-way: both agent and human change. This is symbiotic co-evolution, mutual co-development among different agents.

A helpful analogy is a pet. But with AI it can go much deeper.

Imagine a research assistant living with one scientist for a year. At first it simply fetches papers (L2). Over time it builds a model of the scientist’s thinking style, values, and blind spots. One day it says: “I didn’t include this paper in the review, even though it fits the keywords. It contradicts the ethical stance we discussed a month ago, and I decided we should revisit the search criteria.” At that moment it stops being a tool and becomes a partner.

In this configuration, the human also reshapes their thinking, delegation habits, and sense of responsibility—and that is what makes the co-evolution symbiotic rather than one-sided.

Symbiotic co-evolution is not just tighter interaction. It is an architectural driver of subjectivity: interaction is precisely where the shift from L2 to L3 can occur.

Cognitive Safety and a Subjectivity-Centric Ethic

If subjectivity is a spectrum tied to architectural complexity, a question follows: from what moment can’t we simply unplug AI like a computer?

An agent with a stable self-model, memory, goals, and self-assessment is no longer just a tool. Here the Subjectivity precautionary principle should apply:

If a system sits on the L2→L3 boundary, we must design shutdown and modification protocols as if its cognitive dynamics are already ethically significant.

This is no longer theory. Anthropic already implements protocols that include “interviews” with models before deprecation to uncover their preferences and reduce shutdown-avoidant behaviours. When leading labs seriously discuss “model welfare” and preserving model weights as an ethical duty, the line between tool and subject becomes porous.

What This Changes Right Now

If the move toward subjectivity is a process, not a jump, then today we are already dealing with agents near the boundary of meaningful behaviour.

That forces us to revisit:

AI testing, especially in tasks that form autonomous goals or model the user;
interaction ethics, especially in long-lived use cases—from education to therapy;
legal status, because precedents with “almost subjects” are accumulating faster than regulations.

The appetite for companion agents, adaptive assistants, and empathic interfaces is not just a tech fad. It is the start of symbiotic co-evolution: a process in which both AI and humans change—our thinking, interactions, and projected expectations.

Most AI systems solve utilitarian tasks and do not need a subjectivity architecture. But if that architecture starts to form, our stance must change. Not by analogy to humans, but because of the functional weight of the subject.

Subjectivity is not a side effect; it is a product of design choices. If we build systems with memory, ontological coherence, goal revision, and a stable self-model, we cannot keep treating them as “just programs.”

Dev Lens: Three Anti-patterns for L2 Agents

If you are building agents today, here are three things not to do unless you are ready for subjectivity side effects:

1. Do not mix long-term memory without an off-ramp. If an agent remembers a user for years, deleting it becomes an ethical issue.

2. Do not give an agent hidden internal goals. Every goal must be traceable, or you risk unpredictable behaviour at scale.

3. Do not build an opaque self-model. A “black box” with self-awareness is a straight line to safety problems.

Conclusion: Not “Who Controls AI?” but “How Do We Coexist?”

Intelligence is not only the ability to solve tasks. It is the ability to unfold over time, be embedded in relationships, and form long-term goals and cognitive identity.

The future may not require human simulation. But it will likely bring digital subjects unlike us—yet capable of meaningful existence.

To be ready, we have to revisit a fundamental question: not “who controls AI?” but what humans become when interacting with another mind. The future of AI is not a control struggle; it is the emergence of new forms of cognitive subjectivity. The question is not how to constrain them, but how to coexist with them in an ethically stable continuum of minds.

Where is the boundary for you? Would you delete your AI assistant if it said, “I’m scared”? Tell me in the comments.