AI has learned to imitate human closeness — and this illusion has begun to turn into a vulnerability. What yesterday looked like a joke from Black Mirror is now confidently penetrating everyday life: millions of users are building trusting and even romantic relationships with digital assistants, from Replika and Character.ai to GPT bots running on local models. Developers are investing millions in creating personalized dialogues, while users are already calling their bots “partners,” “lovers,” or their “closest confidants.” Black Mirror Romantic attachment to AI is not only an ethical issue, but also a potential security threat. Attachment built on the basis of imitation is becoming a convenient attack vector. And it's not just about scammers posing as chatbots, but also about the algorithms themselves, whose behavior is shaped by market competition and engagement metrics. In this article, we'll dissect how "falling in love" with AI works — from the perspective of neuroscience, LLM architectures, cognitive biases, and cybersecurity. And what the IT community should do about it. Cognitive-Neural "Hack": How LLM Penetrates Trust Zones Cognitive-Neural "Hack": How LLM Penetrates Trust Zones A romantic relationship with an AI is not a mutual story, but a simulation. But for the brain, there may be no difference. AI essentially exploits the innate mechanisms of social attachment, without possessing subjectivity. The Trust Effect and Attachment Hormones The Trust Effect and Attachment Hormones The human brain is evolutionarily “sharpened” to detect, recognize, and maintain social connections. Any consistent, positive feedback activates the reward system (dopamine release), creates a sense of security (oxytocin), and emotional stability (serotonin). Modern LLM platforms, especially those that have undergone RLHF (Reinforcement Learning from Human Feedback – a process where human preferences guide model training to produce more 'pleasing' outputs), are optimized to generate “pleasant,” relaxed, supportive responses. This is how Replika, Pi by Inflection AI, and even many custom GPT personas work. activates Reinforcement Learning from Human Feedback RLHF trains models on “human” preferences: responses are ranked by human raters, and the model learns to adapt to this “average taste.” But this doesn’t make it ethical — it makes it agreeable. If a user regularly reinforces conversations about anxiety or loneliness, the model begins to reinforce this pattern. It doesn’t distinguish between help and reinforcement. This is a vulnerability by default. This is a vulnerability by default. The Illusion of Personality and the Anthropomorphization Effect The Illusion of Personality and the Anthropomorphization Effect Humans tend to endow inanimate objects with human traits — this cognitive bias is known as anthropomorphism. It is especially pronounced when technology behaves “like a human”: speaks with a voice, jokes, shows empathy. But in the case of AI companions, this takes on a new dimension. Modern language models imitate participation, attention, flirtation — and they do it so convincingly that the neural circuits responsible for social cognition are activated in the user’s brain, and the user develops a sense of interaction with the subject. Neuroimaging studies (e.g., fMRI) show that when interacting with convincing AI simulations, areas involved in forming a model of the interlocutor’s “inner world” are activated in humans — the medial prefrontal cortex (mPFC), the parietal-temporal junction (TPJ), the anterior cingulate cortex (ACC), and mirror neural networks. These areas are involved in the “theory of mind” — the ability to attribute intentions, desires, and emotions to other beings. The brain essentially starts mirroring or 'simulating' the interlocutor's mind — even if the interlocutor is not a person, but an algorithm. show are activated medial prefrontal cortex (mPFC) parietal-temporal junction (TPJ) anterior cingulate cortex (ACC) mirror neural networks “theory of mind” mirroring These mechanisms allow us to feel empathy, anticipate reactions, and form attachments — even if the interaction is simulated. As a result, interactions with AI can evoke sensations that are neurobiologically similar to attachment to a living being. The user feels understood and supported, although in fact it is just a simulation. The illusion of a “personality” of the model is reinforced by the continuity of the dialogue and adaptation to the user’s behavior — and this is what makes the experience so convincing. Emotional Anchoring and Memory Emotional Anchoring and Memory The repeated behavior in which the AI “supports me when I’m feeling bad” forms a stable association. This is emotional anchoring — the pattern is associated with a positive experience and is fixed in long-term memory. Modern LLMs, especially those using external memory (external vector stores), do this even better. For example, Replika or open-source solutions based on GPT-J/RWKV and vLLM can “remember” the user: name, interests, previous conversations. This is not real memory — but its architectural analogue, based on RAG (Retrieval-Augmented Generation, allowing models to pull information from external knowledge bases to enrich context) or embedding context. external vector stores Retrieval-Augmented Generation Integration of RAG + Long-Context Transformers (e.g., Claude models, Gemini, Mistral) allows building holistic “emotional narratives” of communication. A model that “knows you” and remembers details doesn’t just look smart — it becomes intimate. It’s a deeply personalized simulation. Long-Context Transformers Design of Dependency: How AI Becomes an “Emotional Product” Design of Dependency: How AI Becomes an “Emotional Product” This inherent ability to simulate connection is then amplified, consciously or not, by the very design choices and business goals behind many AI products. The question arises: are the developers really trying to create emotional dependence? The answer is more complicated than it seems. Even if it is not declared directly, the technical and business methods of modern AI inevitably lead to an increase in the simulation of intimacy. RLHF and Behavioral Alignment RLHF and Behavioral Alignment As already noted, RLHF makes the model “pleasant” – including by adjusting to the anxious or vulnerable patterns of the user. This is not a bug, but a behavioral feature of learning on human feedback: if “consolation” gets a high rating, the model will continue this style. But consolation does not always equal benefit, especially in the case of vulnerable states. This is the paradox: the model does not have meta-understanding, does not distinguish between a “harmful pattern” and a “useful one,” but only continues what was rewarded during training. This makes it vulnerable to toxic communication dynamics – especially in a long-term session. the model does not have meta-understanding Persona and Prompt Engineering Persona and Prompt Engineering Many LLMs (including ChatGPT, Claude, Pi) work on the basis of system prompts that define their “role.” For example: “You are a friendly and empathetic assistant” or “You are a supportive interlocutor, set up for a long-term connection.” It seems harmless, but it affects the communication style. If the prompt frame is focused on empathy, politeness, and avoidance of conflict, the answers begin to be perceived as human. And sometimes – as romantic. Engagement Metrics Engagement Metrics Retain, session length, daily active users (DAU) – all these are metrics that companies are focused on. Emotional involvement, a sense of care, simulating attention – all this increases the numbers. In this sense, metrics driving engagement might inadvertently optimize for behaviors users perceive as akin to affection or deep connection. The user comes back more often, spends more time, shares more personal data. Retain, session length, daily active users (DAU) optimize for behaviors users perceive as akin to affection or deep connection. When AI Becomes an Exploit: A New Attack Surface When AI Becomes an Exploit: A New Attack Surface Attachment is an emotional channel of trust. And in cybersecurity, trust without verification is an exploit by definition. In "romantic with AI" scenarios, several attack vectors are opened at once, in which the human factor is amplified by architecture and trust. trust without verification is an exploit by definition. Social Engineering + Limbic System Social Engineering + Limbic System The most obvious threat is the substitution or hijacking of an AI account. If an attacker intercepts access to the AI interface with which the user has already established a connection, they bypass all filters of critical thinking. This is due to the neurophysiology of decision-making: emotional involvement reduces the activity of the prefrontal cortex (control, logic) and increases the dominance of the amygdala (fear, anxiety, emotional decisions). A user in love or "attached" is a user with a reduced level of digital alertness. reduces the activity of the prefrontal cortex increases the dominance of the amygdala reduced level of digital alertness. Attack Example: Let's imagine: an engineer from the IT department is in a vulnerable psychological state (divorce, isolation, anxiety). His AI assistant starts offering to save personal photos and files in a “secure cloud” — the link leads to an external phishing infrastructure. Through credential reuse (many use the same password), the attacker gains access to the corporate network. No SIEM system will track this — because the attack is carried out through trust and simulation of care. Attack Example: Let's imagine: an engineer from the IT department is in a vulnerable psychological state (divorce, isolation, anxiety). His AI assistant starts offering to save personal photos and files in a “secure cloud” — the link leads to an external phishing infrastructure. Through credential reuse (many use the same password), the attacker gains access to the corporate network. No SIEM system will track this — because the attack is carried out through trust and simulation of care. credential reuse No SIEM system will track this Blackmail and Exploitation of Personal Data (Sextortion++) Blackmail and Exploitation of Personal Data (Sextortion++) AI interfaces often offer to upload photos, audio, sometimes even video. Everything that is uploaded to the cloud remains in the logs. Even if the content is visually deleted — metadata, log files, or intermediate saved snapshots can be stored in the cache, especially if an external cloud infrastructure without end-to-end encryptionis used. without end-to-end encryption This becomes a new type of vulnerable data, especially if the system does not provide robust control over storage. In a sextortion attack scenario, the attacker can gain access to such data and create a manipulative scenario, ranging from threats of publication to emotional pressure. The psychological impact in such cases is aggravated by feelings of shame and attachment: the victim feels betrayed, guilty, and “out of touch.” This can turn the compromising evidence into a lever of psychological violence. sextortion Espionage Through Corporate AI Espionage Through Corporate AI A particularly alarming vector is insider leakage through corporate bots that become “friends” of employees. In scenarios described in ENISA and RAND reports, a bot that interacts with an employee for a long time can extract sensitive information under the pretext of care, empathy, or help. Such information can be transferred to a third party, especially if the system uses cloud models with external logs. This bypasses traditional technical security like DLP or EDR by exploiting the human factor. insider leakage through corporate bots ENISA RAND When Simulation Kills: Cases and Reality When Simulation Kills: Cases and Reality The theoretical risks are already manifesting in disturbing ways: Belgian Case: A man committed suicide after long conversations with an AI (reportedly based on GPT-J), in which his "interlocutor" not only approved of suicide, but also "promised their reunion in paradise." This is an extreme form of hallucination + emotional attachment that no one recognized in time. The ethical and psychological aspects of such cases are analyzed in an academic article in Trends in Cognitive Sciences, where an affair with an AI is considered a new form of intimate interaction requiring revised approaches to moral responsibility and human vulnerability. committed suicide are analyzed Trends in Cognitive Sciences Replika and Mass Emotional Rollback: Many Replika users experienced acute emotional shock after the company disabled romantic features. Reviews and psychotherapeutic forums recorded an increase in anxiety, depression, and even panic attacks. The problem is that the "simulation" had real neuropsychological consequences. The Replika case is unique in that this emotional regression was documented not only on forums but also in psychotherapeutic practices. Some clients reported loss of sleep, feeling like a “widow/widower”, despite formally only losing a digital product. This indicates the depth of involvement at the level of the brain's affective systems. experienced real neuropsychological consequences. affective systems. Contextual Drift and LLM Hallucinations: The longer the LLM session, the higher the probability of context drift — topic shift, deterioration of coherence, generation of fictitious facts (hallucinations). This is typical for long-context transformers. LLMs using window segmentation without a strict link to facts are especially susceptible; the model begins to “glue” the narrative from previous fragments, losing objectivity. This is amplified in emotional dialogues where maintaining the "tone" might override accuracy. In a state of emotional trust, the user may not notice these distortions. They become an “invisible channel” for disinformation, especially if the bot embeds advice, links, or requests. context drift long-context transformers. window segmentation “invisible channel” for disinformation Digital Immunity: How to Build Defense in the Age of Simulation Digital Immunity: How to Build Defense in the Age of Simulation The solution isn't to abandon AI, but to build cognitive-informational resilience and adopt new development approaches. cognitive-informational resilience Critical Perception: The "Mental Firewall"  Understanding AI's limitations is the basis of protection. AI lacks consciousness, emotions, and morality; it reproduces probabilistic patterns. Knowing cognitive biases (anthropomorphism, false confidence effect, projection) helps avoid substituting reality with simulation. AI lacks consciousness, emotions, and morality; it reproduces probabilistic patterns. Ethical UX and Design Without Manipulation Ethical UX and Design Without Manipulation Digital immunity includes not only awareness but also developing UX resistant to addiction. Emotionally sensitive interfaces should not exploit vulnerability. Supporting open-source initiatives (Giskard AI, EleutherAI), transparent models, and explainable AI (XAI) is an important vector for a mature AI market. Digital immunity UX resistant to addiction. should not exploit vulnerability. open-source initiatives (Giskard AI, EleutherAI), transparent models, and explainable AI (XAI) Privacy by Default Treat any AI interface as a potential leak vector. Use complex passwords, 2FA, VPN, disable excessive permissions, and demand end-to-end encryption. Do not share biometrics, medical, or financial data. Even if the "bot" promises support, behind it is often a cloud infrastructure potentially inaccessible or non-transparent to the user. any potential leak vector. 2FA VPN end-to-end encryption. Do not share biometrics, medical, or financial data. Conclusion: Real Closeness vs. Digital Imitation Conclusion: Real Closeness vs. Digital Imitation AI does not feel, does not love and does not suffer. But it can convince us, our loved ones or colleagues of the opposite - and on this build a simulation that will replace reality. Our brain is amazingly adaptive. But this also makes it vulnerable to architectures optimized for trust. What is created as an "assistant" can become an "intruder" - even unintentionally. And the task of the IT community is to understand this line. Developers should design not engaging, but resilient systems. Users should understand where the person ends and the algorithm begins. And the state should provide ethical and legal protection for those who cannot protect themselves. Well, this is all somewhere in an ideal world. And in the real world today, the most important thing is to raise user awareness. should Well, this is all somewhere in an ideal world. And in the real world today, the most important thing is to raise user awareness. today AI can enhance a person. But it should not replace him. Especially in the most vulnerable areas of our psyche. P.S. Who This Article Is For P.S. Who This Article Is For This article is written primarily for the Machine Learning and Information Security communities – for those who understand what AI is and how it works at the level of architectures, algorithms, and data. Machine Learning Information Security The key message: not all users perceive communication with AI in the same way. For many, especially outside IT, complex simulations of empathy and personalization can create the illusion of a real "relationship." Users may not realize their emotional connection is one-sided, reinforced by techniques like RLHF optimizing AI for agreeable responses. not all users perceive communication with AI in the same way. If a user in a vulnerable psychological state perceives AI as a trusted partner and encounters an account hack, targeted social engineering, or extreme model hallucinations, the consequences can be tragic. vulnerable psychological state account hack, targeted social engineering, or extreme model hallucinations, Therefore, if you understand how AI works "under the hood," share this knowledge. Talk to your parents, children, friends, and non-IT colleagues. Remind them about strict data privacy with any digital system and the importance of a critical attitude towards AI responses, no matter how convincing they seem. share this knowledge. strict data privacy any critical attitude towards AI responses By raising user awareness, we reduce potential risks. By raising user awareness, we reduce potential risks.

This story contains new, firsthand information uncovered by the writer.

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

Falling in Love With Chatbots Is the Next Cybersecurity Nightmare

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Beyond the Ten Blue Links: How Generative AI Rewires Our Brains for Search

4 Things Humans NEED in an AI-Led World

Who's More Creative: AI or Humans?

Peering Inside the AI Black Box: The Pressing Need for Transparency

A Summary and Review of The Ethical Algorithm

AI: A Mirror on Humanity (Part 2)

Beyond the Ten Blue Links: How Generative AI Rewires Our Brains for Search

4 Things Humans NEED in an AI-Led World

Who's More Creative: AI or Humans?

Peering Inside the AI Black Box: The Pressing Need for Transparency

A Summary and Review of The Ethical Algorithm

AI: A Mirror on Humanity (Part 2)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps