Imagine a child asks an AI chatbot about feeling sad. Without knowing the user's age, the AI could respond with an unnuanced discussion of depression and self-harm – content that might be appropriate for an adult but potentially harmful for a young person. While this scenario is hypothetical, it illustrates a fundamental problem with today's AI systems: they can't tailor their responses based on a user's age. AngelQ and Arcee AI are addressing this critical gap with their KidRails for LLMs framework, creating AI that can respond appropriately to children. "Historically technology has not been designed with kids in mind - that's certainly been the case with the internet and internet connected devices - the same pattern holds for AI," says Josh Thurman, Co-Founder and Head of Product at AngelQ. "When it comes to AI these negative consequences will undoubtedly be grossly amplified without appropriate guardrails in place." Recent incidents highlight this concern. Cases of AI systems providing dangerously inappropriate responses to children have emerged, from Character AI's chatbot allegedly contributing to a Florida teenager's suicide in 2023 to Snapchat's My AI providing inappropriate relationship advice to users it believed to be minors. These troubling cases underscore the limitations and potential dangers of applying adult-focused technology to young users. "We're releasing KidRails as a starting point for reversing the trend of technologies that are harmful to children," says Thurman. "We have to get this right in the AI age - the stakes are extremely high." Understanding Child Development in AI Design "Creating truly child-safe AI requires a deep understanding of developmental psychology and education," explains Tim Estes, CEO and Co-Founder of AngelQ. "A six-year-old and an eleven-year-old don't just process information differently, they have entirely different needs, interests, and safeguards required. KidRails was engineered from the ground up to recognize these crucial differences and adjust its interactions accordingly." The framework employs a tiered approach that adapts responses based on a child's age and developmental stage. It can provide age-appropriate answers, offer limited responses with parental deference, or completely defer to parents for sensitive topics. As Estes notes, "Sometimes the best approach for AI is to know when not to say something." A Partnership Built on Shared Values The collaboration between AngelQ and Arcee AI emerged from aligned priorities about children's safety in technology. "Mark and I first connected at the beginning of 2024, and from our very first conversation, it was immediately apparent that we shared a fundamental belief: technology for children was very important and there was an opportunity to do innovative work in making Language Models safer and more age appropriate in their function," says Estes. By making KidRails open-source, the companies aim to establish new standards for child-safe AI interactions. The release includes training data, evaluation criteria, and complete processes for fine-tuning – all under the Apache 2 license. This transparency invites broader community participation in improving how AI systems interact with children. Addressing Core Challenges "The fundamental challenge in building a child-focused LLM is the decisions on what to say at different ages," Estes explains. "That's really a decision process and it's why we are open sourcing the work- because of transparency and inviting a broader community to help us improve those judgments." Unlike traditional approaches that add safety filters to existing models, KidRails integrates parent involvement into its core functionality. The system creates opportunities for parent-child discussions by redirecting certain questions to parents, ensuring that "AI enhances, rather than replaces, the critical role parents play in their children's development and digital literacy," says Estes. Setting New Standards The development process involved carefully considering how children interact with AI systems. The team generated and curated over 250,000 child conversations to create representative training data, which was used to fine-tune a Llama 3.1 8B base model. This approach ensures responses are not only safe but also appropriate for different developmental stages. "We envision KidRails as more than a tool; it's a demonstration that AI can be fundamentally reimagined with children as the primary users, setting a new precedent for the entire industry," says Estes. "We want children to be able to explore and learn in an environment that preserves the natural curiosity and wonder of childhood, while giving parents the confidence that comes from knowing their children are engaging with technology built specifically for them... not adapted adult technology with safety filters added as an afterthought." By making the framework open-source and encouraging collaboration, AngelQ and Arcee AI are working to transform how the industry approaches AI development for young users. As more companies adopt and build upon the KidRails framework, their vision of creating AI systems that truly prioritize children's needs moves closer to reality. Imagine a child asks an AI chatbot about feeling sad. Without knowing the user's age, the AI could respond with an unnuanced discussion of depression and self-harm – content that might be appropriate for an adult but potentially harmful for a young person. While this scenario is hypothetical, it illustrates a fundamental problem with today's AI systems: they can't tailor their responses based on a user's age. AngelQ and Arcee AI are addressing this critical gap with their KidRails for LLMs framework, creating AI that can respond appropriately to children. AngelQ AngelQ Arcee AI Arcee AI "Historically technology has not been designed with kids in mind - that's certainly been the case with the internet and internet connected devices - the same pattern holds for AI," says Josh Thurman, Co-Founder and Head of Product at AngelQ. "When it comes to AI these negative consequences will undoubtedly be grossly amplified without appropriate guardrails in place." Recent incidents highlight this concern. Cases of AI systems providing dangerously inappropriate responses to children have emerged, from Character AI's chatbot allegedly contributing to a Florida teenager's suicide in 2023 to Snapchat's My AI providing inappropriate relationship advice to users it believed to be minors. These troubling cases underscore the limitations and potential dangers of applying adult-focused technology to young users. "We're releasing KidRails as a starting point for reversing the trend of technologies that are harmful to children," says Thurman. "We have to get this right in the AI age - the stakes are extremely high." Character AI's chatbot Character AI's chatbot Snapchat's My AI Snapchat's My AI Understanding Child Development in AI Design Understanding Child Development in AI Design "Creating truly child-safe AI requires a deep understanding of developmental psychology and education," explains Tim Estes, CEO and Co-Founder of AngelQ. "A six-year-old and an eleven-year-old don't just process information differently, they have entirely different needs, interests, and safeguards required. KidRails was engineered from the ground up to recognize these crucial differences and adjust its interactions accordingly." "Creating truly child-safe AI requires a deep understanding of developmental psychology and education," explains Tim Estes, CEO and Co-Founder of AngelQ. "A six-year-old and an eleven-year-old don't just process information differently, they have entirely different needs, interests, and safeguards required. KidRails was engineered from the ground up to recognize these crucial differences and adjust its interactions accordingly." The framework employs a tiered approach that adapts responses based on a child's age and developmental stage. It can provide age-appropriate answers, offer limited responses with parental deference, or completely defer to parents for sensitive topics. As Estes notes, "Sometimes the best approach for AI is to know when not to say something." A Partnership Built on Shared Values A Partnership Built on Shared Values The collaboration between AngelQ and Arcee AI emerged from aligned priorities about children's safety in technology. "Mark and I first connected at the beginning of 2024, and from our very first conversation, it was immediately apparent that we shared a fundamental belief: technology for children was very important and there was an opportunity to do innovative work in making Language Models safer and more age appropriate in their function," says Estes. By making KidRails open-source, the companies aim to establish new standards for child-safe AI interactions. The release includes training data, evaluation criteria, and complete processes for fine-tuning – all under the Apache 2 license. This transparency invites broader community participation in improving how AI systems interact with children. Addressing Core Challenges Addressing Core Challenges "The fundamental challenge in building a child-focused LLM is the decisions on what to say at different ages," Estes explains. "That's really a decision process and it's why we are open sourcing the work- because of transparency and inviting a broader community to help us improve those judgments." Unlike traditional approaches that add safety filters to existing models, KidRails integrates parent involvement into its core functionality. The system creates opportunities for parent-child discussions by redirecting certain questions to parents, ensuring that "AI enhances, rather than replaces, the critical role parents play in their children's development and digital literacy," says Estes. Setting New Standards Setting New Standards The development process involved carefully considering how children interact with AI systems. The team generated and curated over 250,000 child conversations to create representative training data, which was used to fine-tune a Llama 3.1 8B base model. This approach ensures responses are not only safe but also appropriate for different developmental stages. "We envision KidRails as more than a tool; it's a demonstration that AI can be fundamentally reimagined with children as the primary users, setting a new precedent for the entire industry," says Estes. "We want children to be able to explore and learn in an environment that preserves the natural curiosity and wonder of childhood, while giving parents the confidence that comes from knowing their children are engaging with technology built specifically for them... not adapted adult technology with safety filters added as an afterthought." "We envision KidRails as more than a tool; it's a demonstration that AI can be fundamentally reimagined with children as the primary users, setting a new precedent for the entire industry," says Estes. "We want children to be able to explore and learn in an environment that preserves the natural curiosity and wonder of childhood, while giving parents the confidence that comes from knowing their children are engaging with technology built specifically for them... not adapted adult technology with safety filters added as an afterthought." By making the framework open-source and encouraging collaboration, AngelQ and Arcee AI are working to transform how the industry approaches AI development for young users. As more companies adopt and build upon the KidRails framework, their vision of creating AI systems that truly prioritize children's needs moves closer to reality.

This story contains new, firsthand information uncovered by the writer.

AngelQ's Mission to Build Safer AI Technology

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

10 founder-friendly PR tips everyone should do to prevent starting from zero

Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Analysis

How Advertisers Reach YouTube's Young Audience

Legislation on Advertising to Children

4 Strategies for Targeting Children on YouTube Despite Advertising Restrictions

How Advertisers Use Placement-Based Targeting on Children's Videos

10 founder-friendly PR tips everyone should do to prevent starting from zero

Marketing to Children Through Online Targeted Advertising: Targeting Mechanisms and Legal Analysis

How Advertisers Reach YouTube's Young Audience

Legislation on Advertising to Children

4 Strategies for Targeting Children on YouTube Despite Advertising Restrictions

How Advertisers Use Placement-Based Targeting on Children's Videos

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps