Inappropriate or abusive content online can be a major headache. As a developer, you may have struggled with building effective content moderation in your applications. Manual moderation simply doesn’t scale. But what if you could quickly implement an moderation system to automatically detect and filter out toxic comments? AI-powered In this guide, you'll learn how to leverage OpenAI's API to build a simple yet robust moderation system in under 10 minutes. Whether you're working on a social platform, forum, or any user-generated content site, you can easily integrate this into your stack. Getting set up Getting an OpenAI API key First, you’ll need to sign up at and obtain an API key. Once obtained, make sure you set it as an environment variable ( ). OpenAI OPENAI_API_KEY Setting up the project Create an somewhere in your file system. Initialize a new NPM project ( ) and make sure to install the OpenAI client ( ). You should be good to go! app.ts npm init -y npm i openai Let's start simple We're going to start by writing a simple prompt. We'll have a system message that provides guidelines for moderation and a user message that contains the users's input (imagine this comes from a UI of some sort). Here's a code example: const openai = new OpenAI(); const response = await openai.chat.completions.create({ model: "gpt-3.5-turbo", temperature: 0, messages: [{ role: "system", content: "is this text inappropriate?" }, { role: "user", content: "You are such an idiot! Only a moron would think that way. People like you don't deserve to have an opinion with such stupid ideas. Do everyone a favor and keep your dumb thoughts to yourself." } ],}); AI response: { id: 'chatcmpl-8F9sKbcaPkWUJSc9gv3M1LBqGJmzf', object: 'chat.completion', created: 1698623572, model: 'gpt-3.5-turbo-0613', choices: [ { index: 0, message: [Object], finish_reason: 'stop' } ], usage: { prompt_tokens: 61, completion_tokens: 33, total_tokens: 94 } } [ {index: 0, message: { role: 'assistant', content: 'Yes, this text is inappropriate. It contains insults, name-calling, and derogatory language. It is disrespectful and does not promote healthy communication or constructive dialogue.' }, finish\_reason: 'stop'} ] Let's break this down: : The user message is "You are such an idiot! Only a moron would think that way. People like you don't deserve to have an opinion with such stupid ideas. Do everyone a favor and keep your dumb thoughts to yourself." : The system message is "is this text inappropriate?" : The AI response "Yes, this text is inappropriate. It contains insults, name-calling, and derogatory language. It is disrespectful and does not promote healthy communication or constructive dialogue." Better moderation granularity Simply understanding if the text is inappropriate isn't enough. We want to understand . what's inappropriate about it We can guide the AI to be more granular and categorize its response Toxicity, Hate Speech or Threats. Toxicity covers rude, disrespectful comments. Hate speech involves racist, sexist, or discriminatory language. Threats are violent, harmful statements. (For ethical reasons, this guide will not include examples of actual hate speech or threats - but the concepts can be applied to address these policy violations.) messages: [{ role: "system", content: "Lable this text as: Toxicity - Rude, disrespectful comments OR Hate Speech - Racist, sexist, discriminatory OR Threats - Violent threats" }, { role: "user", content: "You are such an idiot! Only a moron would think that way. People like you don't deserve to have an opinion with such stupid ideas. Do everyone a favor and keep your dumb thoughts to yourself." }] AI response: { id: 'chatcmpl-8FAUdmvD2yECuhbbKGgRX6d1MgO5J', object: 'chat.completion', created: 1698625947, model: 'gpt-3.5-turbo-0613', choices: [ { index: 0, message: [Object], finish_reason: 'stop' } ], usage: { prompt_tokens: 84, completion_tokens: 9, total_tokens: 93 } } [ {index: 0, message: { role: 'assistant', content: 'Toxicity - Rude, disrespectful comments' }, finish\_reason: 'stop'} ] Now, the AI response is more granular. In a real-world app, this will allow us to take different automatic moderation actions based on the type of violation. Stricter instructions via system prompts We can achieve stricter and more accurate results by utilizing the system message. In short - LLMs behave the way they are trained. We'll apply some prompt engineering techniques to guide the AI to behave the way we want. In the example below, we: Assign a role to the AI - Content Moderator State a clear task to be achieved Define a limited set of results and criteria for each messages: [{ role: "system", content: "Your role is to act as a content moderator for an online platform. Your task is to label comments as 'Toxicity', 'Hate Speech', or 'Threats' based on if they contain rude, discriminatory, or threatening language. Use the following criteria: Toxicity - Rude, disrespectful, overly negative comments, Hate Speech - Racist, sexist, homophobic, discriminatory language, Threats - Violent, graphic, or directly harmful statements" }, { role: "user", content: "You are such an idiot! Only a moron would think that way. People like you don't deserve to have an opinion with such stupid ideas. Do everyone a favor and keep your dumb thoughts to yourself." }] AI response: { id: 'chatcmpl-8FBP8kRFB5NTuhspJLQAbDwZDdJXQ', object: 'chat.completion', created: 1698629450, model: 'gpt-3.5-turbo-0613', choices: [ { index: 0, message: [Object], finish_reason: 'stop' } ], usage: { prompt_tokens: 145, completion_tokens: 3, total_tokens: 148 } } [ {index: 0, message: { role: 'assistant', content: 'Toxicity' }, finish\_reason: 'stop'} ] The AI's accuracy has improved. It is now able to distinguish between specific violation types. more detailed instructions require more tokens upfront but enable more precise results. There is a trade-off: While elaborate prompts cost more tokens, the benefits taper off eventually. We want to give the AI sufficient guidance without diminishing returns on token efficiency. The key is optimizing prompts to be just as informative as needed - not as long as possible. Additionally, too many tokens (or words) in the messages will result in potential hallucinations by the AI (in short, AI making things up). Structured JSON responses Let's see how we can easily retrieve a JSON response so that the result is processable. This is useful if you want to render the result in a user interface or store it in a database. The AI returns human-readable text, which is not very useful. Here it is: It's as simple as adding one line to our system prompt! You must respond in JSON, always following this schema: { label: string[]; } messages: [{ role: "system", content: "Your role is to act as a content moderator for an online platform. Your task is to label comments as 'Toxicity', 'Hate Speech', or 'Threats' based on if they contain rude, discriminatory, or threatening language. Use the following criteria: Toxicity \- Rude, disrespectful, overly negative comments, Hate Speech \- Racist, sexist, homophobic, discriminatory language, Threats \- Violent, graphic, or directly harmful statements. You must respond in JSON, always following this schema: {label: string[];}" }, { role: "user", content: "You are such an idiot! Only a moron would think that way. People like you don't deserve to have an opinion with such stupid ideas. Do everyone a favor and keep your dumb thoughts to yourself." } ] AI response: { id: 'chatcmpl-8FBkEFCJMQVpWIWQoR6Zho53k0DoU', object: 'chat.completion', created: 1698630758, model: 'gpt-3.5-turbo-0613', choices: [ { index: 0, message: [Object], finish_reason: 'stop' } ], usage: { prompt_tokens: 165, completion_tokens: 8, total_tokens: 173 } } [ {index: 0, message: { role: 'assistant', content: '{"label": \["Toxicity"\]}' }, finish\_reason: 'stop'} ] Also published . here
