The interactivity and unpredictability of user-generated content (UGC) platforms are a big part of why they are so popular. But, that unpredictability means that communities must be diligent in monitoring their content to make sure that it meets their community guidelines or acceptable use policy and is appropriate, safe, and welcoming for all users. This often results in a moderation system where users report potential offenses to the community guidelines and moderators or admins take action as necessary. This is often a manual process that leaves much to be desired. live-streaming (AI) and Machine Learning (ML) tools have improved in recent years, and developers can use these tools to assist in moderating their communities. In this post, we'll look at one way to do that with Amazon Interactive Video Service (Amazon IVS) and Amazon Rekognition. Artificial Intelligence Solution Overview Analyzing every frame of every live stream in an application with would be a very expensive and difficult task. Instead, developers can analyze samples of the live streams in their applications on a specified frequency to assist their moderators by alerting them if there is content in need of further review by a human moderator. It isn't a 100% perfect solution, but it's one way to automate content moderation and help make moderators' jobs easier. AI/ML This solution involves the following steps: Configure on your Amazon IVS Channels to save thumbnail images on a specified frequency auto recording of live streams to Amazon Simple Storage Service (Amazon S3) Create an Amazon EventBridge rule that fires when a new object is created in the Amazon S3 bucket Create an AWS Lambda function that gets triggered by the EventBridge rule and uses Amazon Rekognition to detect content like nudity, violence or gambling that might need to be moderated by a human moderator Create an AWS Lambda function and expose it via Amazon API Gateway to provide a means to stop a live stream, if necessary Send a custom event to an containing the results of the analysis Amazon IVS chat room Creating the Amazon EventBridge Rule and AWS Lambda Functions We'll use AWS Serverless Application Model (SAM) to make it easy to create the rule and functions. Here's the entire file that describes the necessary permissions, the Amazon EventBridge rule, the AWS Lambda layer (for the AWS SDK dependency), and the function definitions. We'll break this down below. template.yaml AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Amazon IVS Moderation Functions
Globals:
  Function:
    Runtime: nodejs18.x
    Timeout: 30
    MemorySize: 128
  Api:
    EndpointConfiguration: 
      Type: REGIONAL
    Cors:
      AllowMethods: "'GET, POST, OPTIONS'"
      AllowHeaders: "'Content-Type'"
      AllowOrigin: "'*'"
      MaxAge: "'600'"
Resources:
  IvsChatLambdaRefLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: sam-app-dependencies
      Description: Dependencies for sam app
      ContentUri: dependencies/
      CompatibleRuntimes:
        - nodejs18.x
      LicenseInfo: "MIT"
      RetentionPolicy: Retain
  IVSAccessPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: IVSModerationAccessPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - 's3:GetObject'
              - 's3:GetObjectAcl'
              - 'ivschat:SendEvent'
              - 'ivs:StopStream'
              - 'rekognition:DetectModerationLabels'
            Resource: '*'
      Roles:
        - Ref: ModerateImageRole
        - Ref: StopStreamRole
  ApiAccessPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: ApiAccessPolicy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Action:
              - 'sts:AssumeRole'
            Resource: '*'
      Roles:
        - Ref: ModerateImageRole
        - Ref: StopStreamRole
  EventRule:
    Type: AWS::Events::Rule
    Properties:
      Description: EventRule
      State: ENABLED
      EventPattern: 
        source:
          - aws.s3
        detail-type:
          - "Object Created"
        detail:
          bucket:
            name:
              - ivs-demo-channel-stream-archive
          object:
            key:
              - suffix: .jpg
      Targets:
        - Arn: !GetAtt ModerateImage.Arn
          Id: MyLambdaFunctionTarget
  PermissionForEventsToInvokeLambda:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref ModerateImage
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt EventRule.Arn
  ModerateImage:
    Type: 'AWS::Serverless::Function'
    Properties:
      Environment:
        Variables:
          DEMO_CHAT_ARN: 'arn:aws:ivschat:us-east-1:[redacted]:room/[redacted]'
          DEMO_CHANNEL_ARN: 'arn:aws:ivs:us-east-1:[redacted]:channel/[redacted]'
      Handler: index.moderateImage
      Layers:
        - !Ref IvsChatLambdaRefLayer
      CodeUri: lambda/
  StopStream:
    Type: 'AWS::Serverless::Function'
    Properties:
      Environment:
        Variables:
          DEMO_CHAT_ARN: 'arn:aws:ivschat:us-east-1:[redacted]:room/[redacted]'
          DEMO_CHANNEL_ARN: 'arn:aws:ivs:us-east-1:[redacted]:channel/[redacted]'
      Handler: index.stopStream
      Layers:
        - !Ref IvsChatLambdaRefLayer
      CodeUri: lambda/
      Events:
        Api1:
          Type: Api
          Properties:
            Path: /stop-stream
            Method: POST
Outputs:
  ApiURL:
    Description: "API endpoint URL for Prod environment"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/" There's a lot going on in that file, so let's break it down a bit. First, we create a layer to enable the inclusion of the AWS SDK for JavaScript (v3) in our function. IvsChatLambdaRefLayer:
  Type: AWS::Serverless::LayerVersion
  Properties:
    LayerName: sam-app-dependencies
    Description: Dependencies for sam app
    ContentUri: dependencies/
    CompatibleRuntimes:
      - nodejs18.x
    LicenseInfo: "MIT"
    RetentionPolicy: Retain In the directory, there is a file that includes the modules that our function needs. dependencies/nodejs package.json {
  "dependencies": {
    "@aws-sdk/client-ivs": "^3.289.0",
    "@aws-sdk/client-ivschat": "^3.289.0",
    "@aws-sdk/client-rekognition": "^3.289.0"
  }
} The next section, identified by the keys and gives our serverless application the ability to access the necessary APIs ( , , , , and ) and expose the stop stream method that we'll create below via Amazon API Gateway. IVSAccessPolicy APIAccessPolicy s3:GetObject s3:GetObjectAcl ivschat:SendEvent ivs:StopStream rekognition:DetectModerationLabels Next, we create the Amazon EventBridge rule. The property under should match the name of the Amazon S3 bucket that you configured in your recording configuration. Recording to Amazon S3 creates various files, including playlists and HLS media, so we can filter this rule to only fire for our thumbnails by setting the under to be . name bucket key object suffix: jpg EventRule:
  Type: AWS::Events::Rule
  Properties:
    Description: EventRule
    State: ENABLED
    EventPattern: 
      source:
        - aws.s3
      detail-type:
        - "Object Created"
      detail:
        bucket:
          name:
            - ivs-demo-channel-stream-archive
        object:
          key:
            - suffix: .jpg
    Targets:
      - Arn: !GetAtt ModerateImage.Arn
        Id: MyLambdaFunctionTarget Next, we give the rule the necessary permissions to invoke the AWS Lambda function. PermissionForEventsToInvokeLambda:
  Type: AWS::Lambda::Permission
  Properties:
    FunctionName: !Ref ModerateImage
    Action: lambda:InvokeFunction
    Principal: events.amazonaws.com
    SourceArn: !GetAtt EventRule.Arn Now we can define our function that will be invoked by the Amazon EventBridge rule. ModerateImage:
  Type: 'AWS::Serverless::Function'
  Properties:
    Environment:
      Variables:
        DEMO_CHAT_ARN: 'arn:aws:ivschat:us-east-1:[redacted]:room/[redacted]'
        DEMO_CHANNEL_ARN: 'arn:aws:ivs:us-east-1:[redacted]:channel/[redacted]'
    Handler: index.moderateImage
    Layers:
      - !Ref IvsChatLambdaRefLayer
    CodeUri: lambda/ I'm declaring the and as environment variables, but your application would likely derive the ARN values from the event passed into the function since you would likely use this functionality with more than just a single Amazon IVS channel. Note: DEMO_CHAT_ARN DEMO_CHANNEL_ARN Finally, we can define the function that will be used to stop a stream, if necessary. StopStream:
  Type: 'AWS::Serverless::Function'
  Properties:
    Environment:
      Variables:
        DEMO_CHAT_ARN: 'arn:aws:ivschat:us-east-1:[redacted]:room/[redacted]'
        DEMO_CHANNEL_ARN: 'arn:aws:ivs:us-east-1:[redacted]:channel/[redacted]'
    Handler: index.stopStream
    Layers:
      - !Ref IvsChatLambdaRefLayer
    CodeUri: lambda/
    Events:
      Api1:
        Type: Api
        Properties:
          Path: /stop-stream
          Method: POST Creating the AWS Lambda Functions Now that we've described our infrastructure with AWS SAM, let's create the functions that we described. In , we the SDK classes, retrieve the values from the environment variables that we passed in, and create instances of the clients needed for our functions. index.mjs import Arn import { IvsClient, StopStreamCommand } from "@aws-sdk/client-ivs";
import { IvschatClient, SendEventCommand } from "@aws-sdk/client-ivschat";
import { RekognitionClient, DetectModerationLabelsCommand } from "@aws-sdk/client-rekognition";

const chatArn = process.env.DEMO_CHAT_ARN;
const channelArn = process.env.DEMO_CHANNEL_ARN;

const ivsClient = new IvsClient();
const ivsChatClient = new IvschatClient();
const rekognitionClient = new RekognitionClient(); The function will receive the Amazon EventBridge event, extract the and from the event, and send a via the to detect any inappropriate or offensive content in images based on the categories . moderateImage bucket key DetectModerationLabelsCommand rekognitionClient listed here export const moderateImage = async (event) => {
  console.log('moderateImage:', JSON.stringify(event, null, 2));
  const bucket = event.detail.bucket.name;
  const key = event.detail.object.key;

  const detectLabelsCommandInput = {
    Image: {
      S3Object: {
        Bucket: bucket,
        Name: key,
      }
    },
  };
  const detectLabelsRequest = new DetectModerationLabelsCommand(detectLabelsCommandInput);
  const detectLabelsResponse = await rekognitionClient.send(detectLabelsRequest);

  if (detectLabelsResponse.ModerationLabels) {
    sendEvent('STREAM_MODERATION', detectLabelsResponse.ModerationLabels);
  }
}; If necessary, the function calls to publish a custom event to any front end connected clients to a given Amazon IVS chat room. moderateImage sendEvent const sendEvent = async (eventName, eventDetails) => {
  const sendEventInput = {
    roomIdentifier: chatArn,
    attributes: {
      streamModerationEvent: JSON.stringify(eventDetails),
    },
    eventName,
  };
  const sendEventRequest = new SendEventCommand(sendEventInput);
  await ivsChatClient.send(sendEventRequest);
}; Your front end can decide how to handle this event and the logic for publishing this event will depend on your business needs. Maybe you'd rather trigger a custom alarm in CloudWatch, send an email, or publish a notification via Amazon SNS? Every application's needs differ, but the moderation data is available at this point to do with it what you need. The method uses the to send a . Again, the implementation of this is up to you. You could even potentially fully automate this command if the Amazon Rekognition result matches a certain category or exceeds a confidence level. stopStream ivsClient StopStreamCommand export const stopStream = async (event) => {
  console.log('stopStream:', JSON.stringify(event, null, 2));
  try {
    const stopStreamRequest = new StopStreamCommand({ channelArn });
    const stopStreamResponse = await ivsClient.send(stopStreamRequest);
    responseObject.body = JSON.stringify(stopStreamResponse);
  }
  catch (err) {
    responseObject.statusCode = err?.name === 'ChannelNotBroadcasting' ? 404 : 500;
    responseObject.body = JSON.stringify(err);
  }
  return responseObject;
}; Demo In my demo, I decided to listen for the custom events and display the results in a moderator view that shows the detected item and the confidence level. I also present the moderator with a 'Stop Stream' button that invokes the method via the exposed Amazon API Gateway. stopStream Summary In this post we learned how to use Amazon Rekognition to assist human moderators moderate the content in the applications that they build using Amazon IVS. If you'd like to learn more about how Amazon IVS can help create safer UGC communities, check out the following blog posts: Moderating Amazon IVS Chat Messages with an AWS Lambda Function Manually Moderating Amazon IVS Chat Messages

Start building interactive live video experiences. 

Read My Stories

Too Long; Didn't Read

Everything you need for scraping at scale

Making Online Communities Safer with AI/ML Content Moderation

Too Long; Didn't Read

@amazonivs

RELATED STORIES