How to Get Started with Amazon IVS and Amazon Personalize in Retail

The ever-increasing popularity of live streaming is opening up new opportunities for online customer interactions. With live streaming providing engaging content, it can facilitate the flow of customers into online shopping, joining the shopping experience with other forms of online engagement. Personalized recommendations also allow online stores to pick the right products at the right time to maximize engagement and conversions.

This blog will walk through a detailed example of how to configure and deploy live streaming and personalization into an ecommerce site. To start this, we’ll be using the AWS retail ecommerce demo environment, our open source platform, to highlight how highly scalable cloud services can enable you to transform customer experiences.

Amazon Interactive Video Service (Amazon IVS) is a managed live streaming solution that is quick and easy to set up, and ideal for creating interactive video experiences. By streaming video to Amazon IVS, you can make low-latency live video available to any viewer in the world. The service is easily customizable using the Amazon IVS Player SDK, and a timed metadata API allows metadata to be distributed to stream consumers using the same video feed. Additionally, Amazon Personalize delivers tailored product recommendations to customers.

Amazon Personalize is a fully managed recommender system that can be used in conjunction with other AWS services, like AWS Amplify and Amazon Pinpoint, to deliver a wide variety of personalized experiences to customers at scale. You can provide Amazon Personalize with historical records of customer-product interactions, in addition to product and customer metadata, and it uses machine learning (ML) to look for patterns in those interactions to recommend products to users that are likely to elicit positive responses in the future. It is also capable of recommending related products and doing personalized re-ranking of search results. Amazon Personalize can respond in real time to trends in the data and to individual behavioral changes by exploiting a runtime interface for ingesting new interactions.

Starting Point: The AWS Retail Demo Store

The AWS Retail Demo Store is an open-source sample retail web application and workshop platform intended as an educational tool for demonstrating how AWS infrastructure and services can be used to build compelling customer experiences for ecommerce, retail, and digital marketing use cases

The core of the AWS Retail Demo Store is a microservices-based architecture deployed as a collection of RESTful web services in Amazon Elastic Container Service (Amazon ECS). Several AWS Managed Services are leveraged to provide build, deployment, authentication, messaging, search, and personalization capabilities. The web user interface (UI) is a single-page application built using responsive web design frameworks and techniques, producing a native app-like experience tailored to the user’s device.

The architecture uses AWS CloudFormation’s custom resources to deploy Amazon IVS channels, update the UI, and add the new “videos” service to Amazon ECS to stream the videos and expose details of the streams to the UI via a Flask API. Amazon IVS resource types in AWS CloudFormation can also be used.

The following diagram highlights the addition of IVS to the AWS Retail Demo Store.

The new UI view displays Amazon IVS video streams of vlog-style videos, with a vlogger presenter interacting with products. Product details are embedded as metadata in the Amazon IVS stream displays details about the product being demonstrated at a given point in time, in addition to related products suggested using Amazon Personalize.

The following sections walk through the main steps to add this functionality and integrate it into the existing ecommerce platform.

Amazon IVS and AWS CloudFormation

To broadcast video with Amazon IVS, the first step is to create a channel. Each channel has capacity for a single video stream at a given point, and outputs a set of streams wrapped in a .m3u8 file, which can be consumed using the Amazon IVS Player. Amazon IVS resources are created through the console, CLI, or with the standard AWS SDKs. To integrate with the infrastructure as code framework of the existing ecommerce backend, Python boto3 library is used to create an AWS CloudFormation custom resource. Custom resources allow AWS Lambda functions to be used in place of typical pre-defined AWS CloudFormation resources, enabling bespoke implementations of AWS resources that may not be possible in default templates.

Any number of videos can be streamed for display in the ecommerce site with a custom resource to create Amazon IVS channels based on the presence of video files in a specified Amazon S3 location. A parameter mapping each video file to a stream is automatically saved in AWS Systems Manager (SSM) Parameter Store, meaning that the application streaming the video to the channel can consistently stream the same video to the same channel.

ivs_client = boto3.client('ivs')
ssm_client = boto3.client('ssm')

channel_name = 'retail-demo-store-' + \

created_channel_arn = ivs_client.create_channel(name=channel_name,latencyMode=’NORMAL’)[‘channel’][‘arn’]
logger.info(f"IVS channel created with ARN {created_channel_arn}")


if is_ssm_parameter_set(SSM_VIDEO_CHANNEL_MAP_PARAM): 
video_channel_param_value = ssm_client.get_parameter (Name=SSM_VIDEO_CHANNEL_MAP_PARAM)[‘Parameter’][‘Value’]
	video_channel_map = json.loads(video_channel_param_value)

	video_channel_map[video_file_key] = created_channel_arn

	ssm_client.put_parameter(
		Name=SSM_VIDEO_CHANNEL_MAP_PARAM,
		Value=json.dumps(video_channel_map),
		Type=’String’,
		Overwrite=True
	)

(from src/aws-lambda/ivs-create-channels/ivs-create-channels.py)

Metadata Encoding

One of the key features of Amazon IVS is the ability to embed metadata directly within a stream, allowing for synchronized content to be distributed within a video stream. This metadata is sent to the Amazon IVS API for a given channel using the PutMetadata operation, and the metadata is distributed to stream consumers in real time.

ivs_client.put_metadata(
  channelArn=channel_arn,
  metadata=line
)

(from src/videos/src/videos-service/app.py)

To stream metadata alongside pre-recorded video for demonstration purposes, metadata is encoded into the same file as the video being streamed to allow for the best possible synchronization between the video content and the metadata being sent. For this reason, the Matroska Multimedia Container (.mkv) format is used, with metadata included within a video container as a subtitle file.

The SRT subtitle format is chosen for its simple format, readability, and ease of editing for developers; files can be edited as plaintext or when using open-source software. The following snippet shows a sample of how the SRT format is used to capture product metadata.

1
00:00:04,000 --> 00:00:08,000
{"productId": 1}

2
00:00:08,000 --> 00:00:12,000
{"productId": 2}

3
00:00:12,000 --> 00:00:16,000
{"productId": 3}

4
00:00:16,000 --> 00:00:20,000
{"productId": 4}

(from videos/sample.srt)

Here, the product ID is saved in a simple JSON object, with the initial timestamp for each entry being the point at which the product first appears in a video stream.

This subtitles-as-metadata implementation also means that the files can be handled using standard video processing and streaming software. The following command shows how FFmpeg is used to combine a metadata file and a video into a single MKV container. While MP4 is the format mentioned here, this same command is compatible with a number of video formats.

ffmpeg -i video.mp4 -i metadata.srt output.mkv

The file output by this command contains everything required to run a metadata-enriched Amazon IVS stream using a pre-recorded video. While FFmpeg is used for recorded videos here, Open Broadcast Software (OBS) can also be used for recording and live streaming events to Amazon IVS.

Video Encoding for Efficiency

Encoding video can be CPU intensive, so in a scenario where pre-recorded video is being repeatedly streamed on a loop, it is beneficial to pre-encode the video as a one-off action. This reduces the CPU requirements of the streaming infrastructure since the video can be directly copied to a stream without the need for additional processing. It also allows for any additional manipulation that may be required to optimize the video for a specific presentational use case. A number of encoding parameters are added to the preceding FFmpeg command to achieve this, as shown:

ffmpeg -i video.mp4 -i metadata.srt -vf scale=640x360 -c:v libx264  \
-pix_fmt yuv420p -profile:v main -tune fastdecode -x264opts “nal-hrd=cbr:no-scenecut” -minrate 3000 \
-maxrate 3000  -g 60 -c:a aac -b:a 160k -ac 2 -ar 44100 output.mkv

By pre-encoding the video like this, it is possible to stream the videos to Amazon IVS without any real-time encoding, meaning that a number of video streams can be streamed using a container with minimal CPU, thus keeping compute costs to a minimum.

Manually Streaming Video to Amazon IVS

To test the compatibility of the video content with Amazon IVS, videos are manually streamed to the service using the following FFmpeg command:

ffmpeg -loglevel panic -hide_banner -re -stream_loop -1 -i \"{video_filepath}\" -r 30 -c:v copy -f flv rtmps://{ingest_endpoint}:443/app/{stream_key}

The Amazon IVS console is used to view the stream and verify that the video is appearing in the stream as anticipated.

Amazon ECS

Video files are streamed to Amazon IVS from a new service, which is added to the existing cluster in the AWS Retail Demo Store. To allow for full integration with a UI, the service provides endpoints returning the stream ingest URLs and the full list of products contained within each stream. The code is wrapped as a Flask API written in Python 3.8.

Upon service startup, the video to Amazon IVS channel mapping is retrieved from SSM and in separate threads each video is streamed to its corresponding Amazon IVS channel on a repeating loop. As this happens, the SRT data in the file is read and processed, with the metadata contained being sent to Amazon IVS using a PutMetadata operation at the appropriate time. The following code snippet gives a sample of how this is done in Python.

ffmpeg_stream_cmd = """ffmpeg -loglevel panic -hide_banner -re -stream_loop -1 -i \"{video_filepath}\" \ -r 30 -c:v copy -f flv rtmps://{ingest_endpoint}:443/app/{stream_key} -map 0:s -f {subtitle_format} -"""

process = subprocess.Popen(
  	ffmpeg_stream_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True, shell=True)

lines = iter(process.stdout)
while True:
   int(next(lines).strip())
   time_range = next(lines).strip()
   if not '-->' in time_range:
       raise ValueError(f'Expected a time range instead of {time_range}')
   send_text = ''
   while True:  # SRT can come in multiple lines
       text = next(lines).strip()
       if len(text) == 0: break  # SRT ends with a newline
       if len(send_text) > 0: send_text += '\n'
       send_text += text

   ivs_client.put_metadata(
       channelArn=ivs_channel_arn,
       metadata=send_text
   )

With the pre-encoding of the videos, the deployed demo is able to stream five videos simultaneously on a container with 0.25vCPU and 0.5GB of memory.

 A “stream_details” endpoint is also implemented, which returns the ingest URLs of each stream, in addition to a list of product IDs for the products shown in each stream, allowing a UI to prefetch any data relating to products contained within the stream, if desired.

{
"streams": [
  {
    "playback_url": "https://2f185ac93237.us-west-2.playback.live-video.net/api/video/v1/us-west-2.266916629424.channel.Kyr5LYOURnWt.m3u8",
    "products": [
      101,
      102,
      103,
      104,
      105,
      106,
      107
    ],
    "thumb_url": "/static/default_stream1_thumb.png"
  },
  {
    "playback_url": "https://2f185ac93237.us-west-2.playback.live-video.net/api/video/v1/us-west-2.266916629424.channel.kFHQJjo6tk1V.m3u8",
    "products": [
      301,
      302,
      303,
      304,
      201,
      1000567890
    ],
    "thumb_url": "/static/default_stream3_thumb.png"
  }
]
}

Ecommerce UI and Metadata-driven Personalization

Displaying the m3u8 stream containing metadata involves an out-of-the box implementation of Amazon’s IVS player, which can be added to a webpage similarly to any other video. A listener for the player’s TEXT_METADATA_CUE event then allows handing of metadata to be built using a callback function.

player.addEventListener(PlayerEventType.TEXT_METADATA_CUE, (cue) => {
console.log('Timed metadata: ', cue.text);
this.handleMetadata(cue.text);
});

For the Retail Demo Store, the IVS streams are presented in a format combining vlog-style videos with UI components highlighting the products being presented in the stream at a given point. As the currently displayed product changes, a selection of related products and a personalized discount (determined using Amazon Personalize) are also shown.

Amazon Personalize for Personalized Discounts

We are leveraging Amazon Personalize to give us the discount that would maximize user uptake. Amazon Personalize gives us personalized re-ranking of available products looking at the user’s and collective historical uptake of discounted products. Then, we would offer the top-ranked products with a personalized discount.

Using Python to Train an Amazon Personalize Solution

For a full tutorial on how to set up Amazon Personalize with your dataset, see the official documentation.

Here, we use the “aws-personalized-ranking” recipe to train a reranking solution, and we tell Amazon Personalize to train against the “DiscountedProductViewed” event type (which is unique to our dataset—your dataset would likely have a different set of event types), to ensure that the solution predicts discounted product view events. An alternative approach would be to train against conversion events (such as “DiscountedProductPurchased”).

personalize = boto3.client("personalize")
create_solution_response = personalize.create_solution(
   name="personalized_discounts",
   datasetGroupArn="retail_demo_store",
   recipeArn="arn:aws:personalize:::recipe/aws-personalized-ranking",
   eventType="DiscountedProductViewed",
   performHPO=True
)

(from src/aws-lambda/personalize-pre-create-campaigns/personalize-pre-create-campaigns.py)

Having configured the solution, we can train it and deploy a “campaign” that gives us re-ranking at runtime. We wrap the campaign in an endpoint that contains logic for managing A/B testing, multiplexing recommender systems, and so forth. That endpoint uses the re-ranking to determine which products to offer a discount on. Roughly like this:

for item in items:
            item_id = item['itemId']
            if item_id in discount_item_map:
                discounted_item = discount_item_map[item_id]
                discounted_item['discounted'] = True
                
            else:
                
                item['discounted'] = False

Using Recommendation Filters in Amazon Personalize, products that were previously purchased can be automatically removed. Additional custom logic can also be written to filter out recommendations pertaining to specific business KPIs.

Conclusion

The combination of Amazon IVS and Amazon Personalize gives endless opportunities for development in the retail space. By using these hosted, scalable services, you are able to focus on your differentiating business applications. All code referenced in this demo is integrated into the Retail Demo Store GitHub repository, which is available hereand can be deployed to your own AWS account for exploration and experimentation in a matter of minutes.