In my last few posts, we've been taking an extended look at moderating Amazon Interactive Video Service (Amazon IVS) chat rooms. Two weeks ago, we learned how to perform , and last week we saw how to . In this post, I'd like to switch gears and talk about an interesting use case for Amazon IVS chat rooms - live, interactive whiteboards. automated chat moderation with AWS Lambda functions manually moderate chat rooms Live streaming is mostly known for entertainment-related streams like gaming or sports, but it is a perfect tool for delivering educational content too. I've blogged before about on a stream, but whiteboarding is a unique way to provide a broadcaster and the viewers with a live, interactive way to visualize and even collaborate on a given topic. The demo in this post is very basic - just a "pen" tool for freehand drawing - but it has the potential to be expanded for shapes and images, making it the perfect starting point for creating your own whiteboard experience to enhance your Amazon IVS live streams. screen sharing and overlaying canvas elements Try it Out! Before we get into how to build the whiteboard, check out how it works below. You'll need to generate a chat token for an existing Amazon IVS chat room for two unique users and enter the tokens and respective values in the two CodePen embeds. In production, you'd use one of the AWS SDKs to generate your tokens, but for this demo, you can generate them with the AWS CLI to see how it works (see for more information). user-id this post $ aws ivschat create-chat-token \
  —room-identifier [CHAT ARN] \
  —user-id "1" \
  —capabilities "SEND_MESSAGE" \
  —query "token" \
  —output text Using the CLI command above, generate and enter the token for user "1" and enter that in the first CodePen. Then, repeat the process for user "2" in the second CodePen below. If your chat room was not created in , update the value to match your chat room's region. Once both users are connected, you can draw on one canvas and observe the drawing on the other canvas. user-id us-east-1 Endpoint If you don't have any Amazon IVS chat rooms to test things out with - WHY NOT?? Just kidding, of course. Here's a gif showing it in action where you can see my amazing art skills on display: Setting Things Up For this demo, I'm collecting the user id, pen color, chat token, and chat endpoint via a . In production, you'll have a dedicated user id, and your chat token and endpoint will come from a call to a server (or serverless function). It's important to track the user id to prevent the drawing from being replicated on the of the user who is currently drawing. Here is the HTML markup for the collection form and the drawing canvas. I've removed the Bootstrap classes used in the CodePen to make the code easier to read here. <form> <canvas> <div id="settings">
  <div>Settings</div>
  <div>
    <div>
      <label for="chat-userid">Chat UserId</label>
      <div>
        <input type="text" id="chat-userid" required />
      </div>
    </div>
    <div>
      <label for="pen-color">Pen Color</label>
      <div>
        <input type="color" id="pen-color" required />
      </div>
    </div>
    <div>
      <label for="chat-token">Chat Token</label>
      <div>
        <input type="text" id="chat-token" required />
      </div>
    </div>
    <div>
      <label for="chat-endpoint">Endpoint</label>
      <div>
        <input type="text" id="chat-endpoint" required />
      </div>
    </div>
    <div>
      <div>
        <button type="button" id="submit-settings">Submit</button>
      </div>
    </div>
  </div>
</div>
<div id="whiteboard-container">
  <canvas id="whiteboard">
    Sorry, your browser does not support HTML5 canvas technology.
  </canvas>
</div> Next, I've added a listener to set a random pen color and listen for the button click. DOMContentLoaded Submit document.addEventListener('DOMContentLoaded', () => {
  document.getElementById('pen-color').value = `#${Math.floor(Math.random()*16777215).toString(16)}`;
  document.getElementById('submit-settings').addEventListener('click', () => {
    init();
  })
}); In the function, I set some global values for the information that we'll need later on to initialize the chat connection. In your application, you'll most likely be working with a modern JS framework and, therefore would avoid using global variables like this. init() const init = () => {
  window.chatEndpoint = document.getElementById('chat-endpoint').value;
  window.userId = document.getElementById('chat-userid').value;
  window.chatToken = document.getElementById('chat-token').value;
  window.penColor = document.getElementById('pen-color').value;
  if (!window.chatEndpoint || !window.chatToken || !window.userId) {
    alert('Chat Endpoint, Token and UserId are required!');
    return;
  }
  document.getElementById('settings').classList.add('d-none');

  // init chat connection
} Initializing a Chat Connection Now that we have the , , and , we can initialize our WebSocket connection to the Amazon IVS chat room by adding it to the function: userId chatToken chatEndpoint init() window.connection = new WebSocket(window.chatEndpoint, window.chatToken);
window.connection.addEventListener('message', (e) => {
  // todo: handle message
}); We'll populate the message handler in just a bit. For now, let's look at how to draw on the canvas. Drawing on the Canvas Before we can draw on the canvas, we'll add a bit of configuration to the canvas element to the function. init() const whiteboardContainer = document.getElementById('whiteboard-container');
const canvasEl = document.getElementById('whiteboard');
canvasEl.width = whiteboardContainer.offsetWidth;
canvasEl.height = whiteboardContainer.offsetHeight;
const ctx = canvasEl.getContext('2d');
ctx.lineWidth = 5;
ctx.fillStyle = '#fff';
ctx.fillRect(0, 0, canvasEl.width, canvasEl.height); The snippet above sets the width and height of the and sets the default background color. <canvas> Handling Mouse Events We'll add three listeners to our element: , , and . The handlers for these events will need to do two things: handle drawing on the current user's canvas, and publish an event via the WebSocket connection to other users so that the drawing can be replicated on all connected clients. <canvas> mousedown mousemove mouseup Consider using pointer events instead of mouse events ( , , and ) to make your whiteboard respond to both touch and mouse events. pointerdown pointermove pointerup canvasEl.addEventListener('mousedown', (e) => {
  window.isDrawing = true;
  const evt = { x: e.offsetX, y: e.offsetY, type: 'mousedown' };
  onMouseDown(evt);
  // queue event for publishing
});
canvasEl.addEventListener('mousemove', (e) => {
  if (window.isDrawing) {
    const evt = { x: e.offsetX, y: e.offsetY, type: 'mousemove' };
    // queue event for publishing
    onMouseMove(evt);
  }
});
canvasEl.addEventListener('mouseup', (e) => {
  window.isDrawing = false;
  onMouseUp({});
  // queue event for publishing
}); Now let's look at each of the functions that perform the actual canvas drawing. First, which gets the context for the canvas, begins a path, and moves to the proper and coordinates. onMouseDown() 2d x y const onMouseDown = (e) => {
  const canvasEl = document.getElementById('whiteboard');
  const ctx = canvasEl.getContext('2d');
  ctx.beginPath();
  const x = e.x;
  const y = e.y;
  ctx.moveTo(x, y);
}; Next, , which draws a line to the current and . Because we set a global variable to before calling , this method will be called continuously until the event sets the flag back to false. This means that will draw a line until we release the mouse button. onMouseMove() x y isDrawing true onMouseDown() mouseup isDrawing onMouseMove() const onMouseMove = (e, color) => {
  const canvasEl = document.getElementById('whiteboard');
  const ctx = canvasEl.getContext('2d');
  const x = e.x;
  const y = e.y;
  ctx.lineTo(x, y);
  ctx.strokeStyle = color || window.penColor;
  ctx.stroke();
}; Finally, closes the path that we started in . onMouseUp() onMouseUp() const onMouseUp = (e) => {
  const canvasEl = document.getElementById('whiteboard');
  const ctx = canvasEl.getContext('2d');
  ctx.closePath();
} At this point, each connected user can draw on their local , but none of the other connected users will be able to see what they have drawn. For this, we need to publish the events via the WebSocket connection. <canvas> Publishing Drawing Events There is no guarantee for how often a mouse event will get fired, but most browsers will fire . If we look at the , we can see that we're limited to 10 transactions per second. It would be pretty easy to hit quota limits if we tried to publish every single event for all connected chat users. To workaround this, we can do do two things: send events in a batch, and only publish a sample of the events to other connected clients. mousemove quite often service quotas for Amazon IVS chat mousemove mousemove Queuing Mouse Events Let's start by looking at how we can batch the events. First, we'll set up a few global variables to manage a queue of events. window.queue = [];
window.maxQueueSize = 20; Next, we'll create a method to build up a batch of events. When the batch size is greater than our configured (or when a event is received), we'll send the current batch. handleQueue() window.maxQueueSize mouseup const handleQueue = (event) => {
  if (window.queue.length <= window.maxQueueSize) {
    window.queue.push(event);
  }
  if (window.queue.length === window.maxQueueSize || event.type == 'mouseup') {
    sendEvents();
  }
}; Publishing Mouse Events The method builds a payload containing a JSON serialized version of the event queue and sends it just like we would normally post a chat message to a chat room with . Notice the object contains a of which we can use to differentiate whiteboard messages from normal chat messages in the handler (we'll look at that handler below). sendEvents() SEND_MESSAGE Attribute type whiteboard message const sendEvents = () => {
  const payload = {
    'Action': 'SEND_MESSAGE',
    'Content': '[whiteboard event]',
    'Attributes': {
      'type': 'whiteboard',
      'color': window.penColor,
      'events': JSON.stringify(window.queue),
    }
  }
  try {
    window.connection.send(JSON.stringify(payload));
    window.queue = [];
  }
  catch (e) {
    console.error(e);
  }
} Now we can modify the event handlers to call . mouse handleQueue() canvasEl.addEventListener('mousedown', (e) => {
  window.isDrawing = true;
  const evt = { x: e.offsetX, y: e.offsetY, type: 'mousedown' };
  onMouseDown(evt);
  handleQueue(evt);
});
canvasEl.addEventListener('mousemove', (e) => {
  if (window.isDrawing) {
    const evt = { x: e.offsetX, y: e.offsetY, type: 'mousemove' };  
  handleQueue(evt);
  onMouseMove(evt);
  }
});
canvasEl.addEventListener('mouseup', (e) => {
  window.isDrawing = false;
  onMouseUp({});
  handleQueue(evt);
}); Sampling Mouse Move Events If we were to run this application at this point, we'd probably quickly hit our service quota even though we're sending the events in batches. As mentioned above, we can sample the event to prevent this. We'll add a method and limit our call to in the handler to being called every 50-100ms. In my tests, I found this to be an acceptable range that both prevents hitting the service quota and provides a reasonably good recreation of the event sequence on the other client's . mousemove throttle() handleQueue() mousemove <canvas> window.throttlePause;

const throttle = (callback, time) => {
  if (window.throttlePause) return;
  window.throttlePause = true;
  setTimeout(() => {
    callback();
    window.throttlePause = false;
  }, time);
}; The only thing left to do to implement this sampling is to modify the handler to only queue the event every 50ms. mousemove canvasEl.addEventListener('mousemove', (e) => {
  if (window.isDrawing) {
    const evt = { x: e.offsetX, y: e.offsetY, type: 'mousemove' };
    throttle(() => {
      handleQueue(evt);
    }, 50);
    onMouseMove(evt);
  }
}); Handling Incoming Mouse Events Now that we have implemented the local drawing, and the logic to queue and publish events, we just need to add our handler for the WebSocket connection to handle the published events and recreate the drawings from other connected clients. Back inside of our function, right after we create the connection, add the following: message init() WebSocket window.connection.addEventListener('message', (e) => {
  const data = JSON.parse(e.data);
  const msgType = data.Attributes.type;

  if(msgType == 'whiteboard') {
    const events = JSON.parse(data.Attributes.events);
    const color = data.Attributes.color;
    const eventUserId = data.Sender.UserId;
    events.forEach(e => {
      const type = e.type;
      if(eventUserId != window.userId) {
        switch(type){
          case 'mousedown':
            onMouseDown({x: e.x, y: e.y});
            break;
          case 'mousemove':
            onMouseMove({x: e.x, y: e.y}, color);
            break;
          case 'mouseup':
            onMouseUp({});
            break;
        };  
      }      
    });  
  }

  // otherwise, handle as an incoming chat...
}); In the handler above, we first check the object for the key that we used when publishing the messages to differentiate these events from "normal" chat messages. If that is , we parse the JSON string which contains an array of events and loop over the array. Within the loop, we check to make sure the - the person publishing the event - is not the current . If not, we recreate the drawing operation on the local by calling the appropriate function. Attributes type type whiteboard events eventUserId userId <canvas> Potential Improvements In a production application, a whiteboard might need additional features like shapes, images, and text. To add such features, your application can utilize the same architecture that we have seen in the demo above. Improving Performance Since this demo uses JSON to publish events, the payload that we are publishing is larger than it has to be. To improve on the number of events that could be published in a single message, your application can publish events in a delimited text format and use identifiers for the events. For example, instead of a JSON array like this that publishes a payload of 94 bytes: [
  {
    "x": 100,
    "y": 100,
    "type": "mousedown"
  },
  {
    "x": 100,
    "y": 100,
    "type": "mousemove"
  },
  {
    "type": "mouseup"
  }
] You could instead format the data like this: 0,100,100|1,100,100,#ff9911|2 Enter fullscreen mode Exit fullscreen mode Here we have delimited each event with a pipe ( ) and the data within each event by a comma. The first character of the event ( , , ) represents the event type ( for a , for , and for ). The second and third characters are the and positions respectively. The fourth character, only necessary for events, is the pen color. This format results in a payload of 31 bytes - a 67% decrease in size. | 0 1 2 0 mousedown 1 mousemove 2 mouseup x y mousemove You could then handle this new payload format with this: payload = '0,100,100|1,100,100,#ff9911|2';
events = payload.split('|');
events.forEach((e) => {
  let type = e[0];
  let x = e[1];
  let y = e[2];
  let color = e[3];
  switch(type){
    case 0:
      onMouseDown({x, y});
      break;
    case 1:
      onMouseMove({x,y}, color);
      break;
    case 2:
      onMouseUp({});
      break;
  }
}) Summary In this post, we created a basic proof of concept that uses Amazon IVS chat messages to give chat developers the ability to add whiteboarding to their interactive live-streaming applications. We also discussed some potential ways to improve the application in production. If you have any questions, leave a comment or reach out to me on . Twitter Also Published Here

This story contains new, firsthand information uncovered by the writer.

Start building interactive live video experiences. 

Read My Stories

Too Long; Didn't Read

Developers: Build Less. Deliver More. [Learn how]	

Using Amazon IVS Chat to Create an Interactive Whiteboard

Too Long; Didn't Read

Amazon Interactive Video Service (IVS)

STORY’S CREDIBILITY

Original Reporting

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Using Amazon IVS Chat to Create an Interactive Whiteboard

Too Long; Didn't Read

Amazon Interactive Video Service (IVS)

STORY’S CREDIBILITY

Original Reporting

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES