Adaptrack

adaptrack

I've written a few posts lately about multi-host live streams with Amazon Interactive Video Service (Amazon IVS). It's an exciting feature that opens up worlds of possibilities that simply weren't available until recently. We first looked at how to . create a multi-host live chat application Next, we saw how to broadcast that live chat session to an Amazon IVS channel. When we looked at adding chat participants to the broadcast client in that last post, you probably noticed that I cheated a bit and hardcoded the values that tell the broadcast client the size and position of the participant's video on the client. VideoComposition Well - cheated is a strong word - let's say that I intentionally simplified the code to focus on the process of broadcasting a live chat session. Essentially what we're looking for here is modifying the size and position of the participant's video in the broadcast so that when there is one video, the layout will look something like this: But when there are two videos, the layout will change to something like this: And when there are five: You get the idea - a dynamic layout that changes based on the amount of participants. In this post, we'll take a look at one approach you could utilize to make creating a dynamic layout a bit easier. We'll build off of the solution in the last post, so if you haven't read that post yet, it's probably a good idea to do that now. In the last post, we listened for an event called . In the event handler for that event, we added our participants to the DOM and rendered the audio and video to the instance. STAGE_PARTICIPANT_STREAMS_ADDED IVSBroadcastClient In order to render a dynamic layout, we'll need to track how many participants are currently in the session, so we'll add an array called as a global variable. Let's modify the event handler to push the current participant id to that array. participantIds stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_ADDED, (participant, streams) => {

  //add participant id to array
  participantIds.push(participant.id);

  renderParticipant(participant, streams);
  renderVideosToClient(participant, streams.find(s => s.streamType === StreamType.VIDEO));
  renderAudioToClient(participant, streams.find(s => s.streamType === StreamType.AUDIO));
  updateVideoCompositions();
}); In the last post, I mentioned that the method was not shown because the implementation would vary. We'll talk about one possible implementation in just a bit. updateVideoCompositions() For now, let's take a look at how we can get a dynamic layout config instead of hardcoding it as we did in the last post. One way to obtain a dynamic size and position is to loop over the participant array and calculate them based on the number of participants, the size of the , and the desired amount of rows, columns, and padding. But, ? <canvas> why That sounds like a lot of difficult code and unnecessary work when you realize that these values never change. If you have one participant, the video will be a fixed size and centered in the . <canvas> It doesn't matter how many participants get added - the layout for each video will always be the same for a given number of participants. So why waste time and CPU cycles when we could pre-calculate these values and store them in an array of arrays? For my demo, I spent some time determining the best values with an intensive 30 minutes with a pen, paper, and calculator to determine the composition values for each possible layout. Please note: I was a maths or art major as evidenced by the following sketch. not For this demo, I capped my live stream at only showing videos for the first 6 participants. Your use case may dictate something different, but having more than 6 participant videos in one live stream becomes a bit too busy in my experience. Here is the result of my calculations: const layouts = [
  [{ height: 720, width: 1280, x: 320, y: 180 }],
  [{ height: 450, width: 800, x: 80, y: 315 }, { height: 450, width: 800, x: 1040, y: 315 }],
  [{ height: 450, width: 800, x: 80, y: 45 }, { height: 450, width: 800, x: 1040, y: 45 }, { height: 450, width: 800, x: 560, y: 585 }],
  [{ height: 450, width: 800, x: 80, y: 45 }, { height: 450, width: 800, x: 1040, y: 45 }, { height: 450, width: 800, x: 80, y: 585 }, { height: 450, width: 800, x: 1040, y: 585 }],
  [{ height: 337, width: 600, x: 20, y: 100 }, { height: 337, width: 600, x: 650, y: 100 }, { height: 337, width: 600, x: 1280, y: 100 }, { height: 337, width: 600, x: 340, y: 640 }, { height: 337, width: 600, x: 980, y: 640 }],
  [{ height: 337, width: 600, x: 20, y: 100 }, { height: 337, width: 600, x: 650, y: 100 }, { height: 337, width: 600, x: 1280, y: 100 }, { height: 337, width: 600, x: 20, y: 640 }, { height: 337, width: 600, x: 650, y: 640 }, { height: 337, width: 600, x: 1280, y: 640 }]
]; That might look overwhelming, but consider that each element in the outer array element contains an array of compositions for each video. If there are 3 participants, we can reference the third element in the outer array, and the position of the participant id in the array will determine which composition will apply to that video. participantIds We can modify our function to grab the proper composition and use those values when we add the video to the broadcast client. renderVideosToClient() const renderVideosToClient = async (participant, stream) => {
  const participantId = participant.id;
  const videoId = `video-${participantId}`;

  // get the index of this participantId
  const pIdx = participantIds.indexOf(participantId);

  let composition = layouts[participantIds.length - 1][pIdx];
  config.index = 2;

  const mediaStream = new MediaStream();
  mediaStream.addTrack(stream.mediaStreamTrack);
  broadcastClient.addVideoInputDevice(mediaStream, videoId, composition);
}; But remember - if we only do this when a participant is added, the previous video compositions will still reflect the composition that was applied when they were added. That is where the function comes into the picture. updateVideoCompositions() Here we loop over the array, grab the proper composition from , and use the ( ) method of the . participantIds layouts updateVideoDeviceComposition() docs broadcastClient const updateVideoCompositions = async () => {
  let idx = 0;
  for (const p of participantIds) {
    const videoId = `video-${p}`;
    let config = layouts[filteredParticipantIds.length - 1][idx];
    config.index = 2;
    broadcastClient.updateVideoDeviceComposition(videoId, config);
    idx = idx + 1;
  }
}; We should also make sure that when a participant leaves the stage that we remove the participant id from the array and again update the composition for all videos. stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_REMOVED, (participant, streams) => {
  const participantId = participant.id;

  // remove participant id from array
  const pIdx = participantIds.findIndex(id => id === participantId);
  participantIds.splice(pIdx, 1);

  const videoTrackId = `video-${participantId}`;
  const audioTrackId = `audio-${participantId}`;
  if (broadcastClient.getVideoInputDevice(videoTrackId)) broadcastClient.removeVideoInputDevice(videoTrackId);
  if (broadcastClient.getAudioInputDevice(audioTrackId)) broadcastClient.removeAudioInputDevice(audioTrackId);
  const videoId = `${participantId}-video`;
  document.getElementById(videoId).closest('.participant-col').remove();
  updateVideoCompositions(); 
}); As mentioned above, you'll most likely want to limit the amount of videos that are added to the live stream via the broadcast client. You might want to add a static image instead of the final video to show that there are more participants than what is shown: Summary In this post, we learned one approach for dynamic layouts when broadcasting a multi-host stage with Amazon IVS. In a future post, we'll look at additional options for broadcasting with multiple hosts. As always, if you have any questions or comments, please leave them below. Also published here

How to Create a Dynamic Layout for Multi Host Broadcasts With Amazon IVS

Cómo crear un diseño dinámico para transmisiones de varios hosts con Amazon IVS

Comment créer une mise en page dynamique pour les diffusions multi-hôtes avec Amazon IVS

अमेज़ॅन आईवीएस के साथ मल्टी होस्ट ब्रॉडकास्ट के लिए डायनामिक लेआउट कैसे बनाएं

Amazon IVS を使用してマルチホスト ブロードキャスト用の動的レイアウトを作成する方法

Como criar um layout dinâmico para transmissões de vários hosts com o Amazon IVS

Cách tạo bố cục động cho truyền phát nhiều máy chủ với Amazon IVS

如何使用 Amazon IVS 创建多主机广播的动态布局

In our we learned how to create a virtual "stage" to create a real-time video chat experience for up to 12 participants with Amazon Interactive Video Service (Amazon IVS). As a standalone feature, that's pretty powerful and enables us to add real-time collaboration to our applications. However, this feature was created to empower developers to easily create collaborative live streaming experiences similar to the "Guest Star" feature on Twitch. In this post, we'll build on the previous demo to combine the audio and video feeds from all participants into a single feed and broadcast that feed to an Amazon IVS channel. last post If you haven't yet read the previous post, you should do that before moving forward with this post. To recap, in that post we learned how to: Create a stage resource with the AWS SDK for JavaScript (v3) Create a stage participant tokens with the AWS SDK for JavaScript (v3) Use the Web Broadcast SDK to connect to the virtual stage for real-time video chat between participants The next step to creating a collaborative live streaming experience is to combine (or "composite") both the local and remote participants into a single stream that can be published to an Amazon IVS channel. For this we can also use the Web Broadcast SDK, so let's see how it's done. Creating a Broadcast Client If you recall, in the last post we had several functions called inside of a handler that enabled permissions, obtained devices, configured the instance, and handled joining the stage. We'll add one more method to this flow called which we can use to create an instance of the . We'll need a element in our markup for the combined stream so that our participants can preview what will ultimately be broadcast to the Amazon IVS channel. DOMContentLoaded Stage initBroadcastClient() IVSBroadcastClient <canvas> const initBroadcastClient = async () => {
  broadcastClient = IVSBroadcastClient.create({
    streamConfig: IVSBroadcastClient.STANDARD_LANDSCAPE,
    ingestEndpoint: '[YOUR INGEST ENDPOINT]',
  });

  const previewEl = document.getElementById('broadcast-preview');
  broadcastClient.attachPreview(previewEl);

  const bgImage = new Image();
  bgImage.src = '/images/stage_bg.png';
  broadcastClient.addImageSource(bgImage, 'bg-image', { index: 0 });
}; To make things a little more visually appealing, I've used to add a background image to the stream. The method receives three arguments: the image, a unique name for the source, and a object that is used to define the (or 'layer') for the source. If you check the for , you'll also note that it can contain values for the , , , and position for the source. We'll take advantage of those properties in just a bit when we add our video layers for each participant. addImageSource() addImageSource() VideoComposition index docs VideoComposition height width x y Adding Participant Audio and Video to the Broadcast Client Next, we're going to add the audio and video for each participant to the broadcast client. We'll do this inside of the handler that we defined in the previous post. Modify that function to add calls to two new functions. StageEvents.STAGE_PARTICIPANT_STREAMS_ADDED stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_ADDED, (participant, streams) => {
  renderParticipant(participant, streams);
  renderVideosToClient(participant, streams.find(s => s.streamType === StreamType.VIDEO));
  renderAudioToClient(participant, streams.find(s => s.streamType === StreamType.AUDIO));
}); Now let's create the function. Here, I'm hardcoding the to values appropriate for a single participant. In your application, you'll want to dynamically calculate the , , , and values depending on the amount of users currently participating in the conversation. renderVideosToClient() VideoComposition height width x y const renderVideosToClient = async (participant, stream) => {
  const participantId = participant.id;
  const videoId = `video-${participantId}`;
  const composition = { 
    index: 1,
    height: 984, 
    width: 1750, 
    x: 85, 
    y: 48 
  };
  const mediaStream = new MediaStream();
  mediaStream.addTrack(stream.mediaStreamTrack);
  broadcastClient.addVideoInputDevice(mediaStream, videoId, composition);
}; The function looks similar, but uses the method of the SDK to add the audio track. renderAudioToClient() addAudioInputDevice() const renderAudioToClient = async (participant, stream) => {
  const participantId = participant.id;
  const audioTrackId = `audio-${participantId}`;
  const mediaStream = new MediaStream();
  mediaStream.addTrack(stream.mediaStreamTrack);
  broadcastClient.addAudioInputDevice(mediaStream, audioTrackId)
}; At this point, the stage is ready to be broadcast to a channel by calling . We'll also need to handle removing participants from the when they leave a stage. For this, update the handler for . broadcastClient.startBroadcast('[YOUR STREAM KEY]') broadcastClient StageEvents.STAGE_PARTICIPANT_STREAMS_REMOVED stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_REMOVED, (participant, streams) => {
  const videoTrackId = `video-${participant.id}`;
  const audioTrackId = `audio-${participant.id}`;
  if (broadcastClient.getVideoInputDevice(videoTrackId)) broadcastClient.removeVideoInputDevice(videoTrackId);
  if (broadcastClient.getAudioInputDevice(audioTrackId)) broadcastClient.removeAudioInputDevice(audioTrackId);
  const videoId = `${participant.id}-video`
  document.getElementById(videoId).closest('.participant-col').remove();
  updateVideoCompositions(); // function not defined in demo, implementation will vary
}); Here's how an implementation might look with a single participant. Note that each stage participant would be shown on the bottom screen, and the composite view to be broadcast is shown above. And when multiple participants have joined the virtual stage, the application adjusts the layout to accommodate each participant. When the 'host' participant clicks the 'Broadcast' button, the combined conversation will be broadcast to the Amazon IVS channel as a composite view with all participants audio and video combined into a single stream. Summary In this post, we learned how to create a live stream broadcast with the audio and video from multiple remote participants. In a future post, we'll examine alternative options for creating the composite stream and broadcasting it to an Amazon IVS channel.

How to Live Stream With Multiple Hosts using a Browser and Amazon IVS

Cómo transmitir en vivo con múltiples hosts usando un navegador y Amazon IVS

Comment diffuser en direct avec plusieurs hôtes à l'aide d'un navigateur et d'Amazon IVS

एक ब्राउज़र और अमेज़ॅन आईवीएस का उपयोग करके एकाधिक होस्ट के साथ लाइव स्ट्रीम कैसे करें

ブラウザと Amazon IVS を使用して複数のホストでライブ ストリーミングする方法

Como transmitir ao vivo com vários hosts usando um navegador e o Amazon IVS

Cách phát trực tiếp với nhiều máy chủ bằng Trình duyệt và Amazon IVS

如何使用浏览器和 Amazon IVS 与多个主机进行直播

We recently support for collaborative live streams via a new virtual resource called a stage with Amazon Interactive Video Service (Amazon IVS). This feature opens up a multitude of possibilities that were not easy (or even possible) to implement in the past. announced This feature gives live streamers the ability to invite guests into their stream, collaborate with other content creators, or even create a "call-in" style show with audio-only participants. In this post, we'll look at how to get started creating a web application with multiple hosts with the Web Broadcast SDK. As always, the are a great starting point, but it always helps to walk through a demo application, so that's what we'll focus on in this post. docs For this demo, we'll build a web application that allows multiple participants to view and talk to one another. In a future post, we'll add the ability to broadcast the conversation to an Amazon IVS channel where others can view the conversation. At a high level, here are the steps that we'll take to create a virtual "stage" application for real-time collaboration between multiple hosts. Create a via the SDK stage Issue token(s) for stage participants Connect to the on the client stage Render participant's audio and video when they join a stage The Server Side We will utilize the new module ( ) of the AWS SDK for JavaScript (v3) to create the stage resource and participant tokens. @aws-sdk/client-ivs-realtime docs Here is an that you can use to handle these steps, but for my demo, I decided to simplify things a bit and created an Express application that can be used to host the API and my front-end. example of a serverless application Creating a Stage Resource via the AWS SDK for JavaScript (v3) In my Express app, I created a service called . This service caches existing and in memory to avoid the need for a database to persist them. IvsRealtimeService stages stageParticipants In production, you'd want to store these values in a persistent data store to avoid the volatility and limitations of caching the values in memory (again, refer to the to learn more about using Amazon DynamoDB to persist stages and tokens). serverless demo My app has one route called that can be used to a stage . This route uses my to check for an existing stage by that name, and either returns the existing stage or creates a new stage. /ivs-stage POST name IvsRealtimeService router.post('/ivs-stage', async function (req, res, next) {
  const body = req.body;
  const stageName = body.name;
  let stage = 
    ivsRealtimeService.stageExists(stageName) ? 
      ivsRealtimeService.getStage(stageName) : 
      await ivsRealtimeService.createStage(stageName);
  res.json(stage);
}); The and methods look like so: stageExists() getStage() import { CreateParticipantTokenCommand, CreateStageCommand, IVSRealTimeClient } from "@aws-sdk/client-ivs-realtime";

const config = {
  credentials: {
    accessKeyId: process.env.ACCESS_KEY,
    secretAccessKey: process.env.SECRET_KEY,
  }
};

export default class IvsRealtimeService {
  ivsRealtimeClient = new IVSRealTimeClient(config);
  stages = {};
  stageParticipants = {};

  getStage(name) {
    return this.stages[name];
  }

  stageExists(name) {
    return this.stages.hasOwnProperty(name);
  }
} And the method sends a via the . createStage() CreateStageCommand ivsRealtimeClient async createStage(name) {
  const createStageRequest = new CreateStageCommand({ name });
  const createStageResponse = await this.ivsRealtimeClient.send(createStageRequest);
  this.stages[name] = createStageResponse.stage;
  this.stageParticipants[name] = [];
  return this.stages[name];
}; Generating Stage Participant Tokens I created another route in my application to handle generating participant tokens called . /ivs-stage-token router.post('/ivs-stage-token', async function (req, res, next) {
  const body = req.body;
  const username = body.username;
  const stageName = body.stageName;
  const userId = uuid4();
  let token = await ivsRealtimeService.createStageParticipantToken(stageName, userId, username);
  res.json(token);
}); Note that this endpoint requires a , , and a and invokes . username stageName userId ivsRealtimeService.createStageParticipantToken() async createStageParticipantToken(stageName, userId, username, duration = 60) {
  let stage;
  if (!this.stageExists(stageName)) {
    stage = await this.createStage(stageName)
  }
  else {
    stage = this.getStage(stageName);
  }
  const stageArn = stage.arn;

  const createStageTokenRequest = new CreateParticipantTokenCommand({
    attributes: {
      username,
    },
    userId,
    stageArn,
    duration,
  });
  const createStageTokenResponse = await this.ivsRealtimeClient.send(createStageTokenRequest);
  const participantToken = createStageTokenResponse.participantToken;
  this.stageParticipants[stageName].push(participantToken);
  return participantToken;
}; In this method, we're creating a which expects an input object containing the , the , the token duration (default: 60 minutes), and an object which can be used to store arbitrary application-specific values (in my case, the ). CreateParticipantTokenCommand userId stageArn attributes username The will be available later on when we create our front-end, so it's a nice way to include participant-specific information. But, like our docs say: attributes This field is exposed to all stage participants and should not be used for personally identifying, confidential, or sensitive information. Now that we've created a few endpoints to help us create stage resources and participant tokens, let's look at creating the web app. Building the Web Application The front-end will be a straightforward, vanilla JavaScript and HTML application to keep things simple and focus on learning the Web Broadcast SDK. For this demo, we can add a route that returns an HTML file. In that file, include the Web Broadcast SDK (version ). 1.3.2 The Includes and Markup <script src="https://web-broadcast.live-video.net/1.3.2/amazon-ivs-web-broadcast.js"></script> Because the number of participants in this virtual stage is dynamic (up to 12), it makes sense to create a that contains a tag and any other buttons, labels, or markup that we need. Here's how that might look. <template> <video> <template id="stages-guest-template">
  <video class="participant-video" autoplay></video>
  <div>
    <small class="participant-name"></small>
  </div>
  <div>
    <button type="button" class="settings-btn">Cam/Mic Settings</button>
  </div>
</template> I also have an empty that will be used to render the participants as they join the virtual stage. <div> <div id="participants"></div> The JavaScript Now that we have the Web Broadcast SDK dependency and the markup ready to go, we can look at the JavaScript required to join a virtual "stage" and render the participants. This involves several steps, but we can break them out into manageable functions to keep things simple. When the DOM is ready, we can call the following functions (we'll look at each below). let
  audioDevices,
  videoDevices,
  selectedAudioDeviceId,
  selectedVideoDeviceId,
  videoStream,
  audioStream,
  username,
  stageConfig,
  username = '[USERNAME]',
  stageName = '[STAGE NAME]',
  stageParticipantToken,
  stage;

document.addEventListener('DOMContentLoaded', async () => {
  await handlePermissions();
  await getDevices();
  await createVideoStream();
  await createAudioStream();
  stage = await getStageConfig(stageName);
  stageParticipantToken = await getStageParticipantToken(stage.name, username);
  await initStage();
}); Devices and Permissions The first 4 method calls ( , , , ) should look familiar if you've worked with the Amazon IVS Web Broadcast SDK in the past. handlePermissions() getDevices() createVideoStream() createAudioStream() We'll quickly look at each function below, but you can always refer to the for more information. docs First, lets us prompt the user for permission to access their webcam and microphone. handlePermissions() const handlePermissions = async () => {
  let permissions;
  try {
    const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true });
    for (const track of stream.getTracks()) {
      track.stop();
    }
    permissions = { video: true, audio: true };
  }
  catch (err) {
    permissions = { video: false, audio: false };
    console.error(err.message);
  }
  if (!permissions.video) {
    console.error('Failed to get video permissions.');
  } else if (!permissions.audio) {
    console.error('Failed to get audio permissions.');
  }
}; Next, retrieves a list of webcams and microphones and stores them. getDevices() In this demo, I'm defaulting the selected video and audio device to the first available device, but in your application, you'd likely be presenting these in a to let the user pick which broadcast device they'd like to use. <select> const getDevices = async () => {
  const devices = await navigator.mediaDevices.enumerateDevices();
  videoDevices = devices.filter((d) => d.kind === 'videoinput');
  audioDevices = devices.filter((d) => d.kind === 'audioinput');
  selectedVideoDeviceId = videoDevices[0].deviceId;
  selectedAudioDeviceId = audioDevices[0].deviceId;
}; Now that we have the devices listed, we can create a video and audio stream from them. For multi-host videos, we should make sure that we keep in mind the for framerate and video size. recommended limits const createVideoStream = async () => {
  videoStream = await navigator.mediaDevices.getUserMedia({
    video: {
      deviceId: {
        exact: selectedVideoDeviceId
      },
      width: {
        ideal: 1280,
        max: 1280,
      },
      height: {
        ideal: 720,
        max: 720,
      },
      frameRate: {
        max: 30,
      },
    },
    });
};

const createAudioStream = async () => {
  audioStream = await navigator.mediaDevices.getUserMedia({
    audio: {
      deviceId: selectedAudioDeviceId
    },
  });
}; Configuring and Joining The Stage Now that we have permissions, devices, and streams sorted we can focus on the virtual stage. First, declare some necessary variables. const { Stage, SubscribeType, LocalStageStream, StageEvents, StreamType } = IVSBroadcastClient; Now we can call the method which calls the API endpoint that we created above to create (or retrieve) the stage resource. getStageConfig() The value of the stage here is how our application will allow multiple participants to join the same virtual stage, so we'd probably want to pass this in somehow (maybe via URL variables or retrieved from a backend). name const getStageConfig = async (name) => {
  const stageRequest = await fetch('/ivs-stage', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ name })
  });
  const stage = await stageRequest.json();
  return stage;
}; Next, retrieves a token from the other API endpoint that we created above. getStageParticipantToken() The variable is a property of the object returned from immediately above, and the depends on your application logic (maybe you have access to a current logged-in user property). stageName stage getStageConfig() username const getStageParticipantToken = async (stageName, username) => {
  const stageTokenRequest = await fetch('/ivs-stage-token', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ username, stageName }),
  });
  const token = await stageTokenRequest.json();
  return token;
}; Now, we're ready to configure and join the stage. The method will create an instance of the object, which expects the stage participant token and a object. initStage() Stage strategy The object contains three functions that express the desired state of the stage (refer to the for all of the different possibilities for strategy). Within the method, we can define it like so. strategy docs initStage() const initStage = async () => {
  const strategy = {
    shouldSubscribeToParticipant: (participant) => {
      return SubscribeType.AUDIO_VIDEO;
    },
    shouldPublishParticipant: (participant) => {
      return true;
    },
    stageStreamsToPublish: () => {
      const videoTrack = videoStream.getVideoTracks()[0]
      const audioTrack = audioStream.getAudioTracks()[0]
      const streamsToPublish = [
        new LocalStageStream(videoTrack),
        new LocalStageStream(audioTrack)
      ];
      return streamsToPublish;
    },
  };
} Before we move forward with the rest of the method, let's break down the strategy object. The first function ( ) expresses how to handle each participant that joins the application. initStage() shouldSubscribeToParticipant() This gives us the freedom to add participants who may act as moderators ( ), audio-only participants ( ), or participants with full video and audio ( ) as shown above. NONE AUDIO AUDIO_VIDEO Next, expresses whether or not the participant should be published. You might want to check the state of a participant based on a button click or a check box to give participants the ability to remain unpublished until they are ready. shouldPublishParticipant() Finally, expresses an array of objects containing the elements that should be published. In this demo, we'll use both the and that we created above to generate these. stageStreamsToPublish() LocalStageStream MediaStream videoStream audioStream Next, inside of the method, we create an instance of the class, passing it the participant token and the strategy. initStage() Stage stage = new Stage(stageParticipantToken.token, strategy); Now that we have a instance, we can attach listeners to the various events that are broadcast on the stage. See the . In this demo, we'll listen for when participants are added or removed. Stage docs for all of the possible events When participants are added, we will render the participant. stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_ADDED, (participant, streams) => {
  renderParticipant(participant, streams);
}); The method, defined outside of the method, clones the that we defined above and customizes it for the given participant. renderParticipant() initStage() <template> Note that the object contains a boolean value ; we can use this to only add the video stream for the current local participant to avoid echoing their own voice back to them. participant isLocal const renderParticipant = (participant, streams) => {
  // clone the <template>
  const guestTemplate = document.getElementById('stagesGuestTemplate');
  const newGuestEl = guestTemplate.content.cloneNode(true);

  // populate the template values
  newGuestEl.querySelector('.participant-col').setAttribute('data-participant-id', participant.id);
  newGuestEl.querySelector('.participant-name').textContent = participant.attributes.username;

  // get a list of streams to add
  let streamsToDisplay = streams;
  if (participant.isLocal) {
    streamsToDisplay = streams.filter(stream => stream.streamType === StreamType.VIDEO)
  }

  // add all audio/video streams to the <video>
  const videoEl = newGuestEl.querySelector('.participant-video');
  videoEl.setAttribute('id', `${participant.id}-video`);
  const mediaStream = new MediaStream();
  streamsToDisplay.forEach(stream => {
    mediaStream.addTrack(stream.mediaStreamTrack);
  });
  videoEl.srcObject = mediaStream;

  // add the cloned template to the list of participants
  document.getElementById('participants').appendChild(newGuestEl);
}; Back in the , we can listen for when a participant leaves the stage so that we can remove their video from the DOM. initStage() stage.on(StageEvents.STAGE_PARTICIPANT_STREAMS_REMOVED, (participant, streams) => {
  const videoId = `${participant.id}-video`
  document.getElementById(videoId).closest('.participant-col').remove();
}); For this demo, we won't add any additional listeners. Your business needs will dictate additional listeners, and your application can respond to those as necessary. For example, if we wanted to update an indicator on the client side with the current connection state, we could listen for and set the state indicator each time the handler is invoked. StageEvents.STAGE_CONNECTION_STATE_CHANGED The final step inside is to join the stage. initStage() try {
   await stage.join();
} 
catch (error) {
   // handle join exception
} Leaving the Stage It's not completely necessary, but we can improve the user experience by explicitly leaving a stage when the participant exits the application. This will ensure that the remaining participants’ UI will be updated sooner than later. For this, we can use a handler to invoke to ensure a clean disconnection. beforeunload stage.leave() const cleanUp = () => {
  if (stage) stage.leave();
};

document.addEventListener("beforeunload", cleanUp); Now, our application is ready to test. Running the application gives us a real-time video chat experience between up to 12 participants. Summary In this post, we learned how to create a real-time video chat experience for up to 12 participants. In our next post, we'll learn how to take the next step and broadcast a real-time chat to an Amazon IVS channel so that end viewers can watch the conversation with high quality and low latency. Also published here

How to Create a Real Time Multi Host Video Chat in a Browser With Amazon IVS: A Guide

Access User Location with Permission Handler in Flutter

The Death of App Attribution

Imagine if you could get all the tips and tricks you need to tackle a binary classification problem on Kaggle or anywhere else. I have gone over 10 Kaggle competitions including: $35,000 Toxic Comment Classification Challenge $25,000 TalkingData AdTracking Fraud Detection Challenge $20,000 IEEE-CI S Fraud Detection $50,000 Jigsaw Multilingual Toxic Comment Classification $25,000 RSNA Intracranial Hemorrhage Detection $30,000 SIIM-ACR Pneumothorax Segmentation $65,000 Jigsaw Unintended Bias in Toxicity Classification $65,000 Santander Customer Transaction Prediction $25,000 Microsoft Malware Prediction $25,000 Humpback Whale Identification – and pulled out that information for you. Dive in. Modeling Use two BiGru layers layers feeding into two final Dense Decide on the of 250 runs with Bayesian optimization best parameters by selecting the best out Use a 2-level by max-pooling and 2 fully-connected layers bidirectional GRU followed Dealing with imbalance problems Check this extensive notebook on handling imbalanced classes Class balancing of one-shots: Getting top-1 frequencies of classes and replacing “new whale” class with classes that are one-shots and not presented @ top-1 Metrics Global AUC explained (+alternatives) ROC AUC score for fraud detection Good Old Accuracy Mean Average Precision (MAP) Binary Log Loss F beta score, where beta is equal to 0.5 Loss BCE and Dice Based BCE with Logit Loss DICE loss for segmentation BCE + DICE Combinations of BCE, Dice and Focal loss Focal Loss Based Focal Loss Function BCE + focal loss Weighted Focal Loss Custom Losses Weighted subgroup negative samples Others Lovasz Loss Weighted sigmoid cross-entropy loss Hard triplet loss Center Loss Additive Angular Margin Loss Margin loss CosFace Loss KL-Div loss with soft label Cross-validation + proper evaluation Use Adversarial validation Apply GroupKFold cross-validation about last 100k records as a validation set Simple time-split and using Generate predictions using unshuffled KFold Use stratified test data 5 fold without early stopping for predicting Implement LightGBM on 10 KFolds with no shuffle If using pseudo labeling, to avoid overfitting don’t validate on the pseudo labels Use the for the final blend Standard 10 fold Stratified cross-validation with multiple seeds Post-processing Use of the history of submissions to tweak our test set predictions Select random 30% of CV, optimize the thresholds for the 30% and apply them to the remaining 70% and check how far off they are from the optimal thresholds of the 70% Use a re-scaling factor for predictions >0.8 as well as <0.01 through the use of probabilistic random noise that introduces a small penalty Scale up the predicted probability of comments that contain cursed words of different languages Label the test samples , adding them to the train set, and training to convergence using the best-performing ensemble Ensembling Averaging Averaging over multiple seeds Average 10 out-of-fold predictions Average multiple seeds Add model and bagging models with different folds diversity by seed averaging Geometric mean An ensemble of LightGBM, CatBoost weighted geometric mean Average different models of An average ensemble XLM-R models Average predictions for 7 language-specific models An ensemble of XLM-R models An ensemble of CatBoost, XGBoost, and LightGBM Stacking Stack Bi-LSTM, Bert-Large-Uncased with WWM, XLNET, with the meta model as ExtraTreesClassifier LightGBM Stacking Stack LightGBM with heavy bayesian optimization using and Stack models PyStackNet MlXtend An of RNN, CNN, LightGBM, and NBSV ensemble Use 5 time bagged XGB CV scores with heavy bayesian optimization Blending Use power blending Blend using and Hyperopt OOF to find optimal weights Others ensembling Implement Hillclimb Apply LGB g data samples bagged 10 times with different trainin Repositories and open solutions Repos with open source solutions Image based solutions Humpback Whale Identification 1st Place Code Data Science Bowl 2nd Place Solution Forecasting Lung Cancer Diagnoses with Deep Learning Kaggle data science bowl 2017 RSNA Intracranial Hemorrhage Detection 1st Place Solution 2nd Place Solution — RSNA Intracranial Hemorrhage Detection 3rd place solution RSNA Intracranial Hemorrhage Detection 4th Place Solution with code RSNA Intracranial Hemorrhage Detection 5th place solution for RSNA Intracranial Hemorrhage Detection RSNA Intracranial Hemorrhage Detection Entrypoint for the 5th-place-solution SIIM-ACR Pneumothorax Segmentation 1st Place Solution SIIM-ACR Pneumothorax Segmentation 3rd Place Solution 5th place solution SIIM-ACR Pneumothorax Segmentation Humpback Whale Identification 5th Place Solution Humpback Whale Identification 4th Place Solution Kaggle Humpback Whale Identification Challenge 2019 2nd place code Tabular based solutions How to implement and how it was used in the LibFM in Keras Talking Data competition on Kaggle XGB Fraud Detection Solution Fraud Detection Feature Engineering 2nd Place Solution Santander Customer Transaction Prediction Santander Customer Transaction Prediction 5th Place Solution S olution to the Kaggle Santander Customer Transaction Prediction competition 2nd place Solution the Microsoft Malware Prediction Challenge on Kaggle Text classification based solutions Toxic Comment Classification Challenge, 12th p lace solution for the Code and write-up Kaggle Toxic Comment Classification Challenge Jigsaw Unintended Bias in Toxicity Classification 4th Place Solution An open solution to the Toxic Comment Classification Challenge Challenge TalkingData AdTracking Fraud Detection 4th Place Solution Bronze medal Jigsaw Solution 2nd place solution for the 2017 national data science bowl Jigsaw Unintended Bias in Toxicity Classification 10th Place Solution Code for 3rd place solution in Kaggle Humpback Whale Identification Challenge Final thoughts Hopefully, this article gave you some background into binary classification tips and tricks, as well as, some tools and frameworks that you can use to start competing. We’ve covered tips on: architectures, losses, post-processing, ensembling, tools and frameworks. If you want to go deeper, simply follow the links and see how the best binary classification models are built. See also: Image Segmentation: Tips and Tricks from 39 Kaggle Competitions Text Classification: All Tips and Tricks from 5 Kaggle Competitions Tabular Data Binary Classification: All Tips and Tricks from 5 Kaggle Competitions This article was originally written by Derrick Mwiti and posted on the Neptune blog . You can find more in-depth articles for machine learning practitioners there.

Practical Tips For Binary Classification Excellence

What is WebRTC? WebRTC stands for Web Real Time Communication. From webrtc.org: "With WebRTC, you can add real-time communication capabilities to your application that works on top of an open standard. It supports video, voice, and generic data to be sent between peers, allowing developers to build powerful voice and video communication solutions" The API is actually not so big and complicated as it sounds. The MDN website is, as always, very complete . https://developer.mozilla.org/en-US/docs/Glossary/WebRTC We have 3 main parts: getUserMedia(): sends a signal to the browser to ask the user for permission and returns the "stream" of audio/video/data RTCPeerConnection: two participants are called peers, local and remote. Not necessarily physically remote, can be two browsers in the same computer. RTCDataChannel: two peers connect via a channel in a bidirectional way to transfer data. A peer can have up to 65,534 data channels (depends on the browser) In webrtc.org we can find many useful samples https://webrtc.github.io/samples/ Let's dissect a basic peer-to-peer connection. demo: https://webrtc.github.io/samples/src/content/peerconnection/pc1/ code: https://github.com/webrtc/samples/tree/gh-pages/src/content/peerconnection/pc1 First, we will need two video elements, one to see the other person and another one to see ourselves: < = > video id "localVideo" playsinline autoplay muted </ > video < = > video id "remoteVideo" playsinline autoplay </ > video Then the controls to start, call and hangup: Start Call Hang Up < = > button id "startButton" </ > button < = > button id "callButton" </ > button < = > button id "hangupButton" </ > button Now we can start using the WebRTC API to handle all the peer-to-peer connection events. First we set some global variables, get GUI elements and add click handlers: startButton = .getElementById( ); callButton = .getElementById( ); hangupButton = .getElementById( );

startButton.addEventListener( , start);
callButton.addEventListener( , call);
hangupButton.addEventListener( , hangup); localVideo = .getElementById( ); remoteVideo = .getElementById( ); localStream; pc1; pc2; offerOptions = { : , : ,
}; const document "startButton" const document "callButton" const document "hangupButton" "click" "click" "click" const document "localVideo" const document "remoteVideo" let let let const offerToReceiveAudio 1 offerToReceiveVideo 1 When we click start, basically we want to ask the user for permission to use the camera and microphone with getUserMedia(): { { stream = navigator.mediaDevices.getUserMedia({ : , : ,
    });

    localVideo.srcObject = stream;
    localStream = stream;
  } (e) {
    alert( );
  }
} async ( ) function start try const await audio true video true catch `getUserMedia() error: ` ${e.name} When starting a call, we are going to establish a connection between two peers, adding each other ICECandidates, set local audio/video tracks, create a connection offer from peer 1 and create an answer from peer 2: { configuration = {};

  pc1 = RTCPeerConnection(configuration);
  pc1.addEventListener( , (e) => onIceCandidate(pc1, e));

  pc2 = RTCPeerConnection(configuration);
  pc2.addEventListener( , (e) => onIceCandidate(pc2, e));
  pc2.addEventListener( , gotRemoteStream);

  localStream.getTracks().forEach( pc1.addTrack(track, localStream)); { pc1.createOffer(offerOptions); onCreateOfferSuccess()
  } (e) {
    onCreateSessionDescriptionError(e);
  }
} { { pc2.createAnswer();
  } (e) {
    onCreateSessionDescriptionError(e);
  }
} { .log( );
} async ( ) function call const new "icecandidate" new "icecandidate" "track" ( ) => track try await await catch async ( ) function onCreateOfferSuccess try await catch ( ) function onCreateSessionDescriptionError error console `Failed to create session description: ` ${error.toString()} From the code above, we notice that we need an RTCIceCandidate, it represents a Internet Connectivity Establishment. It describes all under-the-hood logic needed for two peers to communicate, protocols, routing, error correction, etc. Here we will add the candidates on both sides: { pc === pc1 ? : ;
} { pc === pc1 ? pc2 : pc1;
} { { getOtherPc(pc).addIceCandidate(event.candidate);
  } (e) {
    onAddIceCandidateError(pc, e);
  }
} { .log( );
} ( ) function getName pc return "pc1" "pc2" ( ) function getOtherPc pc return async ( ) function onIceCandidate pc, event try await catch ( ) function onAddIceCandidateError pc, error console ` failed to add ICE Candidate: ` ${getName(pc)} ${error.toString()} After adding ICE candidates, we added a listener to the "track" event. We will add the event stream to the srcObject of the video element: { (remoteVideo.srcObject !== e.streams[ ]) {
    remoteVideo.srcObject = e.streams[ ];
  }
} ( ) function gotRemoteStream e if 0 0 This should give us video from both sides. Finally, to hangup a call, we close the peer connections and set them to null: {
  pc1.close();
  pc2.close();
  pc1 = ;
  pc2 = ;
} ( ) function hangup null null I hope I simplified enough the webrtc.org basic peer connection example https://github.com/webrtc/samples/tree/gh-pages/src/content/peerconnection/pc1 so you can start creating awesome video call applications. Of course for a production ready service, we will have to consider a lot more. Happy coding.

Understanding WebRTC 

I just finished working on , a website which lets you sort your favourite Spotify music by different moods, and then save them as playlists. moooodify This article will take you through my journey (briefly). I am no expert coder by any means, so I mostly learned as I went along. Alright, without further ado, let's get into it. You can check it out on . Product Hunt here Step 1: Registering my app First, I created an account on the . Registering my app there gave me a client ID which would be important for the next step. I could also whitelist redirect URIs. This URI enables the Spotify authentication service to automatically re-launch your app every time the user logs in. Developer's Dashboard You can learn more about registering your app . here Step 2: Authorization To have the end user approve your app for access to their Spotify data and features, or to have your app fetch data from Spotify, I needed to authorize your application. I used the method of authorization, which is shown below. Implicit Grant It's quite straightforward. I redirect to a URL with Spotify login, sending my  Client ID and the that were relevant, which I got from the previous step. permission scopes After the user has logged in, I get the access token which I can use to access the Web API. To learn more about what exact commands to use, you can see the . Spotify Authorization Guide Step 3: Using the Spotify Web API Sorting songs by mood Once my app is authorized, I first get the , depending on which tab the user is on. For example, if the user selects their top tracks of the last 6 months, this is the GET request: user's top tracks $.ajax({ : , : { : + access_token
    }, : {
        getTrackIDs(response);

    }, url 'https://api.spotify.com/v1/me/top/tracks?limit=50&time_range=medium_term' headers 'Authorization' 'Bearer ' success ( ) function response In case of or , the time_range query changes to short_term or long_term respectively. last month all time Once I get the response, I use the track IDs to then get the album art for each of the tracks. For that, I have to . This is how the request looked: get several albums { $.ajax({ : + (trackIDString), : { : + access_token
    }, : {
  setImages();
})
} ( ) function getAlbumArt trackIDString return url 'https://api.spotify.com/v1/albums?ids=' encodeURIComponent headers 'Authorization' 'Bearer ' success ( ) function response I then used the images attribute of the response to store them in an array. Having got all the info we need to display, that is, track name, artist name, and album art, we can now go ahead and sort our tracks into moods and display them. However, we need to get the first. We use the trackIDString we used for the albums as a query here too. audio features for each track $.ajax({ : + (trackIDString), : { : + access_token
            }, : {
                setTrackInfo(response);
            },
            }) url 'https://api.spotify.com/v1/audio-features?ids=' encodeURIComponent headers 'Authorization' 'Bearer ' success ( ) function response Alright! Now we have everything to sort the songs into different moods. Spotify provides various audio features like danceability, energy, acousticness, valence, tempo etc. I used a few of these in different combinations to come up with five different moods: Mellow, Upbeat, Chill, Party, High Energy. I then used JavaScript's method to insert the new track info on to the website. appendChild() I also created a tool where you can apply your own custom moods, in case you aren't feeling like any of the pre-generated ones. The principle is the same. The AJAX calls are made in the same order to fetch the tracks, their album art, and audio features. Creating Playlists Once the songs are sorted into moods, the user has the option to save these as playlists on to Spotify. For this, first I made a call to . We will need this to modify their playlists. get the user ID $.ajax({ : , : { : + access_token
      }, : {
          createPlaylist(response);

      }
  }) url 'https://api.spotify.com/v1/me/' headers 'Authorization' 'Bearer ' success ( ) function response We can then use the user ID to create a new playlist for them. For this, we use a POST request. $.ajax({ : , : + (user.id) + , : .stringify({ : playlist-name, : playlist-description
       }), : { : + access_token, : }, : {
        addTracks(response);     
      },
      }
    }) type 'POST' url 'https://api.spotify.com/v1/users/' encodeURIComponent '/playlists' data JSON "name" "description" headers 'Authorization' 'Bearer ' 'Content-Type' 'application/json' success ( ) function response This will create an empty playlist with a name and description of our choice. Now that we have a playlist, the only thing remaining is to add tracks to it. $.ajax({ : , : + (playlist.id) + + trackURIString, : { : + access_token, : }, : { .log( );
        },
        }
      }) type 'POST' url 'https://api.spotify.com/v1/playlists/' encodeURIComponent '/tracks?uris=' headers 'Authorization' 'Bearer ' 'Content-Type' 'application/json' success ( ) function result console 'yay' That's it! A filled playlist will be available in the user's Spotify library. Learnings It was the first time I was using the FTP method to host the website. Making it responsive, and having it work on all browsers was also a learning point. It would sometimes work only on Chrome, sometimes on every browser but Safari, so that was a challenge. The JavaScript was almost completely new. I didn't know about AJAX calls before this. Apart from that, I also tried my hand at CSS keyframe animations to give the website a smooth feel. All in all, this has been a great learning experience. Going through the whole flow of designing it, coding it, setting up tracking, and then releasing it has been extremely insightful. Try the website out yourself! Find me on and . Thanks for reading! Twitter Medium

My App Lets You Sort your Music by Mood [with some help from Spotify's Web API]

Build a React Shuffle Blog using the Spotify API

that provides browsers and mobile applications with Real-Time Communications (RTC) capabilities via simple APIs. WebRTC is a free, open project Webrtc is a cross platform solution with RTC capabilities. One can stream his own video stream be it from camera or screen recording or any other video to any peer via webrtc. If one is developing a real time peer to peer game with real time data sharing between the peers, webrtc is one of the options. Let’s understand how video calling from one peer to another works in webrtc. Getting started Since September 2017 Google started to distributed precompiled versions of webrtc for android via maven. If you want to play with the source and compile it yourself you can do it easily as per (Earlier you could compile it only on linux but now all the three major OS are supported). To use the pre compiled version simply add the following dependency. these steps compile 'org.webrtc:google-webrtc:1.0.22379' How does webrtc work? Before we start exchanging data between two peers via webrtc, we need to provide info to the peers about each other for media format negotiation and discovery. This is done via following protocols. is used for connecting peer to peer. is used to provide the metadata of the media content like resolution, encoding, bitrate, etc. If the two peers are not on same network then we will need to provide a server to provide the public address of the peers. If any of the network is firewall protected the we need to provide servers also. You can learn more about these protocols The mechanism to exchange ICE and SDP between the peers is called Signalling System and is usually done via websockets. Interactive Connectivity Establishment (ICE) Session Description Protocol (SDP) Session Traversal Utilities for NAT (STUN) Traversal Using Relays around NAT (TURN) here. Webrtc android API Most of our webrtc code will make use of and apis. is the equivalent of RTCPeerConnection in web world and is used to establish peer to peer connection. is used to create and objects_._ PeerConnection PeerConnectionFactory PeerConnection PeerConnectionFactory PeerConnection, MediaStream MediaStreamTrack ** **Before creating the factory object we need to initialize webrtc. As you can here provides options for enabling/disabling hardware accelerations(I had to disable in on certain devices) and setting field trials. are for enabling experimental webrtc features. Creating PeerConnectionFactory InitializationOptions Field trials val fieldTrials = (PeerConnectionFactory. + "/" + PeerConnectionFactory. + "/")val options = InitializationOptions.builder(application).setFieldTrials(fieldTrials).setEnableVideoHwAcceleration(videoAccelerationEnabled).createInitializationOptions()PeerConnectionFactory.initialize(options)factory = PeerConnectionFactory(PeerConnectionFactory.Options())val rootEglBase = EglBase.create()factory?.setVideoHwAccelerationOptions(rootEglBase. , rootEglBase. ) VIDEO_FRAME_EMIT_TRIAL TRIAL_ENABLED eglBaseContext eglBaseContext ** **Once we have the object we can now create a object which has and associated with it. Creating Media Streams PeerConnectionFactory MediaStream AudioTrack VideoTrack val localMediaStream = factory.createLocalMediaStream(MEDIA_ID)val audioSource = factory.createAudioSource(MediaConstraints())val audioTrack = factory.createAudioTrack(AUDIO_ID, audioSource)localMediaStream.addTrack(audioTrack) videoCapturer = if (Build.VERSION. > Build.VERSION_CODES. ) {createCameraCapturer(Camera2Enumerator(application))} else {createCameraCapturer(Camera1Enumerator(videoAccelerationEnabled))}val videoTrack = factory.createVideoTrack("VideoTrack", factory.createVideoSource(videoCapturer))localMediaStream.addTrack(videoTrack) SDK_INT LOLLIPOP Webrtc provides us a very easy way to use and API depending on the support. On supported devices we can use either of the APIs CameraVideoCapturer Camera Camera2 private fun createCameraCapturer(enumerator: CameraEnumerator): CameraVideoCapturer? {val deviceNames = enumerator._deviceNames_   for (deviceName in deviceNames) {if (enumerator.isFrontFacing(deviceName)) {val videoCapturer = enumerator.createCapturer(deviceName, null)if (videoCapturer != null) {return videoCapturer}}} for (deviceName in deviceNames) {if (!enumerator.isFrontFacing(deviceName)) {Timber.d("Creating other camera capturer.")val videoCapturer = enumerator.createCapturer(deviceName, null)if (videoCapturer != null) {return videoCapturer}}}return null} ** **Since we have the local ready, we need to render it on a view so that its visible to the user. is a in webrtc libray that does the rendering of webrtc frames for us. We can directly add it in layout xml. SurfaceViewRenderer MediaStream SurfaceViewRenderer View localViewRenderer.init(rootEglBase.getEglBaseContext(), null)localViewRenderer.setEnableHardwareScaler(true)localViewRenderer.setMirror(true)localViewRenderer.setScalingType(ScalingType.SCALE_ASPECT_FILL)val localVideoRenderer = VideoRenderer(localViewRenderer)videoTrack.addRenderer(localVideoRenderer) We have everything ready to transmit now, its time to create the We pass in a PeerConnection#Observer instance in the factory method, which is notified when a ICE candidate is generated (we need to send that to the other peer). The observer is also notified when the remote is available. we will attach a renderer to it just like the local . It also requires a list IceServer(STUN and TURN servers) which can be empty if testing on local network PeerConnection PeerConnection. MediaStream MediaStream val peerConnectionObserver = object : PeerConnection.Observer {override fun onIceCandidate(iceCandidate: IceCandidate) {localIceCandidatesSource.onNext(iceCandidate)}override fun onAddStream(mediaStream: MediaStream) {mediaStream.addRenderer(remoteRenderer)}...}peerConnection = factory?.createPeerConnection(getIceServers(), peerConnectionObserver) In Observer’s onIceCandidate we shouldn’t add the IceCandidate to PeerConnection (peerConnection.addIceCandidate(iceCandidate) till we have the remote SDP NOTE: Once the PeerConnection is created we need to initiate video call. The peer which initiates the call will and set its local SDP ( ). When the local SDP is set it will send that SDP to the other peer which will set its remote SDP ( ). Once the remote SDP is set it will with that sdp and set its local sdp when asnwer is created and send the local sdp to the peer which created the offer. The initiator peer will now set its remote SDP. CreateOffer/CreateAnswer createOffer peerConnection#setLocalSdp peerConnection#setRemoteSdp createAnswer fun createOffer() {peerConnection.createOffer(object : SdpObserver {override fun onCreateSuccess(sdp: SessionDescription) {setLocalSdp(sdp)}...}, getPeerConnectionConstraints())}fun setLocalSdp(sdp: SessionDescripton) {peerConnection.setLocalDescription(object : SdpObserver {override fun onSetSuccess() {api.sendSdp(peerConnection. )drainIceCandidates()}...}, sdp)} localDescription fun onOffer(sdp: SessionDescription) {peerConnection.setRemoteDescription(object : SdpObserver {override fun onSetSuccess() {createAnswer()}}, sdp)} fun createAnswer() {peerConnection?.createAnswer(object : SdpObserver {override fun onCreateSuccess(sdp: SessionDescription) {setLocalSdp(sdp)}}, getPeerConnectionConstraints())} ** **When the peers decide to end the connection its pretty important to do clean up in particular order as the C layer does a reference count check and will crash with assertion failures if the objects are not properly disposed. Dispose/ Clean up fun cleanUp() {peerConnection.dispose()viceoCapturer.dispose()videoSource.dispose()factory.dispose()localVideoRenderer.dispose()remoteRenderer.dispose()localViewRenderer.release()remoteViewRenderer.release()rootEglBase.release()} **Next Up**Basic socket implementation for webrtc using and video filters for webrtc. ktor.io References https://tech.appear.in/2015/05/25/Introduction-to-WebRTC-on-Android/ https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling

Stories, News & Mentions about ADAPTRACK

Adaptrack’s Stories on HackerNoon