Real time communication with Webrtc on Android

WebRTC is a free, open project that provides browsers and mobile applications with Real-Time Communications (RTC) capabilities via simple APIs.

Webrtc is a cross platform solution with RTC capabilities. One can stream his own video stream be it from camera or screen recording or any other video to any peer via webrtc. If one is developing a real time peer to peer game with real time data sharing between the peers, webrtc is one of the options. Let’s understand how video calling from one peer to another works in webrtc.

Getting started

Since September 2017 Google started to distributed precompiled versions of webrtc for android via maven. If you want to play with the source and compile it yourself you can do it easily as per these steps(Earlier you could compile it only on linux but now all the three major OS are supported). To use the pre compiled version simply add the following dependency.

compile 'org.webrtc:google-webrtc:1.0.22379'

How does webrtc work?

Before we start exchanging data between two peers via webrtc, we need to provide info to the peers about each other for media format negotiation and discovery. This is done via following protocols. Interactive Connectivity Establishment (ICE) is used for connecting peer to peer. Session Description Protocol (SDP) is used to provide the metadata of the media content like resolution, encoding, bitrate, etc. If the two peers are not on same network then we will need to provide a Session Traversal Utilities for NAT (STUN) server to provide the public address of the peers. If any of the network is firewall protected the we need to provide Traversal Using Relays around NAT (TURN) servers also. You can learn more about these protocols here. The mechanism to exchange ICE and SDP between the peers is called Signalling System and is usually done via websockets.

Webrtc android API

Most of our webrtc code will make use of PeerConnection and PeerConnectionFactory apis. PeerConnection is the equivalent of RTCPeerConnection in web world and is used to establish peer to peer connection. PeerConnectionFactory is used to create PeerConnection, MediaStream and MediaStreamTrack objects_._

**Creating PeerConnectionFactory**Before creating the factory object we need to initialize webrtc. As you can InitializationOptions here provides options for enabling/disabling hardware accelerations(I had to disable in on certain devices) and setting field trials. Field trials are for enabling experimental webrtc features.

val fieldTrials = (PeerConnectionFactory.VIDEO_FRAME_EMIT_TRIAL + "/" + PeerConnectionFactory.TRIAL_ENABLED + "/")val options = InitializationOptions.builder(application).setFieldTrials(fieldTrials).setEnableVideoHwAcceleration(videoAccelerationEnabled).createInitializationOptions()PeerConnectionFactory.initialize(options)factory = PeerConnectionFactory(PeerConnectionFactory.Options())val rootEglBase = EglBase.create()factory?.setVideoHwAccelerationOptions(rootEglBase.eglBaseContext, rootEglBase.eglBaseContext)

**Creating Media Streams**Once we have the PeerConnectionFactory object we can now create a MediaStream object which has AudioTrack and VideoTrack associated with it.

val localMediaStream = factory.createLocalMediaStream(MEDIA_ID)val audioSource = factory.createAudioSource(MediaConstraints())val audioTrack = factory.createAudioTrack(AUDIO_ID, audioSource)localMediaStream.addTrack(audioTrack)

videoCapturer = if (Build.VERSION.SDK_INT > Build.VERSION_CODES.LOLLIPOP) {createCameraCapturer(Camera2Enumerator(application))} else {createCameraCapturer(Camera1Enumerator(videoAccelerationEnabled))}val videoTrack = factory.createVideoTrack("VideoTrack", factory.createVideoSource(videoCapturer))localMediaStream.addTrack(videoTrack)

CameraVideoCapturerWebrtc provides us a very easy way to use Camera and Camera2 API depending on the support. On supported devices we can use either of the APIs

private fun createCameraCapturer(enumerator: CameraEnumerator): CameraVideoCapturer? {val deviceNames = enumerator._deviceNames_ for (deviceName in deviceNames) {if (enumerator.isFrontFacing(deviceName)) {val videoCapturer = enumerator.createCapturer(deviceName, null)if (videoCapturer != null) {return videoCapturer}}}

for (deviceName in deviceNames) {if (!enumerator.isFrontFacing(deviceName)) {Timber.d("Creating other camera capturer.")val videoCapturer = enumerator.createCapturer(deviceName, null)if (videoCapturer != null) {return videoCapturer}}}return null}

**SurfaceViewRenderer**Since we have the local MediaStream ready, we need to render it on a view so that its visible to the user. SurfaceViewRenderer is a View in webrtc libray that does the rendering of webrtc frames for us. We can directly add it in layout xml.

localViewRenderer.init(rootEglBase.getEglBaseContext(), null)localViewRenderer.setEnableHardwareScaler(true)localViewRenderer.setMirror(true)localViewRenderer.setScalingType(ScalingType.SCALE_ASPECT_FILL)val localVideoRenderer = VideoRenderer(localViewRenderer)videoTrack.addRenderer(localVideoRenderer)

PeerConnectionWe have everything ready to transmit now, its time to create the PeerConnection. We pass in a PeerConnection#Observer instance in the factory method, which is notified when a ICE candidate is generated (we need to send that to the other peer). The observer is also notified when the remote MediaStream is available. we will attach a renderer to it just like the local MediaStream. It also requires a list IceServer(STUN and TURN servers) which can be empty if testing on local network

val peerConnectionObserver = object : PeerConnection.Observer {override fun onIceCandidate(iceCandidate: IceCandidate) {localIceCandidatesSource.onNext(iceCandidate)}override fun onAddStream(mediaStream: MediaStream) {mediaStream.addRenderer(remoteRenderer)}...}peerConnection = factory?.createPeerConnection(getIceServers(), peerConnectionObserver)

NOTE: In Observer’s onIceCandidate we shouldn’t add the IceCandidate to PeerConnection (peerConnection.addIceCandidate(iceCandidate) till we have the remote SDP

CreateOffer/CreateAnswerOnce the PeerConnection is created we need to initiate video call. The peer which initiates the call will createOffer and set its local SDP (peerConnection#setLocalSdp). When the local SDP is set it will send that SDP to the other peer which will set its remote SDP (peerConnection#setRemoteSdp). Once the remote SDP is set it will createAnswer with that sdp and set its local sdp when asnwer is created and send the local sdp to the peer which created the offer. The initiator peer will now set its remote SDP.

fun createOffer() {peerConnection.createOffer(object : SdpObserver {override fun onCreateSuccess(sdp: SessionDescription) {setLocalSdp(sdp)}...}, getPeerConnectionConstraints())}fun setLocalSdp(sdp: SessionDescripton) {peerConnection.setLocalDescription(object : SdpObserver {override fun onSetSuccess() {api.sendSdp(peerConnection.localDescription)drainIceCandidates()}...}, sdp)}

fun onOffer(sdp: SessionDescription) {peerConnection.setRemoteDescription(object : SdpObserver {override fun onSetSuccess() {createAnswer()}}, sdp)}

fun createAnswer() {peerConnection?.createAnswer(object : SdpObserver {override fun onCreateSuccess(sdp: SessionDescription) {setLocalSdp(sdp)}}, getPeerConnectionConstraints())}

**Dispose/ Clean up**When the peers decide to end the connection its pretty important to do clean up in particular order as the C layer does a reference count check and will crash with assertion failures if the objects are not properly disposed.

fun cleanUp() {peerConnection.dispose()viceoCapturer.dispose()videoSource.dispose()factory.dispose()localVideoRenderer.dispose()remoteRenderer.dispose()localViewRenderer.release()remoteViewRenderer.release()rootEglBase.release()}

**Next Up**Basic socket implementation for webrtc using ktor.io and video filters for webrtc.

References

• https://tech.appear.in/2015/05/25/Introduction-to-WebRTC-on-Android/
• https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling