Getting started with WebRTC for Android — Loopback P2P calls

By Vivek Chanddru，Jul 15, 2017

Peer to Peer video calling is on the rage for sometime and now every app has audio/video calling inbuilt. This tutorial series is all about exploring the possibility of providing such feature in an easy and understandable manner for beginners.

This tutorial series is loosely based on the Codelabs for WebRTC.

This part of the series is primarily based on Step 2 of the above mentioned codelab where we use PeerConnection in WebRTC to transfer audio and video data between two peers in a same device.

This is Part 3 of the series “Getting started with WebRTC for Android” and if you are new to this article, please make sure that you have read the previous parts of this series before continuing with this part.

Part 1: Introduction to WebRTC
Part 2: Introduction to PeerConnection
Part 3: Peer-to-Peer Video Calling — Loopback(this article)
Part 4: Peer-to-Peer Video Calling with socket.io

The Concept

Let us consider the following scenario where our courageous boy wishes to call the lady (to propose, of course!)
He plans to use an app which provides awesome video and audio calling feature. He wants to see her reaction when he proposes (awww). The following steps happens at both the ends of the peer connection.
[外鏈圖片轉存失敗,源站可能有防盜鏈機制,建議將圖片保存下來直接上傳(img-TqhergqL-1583665359573)(https://miro.medium.com/max/2288/1*X4iOI4qIKwoC8oK7AFdr6w.jpeg)]

• Initially, our app creates an peer connection and an offer SDP. This offer contains the data about the calling peer and it is used to identify the codecs and other entities of the peer.
• This offer is then stored as “Local Description” at the calling peer and is then sent over to the callee over some signaling mechanism (Typically, most of the systems use Websockets as the signaling medium. It might vary depending on your usage and requirements).
• Once our app at the callee side receives the offer, it knows that there is a call to be established. It will store the “offer” as its “Remote Description” and creates the corresponding “Answer” SDP.
• This answer SDP is similar to the caller’s offer SDP in that it will point out the details specific to that peer.
• The app at the callee side, stores this “Answer SDP” as its “Local Description” and then sends it over through the signaling channel to the caller.
• The caller receives this answer and stores it as its “Remote Description”.
• Caller and Callee then transfers the Ice Candidates pertaining to them through the signaling channel. Upon receiving these candidates, the peer would add these candidates to their PeerConnection instance.
• Once the transfer of Ice candidates are completed, the peers are effectively aware of how to transfer the media data between them. Transfer of media occurs through RTP and is taken care by the WebRTC framework.

Loopback PeerConnection

If you had a look at the Step 2 of WebRTC codelabs, You might notice that they have created a loop between the caller and the callee.

A single device acts as both the local peer and the remote peer thereby doing both offering and answering parts. Though it does not have any practical usage, it is better to pass through this step to have a deep understanding of how stuffs work.

Have a look at the code below to figure out how it is implemented.

public void start() {
    //Initialize PeerConnectionFactory globals.
    //Params are context, initAudio,initVideo and videoCodecHwAcceleration
    PeerConnectionFactory.initializeAndroidGlobals(this, true, true, true);

    //Create a new PeerConnectionFactory instance.
    PeerConnectionFactory.Options options = new PeerConnectionFactory.Options();
    peerConnectionFactory = new PeerConnectionFactory(options);

    //Now create a VideoCapturer instance. Callback methods are there if you want to do something! Duh!
    VideoCapturer videoCapturerAndroid = getVideoCapturer(new CustomCameraEventsHandler());

    //Create MediaConstraints - Will be useful for specifying video and audio constraints.
    audioConstraints = new MediaConstraints();
    videoConstraints = new MediaConstraints();

    //Create a VideoSource instance
    videoSource = peerConnectionFactory.createVideoSource(videoCapturerAndroid, videoConstraints);
    localVideoTrack = peerConnectionFactory.createVideoTrack("100", videoSource);

    //create an AudioSource instance
    audioSource = peerConnectionFactory.createAudioSource(audioConstraints);
    localAudioTrack = peerConnectionFactory.createAudioTrack("101", audioSource);
    localVideoView.setVisibility(View.VISIBLE);

    //create a videoRenderer based on SurfaceViewRenderer instance
    localRenderer = new VideoRenderer(localVideoView);
    // And finally, with our VideoRenderer ready, we
    // can add our renderer to the VideoTrack.
    localVideoTrack.addRenderer(localRenderer);
}

private void call() {
    //we already have video and audio tracks. Now create peerconnections
    List<PeerConnection.IceServer> iceServers = new ArrayList<>();

    //create sdpConstraints
    sdpConstraints = new MediaConstraints();
    sdpConstraints.mandatory.add(new MediaConstraints.KeyValuePair("offerToReceiveAudio", "true"));
    sdpConstraints.mandatory.add(new MediaConstraints.KeyValuePair("offerToReceiveVideo", "true"));

    //creating localPeer
    localPeer = peerConnectionFactory.createPeerConnection(iceServers, sdpConstraints, new CustomPeerConnectionObserver("localPeerCreation") {
        @Override
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            onIceCandidateReceived(localPeer, iceCandidate);
        }
    });

    //creating remotePeer
    remotePeer = peerConnectionFactory.createPeerConnection(iceServers, sdpConstraints, new CustomPeerConnectionObserver("remotePeerCreation") {

        @Override
        public void onIceCandidate(IceCandidate iceCandidate) {
            super.onIceCandidate(iceCandidate);
            onIceCandidateReceived(remotePeer, iceCandidate);
        }

        @Override
        public void onAddStream(MediaStream mediaStream) {
            super.onAddStream(mediaStream);
            gotRemoteStream(mediaStream);
        }
    });

    //creating local mediastream
    MediaStream stream = peerConnectionFactory.createLocalMediaStream("102");
    stream.addTrack(localAudioTrack);
    stream.addTrack(localVideoTrack);
    localPeer.addStream(stream);

    //creating Offer
    localPeer.createOffer(new CustomSdpObserver("localCreateOffer"){
        @Override
        public void onCreateSuccess(SessionDescription sessionDescription) {
            //we have localOffer. Set it as local desc for localpeer and remote desc for remote peer.
            //try to create answer from the remote peer.
            super.onCreateSuccess(sessionDescription);
            localPeer.setLocalDescription(new CustomSdpObserver("localSetLocalDesc"), sessionDescription);
            remotePeer.setRemoteDescription(new CustomSdpObserver("remoteSetRemoteDesc"), sessionDescription);
            remotePeer.createAnswer(new CustomSdpObserver("remoteCreateOffer") {
                @Override
                public void onCreateSuccess(SessionDescription sessionDescription) {
                    //remote answer generated. Now set it as local desc for remote peer and remote desc for local peer.
                    super.onCreateSuccess(sessionDescription);
                    remotePeer.setLocalDescription(new CustomSdpObserver("remoteSetLocalDesc"), sessionDescription);
                    localPeer.setRemoteDescription(new CustomSdpObserver("localSetRemoteDesc"), sessionDescription);
                }
            },new MediaConstraints());
        }
    },sdpConstraints);
}

private void hangup() {
    localPeer.close();
    remotePeer.close();
    localPeer = null;
    remotePeer = null;
}

private void gotRemoteStream(MediaStream stream) {
    //we have remote video stream. add to the renderer.
    final VideoTrack videoTrack = stream.videoTracks.getFirst();
    AudioTrack audioTrack = stream.audioTracks.getFirst();
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            try {
                remoteRenderer = new VideoRenderer(remoteVideoView);
                remoteVideoView.setVisibility(View.VISIBLE);
                videoTrack.addRenderer(remoteRenderer);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    });
}

public void onIceCandidateReceived(PeerConnection peer, IceCandidate iceCandidate) {
    //we have received ice candidate. We can set it to the other peer.
    if (peer == localPeer) {
        remotePeer.addIceCandidate(iceCandidate);
    } else {
        localPeer.addIceCandidate(iceCandidate);
    }
}

If you take a look at the above code, We have 3 methods.

start() method essentially creates the local audio and video source and adds them to a SurfaceViewRenderer (this is provided by WebRTC so that we can draw our video frames directly to this view).

hangup() method is just a simple piece of code that clears off all the PeerConnection instances

call() is where all the fun happens.

• We create two peer connection instances (one for local peer and another for remote peer). Once those peers are created, we make the local peer create an Offer that is set as its Local Description and also the Remote Description of the remote peer.
• We then make the remote peer to create answer which is set as its Local Description and the local peer’s Remote Description.

We also have the onIceCandidateReceived() method whose work is to set the Ice candidates received from one peer to another peer.

You can take a look at Step-2 folder of the Git repository for the full working code of Loopback peerconnection.

All is going well now. Our app will be able to show your face to you, just like we did in the first part. Except that, this time our app is showing the face through a peer connection.

We got the basics working. We have two peers which transfer the offer and answer SDPs along with the candidates and now they both can transfer the data. But this is not the real case. Real people call each other and not themselves. Extending this solution to a real use case is easy. We just need a medium to transfer the SDPs and some STUN and TURN setup.

This has become too large for a single post. See you soon(hopefully!) with the grand finale where you will have a working Peer-to-Peer enabled video calling app.