Skip to content

Instantly share code, notes, and snippets.

@mayanktg

mayanktg/GSoC_proposal.md

Last active August 29, 2015 13:57
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mayanktg/9415556 to your computer and use it in GitHub Desktop.
Save mayanktg/9415556 to your computer and use it in GitHub Desktop.
Download ZIP
Instantbird: WebRTC support
Raw
GSoC_proposal.md

Personal Details

  • Name: Mayank Kumar
  • Email: mayanktg@gmail.com
  • IRC Nic: mayanktg
  • Telephone: +91 9424611674
  • Other contact methods:
  • Country of Residence: India
  • Timezone: UTC +5:30
  • Primary Language: English

Project Proposal

Introduction

The project aims in adding voice and video support to the Instantbird chat keeping in mind the existing user interface of the application and thus have a seamless integration. WebRTC makes it easy for us to have real-time communication, audio & video to mention, without the use of additional plugins.

Important points to consider:

  • How will the integration take place? What are the prerequisites, which part needs to be changed, using the WebRTC API and other details.
  • How to have a minimal and easy user interface which doesn't bring major changes to current settings.
  • What could possibly go wrong? What problems may arise as a result of this feature for the users?

I will try to answer these questions in the best possible way in my proposal.

Current Research (PeerConnection)

Once the media is obtained using the mozGetUserMedia, RTCPeerConnection is used to send this to others. After the caller and callee initialization is done we need to connect the two remote servers. For this WebRTC uses ICE to connect to the peers. WebRTC needs server support for the following functionality:

  • User Discovery
  • Signaling
  • Firewall/NAT traversal
  • Relay Servers in case P2P communication fails

User Identification: The media data should be sent to that address on the relay server (i.e. of the callee) that has been allocated to be used by the caller.

Signaling using XMPP: Apart from streaming data, to send control signals we need a process called as signalling which isn't specified by WebRTC. For this we have to use a two way messaging protocol such as XMPP. XMPP uses XEP-0320 specification to explain use of DTLS-SRTP for Real-Time Transport Protocol (RTP). Apart from media signaling authentication and security is also handled by XMPP.

Firewall and NAT Traversal: ICE tries to connect to the peers directly but if connection failure occurs due to NAT, then STUN server is needed. If it happens because of Firewall TURN server is needed. The latest Firefox has an enabled STUN and TURN server by default and a domain stun.services.mozilla.com for the ICE.

For further information:
https://www.webrtc-experiment.com/docs/webrtc-for-newbies.html
https://www.webrtc-experiment.com/docs/WebRTC-PeerConnection.html
http://www.slideshare.net/amiteshawa/web-rtc-media-stra
http://io13webrtc.appspot.com/#40

Implementation (via XMPP)

The project is implemented using Jingle which specifies how to use one-to-one, peer-to-peer "media" sessions between XMPP entities. For the media exchange we have to use a non-XMPP transport, here its WebRTC (support explained in XEP-0320). This has been simplified and divided into following steps:

1. Media Detection:

When a user starts a chat, the video/audio recording device is to be detected on both sides and the video/voice call support is to be enabled/disabled only when both ends have the device ready. For this we can use navigator.mozGetUserMediaDevices to detect device and send the information to each other about the device status whenever chat window is opened.

2. Initiate call session:

A call request is made by the caller and a notification about this to accept or reject call is sent to the callee. As per the response provided an ACK is sent to the caller. If call request is accepted Ack is sent to the callee and the next step is to be proceeded, else the call disconnects.

3. The call:

Now since call has been connected a video frame is created for receiving the video call. This is implemented using RTCPeerConnection, which handles stable and efficient communication of streaming data between peers.

4. Call Disconnect:

The call gets disconnected if

  • the caller/callee ends the connection by clicking the "End Call" button.
  • the chat window of either looses connection or is closed.

Which parts of Instantbird will be effected?

  • Firstly we will focus in adding it for XMPP. The code for XMPP protocol lies in the chat/protocols/xmpp directory of Instantbird code-base and similarly for all other protocols lies inside chat/protocols/.
  • We will have to add the interface too, which is found in the im/content directory. Here the conversation.xml file is to be touched since it is the one concerned with the conversation window we will be using.
  • Notifications should be added when a user receives a call request.
  • Add logging of call events. This can be again found inside im/content/. This is to make the details about the call registered in the chat logs. Details about the call duration, start and end of call should be added.
  • Also we would be supposed to write tests to check for the proper functioning of the project.

Design Prototype

Corner Cases

  • The voice/audio call is only possible between the users using WebRTC enabled Instantbird. So in the upcoming versions WebRTC should be enabled by default.
  • Currently Mozilla doesn't supports mediaConstraints, i.e. the ability to restrict video sizes. This is important for controlling the bandwidth usage, but not necessary requirement.
  • We should use different notification methods as per the current mood of the user such that he doesn't gets distracted in the middle of an important work.
  • Protocols such as Facebook Chat, GTalk, Odnoklassniki and Vkontakte inherit and customize XMPP. We should check that the implementation is working for these protocols too.

Schedule of Deliverables

The project can be divided into following subparts

  1. media detection
  2. creating the platform to use RTCPeerConnection
  3. Video call support
  4. Audio calll support
  5. customizing the UI to smoothly integrate with current Instantbird.

Below is the proposed timeline of the deliverables for this project.

Apr 21st - May 18th (Community Bonding): This one month duration is set by Google as a bonding period with the organization. So I'll learn about the code of Instantbird, organization policies and also study in further details about the project. The discussion about the design of the required interface would be finalized and properly drafted.

1-2 Week (May 19th - June 2nd): During first two week of coding period, I will set up the media detection as mentioned above, add calling buttons and create a platform to use the RTCPeerConnection so that the media transmitted from one end can be successfully received at the other end.

3-5 Week (June 3rd - June 24th): Add the basic functional video call support with ability to make, receive and end call (i.e. use WebRTC Peer Connection API --> write one-to-one video sharing --> use XEP-0166 for media signaling). Add options to mute audio, enable/disable microphone and camera. Add notification options. Perform integration test. This can be considered as the first patch. Also this marks the mid term evaluation.

6-7 Week (June 25th - July 9th): Similarly add audio call functionality to the chat.

8-9 Week (July 10th - July 24th): Add logging of call events, better notification options, other XMPP based protocols would be studied and customization needed to support them will be done and also add a setting option to view the default device being used.

10th Week (July 25th - Aug 1st): Work on UI improvement. A stable interface to be made for both audio and video calling. Prepare documentation for the project.

11th Week (Aug 2nd - Aug 9th): This part is dedicated to the complete testing of the components. Bugs if any shall be fixed. Integration testing to be performed.

12th Week (Aug 10th - Aug 18th): Buffer week. I will keep it reserved if certain code requires improvement or needs to be modified. This will ensure the best code is going in the code-base and a final working build of Instantbird is created.

NOTE: If the project is completed ahead of the schedule, the work of WebRTC implementation for other protocols will be started.

Open Source Development Experience

  • I have open sourced most of my college projects and technology club works so that other students can review, improve the code and get benefited.
  • I volunteered WordCamp Jabalpur 2011 (Unconference for Wordpress development). Here everyone from casual users to core developers from all around the country participated and shared ideas.
  • I have done a bug fix for Instantbird which is included in the Nightlies.
    • Bug 954216: Add context menu to set/remove user icon.
    • Bug 975542: Currently working on this. Adding user icon by taking picture using webcam.

Work/Internship Experience

I have done two internships before, one at an early stage start-up Verloop and second one at an enterprise Venturesity.

  • Verloop (March 2012 - May 2012): It was a Non-Profit Organization which equipped State funded Schools of Central India with a Wikipedia like Verloop DVD comprising of articles, images and videos to assist learning.
  • Venturesity (May 2013 - June 2013): It is a US cum Bangalore based Education Company working on providing quality courses in the field of business and technology delivered online (live tutoring) and as classroom courses. It also helps in simplifying the hiring process. My job was to create the website for their learning platform. Also to implement their stealth mode product in which WebRTC is to be used as an LMS and a hiring tool to conduct coding interviews.

Academic Experience

I am a third year undergraduate Computer Science student pursuing my Bachelor of Engineering degree from Jabalpur Engineering College, India. I have loved programming since my early school days. I'm in the core team of college's Technical Learning Group, [Curiosity Club] (http://www.jec-jabalpur.org/curiosity/) and I also manage the web part of the Training and Placement team's work. I presented a National level paper on "World Unique Identification Card".

  • I built an on-line payslip viewer for the college staffs so that they can manage their monthly salary and Form-16 efficiently.
  • Technical Head, JEC Photographic Society (Feb, 2012 - Mar, 2013): I was the head of the Technical team of college's 41 year old Photographic Society.

Why Me

I have been actively learning the Instantbird code-base and have an understanding of how the product's development happen. I have worked on WebRTC during my intern and I have a decent understanding of the API.

Why Mozilla

Mozilla has always been a supporter of an open web and also open source education. In a country where most of the students have to haul for proper education, I see Mozilla as a medium to provide good essential technical education to the individuals. I want to work to work with Mozilla so as to make education a medium to change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment