webrtc-the-video-parts.txt

# WebRTC: The video bits

Quality of Service is a difficult problem in WebRTC. You spend hours perfecting
your design in the lab. You test in different network conditions and try your best.
A week later the reports trickle in of different customers having unreproducable issues.
Then when you go to research you can't even find consensus on how to solve these problems.
Most companies don’t discuss how they do it, and those that do don’t agree.

This talk is about my time building media experiences with WebRTC.  The lessons I
have learned, and mistakes along the way. This talk doesn’t discuss signaling, connectivity
or security. If you are interested in those topics check out WebRTC for the Curious.

# Nothing can make up for a bad call
* Good UI, Easy Signup etc... Lack of QoS will hit you eventually

* So... packetize the media and send it, what’s the problem?!

# Constraints of the real world
  - Video is huge
    * Uncompressed 1280x720 8-Bit needs ~80 MB/s
  - Codecs
    * Compressed video needs careful transport
    * Support differs everywhere
  - Network
    * Congestion
    * Errors

# Video Compression
  - intra frame vs inter frame

  - Picture Types
    * Intra-code Frame (Keyframe)
    * Predicted Frame
    * Bi-directional Predicted Frame
  (Include picture from Wikipedia)

  - keyframe interval

# Real World Networks
  - Congestion
    * Can manifest as loss or latency
    * Every network is different
  - Jitter
  - Networks are dynamic
  - Errors can be unexpected at every step
    * Wifi
    * Gateways
    * ....

# How does WebRTC transport media
* RTP
  - Sent over UDP
  - Each packet has a header with follow details
    * SSRC (what session is this)
    * Sequence Number (unique id rolls over)
    * TimeStamp (relative, random start
    * Extensions (can add w/e you want)

* RTCP
  - Used to send control messages about media
  - Can send w/e you want, but a common subset is what everyone uses

# Fixing Congestion
  - Adjust the encoder to use the amount of bandwidth we have available

  - We need to use a congestion control to figure out bandwidth!

  - `Google Congestion Controller`, `NADA`, `SCReAM`....

# Flow of congestion controller
  - Slow start, then probe

  - Occasionally we probe again, taking into account Loss+RTT


  - Taken from BBR RFC (TCP Congestion Controller)
```
                |
                V
       +---> Startup  ----+
       |        |         |
       |        V         |
       |      Drain   ----+
       |        |         |
       |        V         |
       +---> ProbeBW -----+
       |      ^    |      |
       |      |    |      |
       |      +----+      |
       |                  |
       +---- ProbeRTT <---+
```

# How do we get data for the congestion controller?
  - Old way, Receiver Reports
    * Gives percentage loss and jitter
    * Doesn't give per packet RTT, hard to catch bufferbloat

  - New way Transport Wide Congestion Control
    * Gives you exactly what packets were lost
    * Also tells you exactly when they arrived

  - Using this data we can drive our congestion controller

# Fixing Jitter
  - If we play as they arrive we will have stuttering

  - JitterBuffer
    * Trade-off between latency and quality
    * This concept will be re-used in the future

# Fixing Audio/Video sync
  - Buffering/Network messes up your timing

  - Sender Reports to the rescue
    * Align relative RTP timestamps with NTP timestamps

# Fixing loss
  - Some loss is unavoidable even without congestion
    * Wifi

  - Not all loss is equal
    * Do you care about a P-Frame right before an I-Frame?

# Fixing loss by reacting
  - RTCP Negative Acknowledgments (NACK)
    * SequenceNumber `I` of SSRC `J` was lost
    * Re-use that JitterBuffer and add a delay to wait

  - Also know as Automatic repeat request (ARQ)

# Fixing loss pre-emptively
  - Forward Error Correction (FEC)
    * Redundant Data is sent in another stream
    * Is dynamic/protects the important bits

  - Less RTT then NACK, but wasting bandwidth

# QoS and WebRTC Servers
  - Simulcast
    * Send multiple video feeds (high, medium, low)
    * Server then distributes the feed to each viewer

  - Scalable Video Coding
    * Layered coding
    * high-quality video stream that contains one or more subset streams

# My time with RTP
  - It's possible to ship without all this
    * Don't let perfect be enemy of the good
    * Do a high keyframe interval and ship

  - Always something you can improve

# My time with RTP
  - Moving any knob can effect the others
    * Congestion equals more NACKs, more NACKs equals more congestion....

  - Lab/Demos are fun, but not very useful
    * Network simulators are determinstic
    * User metrics are what really matter