Media Streams and Codecs — Vocabulary

getUserMedia() is the gateway to device media in WebRTC — it returns a MediaStream and lets you specify quality and device preferences via constraints.

Example call: const stream = await navigator.mediaDevices.getUserMedia({ video: { width: { ideal: 1280 }, height: { ideal: 720 }, facingMode: 'user' }, audio: { echoCancellation: true, noiseSuppression: true } }). The constraints are hints, not hard requirements — the browser selects the closest available option. The returned MediaStream contains separate tracks: one video MediaStreamTrack and one audio MediaStreamTrack. Each track can be added to an RTCPeerConnection individually (pc.addTrack(track, stream)) for transmission.

Key vocabulary:
• getUserMedia() — browser API requesting camera/microphone access; returns MediaStream
• MediaStreamTrack — a single audio or video track within a MediaStream; can be enabled/disabled independently
• constraints — video/audio quality and device preferences passed to getUserMedia; treated as hints by the browser

setCodecPreferences() puts the application in control of codec selection — critical for optimising quality and bandwidth for your use case.

VP9 advantages: better compression (30-50% lower bitrate vs. VP8 for equivalent quality), hardware acceleration on modern devices, native support in Chromium-based browsers, Firefox, and Safari (since 2021). VP8 advantages: wider compatibility (older browsers), lower CPU usage on very old hardware. For video conferencing where bandwidth efficiency matters, VP9 is typically preferred. For one-to-one calls on mobile where battery life matters, VP8 may be better due to lower CPU usage. H.264 has the best hardware encoder support (hardware chips in most phones) but has licensing considerations. AV1 is emerging as the next step (better than VP9) but has limited hardware acceleration today.

Key vocabulary:
• setCodecPreferences() — RTCRtpTransceiver method setting codec priority order for SDP negotiation
• VP9 — Google's open-source codec; better compression than VP8; preferred for bandwidth-constrained video
• codec negotiation — offer lists supported codecs in priority order; answer selects the highest-priority mutually supported codec

Simulcast shifts the per-receiver encoding burden from the sender to the SFU — enabling scalable video quality adaptation without overloading the sending browser.

Without simulcast in a group call: if peer A has 10 receivers with different bandwidth, the sender would need to encode 10 different streams at 10 different quality levels. Browsers can't handle this CPU load. With simulcast: the sender encodes 3 layers once (e.g., 1080p at 2Mbps, 720p at 800kbps, 360p at 200kbps). The SFU receives all 3 layers for every participant and selects the best layer per receiver: a user on a good broadband connection gets 1080p; a mobile user on cellular gets 360p; users on moderately good connections get 720p. The SFU adapts in real time as bandwidth changes. Simulcast is enabled via RTCRtpSender encodings with multiple entries, each specifying rid, maxBitrate, and scaleResolutionDownBy.

Key vocabulary:
• simulcast — encoding and sending multiple quality layers simultaneously; SFU selects the right layer per receiver
• RID (Restriction Identifier) — label for each simulcast layer (e.g., 'h', 'm', 'l' for high/medium/low)
• SFU layer selection — SFU switches between simulcast layers based on receiver bandwidth; seamless for the viewer

getDisplayMedia() captures display content with mandatory user picker interaction — the application cannot silently capture the screen without explicit user consent for each session.

Key differences from getUserMedia(): (1) the user must actively choose what to share via the browser's screen picker (entire screen, specific app window, or browser tab) — the application cannot bypass this; (2) no camera-style constraints — you can't specify which screen to capture programmatically; (3) the video track fires the ended event when the user clicks "Stop sharing" in the browser's persistent sharing indicator — the application must listen for this to update UI; (4) recommended settings for screen share: frameRate 5-15fps (screens change less than cameras), high resolution (1920x1080 for readability), cursor capture enabled. Screen share tracks can be added to an existing call using addTrack() and replacing (or adding to) the camera track.

Key vocabulary:
• getDisplayMedia() — captures screen/window/tab; requires browser picker interaction; different from getUserMedia()
• display surface — what is being captured: monitor (entire screen), window (app window), or browser (tab)
• cursor capture — optional setting to include the mouse cursor in the screen capture stream

Opus's built-in FEC and comfort noise make audio remarkably resilient — voice remains intelligible even with 20% packet loss, while video freezes noticeably at 3-5% loss.

Opus FEC mechanism: when encoding a packet, Opus includes a low-bitrate copy of the previous packet in the current packet's header. If a packet is lost, the receiver can partially recover the audio from the FEC data in the next packet. This "in-band FEC" trades a small bandwidth increase (5-10%) for significant loss resilience. Video has no equivalent cheap recovery mechanism — a lost keyframe requires a full refresh (PLI/FIR request), causing visible freezing. Video uses NACK (Negative Acknowledgement) to request retransmission, but retransmission adds latency. This is why in a degraded network, a WebRTC call maintains voice quality while video freezes — by design.

Key vocabulary:
• Opus — mandatory WebRTC audio codec; supports 8-510kbps, 8-48kHz; includes built-in FEC
• in-band FEC — forward error correction data embedded in the Opus packet header; recovers from single-packet loss
• PLI (Picture Loss Indication) — receiver signal requesting the sender to send a new video keyframe after packet loss causes decoding failure