|
| 1 | +# Decoder Error |
| 2 | + |
| 3 | +**Authors** |
| 4 | +* [Nishitha Burman Dey](https://github.com/nishitha-burman) |
| 5 | +* [Steve Becker](https://github.com/SteveBeckerMSFT) |
| 6 | +* [Diego Perez Botero](https://github.com/Diego-Perez-Botero) |
| 7 | +* [Philipp Hancke](https://github.com/fippo) |
| 8 | + |
| 9 | +Much of this explainer synthesizes and consolidates prior discussions and contributions from members of the WebRTC working group. |
| 10 | + |
| 11 | +## Participate |
| 12 | +* [Issue tracker](https://github.com/MicrosoftEdge/MSEdgeExplainers/labels/DecoderError) |
| 13 | +* [Discussion forum](https://github.com/w3c/webrtc-extensions/issues/146) |
| 14 | + |
| 15 | +## Introduction |
| 16 | +Game streaming platforms like Xbox Cloud Gaming and Nvidia GeForce Now rely on hardware decoding in browsers to deliver low-latency, power efficient experiences. However, there is currently no reliable way for these applications to detect when decoding silently falls back to software during a stream. |
| 17 | + |
| 18 | +This proposal introduces a runtime event to notify applications when a decoder error or fallback occurs. The goal is to give developers actionable visibility into runtime behavior without exposing new fingerprinting vectors or hardware details. |
| 19 | + |
| 20 | +## User-Facing Problem |
| 21 | +End users of game streaming services may experience increased latency, degraded quality, and battery drain when the browser switches from hardware to software decoding. Developers currently lack a way to detect this fallback in real time without prompting users for camera/mic permissions. In the past, developers used to rely on [`decoderImplementation`](https://w3c.github.io/webrtc-stats/#dom-rtcinboundrtpstreamstats-decoderimplementation) info, but as of Chromium M110+ it requires [`getUserMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) permissions. This is not ideal because the UI prompt is invasive, it’s excessive since it grants access to the camera and mic hardware when apps don’t need it, and it has a high failure rate since users have little reason to grant the permission unless they want to use voice chat. This gap makes it difficult to diagnose performance regressions and provide troubleshooting guidance. |
| 22 | + |
| 23 | +## Goals |
| 24 | +* Enable developers to detect runtime decoder fallback from hardware to software in a non-invasive way without requiring additional permissions (does not require `getUserMedia()` permissions). |
| 25 | +* Allow applications to diagnose regressions (e.g. codec negotiation issues, device specific problems). |
| 26 | +* Support user experience improvements by enabling apps to adapt (e.g. lowering resolution, re-negotiating codecs), alerting end users when software decode occurs, and displaying troubleshooting information. |
| 27 | + |
| 28 | +## Non-goals |
| 29 | +* Exposing vendor-specific hardware information. |
| 30 | +* Exposing deterministic codec support/capabilities beyond what [`MediaCapabilities`](https://developer.mozilla.org/en-US/docs/Web/API/Media_Capabilities_API) already provides. |
| 31 | +* Providing detailed telemetry such as frame-level error counts or decoder identifiers. |
| 32 | + |
| 33 | +## User Research |
| 34 | +Feedback from Xbox Cloud Gaming, Nvidia GeForce Now and similar partners shows: |
| 35 | +* Fallback is common in the field, and developers lack visibility into when/why it occurs. |
| 36 | +* Reliance on `getUserMedia()` to query `decoderImplementation` has a high failure rate because users often deny permissions that are irrelevant to media playback. |
| 37 | +* Previous workarounds (e.g. guessing based on decode times) have proven unreliable and masked bugs. |
| 38 | +* Relying on `MediaCapabilities` is insufficient because it only provides a static capability hint and does not reflect what happens at runtime, for example, when hardware decode fails mid-session and the browser silently falls back to software. |
| 39 | +* Without this signal, developers cannot confidently diagnose or reduce fallback incidence. |
| 40 | + |
| 41 | +## Proposed Approach |
| 42 | +Introduce an event on [`RTCRtpReceiver`](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpReceiver) ([see slide 30](https://docs.google.com/presentation/d/1FpCAlxvRuC0e52JrthMkx-ILklB5eHszbk8D3FIuSZ0/edit?slide=id.g2452ff65d17_0_71#slide=id.g2452ff65d17_0_71)) that fires when a decoder error occurs: |
| 43 | +* The engine falls back from hardware to software decoding |
| 44 | +* No software decoder is available (e.g. in the case of H.265) |
| 45 | + |
| 46 | +This enables applications to alert users, re-negotiate codecs, and debug issues without requiring [`getUserMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) permissions. |
| 47 | + |
| 48 | +### Example |
| 49 | + |
| 50 | +```JavaScript |
| 51 | +const pc = new RTCPeerConnection(); |
| 52 | + |
| 53 | +pc.addEventListener('track', (event) => { |
| 54 | + const receiver = event.receiver; |
| 55 | + |
| 56 | + // Listen for decoder state changes |
| 57 | + receiver.addEventListener('decoderstatechange', (ev) => { |
| 58 | + |
| 59 | + // Adapt application behavior based on power efficiency |
| 60 | + if (!ev.powerEfficient) { |
| 61 | + // Notify the user |
| 62 | + showToast("Playback quality may be reduced"); |
| 63 | + |
| 64 | + // Lower resolution or disable heavy post-processing |
| 65 | + adjustQuality('low'); |
| 66 | + |
| 67 | + // Log telemetry signal with codec and RTP timestamp |
| 68 | + logMetric(`Decoder fallback: codec=${ev.codecString}, rtp=${ev.rtpTimestamp}`); |
| 69 | + } |
| 70 | + }); |
| 71 | +}); |
| 72 | + |
| 73 | +``` |
| 74 | +### Proposed IDL |
| 75 | + |
| 76 | +```JavaScript |
| 77 | +partial interface RTCRtpReceiver { |
| 78 | +attribute EventHandler ondecoderstatechange; |
| 79 | +}; |
| 80 | + |
| 81 | +interface RTCDecoderStateChangeEvent : Event { |
| 82 | +constructor(DOMString type, RTCDecoderStateChangeEventInit eventInitDict); |
| 83 | + |
| 84 | +// Media timeline reference |
| 85 | +readonly attribute unsigned long rtpTimestamp; |
| 86 | + |
| 87 | +// Codec now in effect after the change. |
| 88 | +readonly attribute DOMString codecString; |
| 89 | + |
| 90 | +// Align with MediaCapabilitiesInfo, powerEfficient changes primarily based on hardware/software decoder |
| 91 | +// https://www.w3.org/TR/media-capabilities/#media-capabilities-info |
| 92 | +readonly attribute boolean powerEfficient; |
| 93 | +}; |
| 94 | +``` |
| 95 | +**Note:** The event fires at the beginning of streaming. |
| 96 | +
|
| 97 | +## Alternatives Considered |
| 98 | +1. Use [`decoderImplementation`](https://w3c.github.io/webrtc-stats/#dom-rtcinboundrtpstreamstats-decoderimplementation) info via WebRTC Stats API |
| 99 | + * Rejected because it now requires [`getUserMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) permissions, which are invasive and have a high failure rate. |
| 100 | + * Requires unnecessary permissions (camera/microphone). |
| 101 | +2. Use [`MediaCapabilitiesInfo.powerEfficient`](https://www.w3.org/TR/media-capabilities/#media-capabilities-info) |
| 102 | + * Rejected because this is a static hint, not a runtime signal. |
| 103 | + * Does not update when the browser silently switches from hardware to software. |
| 104 | +3. Guess based on decode times |
| 105 | + * Unreliable and has masked bugs in production. |
| 106 | +4. [Add `decoderFallback` field to `RTCInboundRtpStreamStats`](https://github.com/w3c/webrtc-stats/pull/725) |
| 107 | + * Rejected because relying on stats to trigger a change felt like an anti-pattern and the recommendation was to explore an event driven solution. Additionally, there were concerns around fingerprinting. |
| 108 | + * [WebRTC March 2023 meeting – 21 March 2023](https://www.w3.org/2023/03/21-webrtc-minutes.html) |
| 109 | +
|
| 110 | +## Privacy Considerations |
| 111 | +* The event does not expose hardware vendor or device identity, reducing fingerprinting risk. |
| 112 | +* Does not reveal deterministic codec/hardware support. |
| 113 | +
|
| 114 | +### Counter-argument to fingerprinting concerns: |
| 115 | +* **Information is already exposed via Media Capabilities**: Hardware/software decode status is already partially exposed via the [`MediaCapabilitiesInfo.powerEfficient`](https://www.w3.org/TR/media-capabilities/#media-capabilities-info) attribute. A “common implementation strategy” is to treat hardware usage as indicative of optimal power draw. |
| 116 | +* **Fallback doesn’t directly reveal capability**: The fallback event does not deterministically expose hardware support, as software fallback may occur for various reasons, making it a dynamic and contextual signal rather than a static fingerprint. Software fallback may occur because: |
| 117 | + * Device lacks hardware support for the specific codec. |
| 118 | + * The hardware decoder is temporarily unavailable. |
| 119 | +
|
| 120 | +## Stakeholder Feedback |
| 121 | +* Web Developers: Positive |
| 122 | + * [Xbox Cloud Gaming](https://github.com/w3c/webrtc-stats/pull/725#discussion_r1093134014) & Nvidia GeForce Now have direct use cases. |
| 123 | +* Chromium: Positive; actively pursuing proposal. |
| 124 | +* WebKit & Gecko: Overall positive feedback, but privacy/fingerprinting is a common concern. |
| 125 | +
|
| 126 | +Last discussed in the 2025-09-16 WebRTC WG Call: [Slides 17-21](https://docs.google.com/presentation/d/11rr8X4aOao1AmvyoDLX8o9CPCmnDHkWGRM3nB4Q_104/edit?slide=id.g37afa1cfe47_0_26#slide=id.g37afa1cfe47_0_26) & [minutes](https://www.w3.org/2025/09/16-webrtc-minutes.html) |
| 127 | +
|
| 128 | +## References & Acknowledgements |
| 129 | +Many thanks for valuable feedback and advice from: |
| 130 | +* [Nic Champagne Williamson](https://github.com/champnic) |
| 131 | +* [Gabriel Brito](https://github.com/gabrielsanbrito) |
| 132 | +* [Henrik Boström](https://github.com/henbos) |
| 133 | +* [Sun Shin](https://github.com/xingri) |
| 134 | +
|
| 135 | +Links to past working group meetings where this has been discussed: |
| 136 | +* 2025-09-16 WebRTC WG Call: [Slides 17-21](https://docs.google.com/presentation/d/11rr8X4aOao1AmvyoDLX8o9CPCmnDHkWGRM3nB4Q_104/edit?slide=id.g37afa1cfe47_0_26#slide=id.g37afa1cfe47_0_26) & [minutes](https://www.w3.org/2025/09/16-webrtc-minutes.html) |
| 137 | +* 2023-09-15 WebRTC WG Call: [Slides 25-31](https://docs.google.com/presentation/d/1FpCAlxvRuC0e52JrthMkx-ILklB5eHszbk8D3FIuSZ0/edit?slide=id.g2452ff65d17_0_71#slide=id.g2452ff65d17_0_71) & minutes |
| 138 | +* 2023-03-21 WebRTC WG Call: [Slides 16-18](https://lists.w3.org/Archives/Public/www-archive/2023Mar/att-0004/WEBRTCWG-2023-03-21.pdf) & [minutes](https://www.w3.org/2023/03/21-webrtc-minutes.html) |
| 139 | +
|
0 commit comments