v4.1.0

Release v4.1.0 (2024-07-08)

Quick Links:
📖 API documentation - ⏯ Demo - 🎓 Migration guide from v3

• 🔍 Overview
• 📑 Changelog
• DASH ContentProtection References
• MULTI_THREAD feature improvements
• Better DASH URL path resolution
• DRM: Comprehensively test DRM compatibility on Edge
• DASH/DRM: All 0 key-id now ignored in container files
• A lot of transparent large project updates

🔍 Overview

We're now releasing the v4.1.0.
This release adds multiple fixes and improvements:

• it improves our MULTI_THREAD experimental feature allowing to run most of the RxPlayer logic in another thread - which we're now using on the great majority of devices at Canal+.

• This release also adds support for DASH Content Protection References, which can greatly reduce the size and thus improve parsing time of Multi-Period Manifest with complex DRM configurations.

• After an issue report, we noticed that our DASH URL resolution algorithm did not handle all cases. We thus rewrote it so it completely respect the corresponding standard (the RFC 3986).

• For the Edge browser, after having multiple PlayReady-specific issues, we now perform much more checks before validating the fact that PlayReady SL3000/SL2000 is available on the device. This helped to fix multiple Edge issues we were having.

• Many other smaller fixes, including on the maxVideoBufferSize option which was not always relying on good estimates, on better handling contents with mixed encryption and multiple fixes of minor compatibility issues (mainly on the PlayStation 4 and 5, and on Safari)

📑 Changelog

Features

• DASH: Implement ContentProtection references [#1439]

Bug fixes

• DASH: support absolute path in URL resolution with RFC 3986 implementation [#1443, #1440]
• DASH: fix cases of blinking subtitles [#1416, #1424]
• Fix precision issues of the maxVideoBufferSize API [#1421]
• DASH: Prevent multiple loading of the same segment for some DASH low-latency contents [#1422]
• DRM/Compat: on Edge test comprehensively KeySystems before considering them as usable [#1434]
• DRM/DASH: Ignore 0x0 key id found in DASH initialization segments are they are often linked to unencrypted data. [#1466, #1458]
• DRM/Compat: On the PlayStation 5, reload directly when a decryption key become unusable to prevent fatal errors [#1399]
• MULTI_THREAD: Perform several actions so that our MULTI_THREAD experimental feature now works on older browser and on the Playstation 4 [#1401, #1402]
• Directfile/Compat: On safari on iOS no longer stay stuck in buffering when autoPlay is set to false or not set and the video element has the attribute "playsinline" [#1408, #1390]
• Directfile/compat: On safari mobile in directfile mode, do not stay in an infinite LOADING state if the duration is set to NaN (rare issue in a normally-unsupported multiple RxPlayer-per-media-element scenario) [#1393]
• Fix RxPlay error messages not properly displaying in Chrome's inspector since Chrome 126 [#1474]

Other improvements

• Signal an error if multiple active RxPlayer are linked to the same media element [#1394]
• Undetermined audio and text track language now have a normalized property equal to "und" for better ISO 639-3 compatibility [#1428]
• MULTI_THREAD: The experimental MULTI_THREAD feature does not need a dashWasmUrl anymore nor compatibility to WebAssembly [#1384]
• MULTI_THREAD: The DEBUG_ELEMENT feature now allows to display all debug information even under the "multithreading" mode [#1438]
• Generate TypeScript declaration maps [#1412]
• Do not rely on the performance.now API if not available [#1402]
• DRM: Refactor MediaKeys attachment logic to simplify device support updates [#1357]
• tests: use exponential backoff to speed up integration tests [#1389]
• code: Rely on the TypeScript type keyword at type imports to be sure they have no code impact on our final build [#1365]
• code: Reorganize core RxPlayer code into a src/main_thread and src/core respectively for main thread and worker code in a "multithread" mode [#1365]
• code: Rely on the prettier and rustfmt formatting tools in the codebase [#1387]
• build: remove dependency to webpack [#1435, #1425, #1420]
• tests: migrate all tests to the vitest framework to simplify and unify test-related dependencies and test writing [#1444, #1445]

DASH ContentProtection References

ContentProtection metadata leading to huge MPD

DASH MPD can get quite huge on complex contents with multiple Period elements and various decryption keys depending on the Period and Representation.

One of the bigger part of the MPD in those scenario is the <ContentProtection> element, which contains metadata related to content decryption. This element can get very big as it contains Base64-encoded binary data for various client-side key systems.

Screenshot: For encrypted contents, we can see on this screenshot that encryption-related metadata (in <ContentProtection> elements) - especially base64-encoded data - are one of the main culprit for an MPD large size.
Moreover, note that this example only advertise metadata for PlayReady and Widevine. So this example is even lighter than most actual encrypted contents we play in production.

Each DASH AdaptationSet or Representation linked to that metadata is then supposed to have this element, contributing to the MPD's size. And when the same encryption metadata repeats, for example in another Period, that same huge element has to be repeated there - leading to a very large MPD.

Thankfully, newer iterations of the DASH specification provide a solution to greatly reduce the size of those kind of MPD: ContentProtection references.

ContentProtection references

With ContentProtection referecences we can only declare once (in the MPD) encryption metadata linked to multiple AdaptationSet or Representation and then refer to it through an identifier each time it is needed.

Screenshot: The same MPD than in the first screenshot, but this time making use of ContentProtection references. Here the actual metadata could be defined on top of the MPD, and only refered to through a ref attribute as pictured. You can see that the exact same information would take less space here.

This allows to greatly reduce the size of multi-Period MPD with encrypted media whose encryption key repeats multiple times in the stream - which happens often.

This feature is directly enabled, with nothing to do on the application-side.

MULTI_THREAD feature improvements

The MULTI_THREAD feature

In the v4.0.0, we added the MULTI_THREAD feature which allows to run most of the RxPlayer's logic in a Worker, and thus in another thread.

Doing this has multiple advantages, performance-related ones, yet this isolation also has a considerable effect on the quality of our adaptive algorithms: on some low-end devices where we before observed either a lower quality or frequent transitions between multiple qualities, we're now able to better maintain a higher quality.

Removing the need to add our WebAssembly parser

The main issue with adding the MULTI_THREAD mode was its complex setup, among which the need to add our WebAssembly MPD parser, which implies WebAssembly support and thus preventing its usage on many "old" (year ~2019 and less) devices.

We've since noticed some work done on efficient JavaScript-only XML parsing whose performance was impressive. As the need for optimal XML parsing in a Worker environment was the main reason why we relied on our WebAssembly parser, we wondered if we couldn't take inspiration from this work to much simplify the MULTI_THREAD setup and increase support.

This is now done and performance has been impressive enough that we now consider that this can be the default MPD parser directly included in the worker file provided to the RxPlayer's attachWorker method.

Without this supplementary step, the feature becomes much simpler to profit from:

import RxPlayer from "rx-player/minimal";
import { MULTI_THREAD } from "rx-player/experimental/features";

// To simplify this example, we'll directly import an "embedded" version of the
// supplementary code loaded by the `MULTI_THREAD` feature.
// We could also load it on demand through an URL
import { EMBEDDED_WORKER } from "rx-player/experimental/features/embeds";

RxPlayer.addFeatures([MULTI_THREAD]);
const player = new RxPlayer(/* your usual options */);
player.attachWorker({ workerUrl: EMBEDDED_WORKER }).catch((err) => {
  console.error("An error arised while initializing the worker", err);
});

// Now playing in "multithreading" mode if possible
player.loadVideo({ /* ... */ });

Complete DEBUG_ELEMENT when playing in "multithreading" mode

The DEBUG_ELEMENT experimental feature allows to monitor the RxPlayer behavior with a UI containing a lot of playback-related information, some not even available through our API (such as a graphical representation of the content of the various buffers).

In version 4.0.0 a "multithreading" mode was introduced in the RxPlayer, leveraging the WebWorker API to distribute the Javascript workload between the main thread, and a WebWorker.
While this improved performance, especially on low-end devices, it introduced limitations: WebWorkers do not have access to the DOM, preventing the Debug Element from being updated directly from the WebWorker (where most of the RxPlayer core logic runs in that mode).
Consequently, information available only on the WebWorker side, such as precize information on the buffers' content, was not displayed when using "multithreading" mode.

To address this in version 4.1.0, the WebWorker now communicates playback data back to the main thread (only when the element is displayed). As a result, the Debug Element can now consistently display the same metrics, regardless of whether "multithreading" mode is enabled or not.

Screenshot: The player is playing a video using the multithreading mode, the Debug Element overlay is displayed on top of the video, and provides the same level of information as when multithreading is off..

Notes on the MULTI_THREAD feature

Our tests of the MULTI_THREAD feature in production on most devices have been conclusive, with no MULTI_THREAD-specific issue detected.

We're however still keeping the MULTI_THREAD as an "experimental" feature despite being convinced of its stability to still let us the ability to potentially break its API in the future without having to update the RxPlayer major version - as we're continuing to try improving on this feature.

Better DASH URL path resolution

DASH's Manifest, the MPD, can reference media segments using an absolute URL or a relative URL.
When segments are referenced by a relative URL, we combine it to a "base URL", which is either the URL of the MPD, a specific value indicated by a <BaseURL> XML element in the Manifest, or both.

Example

<!-->MPD downloaded from "http://example.com/manifest.mpd"<-->
<MPD>
  <BaseURL>foo/</BaseURL>
  <Period>
    <AdaptationSet mimeType="video/mp4">
      <BaseURL>video/</BaseURL>
      <SegmentTemplate  initialization="init.mp4" media="seg-$Number$.m4f" startNumber="1"/>
      <Representation bandwidth="686685" id="video/1"/>
    </AdaptationSet>
  </Period>
</MPD>

With the manifest below, the first video segment will be downloaded at http://example.com/foo/video/1/seg-1.m4f because the base URL where the MPD is loaded is http://example.com/, then come two relative BaseURL elements, the first pointing to foo/ and the second pointing to video/ and at last we can see that segment follow a seg-$Number$.m4f template (media attribute) , beginning with the $Number$ 1 (startNumber attribute)

Issue

Thanks to a GitHub issue, we found out that this URL-resolution logic had an issue when relative URLs begin with a slash /.

Taking our previous example, the RxPlayer wouldn't change its logic if the second BaseURL element announced /video/ (instead of just video/ in the example) as a path:

<!-->MPD downloaded from "http://example.com/manifest.mpd"<-->
<MPD>
  <BaseURL>foo/</BaseURL>
  <Period>
    <AdaptationSet mimeType="video/mp4">
      <BaseURL>/video/</BaseURL>
      <SegmentTemplate  initialization="init.mp4" media="seg-$Number$.m4f" startNumber="1"/>
      <Representation bandwidth="686685" id="video/1"/>
    </AdaptationSet>
  </Period>
</MPD>

It however turns out that this was wrong, a starting / should have led us to ignore previous parts of the constructed path until now on the current domain. More simply said: we should here have requested http://example.com/video/1/seg-1.m4f, not http://example.com/foo/video/1/seg-1.m4f.

Rules for relative URLs are in fact defined by an RFC (internet-related standards), the RFC 3986. As such, it follows the exact same rules than many other relative URL you may find in other types of documents, such as HTML pages.
When we initially wrote our URL resolution logic, we basically only relied on what felt natural to us (to make it short: relative URL concatenate), and didn't really refer to this standard to see if we were doing the right thing.

The fix

This issue helped us realizing that our logic wasn't following the set standard.

We thus completely rewrote (and added tests to) our URL resolution logic, which should now be able to be able to handle all edge cases related to relative and absolute URL resolution.

DRM: Comprehensively test DRM compatibility on Edge

DRM, for Digital Rights Management, are technologies allowing to regulate the access to a given digital content. In the context of adaptive video streaming, it most often takes the form of media data encryption and a lot of security mechanisms ensuring that only sufficiently-secret contexts have access to the decryption key.

The standard allowing to decrypt media through such mechanisms on the web is called Encrypted Media Extension (generally just called "EME"). It's a very complex set of API, and sadly is often "badly"-implemented: many environments do not seem to fully respect that standard, especially around its numerous edge cases.

We for example found out that the Edge browser may falsely announce being able to rely on high-security DRM (through the navigator.requestMediaKeySystemAccess API - the first API we need to call to know DRM-capabilities of the current environment) despite the fact that it does not: trying to actually decrypt such contents led to an error.

Screenshot: example usage of the navigator.requestMediaKeySystemAccess API to know which technology is supported. On the tested environment here, Widevine and ClearKey are supported but not PlayReady.
On the aforementioned Edge environment, PlayReady with some attached configuration would be announced as supported but could not actually be relied on.

For the v4.1.0, only when the current environment is the [chromium-based] Edge browser and the chosen DRM technology is PlayReady, we decided to perform what we could call a "dry-run": we not only call navigator.requestMediaKeySystemAccess, but also ALL EME API necessary for the acquisition of a (fake) decryption key just to check if those API work as expected. If they do not, we consider the corresponding technology as unsupported.

This generally only happens with PlayReady SL3000, with a fallback to other DRM technologies (PlayReady SL2000, Widevine...) usually being compatible.

DASH/DRM: All 0 key-id now ignored in container files

At rare occurences, people reported to us issues when playing some contents mixing encrypted and unencrypted media data leading to the RxPlayer thinking it cannot decrypt content that is in fact unencrypted.

Illustration: Timeline of a live content with multiple programs, where some of them - here ad breaks - are unencrypted, yet most of them (movies, football game, TV shows, series...) are encrypted with various keys.

It turns out that some unencrypted media contain encryption-related metadata in their container file (e.g. the so-called "mp4" files) even when they are unencrypted - and thus when this metadata is not even necessary. In this scenario, it seems common to set the "key id", a generally-unique identifier for the wanted key, to bits set to only 0.

Yet the RxPlayer parsed that metadata (and that key id) then thinking that the corresponding content is encrypted and that it just has a weird key id. When it doesn't succeed to obtain the key for this content (because it doesn't exist), it stops playback with a encryption-related error message.

Because we saw it happen multiple times, we tried looking at specifications and especially if an all-0 key id could be legal or if it is reserved for unencrypted contents. It turns out that there can be a link between all-0 key id and unencrypted contents but the specifications we've seen seem to also add multiple other conditions. Those supplementary were sadly not respected in the contents reproducing the issue.

Still, as there's a concrete link between that 0 key id and unencrypted content, as it seems very highly unlikely to actually have encrypted content with that key id, and as we still usually can get the key id inside the Manifest (where no such ignoring happen), we decided to globally ignore key-id entirely set to 0 when found in an ISOBMFF-derived container (like an mp4 file).

This improvement has been developed by an external contributor @el-gringo.

A lot of transparent large project updates

Many changes contained in that release are actually code refactoring that date from before the v4.0.0 release, that we postponed for later (now) to be sure of their impact on the RxPlayer's stability.

This includes:

• a lot of work to make the MULTI_THREAD experimental feature maintainable in the long term.

  This feature forced us to question what is part of the code that will run in the main JS thread and what is part of the code running in a "Worker" environment (wich moreover has some limitations, like has no access to the HTML DOM).

  To make this situation more explicit, the RxPlayer's central logic, previously in the src/core directory has now been splitted in two: src/main_thread for the main thread code and src/core for the code that may run in a Worker.

• We added the prettier tool so external contributors (and us!) do not have to think anymore what style the code should follow. As long as prettier is happy with it, we're happy with it.

  For Rust code, we similarly rely on rustfmt.

• We unified all our tests under a same testing framework, namely here vitest (and webdriverio) which included all features we wanted. Previously we were using jest for our unit tests and a collection of other tools (mocha, chai, sinon, karma) for our other types of tests, but we had multiple issues with some karma-related packages and their compatibility with MacOS.

• We also simplified our build processes by completely removing the need for babel and webpack.
  We now rely on:

  • TypeScript and our own scripts for most of our builds and most probably for the one you're using
  • esbuild for our bundles (at the bottom of release notes), the bundled worker code and our demo page
  • rustc (through cargo) and wasm-opt to build our WebAssembly MPD parser
  • vitest for the builds our tests depend on