Steps to reduce random failures of media mochitests

Matthew Gregan and I have taken some steps to help reduce random orange in the media mochitests. Sayrer pointed out that during some random failures in test_play_events we were timing out with mmap() faliures and almost 130 threads running. It turns out that on 32bit Linux each thread stack is assigned 10MB of virtual address space, and when you're running a lot of threads you can run out of virtual address space, causing the timeouts.

We're probably particularly vulnerable to this since we're still not doing garbage collection in xpcshell during mochitest runs.

I spent some time ensuring we're more proactive at shutting down the media decoder's threads (each media element requires three threads, plus one more per media element on Linux used by the sound server), and I refactored our media tests to reduce the number of tests we run in parallel. This ensures our peak number of threads should be kept down, thus reducing peak memory usage/allocated virtual address space.

Concurrently to this, Matthew Gregan identified and fixed a thread safety issue in our media decoder and another in the underlying libsydneyaudio library on Linux, which we use for audio playback. We expect these two fixes to also reduce some random playback failures.

Since landing on these changes Thursday PDT, we've had a lot less random orange on mozilla-central. We're still getting some random failures, but most of these are on Linux, and many of those are caused by a known bug in the version of pulseaudio running on the Tinderbox mochitest machines. We expect many of these random failures will go away when the releng guys update pulseaudio on the Linux Tinderbox mochitest machines.

We're still getting a fair number of orange in test_progress, mostly on Windows, but we're going to be changing our media progress event to bring it into line with the current spec, and when we do that we'll be removing that test.

HTML5 Video preload attribute supported in Firefox 4, autobuffer attribute removed

Late last week I landed support on Firefox trunk for the HTML5 video 'preload' attribute. This replaces the 'autobuffer' attribute, which we previously supported. If you were previously using the autobuffer attribute on your HTLM5 videos, you need to update to using the preload attribute as well.

The preload attribute provides a hint to the browser as to how much downloading is sensible for a given media. The preload attribute can have three values:

1. "none" - suggests to the browser that it doesn't need to load this media at all until the user plays the resource. The browser will delay any network traffic required to load the media until the users tries to play the resource, or explicitly loads the resource. I suggest using this preload value, along with the poster attribute, when it's unlikely that the user will play the resource. This is probably most useful in a mobile environment, where data can be expensive.
2. "metadata" - suggests to the browser that it isn't necessary to load the entire resource in advance. The browser will suspend the load after loading metadata, displaying the first video frame (if there's no poster image), and ensuring it can play the media. This is the default behaviour, and prevents excess network traffic when the web developer isn't certain the video will definitely be played. If you don't specify a preload value, Firefox will automatically do this. This was also the default behaviour in Firefox 3.5 and 3.6 in the absence of the autobuffer attribute. This default behaviour is a suitable compromise between bandwidth saving and user convenience.
3. "auto" - suggests to the browser that it should load as much of the resource as possible. As long as the browser's own media cache isn't full, it will keep on downloading. I suggest this is most useful in the "YouTube" case, when you've got a media which the user is almost certainly going to watch, and so having the user download the media is not likely to be wasting server bandwidth. This behaviour is the same as using the autobuffer attribute in Firefox 3.5 and 3.6

Since the autobuffer attribute is no longer present in Firefox 4, and the preload attribute is not present in Firefox 3.5 and 3.6, if you want a media to download completely, you should include both preload="auto" and autobuffer in the video element, e.g.:

<video autobuffer preload="auto" src="video.ogg"></video>

If you want the video to only be downloaded if the user actually plays the video, you should not include either the preload or the autobuffer attribute. The default behaviour in Firefox 3.5, 3.6, and the upcoming Firefox 4 is to only load up to the first frame and then suspend the download.

Keyframe indexed Ogg files supported in Firefox 4

Seeking in Ogg files served over the internet has historically been slow. This is because traditional Ogg files do not contain a keyframe index, so you've got no idea where in the media you need to begin decoding from in order to playback from any given time, particularly in variable bitrate media. Seeking typically has to be implemented as a bisection search over the media, which can be slow when performed over a high-latency network such as the internet.

I've been working with the people at the Xiph foundation to develop a keyframe index for Ogg files to alleviate this problem, and I'm delighted to announce that today I finally landed support for indexed Ogg files in Firefox trunk. The keyframe index format I developed has been included in the newly minted Ogg Skeleton 4.0 metadata track.

How much of an impact does seeking with an index make? Download a current Firefox 4 nightly build, and point it at my keyframe indexed Ogg seek demo, and see for yourself. It speeds up seeking dramatically.

If you're using ffmpeg2theora version 0.27 or later to encode your Ogg media, you'll automatically get a Skeleton 4.0 track with keyframe indexes when you encode your media. If you have existing Ogg media which you want add keyframe indexes to, you can use my OggIndex command line tool to add indexes to your Ogg media. The author of ffmpeg2theora, J^, is kindly building and hosting nightly builds of OggIndex, and the OggIndex source code is also available via git under a BSD license.

Indexed Oggs are also supported in GStreamer trunk thanks to the work of Sebastian Dröge.

I've been working on this sporadically for almost a year now, so it's great to finally get this landed!

Buffered attribute landed for HTML5 video

Last night I landed support for the buffered HTML5 video attribute in Firefox. This is cool because we can now accurately determine which time-segments of a video we can play and seek into without needing to pause playback to download more data. Previously you could only get the byte position the download had reached, which often doesn't map to the time ranges which are playable very well, especially in a variable bit rate video. This also can't tell you if there are chunks which we skipped downloading before the downloaded byte position. Once the video controls UI is updated, users will be able to know exactly which segments of their video are downloaded and playable and can be seeked into without pausing playback to download more data.

To see this in action, download last night's Firefox nightly build or later, and point your browser at my  video buffered attribute demo. You'll see something like the screenshot below, including an extra progress bar (implemented using canvas) showing the time ranges which are buffered.

 

I've implemented the buffered attribute for the Ogg and WAV backends. Support for the buffered attribute for WebM is being worked on by Matthew Gregan, and is well underway. At the moment we return empty ranges for the buffered attribute on video elements playing WebM and raw video.

My checkin just missed the cutoff for Firefox 4 Beta 3, so the first beta release that the video buffered attribute will appear in is Firefox 4 Beta 4.