Say you've started playback of an HTML5 audio or video element, and you decide you really want to cancel playback and downloading of the media resource. Stopping playback is easy, just call the pause() method. But the network connection won't be stopped until the media element gets garbage collected, and even if you release all references to the media element, it won't be destroyed until the browser decides to runs its garbage collector. How can you stop the download of the media resource in the meantime? Here's a quick hack to achieve this: just reset the element's src attribute to the empty string. This destroys the element's internal decoder, and stops the network download.
For example:
<audio id="audio" controls>
<source src="funky-music.ogg">
<source src="funky-music.mp3">
</audio> <!-- ... Some time later, we decide we should stop the audio element playing and downloading... --> <script> var audio = document.getElementById("audio");
audio.pause();
audio.src = ""; // Stops audio download. audio.load(); // Initiate a new load, required in Firefox 3.x. </script>
Be aware that this will destroy the media element's decoders, so the element won't be playable anymore, and it will be rendered as an "error cross" if it's in a document. Also in Firefox 3.x you need to call load() after changing the source, whereas in Firefox 4 the load is scheduled to run when you change the src attribute, and the extra load() call is not required.
I've updated the media load() implementation in Firefox 4 to match the current WHATWG media load algorithm specification. There are three main changes that web developers using media elements should be aware of.
Firstly, error reporting has slightly changed. When a media element fails to load from its <source> children, an "error" event is dispatched to every child element which failed to load. Previously in Firefox 3.x you'd receive only one "error" event dispatched to the media element once all of its child <source> elements had failed to load. Now you only receive "error" events in the child <source> elements, and not in the media element itself.
For example, suppose you have the following markup:
Firefox 4 does not support the playback of patent encumbered content, so you'll receive "error" events in the <source> elements with the MP4 and 3GP resources, before the Ogg resource is loaded. Note also that the <source> children are loaded in the order in which they appear in the markup, and if one <source> child successfully loads and is playable, the children after it won't be loaded.
To detect that all child <source> elements have failed to load, check the value of the networkState attribute of the media element; if its value is HTMLMediaElement.NETWORK_NO_SOURCE, you know all child <source> elements have failed to load.
If you add another child <source> element to a media element which is in networkState HTMLMediaElement.NETWORK_NO_SOURCE, it will attempt to load the resource specified by the newly added <source>.
Secondly, when the load begins has changed. When you set the src attribute of a media element, or add a <source> child element to a media element, a load will be scheduled to run asynchronously as soon as the current JavaScript context exits (basically the load starts the next time we return to the browser application's main event loop). So for example, suppose you have some timeout set such as:
<script>
setTimeout(
function() {
var v = document.createElement("video");
v.src = "video.ogg";
document.appendChild(v)
// Do some other stuff... }, 1000);
</script>
When the function runs, the load for the video element won't begin until after the function returns, and control returns to the browser. This is important, because in Firefox 3.x, the load is started (and could even run to completion!) as soon as you set the media element's src attribute.
Lastly, the media element's events now no longer bubble. Previously they bubbled, and this was a bug in Firefox 3.x and in violation of the spec.
A number of people, myself included, have had trouble getting pymake to build Mozilla on Windows. Roc discovered the trick to getting it to work: use relative paths in the mozconfig. For example, my mozconfig (which lives in $srcdir/mozconfig) for a debug build is:
Matthew Gregan and I have taken some steps to help reduce random orange in the media mochitests. Sayrer pointed out that during some random failures in test_play_events we were timing out with mmap() faliures and almost 130 threads running. It turns out that on 32bit Linux each thread stack is assigned 10MB of virtual address space, and when you're running a lot of threads you can run out of virtual address space, causing the timeouts.
I spent some time ensuring we're more proactive at shutting down the media decoder's threads (each media element requires three threads, plus one more per media element on Linux used by the sound server), and I refactored our media tests to reduce the number of tests we run in parallel. This ensures our peak number of threads should be kept down, thus reducing peak memory usage/allocated virtual address space.
Since landing on these changes Thursday PDT, we've had a lot less random orange on mozilla-central. We're still getting some random failures, but most of these are on Linux, and many of those are caused by a known bug in the version of pulseaudio running on the Tinderbox mochitest machines. We expect many of these random failures will go away when the releng guys update pulseaudio on the Linux Tinderbox mochitest machines.
Late last week I landed support on Firefox trunk for the HTML5 video 'preload' attribute. This replaces the 'autobuffer' attribute, which we previously supported. If you were previously using the autobuffer attribute on your HTLM5 videos, you need to update to using the preload attribute as well.
The preload attribute provides a hint to the browser as to how much downloading is sensible for a given media. The preload attribute can have three values:
"none" - suggests to the browser that it doesn't need to load this media at all until the user plays the resource. The browser will delay any network traffic required to load the media until the users tries to play the resource, or explicitly loads the resource. I suggest using this preload value, along with the poster attribute, when it's unlikely that the user will play the resource. This is probably most useful in a mobile environment, where data can be expensive.
"metadata" - suggests to the browser that it isn't necessary to load the entire resource in advance. The browser will suspend the load after loading metadata, displaying the first video frame (if there's no poster image), and ensuring it can play the media. This is the default behaviour, and prevents excess network traffic when the web developer isn't certain the video will definitely be played. If you don't specify a preload value, Firefox will automatically do this. This was also the default behaviour in Firefox 3.5 and 3.6 in the absence of the autobuffer attribute. This default behaviour is a suitable compromise between bandwidth saving and user convenience.
"auto" - suggests to the browser that it should load as much of the resource as possible. As long as the browser's own media cache isn't full, it will keep on downloading. I suggest this is most useful in the "YouTube" case, when you've got a media which the user is almost certainly going to watch, and so having the user download the media is not likely to be wasting server bandwidth. This behaviour is the same as using the autobuffer attribute in Firefox 3.5 and 3.6
Since the autobuffer attribute is no longer present in Firefox 4, and the preload attribute is not present in Firefox 3.5 and 3.6, if you want a media to download completely, you should include both preload="auto" and autobuffer in the video element, e.g.:
If you want the video to only be downloaded if the user actually plays the video, you should not include either the preload or the autobuffer attribute. The default behaviour in Firefox 3.5, 3.6, and the upcoming Firefox 4 is to only load up to the first frame and then suspend the download.
Seeking in Ogg files served over the internet has historically been slow. This is because traditional Ogg files do not contain a keyframe index, so you've got no idea where in the media you need to begin decoding from in order to playback from any given time, particularly in variable bitrate media. Seeking typically has to be implemented as a bisection search over the media, which can be slow when performed over a high-latency network such as the internet.
I've been working with the people at the Xiph foundation to develop a keyframe index for Ogg files to alleviate this problem, and I'm delighted to announce that today I finally landed support for indexed Ogg files in Firefox trunk. The keyframe index format I developed has been included in the newly minted Ogg Skeleton 4.0 metadata track.
How much of an impact does seeking with an index make? Download a current Firefox 4 nightly build, and point it at my keyframe indexed Ogg seek demo, and see for yourself. It speeds up seeking dramatically.
If you're using ffmpeg2theora version 0.27 or later to encode your Ogg media, you'll automatically get a Skeleton 4.0 track with keyframe indexes when you encode your media. If you have existing Ogg media which you want add keyframe indexes to, you can use my OggIndex command line tool to add indexes to your Ogg media. The author of ffmpeg2theora, J^, is kindly building and hosting nightly builds of OggIndex, and the OggIndex source code is also available via git under a BSD license.
Indexed Oggs are also supported in GStreamer trunk thanks to the work of Sebastian Dröge.
I've been working on this sporadically for almost a year now, so it's great to finally get this landed!
Last night I landed support for the buffered HTML5 video attribute in Firefox. This is cool because we can now accurately determine which time-segments of a video we can play and seek into without needing to pause playback to download more data. Previously you could only get the byte position the download had reached, which often doesn't map to the time ranges which are playable very well, especially in a variable bit rate video. This also can't tell you if there are chunks which we skipped downloading before the downloaded byte position. Once the video controls UI is updated, users will be able to know exactly which segments of their video are downloaded and playable and can be seeked into without pausing playback to download more data.
To see this in action, download last night's Firefox nightly build or later, and point your browser at my video buffered attribute demo. You'll see something like the screenshot below, including an extra progress bar (implemented using canvas) showing the time ranges which are buffered.
I've implemented the buffered attribute for the Ogg and WAV backends. Support for the buffered attribute for WebM is being worked on by Matthew Gregan, and is well underway. At the moment we return empty ranges for the buffered attribute on video elements playing WebM and raw video.
My checkin just missed the cutoff for Firefox 4 Beta 3, so the first beta release that the video buffered attribute will appear in is Firefox 4 Beta 4.
If you're looking to create some WebM videos to try out the new WebM support in Firefox 4 Beta 1, an easy tool to use is Miro Video Converter. Because WebM is such a new format, existing video editing software doesn't encode in WebM yet, so if you want to make a WebM video, you must create it in another format and then convert it to WebM using a tool like Miro Video Converter.
Miro Video Converter provides a very simple interface for ffmpeg, one of the leading open source video encoding tools. Miro Video Converter configures ffmpeg to output converted video to the same quality as the input, and it uses the excellent xiph.org libvorbis audio encoder, rather than ffmpeg's own broken Vorbis encoder (if you hear poor quality audio on YouTube WebM videos, it's probably because they're still using ffmpeg's vorbis encoder).
Miro Video Converter is freely available for download on Windows and Mac, and can convert to other formats besides WebM.
The new Firefox 4 Beta 1 includes support for HTML5 video elements playing WebM videos. This is exciting, as much of the industry is getting behind WebM. Opera is shipping WebM support in Opera 10.6, Google Chrome's "early access release channel" builds include WebM support. Microsoft said they'd support WebM in their HTML5 video implementation in their upcoming IE9, provided appropriate codecs are installed on the user's system. Adobe has announced that they'll support WebM playback in Flash, which will provide fallback playback of WebM in any older or otherwise non WebM supporting browsers. Intel says it will move towards hardware support for WebM once it becomes popular.
Google has freely licensed the VP8 video codec used in WebM, and provided a royalty free patent grant. This is great news for the future of the internet. We now have a royalty free video codec, with quality which is competitive with proprietary alternatives. This means anyone can freely use high quality internet video, without having to worry about getting sued or having to negotiate a patent license.
All this will hopefully contribute to increased adoption of WebM and HTML5 video, coupling all the power of the modern web browser's rendering pipeline with high quality video.
Last week we landed patches from Makoto Kato to enable optimized assembly in libvpx on Win64. We also landed a similar patch from Tim Terriberry to enable optimized assembly in libvpx on Linux x86_64. This will improve decoding performance of WebM videos on these platforms. We're still being hit by scaling performance, particularly on high resolution and "full-screen" YouTube HTML5 experiment WebM videos. We're aware of this, and are working to improve it.
Today I landed Firefox's WebM support on mozilla-central, our Firefox development branch. It should appear in nightly builds from tonight onwards.
Credits to the development effort go to Chris Double for writing the decoder backend, Matthew Gregan for writing the ISC style licensed libnestegg WebM demuxer, and myself for integrating all the moving parts into the Mozilla tree and build system. Also thanks need to go to the releng team which promptly installed YASM on the Mozilla Linux and Mac build machines, and to Roc, Shaver and many others for cracking the whip and working behind the scenes.
Firefox should build with WebM support without needing any extra changes to your build configuration, unless you're building on Win32, where you'll need to have MASM installed in order to compile libvpx's optimized assembly. MASM ships with the Windows 7 SDK, and with Visual Studio Pro. If you've got neither of those installed, you can also download MASM directly.
If you're building on Linux x86, Mac x86 or Mac x86_64 and you've got YASM installed, you'll automatically build VP8 decoder's optimized assembly code from libvpx. If you don't have YASM, you'll fallback to using the generic C code, which won't be as fast, but still performs acceptably. We don't have WebM support building on Win64 yet, you can disable if you reconfigure with --disable-webm.
I have discovered a new tool in our toolbox in the fight against random test failures: crashinjectdll.dll on Windows, nptest.so on Linux (I don't know what it's called on MacOS...). I suspect we're randomly injecting crashes into plugins, in order to test recovery from plugin crashes with out-of-process plugins (correct me if I'm wrong). If a mochitest run times out on Tinderbox, and a crash has been injected, when the mochitest run is killed we'll get call stacks in the output log for all the threads that were running. This can give us some details of the state of Firefox when the timeout occurred, and may give us a clue as to why we've timed out.
Maybe we should make always inject a plugin crash at the start of a mochitest run, so that we always have call stacks when we kill a timed-out mochitest run?
Next time someone comments in your random orange bugs with a log, check to see if it's got call stacks in it!
Today I finally landed the new Ogg video decoder for Firefox/Mozilla. It should appear in Firefox nightlies from 2 April 2010 onwards. Chris Double did the initial implementation for the new Ogg decoder, and I took over in December, implemented seeking, and bashed it into shape.
This replaces our old Ogg decoder which used liboggplay. We'll now be able to implement new features easier, and have greater control over our decode pipeline. We can now control our audio and video decode separately, and this allows us to do things like not decoding video interframes if the video decode is taking too much time and causing the audio decode to fall behind. This prevents audio stutter when decoding large-frame-sized videos. The new backend also has more reliable seeking, and can even load a few weirdly formed Ogg files that the previous backend couldn't handle.
It was a large patch, and it's a testament to our robust set of video and audio unit tests that when I checked it into mozilla-central, there was basically only one small fix required. It has been a long grind getting the new decoder to work robustly, but our unit tests have been an invaluable, if fickle, task-mistress.
The new backend should at least be feature-equivalent to the old backend. If you spot a regression in Ogg video playback in new nightly builds, please file a bug in Core in the "Video/Audio" component!
One common way to reduce the load on a web server is to use gzip or deflate compression when serving to a supporting web browser. When Firefox requests an Ogg media, it advertises that it can handle a gzipped or deflated response; the HTTP request includes the Accept-Encoding: gzip,deflate header. But despite Firefox advertising that it supports gzip/deflate, you probably don't want your web server to gzip or deflate Ogg media. If you serve an Ogg media compressed, Firefox won't be able to seek in the media, or determine its duration. Since the video/audio data in Ogg files is already compressed, gzip/deflate won't actually save you much bandwidth anyway, so you probably want to disable compression when serving Ogg files.
I have switched my blog to being on a custom domain hosted at Blogger. The address for my blog is now blog.pearce.org.nz. Previously I was using Blogger's publish via FTP to host my blog on my own domain, but they have discontinued that service. I tried running Wordpress on my own domain, but I couldn't get plugins to work, so without WP-Cache my blog was slow. I'm glad to have my blog restored to its former snappy look and feel.
Seeking in Ogg videos in HTML5 <video> over the network currently can be very slow. This is because when we seek an Ogg/Theora video to a target time, we must perform a bisection search over the file in order to find the target Theora video frame. If this is an interframe (which just records what's changed since its preceding frame), we must then perform another bisection search to find the interframe's keyframe, and then decode forwards to the target frame in order to completely construct it.
A reasonable bisection search implementation may require half a dozen HTTP requests to complete, so if we need to do two bisection searches per seek (once for the target frame, and once for the target frame's keyframe), we actually need to do about a dozen HTTP requests per seek...
If we knew in advance where the keyframes were, we wouldn't need to do any bisection searches; we could make just one HTTP request to the last keyframe preceeding the target frame. Clearly making fewer HTTP requests is faster.
Enter the Skeleton 3.1 with Keyframe Index track. This extends the existing Skeleton 3.0 metadata track to provide an index for every Theora video and Vorbis audio track in an Ogg media. This will enable players to make the optimal HTTP request when seeking in media files served over the internet, resulting in as fast seeking as possible when viewing online video.
It's nice if a video player's UI can display the playback duration of the Ogg media. Unfortunately the raw Ogg format does not store the duration either, so it must be calculated, which requires additional HTTP requests, and slows down the video loading. The Skeleton 3.1 track also includes the playback duration of its Ogg containing file, to eliminate this overhead and speed up loading.
I developed the Skeleton 3.1 with Keyframe Index track in conjunction with the folks at Xiph.org. See the Xiph.org Skeleton 3.1 with Keyframe Index wiki page for the work-in-progress specification. Any comments on the specification would be much appreciated, please send them to the Theora mailing list.
To see how keyframe indexing improves network seeking performance of HTML5 Ogg/Theora <video>, you can download a development version of Firefox which can take advantage of indexes here:
If you already have a Firefox instance running, you'll need to either close your running Firefox instance before starting the index-capable Firefox, or start the index-capable Firefox with the --no-remote command line parameter. To compare the network performance of indexed versus non-indexed seeking, point the index-capable Firefox to the indexed seek demo page.
You should notice a clear speed difference when seeking to an unbuffered position in the indexed media.
The Skeleton 3.1 with Keyframe Index specification is still being developed, but we hope to lock it down soon. We are planning to ship support for keyframe index-assisted seeking in Firefox 3.7.
