Today I finally landed the new Ogg video decoder for Firefox/Mozilla. It should appear in Firefox nightlies from 2 April 2010 onwards. Chris Double did the initial implementation for the new Ogg decoder, and I took over in December, implemented seeking, and bashed it into shape.
This replaces our old Ogg decoder which used liboggplay. We'll now be able to implement new features easier, and have greater control over our decode pipeline. We can now control our audio and video decode separately, and this allows us to do things like not decoding video interframes if the video decode is taking too much time and causing the audio decode to fall behind. This prevents audio stutter when decoding large-frame-sized videos. The new backend also has more reliable seeking, and can even load a few weirdly formed Ogg files that the previous backend couldn't handle.
It was a large patch, and it's a testament to our robust set of video and audio unit tests that when I checked it into mozilla-central, there was basically only one small fix required. It has been a long grind getting the new decoder to work robustly, but our unit tests have been an invaluable, if fickle, task-mistress.
The new backend should at least be feature-equivalent to the old backend. If you spot a regression in Ogg video playback in new nightly builds, please file a bug in Core in the "Video/Audio" component!
Friday, 2 April 2010
Friday, 12 March 2010
Don't serve Ogg media with gzip/deflate compression
One common way to reduce the load on a web server is to use gzip or deflate compression when serving to a supporting web browser. When Firefox requests an Ogg media, it advertises that it can handle a gzipped or deflated response; the HTTP request includes the Accept-Encoding: gzip,deflate header. But despite Firefox advertising that it supports gzip/deflate, you probably don't want your web server to gzip or deflate Ogg media. If you serve an Ogg media compressed, Firefox won't be able to seek in the media, or determine its duration. Since the video/audio data in Ogg files is already compressed, gzip/deflate won't actually save you much bandwidth anyway, so you probably want to disable compression when serving Ogg files.
I have added this tip to the Mozilla Developer Center article on configuring servers for Ogg media.
I have added this tip to the Mozilla Developer Center article on configuring servers for Ogg media.
Sunday, 7 March 2010
Switched blog to custom domain
I have switched my blog to being on a custom domain hosted at Blogger. The address for my blog is now blog.pearce.org.nz. Previously I was using Blogger's publish via FTP to host my blog on my own domain, but they have discontinued that service. I tried running Wordpress on my own domain, but I couldn't get plugins to work, so without WP-Cache my blog was slow. I'm glad to have my blog restored to its former snappy look and feel.
Monday, 11 January 2010
Indexing keyframes in Ogg videos for fast seeking
Seeking in Ogg videos in HTML5 <video> over the network currently can be very slow. This is because when we seek an Ogg/Theora video to a target time, we must perform a bisection search over the file in order to find the target Theora video frame. If this is an interframe (which just records what's changed since its preceding frame), we must then perform another bisection search to find the interframe's keyframe, and then decode forwards to the target frame in order to completely construct it.
A reasonable bisection search implementation may require half a dozen HTTP requests to complete, so if we need to do two bisection searches per seek (once for the target frame, and once for the target frame's keyframe), we actually need to do about a dozen HTTP requests per seek...
If we knew in advance where the keyframes were, we wouldn't need to do any bisection searches; we could make just one HTTP request to the last keyframe preceeding the target frame. Clearly making fewer HTTP requests is faster.
Enter the Skeleton 3.1 with Keyframe Index track. This extends the existing Skeleton 3.0 metadata track to provide an index for every Theora video and Vorbis audio track in an Ogg media. This will enable players to make the optimal HTTP request when seeking in media files served over the internet, resulting in as fast seeking as possible when viewing online video.
It's nice if a video player's UI can display the playback duration of the Ogg media. Unfortunately the raw Ogg format does not store the duration either, so it must be calculated, which requires additional HTTP requests, and slows down the video loading. The Skeleton 3.1 track also includes the playback duration of its Ogg containing file, to eliminate this overhead and speed up loading.
I developed the Skeleton 3.1 with Keyframe Index track in conjunction with the folks at Xiph.org. See the Xiph.org Skeleton 3.1 with Keyframe Index wiki page for the work-in-progress specification. Any comments on the specification would be much appreciated, please send them to the Theora mailing list.
I have developed a prototype Ogg indexer, OggIndex, and also recent ffmpeg2theora nightlies will encode keyframe indexes if you specify the command line option -–seek-index. OggIndex and experimental indexing ffmpeg2theora nightlies are available for download.
To see how keyframe indexing improves network seeking performance of HTML5 Ogg/Theora <video>, you can download a development version of Firefox which can take advantage of indexes here:
http://pearce.org.nz/video/firefox-indexed-seek-linux.tar.bz2
http://pearce.org.nz/video/firefox-indexed-seek-macosx.dmg
http://pearce.org.nz/video/firefox-indexed-seek-win32.zip
If you already have a Firefox instance running, you'll need to either close your running Firefox instance before starting the index-capable Firefox, or start the index-capable Firefox with the --no-remote command line parameter.
To compare the network performance of indexed versus non-indexed seeking, point the index-capable Firefox to the indexed seek demo page.
You should notice a clear speed difference when seeking to an unbuffered position in the indexed media.
The Skeleton 3.1 with Keyframe Index specification is still being developed, but we hope to lock it down soon. We are planning to ship support for keyframe index-assisted seeking in Firefox 3.7.
A reasonable bisection search implementation may require half a dozen HTTP requests to complete, so if we need to do two bisection searches per seek (once for the target frame, and once for the target frame's keyframe), we actually need to do about a dozen HTTP requests per seek...
If we knew in advance where the keyframes were, we wouldn't need to do any bisection searches; we could make just one HTTP request to the last keyframe preceeding the target frame. Clearly making fewer HTTP requests is faster.
Enter the Skeleton 3.1 with Keyframe Index track. This extends the existing Skeleton 3.0 metadata track to provide an index for every Theora video and Vorbis audio track in an Ogg media. This will enable players to make the optimal HTTP request when seeking in media files served over the internet, resulting in as fast seeking as possible when viewing online video.
It's nice if a video player's UI can display the playback duration of the Ogg media. Unfortunately the raw Ogg format does not store the duration either, so it must be calculated, which requires additional HTTP requests, and slows down the video loading. The Skeleton 3.1 track also includes the playback duration of its Ogg containing file, to eliminate this overhead and speed up loading.
I developed the Skeleton 3.1 with Keyframe Index track in conjunction with the folks at Xiph.org. See the Xiph.org Skeleton 3.1 with Keyframe Index wiki page for the work-in-progress specification. Any comments on the specification would be much appreciated, please send them to the Theora mailing list.
I have developed a prototype Ogg indexer, OggIndex, and also recent ffmpeg2theora nightlies will encode keyframe indexes if you specify the command line option -–seek-index. OggIndex and experimental indexing ffmpeg2theora nightlies are available for download.
To see how keyframe indexing improves network seeking performance of HTML5 Ogg/Theora <video>, you can download a development version of Firefox which can take advantage of indexes here:
http://pearce.org.nz/video/firefox-indexed-seek-linux.tar.bz2
http://pearce.org.nz/video/firefox-indexed-seek-macosx.dmg
http://pearce.org.nz/video/firefox-indexed-seek-win32.zip
If you already have a Firefox instance running, you'll need to either close your running Firefox instance before starting the index-capable Firefox, or start the index-capable Firefox with the --no-remote command line parameter.
To compare the network performance of indexed versus non-indexed seeking, point the index-capable Firefox to the indexed seek demo page.
You should notice a clear speed difference when seeking to an unbuffered position in the indexed media.
The Skeleton 3.1 with Keyframe Index specification is still being developed, but we hope to lock it down soon. We are planning to ship support for keyframe index-assisted seeking in Firefox 3.7.
Friday, 20 November 2009
Replay Debugging mochitest failures with VMWare Workstation 7
Ever since my last escapades with Replay Debugging in VMWare Workstation 6.5, I've been looking forward to improvements in this awesome technology. Thankfully the guys at VMWare have been hard at work, and now VMWare Workstation 7 now boasts improved Replay Debugging. I've found it much more robust and reliable, and Roc and I have already used it to debug some random orange bugs.
I've documented how to produce a Replay Debugging setup for debugging intermittent test failures in Mozilla mochitests, and put it up on MDC:
https://developer.mozilla.org/En/Debugging/Record_and_Replay_Debugging_Firefox
Now anyone can setup a machine to record and replay debug intermittent mochitests! A word of warning: you need a modern CPU in order to get good performance. I had poor performance when running on my two-year-old Core2Duo laptop, but replay performance is almost at real-time speeds on my shiny new Intel i7 950 box.
I still have two patches that need to be refined and then checked in, to facilitate replay debugging. The first enables the mochitest harness to loop forever on a test directory. The second enables you to set break points on specific JavaScript dump() calls, so you can break during replay close to where the action is.
We're far from having a fully automated record and replay setup, but we've made a start!
I've documented how to produce a Replay Debugging setup for debugging intermittent test failures in Mozilla mochitests, and put it up on MDC:
https://developer.mozilla.org/En/Debugging/Record_and_Replay_Debugging_Firefox
Now anyone can setup a machine to record and replay debug intermittent mochitests! A word of warning: you need a modern CPU in order to get good performance. I had poor performance when running on my two-year-old Core2Duo laptop, but replay performance is almost at real-time speeds on my shiny new Intel i7 950 box.
I still have two patches that need to be refined and then checked in, to facilitate replay debugging. The first enables the mochitest harness to loop forever on a test directory. The second enables you to set break points on specific JavaScript dump() calls, so you can break during replay close to where the action is.
We're far from having a fully automated record and replay setup, but we've made a start!
Monday, 5 October 2009
Problems building Firefox with Visual Studio 2005 and Windows 7 SDK
When I tried to update my 1.9.2 tree this morning on my Windows box I quickly discovered that Ben Hearsum had recently updated our build system to require the Windows 7 SDK. However once I installed the new SDK and tried to build (using Visual Studio 2005's compiler), I got the following error:
shell32.lib(shguid.obj) : fatal error LNK1103: debugging information corrupt; recompile module.
This is a compatibility issue between Visual Studio 2008 and Visual Studio 2005 which only occurs when you compile in debug mode. There's a hotifx to resolve this issue. Once I installed this, I could magically build again. Thanks to biesi for pointing out the hotfix!
shell32.lib(shguid.obj) : fatal error LNK1103: debugging information corrupt; recompile module.
This is a compatibility issue between Visual Studio 2008 and Visual Studio 2005 which only occurs when you compile in debug mode. There's a hotifx to resolve this issue. Once I installed this, I could magically build again. Thanks to biesi for pointing out the hotfix!
Friday, 18 September 2009
Ogg video seek performance improvements
I've recently landed bug 501031 on mozilla-central and on 1.9.2 which roughly cuts Ogg seeking time in half. In Firefox 3.5, seeking Ogg Internet video was very slow, often taking 20 seconds or more to seek. This patch will make seeking in Ogg media in Firefox 3.6 much faster! This patch should also reduce the likelihood of encountering visual artefacts after a seek.
Currently the Ogg format doesn't contain any kind of keyframe index, so when you want to seek to a given time you typically do a bisection search over the entire file, reading little chunks as you go to figure out where your bisection has landed. This works fine for files on disk, but when seeking in files served over the Internet this can be slow, especially when viewing media which is hosted half a world away.
To play Ogg files, Firefox uses liboggplay, which in turn uses liboggz to seek. Unfortunately liboggz's seek bisection is in need of some maintenance (it's currently being rewritten by the maintainer). Its bisection is erratic and it fails to terminate its search appropriately, so it makes many more bisections than required. Most bisections or non-sequential reads result in a new HTTP connection, which is what makes this process slow for Internet video.
My patch fixes liboggz's seek to bisect sensibly, and to end the bisection search when it lands within 500ms of the seek target. This means that once we land close enough to the seek target, we'll just keep downloading from there. This is typically faster than continuing the bisection search due to the latency in setting up HTTP requests. We subtract 500ms from the seek target before we begin the bisection search, so that we don't finish the seek after the actual seek target.
Theora stores its video data as keyframe and interframes. In order to seek to and display a frame at a given time, we need to seek to the previous keyframe and decode forward to the target frame. The suggested approach is to extract the keyframe's position from the frame's granulepos field, and then seek again to the keyframe. This second seek needs to be exactly right, and that's hard due to some nasty edge cases with regards to stream muxing. Doing another bisection search in our case is also slow due to the latency in setting up HTTP requests. So now we just calculate the maximum possible time-offset that a frame can be from its keyframe, and subtract that from our seek target. This means we will often download more data than necessary, but for us that's typically faster than doing another bisection search.
The moral of the story is that if you want your video to seek quickly, include regular keyframes!
Currently the Ogg format doesn't contain any kind of keyframe index, so when you want to seek to a given time you typically do a bisection search over the entire file, reading little chunks as you go to figure out where your bisection has landed. This works fine for files on disk, but when seeking in files served over the Internet this can be slow, especially when viewing media which is hosted half a world away.
To play Ogg files, Firefox uses liboggplay, which in turn uses liboggz to seek. Unfortunately liboggz's seek bisection is in need of some maintenance (it's currently being rewritten by the maintainer). Its bisection is erratic and it fails to terminate its search appropriately, so it makes many more bisections than required. Most bisections or non-sequential reads result in a new HTTP connection, which is what makes this process slow for Internet video.
My patch fixes liboggz's seek to bisect sensibly, and to end the bisection search when it lands within 500ms of the seek target. This means that once we land close enough to the seek target, we'll just keep downloading from there. This is typically faster than continuing the bisection search due to the latency in setting up HTTP requests. We subtract 500ms from the seek target before we begin the bisection search, so that we don't finish the seek after the actual seek target.
Theora stores its video data as keyframe and interframes. In order to seek to and display a frame at a given time, we need to seek to the previous keyframe and decode forward to the target frame. The suggested approach is to extract the keyframe's position from the frame's granulepos field, and then seek again to the keyframe. This second seek needs to be exactly right, and that's hard due to some nasty edge cases with regards to stream muxing. Doing another bisection search in our case is also slow due to the latency in setting up HTTP requests. So now we just calculate the maximum possible time-offset that a frame can be from its keyframe, and subtract that from our seek target. This means we will often download more data than necessary, but for us that's typically faster than doing another bisection search.
The moral of the story is that if you want your video to seek quickly, include regular keyframes!
Subscribe to:
Posts (Atom)