I have discovered a new tool in our toolbox in the fight against random test failures: crashinjectdll.dll on Windows, nptest.so on Linux (I don't know what it's called on MacOS...). I suspect we're randomly injecting crashes into plugins, in order to test recovery from plugin crashes with out-of-process plugins (correct me if I'm wrong). If a mochitest run times out on Tinderbox, and a crash has been injected, when the mochitest run is killed we'll get call stacks in the output log for all the threads that were running. This can give us some details of the state of Firefox when the timeout occurred, and may give us a clue as to why we've timed out.
Maybe we should make always inject a plugin crash at the start of a mochitest run, so that we always have call stacks when we kill a timed-out mochitest run?
Next time someone comments in your random orange bugs with a log, check to see if it's got call stacks in it!
Friday, 9 April 2010
Friday, 2 April 2010
New Ogg video decoder for Firefox
Today I finally landed the new Ogg video decoder for Firefox/Mozilla. It should appear in Firefox nightlies from 2 April 2010 onwards. Chris Double did the initial implementation for the new Ogg decoder, and I took over in December, implemented seeking, and bashed it into shape.
This replaces our old Ogg decoder which used liboggplay. We'll now be able to implement new features easier, and have greater control over our decode pipeline. We can now control our audio and video decode separately, and this allows us to do things like not decoding video interframes if the video decode is taking too much time and causing the audio decode to fall behind. This prevents audio stutter when decoding large-frame-sized videos. The new backend also has more reliable seeking, and can even load a few weirdly formed Ogg files that the previous backend couldn't handle.
It was a large patch, and it's a testament to our robust set of video and audio unit tests that when I checked it into mozilla-central, there was basically only one small fix required. It has been a long grind getting the new decoder to work robustly, but our unit tests have been an invaluable, if fickle, task-mistress.
The new backend should at least be feature-equivalent to the old backend. If you spot a regression in Ogg video playback in new nightly builds, please file a bug in Core in the "Video/Audio" component!
This replaces our old Ogg decoder which used liboggplay. We'll now be able to implement new features easier, and have greater control over our decode pipeline. We can now control our audio and video decode separately, and this allows us to do things like not decoding video interframes if the video decode is taking too much time and causing the audio decode to fall behind. This prevents audio stutter when decoding large-frame-sized videos. The new backend also has more reliable seeking, and can even load a few weirdly formed Ogg files that the previous backend couldn't handle.
It was a large patch, and it's a testament to our robust set of video and audio unit tests that when I checked it into mozilla-central, there was basically only one small fix required. It has been a long grind getting the new decoder to work robustly, but our unit tests have been an invaluable, if fickle, task-mistress.
The new backend should at least be feature-equivalent to the old backend. If you spot a regression in Ogg video playback in new nightly builds, please file a bug in Core in the "Video/Audio" component!
Friday, 12 March 2010
Don't serve Ogg media with gzip/deflate compression
One common way to reduce the load on a web server is to use gzip or deflate compression when serving to a supporting web browser. When Firefox requests an Ogg media, it advertises that it can handle a gzipped or deflated response; the HTTP request includes the Accept-Encoding: gzip,deflate header. But despite Firefox advertising that it supports gzip/deflate, you probably don't want your web server to gzip or deflate Ogg media. If you serve an Ogg media compressed, Firefox won't be able to seek in the media, or determine its duration. Since the video/audio data in Ogg files is already compressed, gzip/deflate won't actually save you much bandwidth anyway, so you probably want to disable compression when serving Ogg files.
I have added this tip to the Mozilla Developer Center article on configuring servers for Ogg media.
I have added this tip to the Mozilla Developer Center article on configuring servers for Ogg media.
Sunday, 7 March 2010
Switched blog to custom domain
I have switched my blog to being on a custom domain hosted at Blogger. The address for my blog is now blog.pearce.org.nz. Previously I was using Blogger's publish via FTP to host my blog on my own domain, but they have discontinued that service. I tried running Wordpress on my own domain, but I couldn't get plugins to work, so without WP-Cache my blog was slow. I'm glad to have my blog restored to its former snappy look and feel.
Monday, 11 January 2010
Indexing keyframes in Ogg videos for fast seeking
Seeking in Ogg videos in HTML5 <video> over the network currently can be very slow. This is because when we seek an Ogg/Theora video to a target time, we must perform a bisection search over the file in order to find the target Theora video frame. If this is an interframe (which just records what's changed since its preceding frame), we must then perform another bisection search to find the interframe's keyframe, and then decode forwards to the target frame in order to completely construct it.
A reasonable bisection search implementation may require half a dozen HTTP requests to complete, so if we need to do two bisection searches per seek (once for the target frame, and once for the target frame's keyframe), we actually need to do about a dozen HTTP requests per seek...
If we knew in advance where the keyframes were, we wouldn't need to do any bisection searches; we could make just one HTTP request to the last keyframe preceeding the target frame. Clearly making fewer HTTP requests is faster.
Enter the Skeleton 3.1 with Keyframe Index track. This extends the existing Skeleton 3.0 metadata track to provide an index for every Theora video and Vorbis audio track in an Ogg media. This will enable players to make the optimal HTTP request when seeking in media files served over the internet, resulting in as fast seeking as possible when viewing online video.
It's nice if a video player's UI can display the playback duration of the Ogg media. Unfortunately the raw Ogg format does not store the duration either, so it must be calculated, which requires additional HTTP requests, and slows down the video loading. The Skeleton 3.1 track also includes the playback duration of its Ogg containing file, to eliminate this overhead and speed up loading.
I developed the Skeleton 3.1 with Keyframe Index track in conjunction with the folks at Xiph.org. See the Xiph.org Skeleton 3.1 with Keyframe Index wiki page for the work-in-progress specification. Any comments on the specification would be much appreciated, please send them to the Theora mailing list.
I have developed a prototype Ogg indexer, OggIndex, and also recent ffmpeg2theora nightlies will encode keyframe indexes if you specify the command line option -–seek-index. OggIndex and experimental indexing ffmpeg2theora nightlies are available for download.
To see how keyframe indexing improves network seeking performance of HTML5 Ogg/Theora <video>, you can download a development version of Firefox which can take advantage of indexes here:
http://pearce.org.nz/video/firefox-indexed-seek-linux.tar.bz2
http://pearce.org.nz/video/firefox-indexed-seek-macosx.dmg
http://pearce.org.nz/video/firefox-indexed-seek-win32.zip
If you already have a Firefox instance running, you'll need to either close your running Firefox instance before starting the index-capable Firefox, or start the index-capable Firefox with the --no-remote command line parameter.
To compare the network performance of indexed versus non-indexed seeking, point the index-capable Firefox to the indexed seek demo page.
You should notice a clear speed difference when seeking to an unbuffered position in the indexed media.
The Skeleton 3.1 with Keyframe Index specification is still being developed, but we hope to lock it down soon. We are planning to ship support for keyframe index-assisted seeking in Firefox 3.7.
A reasonable bisection search implementation may require half a dozen HTTP requests to complete, so if we need to do two bisection searches per seek (once for the target frame, and once for the target frame's keyframe), we actually need to do about a dozen HTTP requests per seek...
If we knew in advance where the keyframes were, we wouldn't need to do any bisection searches; we could make just one HTTP request to the last keyframe preceeding the target frame. Clearly making fewer HTTP requests is faster.
Enter the Skeleton 3.1 with Keyframe Index track. This extends the existing Skeleton 3.0 metadata track to provide an index for every Theora video and Vorbis audio track in an Ogg media. This will enable players to make the optimal HTTP request when seeking in media files served over the internet, resulting in as fast seeking as possible when viewing online video.
It's nice if a video player's UI can display the playback duration of the Ogg media. Unfortunately the raw Ogg format does not store the duration either, so it must be calculated, which requires additional HTTP requests, and slows down the video loading. The Skeleton 3.1 track also includes the playback duration of its Ogg containing file, to eliminate this overhead and speed up loading.
I developed the Skeleton 3.1 with Keyframe Index track in conjunction with the folks at Xiph.org. See the Xiph.org Skeleton 3.1 with Keyframe Index wiki page for the work-in-progress specification. Any comments on the specification would be much appreciated, please send them to the Theora mailing list.
I have developed a prototype Ogg indexer, OggIndex, and also recent ffmpeg2theora nightlies will encode keyframe indexes if you specify the command line option -–seek-index. OggIndex and experimental indexing ffmpeg2theora nightlies are available for download.
To see how keyframe indexing improves network seeking performance of HTML5 Ogg/Theora <video>, you can download a development version of Firefox which can take advantage of indexes here:
http://pearce.org.nz/video/firefox-indexed-seek-linux.tar.bz2
http://pearce.org.nz/video/firefox-indexed-seek-macosx.dmg
http://pearce.org.nz/video/firefox-indexed-seek-win32.zip
If you already have a Firefox instance running, you'll need to either close your running Firefox instance before starting the index-capable Firefox, or start the index-capable Firefox with the --no-remote command line parameter.
To compare the network performance of indexed versus non-indexed seeking, point the index-capable Firefox to the indexed seek demo page.
You should notice a clear speed difference when seeking to an unbuffered position in the indexed media.
The Skeleton 3.1 with Keyframe Index specification is still being developed, but we hope to lock it down soon. We are planning to ship support for keyframe index-assisted seeking in Firefox 3.7.
Friday, 20 November 2009
Replay Debugging mochitest failures with VMWare Workstation 7
Ever since my last escapades with Replay Debugging in VMWare Workstation 6.5, I've been looking forward to improvements in this awesome technology. Thankfully the guys at VMWare have been hard at work, and now VMWare Workstation 7 now boasts improved Replay Debugging. I've found it much more robust and reliable, and Roc and I have already used it to debug some random orange bugs.
I've documented how to produce a Replay Debugging setup for debugging intermittent test failures in Mozilla mochitests, and put it up on MDC:
https://developer.mozilla.org/En/Debugging/Record_and_Replay_Debugging_Firefox
Now anyone can setup a machine to record and replay debug intermittent mochitests! A word of warning: you need a modern CPU in order to get good performance. I had poor performance when running on my two-year-old Core2Duo laptop, but replay performance is almost at real-time speeds on my shiny new Intel i7 950 box.
I still have two patches that need to be refined and then checked in, to facilitate replay debugging. The first enables the mochitest harness to loop forever on a test directory. The second enables you to set break points on specific JavaScript dump() calls, so you can break during replay close to where the action is.
We're far from having a fully automated record and replay setup, but we've made a start!
I've documented how to produce a Replay Debugging setup for debugging intermittent test failures in Mozilla mochitests, and put it up on MDC:
https://developer.mozilla.org/En/Debugging/Record_and_Replay_Debugging_Firefox
Now anyone can setup a machine to record and replay debug intermittent mochitests! A word of warning: you need a modern CPU in order to get good performance. I had poor performance when running on my two-year-old Core2Duo laptop, but replay performance is almost at real-time speeds on my shiny new Intel i7 950 box.
I still have two patches that need to be refined and then checked in, to facilitate replay debugging. The first enables the mochitest harness to loop forever on a test directory. The second enables you to set break points on specific JavaScript dump() calls, so you can break during replay close to where the action is.
We're far from having a fully automated record and replay setup, but we've made a start!
Monday, 5 October 2009
Problems building Firefox with Visual Studio 2005 and Windows 7 SDK
When I tried to update my 1.9.2 tree this morning on my Windows box I quickly discovered that Ben Hearsum had recently updated our build system to require the Windows 7 SDK. However once I installed the new SDK and tried to build (using Visual Studio 2005's compiler), I got the following error:
shell32.lib(shguid.obj) : fatal error LNK1103: debugging information corrupt; recompile module.
This is a compatibility issue between Visual Studio 2008 and Visual Studio 2005 which only occurs when you compile in debug mode. There's a hotifx to resolve this issue. Once I installed this, I could magically build again. Thanks to biesi for pointing out the hotfix!
shell32.lib(shguid.obj) : fatal error LNK1103: debugging information corrupt; recompile module.
This is a compatibility issue between Visual Studio 2008 and Visual Studio 2005 which only occurs when you compile in debug mode. There's a hotifx to resolve this issue. Once I installed this, I could magically build again. Thanks to biesi for pointing out the hotfix!
Subscribe to:
Posts (Atom)