Simple rate limited HTTP server for testing HTML5 media/streaming

While working on the Firefox HTML5 video and audio support, I've found it extremely useful to have an HTTP server on which the transfer rate is reliably limited. Existing servers are either too heavy weight, like apache, or have inconsistent rate-limiting, like lighttpd which I found to have very "bursty" rate limiting.

I ended up taking the educational route, and implementing a simple HTTP server in C++. It supports the following features:

1. Support for HTTP1.1 Byte Range Requests. This means you can seek into unbuffered data when watching HTML5 video.
2. Rate limiting, configurable on a per request basis by passing the "rate=x" HTTP query parameter, where x is the transfer rate of the connection in kilobytes per second. The server will send x/10 KB ten times per second to maintain this rate smoothly.
3. Simulated live streaming, configurable on a request basis by passing the "live" query parameter. When in "live" mode, no Content-Length header is sent, and the server doesn't advertise or perform byte range requests - so you can't seek into unbuffered video/audio, just like in a live stream.
4. Cross platform; tested on Windows (runs on port 80) and Linux (runs on port 8080). I haven't test it on MacOS yet.
5. Simply serves all files in the program's working directory, making it easy to use (and abuse).
6. Open source! Get the code at https://github.com/cpearce/HttpMediaServer, or download a pre-built win32 binary.

For example, if you wanted to simulate a live stream being served at 100KB/s, your test URL might look something like http://localhost:80/video.ogg?rate=100&live.

I've been using it for quite a while, and over the weekend I finally cleaned it up and put it up on GitHub. Check it out.

How to get Tinderbox logs faster

David Baron pointed out at the "War on Orange" meeting today, you can get Tinderbox logs much faster if you avoid using the "showlog.cgi" program.

For example, if you have a build log URL:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284748812.1284749884.15885.gz

You can remove "showlog.cgi?log=" from the URL, and you'll download the log directly, i.e.:

http://tinderbox.mozilla.org/Firefox/1284748812.1284749884.15885.gz

This will download much faster, and you can grep for the text you're interested in directly in the downloaded log.

Steps to reduce random failures of media mochitests

Matthew Gregan and I have taken some steps to help reduce random orange in the media mochitests. Sayrer pointed out that during some random failures in test_play_events we were timing out with mmap() faliures and almost 130 threads running. It turns out that on 32bit Linux each thread stack is assigned 10MB of virtual address space, and when you're running a lot of threads you can run out of virtual address space, causing the timeouts.

We're probably particularly vulnerable to this since we're still not doing garbage collection in xpcshell during mochitest runs.

I spent some time ensuring we're more proactive at shutting down the media decoder's threads (each media element requires three threads, plus one more per media element on Linux used by the sound server), and I refactored our media tests to reduce the number of tests we run in parallel. This ensures our peak number of threads should be kept down, thus reducing peak memory usage/allocated virtual address space.

Concurrently to this, Matthew Gregan identified and fixed a thread safety issue in our media decoder and another in the underlying libsydneyaudio library on Linux, which we use for audio playback. We expect these two fixes to also reduce some random playback failures.

Since landing on these changes Thursday PDT, we've had a lot less random orange on mozilla-central. We're still getting some random failures, but most of these are on Linux, and many of those are caused by a known bug in the version of pulseaudio running on the Tinderbox mochitest machines. We expect many of these random failures will go away when the releng guys update pulseaudio on the Linux Tinderbox mochitest machines.

We're still getting a fair number of orange in test_progress, mostly on Windows, but we're going to be changing our media progress event to bring it into line with the current spec, and when we do that we'll be removing that test.

A new unwitting ally in the fight against random orange

I have discovered a new tool in our toolbox in the fight against random test failures: crashinjectdll.dll on Windows, nptest.so on Linux (I don't know what it's called on MacOS...). I suspect we're randomly injecting crashes into plugins, in order to test recovery from plugin crashes with out-of-process plugins (correct me if I'm wrong). If a mochitest run times out on Tinderbox, and a crash has been injected, when the mochitest run is killed we'll get call stacks in the output log for all the threads that were running. This can give us some details of the state of Firefox when the timeout occurred, and may give us a clue as to why we've timed out.

Maybe we should make always inject a plugin crash at the start of a mochitest run, so that we always have call stacks when we kill a timed-out mochitest run?

Next time someone comments in your random orange bugs with a log, check to see if it's got call stacks in it!

Replay Debugging mochitest failures with VMWare Workstation 7

Ever since my last escapades with Replay Debugging in VMWare Workstation 6.5, I've been looking forward to improvements in this awesome technology. Thankfully the guys at VMWare have been hard at work, and now VMWare Workstation 7 now boasts improved Replay Debugging. I've found it much more robust and reliable, and Roc and I have already used it to debug some random orange bugs.

I've documented how to produce a Replay Debugging setup for debugging intermittent test failures in Mozilla mochitests, and put it up on MDC:

https://developer.mozilla.org/En/Debugging/Record_and_Replay_Debugging_Firefox

Now anyone can setup a machine to record and replay debug intermittent mochitests! A word of warning: you need a modern CPU in order to get good performance. I had poor performance when running on my two-year-old Core2Duo laptop, but replay performance is almost at real-time speeds on my shiny new Intel i7 950 box.

I still have two patches that need to be refined and then checked in, to facilitate replay debugging. The first enables the mochitest harness to loop forever on a test directory. The second enables you to set break points on specific JavaScript dump() calls, so you can break during replay close to where the action is.

We're far from having a fully automated record and replay setup, but we've made a start!

Setting up VMware to record, replay and debug intermittent Mochitest failures

Edit 16 March 2010: This blog post is now out of date, and very likely wrong. The official documentation for setting up Replay Debugging for Firefox is now on the Mozilla Developer Center wiki: Record and Replay Debugging Firefox

Over the past few days I've been working to get VMware Workstation's Replay Debugging to work on Mozilla’s Mochitest suite. It's been a long process, but I've finally got something that records and can be replay-debugged! Replay debugging allows us to record everything that happens in a virtual machine, and then replay it back and step through the execution in a debugger. Often when we have an intermittent test failure, it's hard to reproduce (hence it's intermittent-ness). Now I can record a VM running Mochitests and if I record a test failure, I can replay the execution and step-through and see exactly what code paths where followed and hopefully figure out why. This is powerful, as it means we can deterministically and repeatedly reproduce an intermittent test failure in a debugger, making them a lot easier to debug.

Using replay debugging to debug intermittent test failures was originally Robert O'Callahan's idea. He had trouble setting this up on Linux, so he suggested I try it on Windows. It took a lot of messing around, but finally it works. The key lessons learned are:

• Create the record-and-replay build on a network drive mapped to the same path on both your host and guest systems. This means that the debug symbols have the same path to source files embedded in them on both systems. Also paths compiled into the executable (e.g.: assertion __FILE__:__LINE__ messages) are valid on both systems.
• When creating a recording of Firefox, start the recording before you start Firefox. I suspect that the replay-debugger must observe the DLLs being loaded at program startup in order to load debug symbols and thus allow the debugger to function.
• The settings for replay debugging and for remote debugging are totally unrelated.
• Project > Properties > Debugging > Command is the path to the executable which ran in the VM recording which the debugger will try to connect to when replaying.
• Your build needs to be --enable-libxul --disable-static.

It took a lot of messing around, so for my own record, and for the use of anyone who also wants to set up recording and replay-debugging of a Mochitest run, the exact steps I went through are:

1. Get a Windows machine with a supported CPU. Originally I had tried to set this up on a boot-camped Mac Mini, but that had a Core Duo processor, which is unsupported. My Vista laptop has a Core 2 Duo processor which is supported, so I've been working on that.
2. Install Visual Studio Professional on your host system. Microsoft has a free 90 day Trial of Visual Studio 2008 Professional available for download. VMware recommend you use 2005 Professional, but I've successfully used both 2005 and 2008. You'll need the build prerequisites for this installed of course.
3. Install VMware Workstation 6.5 on your host system. You must install this after Visual Studio, else its debugger plugin won't show up in Visual Studio.
4. Install a Windows OS in your VM. This is the "guest" system. I installed Windows XP SP3.
5. Install Visual Studio Professional on your guest system. You need this because it installs the Remote Debug Monitor. Visual Studio Express versions don't have this. Edit: Only required for remote debugging, not replay.
6. In your guest Windows OS, disable Windows Firewall. You can do this by running "firewall.cpl" at the command prompt. Edit: Only required for remote debugging, not replay.
7. In your guest Windows OS, set the security policy for "Network access: Sharing and security model for local accounts" to be "Classic - local users authenticate as themselves". You can access this from Control Panel > Administrative Tools > Local Security Policy > Local Policies. This setting allows the remote debugger to log into the VM system. Edit: Only required for remote debugging, not replay.
8. Create a network drive, and map it in both your host and guest system to the same path. This will store the builds you test, and ensure that the builds have symbols which have valid paths for both the host and guest machines. I created a new drive Z: on my host system. It was stored on an external hard disk, as my laptop's always running very low on space. You'll need lots of disk space.
9. It's a good idea to create a VM snapshot after setting up everything, so that these settings can't be lost. Every time you replay, the state of the VM is reset to the start of the recording. The state is also reset to the "initial snapshot" if you try to create a recording from inside Visual Studio. The state is saved if you shutdown the VM normally. This can wipe settings if you're not careful.
10. Check out the appropriate Mozilla source tree to the network drive.
11. Build your tree on the network drive. Ensure your build is an enable-libxul disable-static build, i.e. add to your .mozconfig: "ac_add_options --enable-libxul --disable-static". Without this I found that some the symbols for some DLLs weren't loaded (gklayout.gll in particular), so I couldn't set breakpoints where I wanted. I found building on a network drive took about 2.5 times longer than a normal build.
12. Create a new project in Visual Studio. You can't just create a project by opening an EXE file, the VMware menu is greyed out if you do this. You must create a new project file using the File > New > Project > Win32 > Win32 Console Application. I opted to create an empty project, and that works fine for our purposes.
13. Configure Project > Properties > Debugging and enter the Command as the path to firefox.exe on your network drive.
14. Boot up your guest operating system in your virtual machine. Start a new recording in your VM. We're going to create a recording from inside the VM, rather than initiating the recording from Visual Studio. This is important, because we can't (at least not easily) launch a Mochitest run in an MSYS shell in the guest operating system from inside the Visual Studio debugger. It's much simpler to just record the virtual machine while it's doing a Mochitest run. You must start the recording before firefox.exe starts up however, else the debugger may not connect to it when you replay.
15. In the guest operating system, run Mochitests until you reproduce a failure, timeout etc. Stop the recording.
16. In Visual Studio on the host system, configure VMWare. Open menu item VMWare > Options > Replay Debugging in VM. Set "Virtual Machine" to point to your VMX file for your guest operating system. Set "Recording to Replay" to the name of the recording you just recorded.
17. In Visual Studio on the host system, open the source files you want to put break points in from your network drive. Set breakpoints in them.
18. Press the "Debug an application running inside of a recording" button on the toolbar, or VMware > Start Replay Debugging.
19. The VM will start replaying the recording. It will be slow, and will take a few minutes to start up, but assuming you're configured correctly, it should replay, and execution should break on your break points. If the recording fails to start, check for error messages in the VMware output window in Visual Studio.

That's it! All the black magic required should be outlined above. Now, to fix some intermittent test failures...