I have discovered a new tool in our toolbox in the fight against random test failures: crashinjectdll.dll on Windows, nptest.so on Linux (I don't know what it's called on MacOS...). I suspect we're randomly injecting crashes into plugins, in order to test recovery from plugin crashes with out-of-process plugins (correct me if I'm wrong). If a mochitest run times out on Tinderbox, and a crash has been injected, when the mochitest run is killed we'll get call stacks in the output log for all the threads that were running. This can give us some details of the state of Firefox when the timeout occurred, and may give us a clue as to why we've timed out.
Maybe we should make always inject a plugin crash at the start of a mochitest run, so that we always have call stacks when we kill a timed-out mochitest run?
Next time someone comments in your random orange bugs with a log, check to see if it's got call stacks in it!