Ghost In The Machine
May 24th, 2007
I’ve noticed something odd about the Tp test we run on our performance tinderboxes.
It seems our tinderbox gets a little cranky in the morning. To be fair, 6 am is quite early.
A lot of people asked me where they could get the medications I’ve been prescribed when I claimed that there was a possible Tp regression after an inexplicable performance hiccup on our Windows performance box earlier this week. You see, there was a checkin that we expected would improve perf a bit, and it seemed to. Below, you’ll see where The Checkin occurred. Afterwards, it got some test runs in, and they were pretty closely grouped. The graph below shows the ceiling it established.
It could be that we were just lucky, and the box gave us some good stats before it went nuts. We don’t really know. I heard a lot of claims that the box was “within the range” we would expect after the numbers came back down. Well, yes, if you assume that none of the checkins that occurred while the box was acting strange regressed Tp. That’s the most optimistic assessment, but there is no rational reason to pick that possibility out of many, especially when the other numbers settled to something pretty much identical to what they were before.
It could be that any possible regression is too small to justify the time spent testing builds from before and after. But, I didn’t hear that objection.
May 25th, 2007 at 1:38 am
David Baron told me a story once about how numbers on a Linux box would come back like this. It went something like: Firefox came up in a window, the tests ran, Firefox was closed, the build cycled, Firefox came up in a window, but shifted 10 pixels down and to the right (as window managers are sometimes prone to do).
While this happened, more and more of the window was going off-screen each time the tests ran. The incremental performance changes were due to the fact that X didn’t need to draw those portions of the window.
You’re probably seeing something else, but I thought I’d share that story with the rest of the class.
May 25th, 2007 at 1:55 am
Nightly cron scripts? Fedora always goes into overdrive for a little bit at 4am, and it’s not unreasonable to think something similar might happen on this machine at 6.
May 25th, 2007 at 11:03 am
Yeah, cron is what I would guess. Especially indexing tools like updatedb and whatnot….
May 26th, 2007 at 1:51 pm
My first thought is also “what is specifically run at our around 6am every morning?” and my first place to search would also be cron…
May 27th, 2007 at 7:28 am
I was going to suggest a daily cron script was getting fired but that seems to be too popular. So I’m going with monkeys.