Recently, we ran in to a problem with our web content sync setup.
Old Setup:
- Host all web bits on two admin servers
- Pull via rsync from admin01 server to the apache servers on 5 min staggered cron
That setup worked fairly well for us for quite a long time, but it doesn’t scale. At about 18 or 20 apache servers pulling at 5 min intervals the admin server was constantly pegged from all the rsync processes scanning the file system.
We needed something with state, but something that was simple. Revision control has state, but would the operations be quick enough to be useful? It turns out Git was pretty well suited for the task.
New setup:
- Host all bits on admin servers
- Commit bits on admin01 on a 5 minute cron in to git
- Pull commits via Git to apache servers
This new setup scales very well, because we only need one file system scan per 5 minutes instead of 20+. The Git fetches are very fast.
Initial installation:
admin01:/etc/xinetd.d/git-daemon:
service git
{
disable = no
type = UNLISTED
port = 9418
socket_type = stream
wait = no
user = nobody
server = /usr/bin/git-daemon
server_args = --inetd --export-all --verbose /www
log_on_failure += USERID
}
Admin01: Import /www:
cd /www
git-init
git-add .
git-commit -m 'Initial Import'
Apache Servers initial setup:
git-clone git://admin01/www
Commit Cron on admin01 (add any new files and delete any removed files from the repo):
#!/bin/bash
lockfile="/tmp/git.lock"
if [ -f $lockfile ]; then
if kill -0 $(cat $lockfile); then
echo "$0 appears to be already running."
exit 1
fi
fi
echo $$ > $lockfile
cd /www
/usr/bin/git-add .
/usr/bin/git-commit -a -m "AUTO COMMIT: `date +%FT%T`"
rm -f $lockfile
Fetch Cron on the apache servers (fetches origin and then resets the /www to origin):
#!/bin/bash
cd /www;
/usr/bin/git-fetch && /usr/bin/git-reset --hard origin;
Dan 11:15 am on September 21, 2009 Permalink |
Include the timezone with the time. Looking at the scheduled maintenance for the 15th is ambiguous on 2 counts:
Time Description
00:27:10 Site maintenance @7:10pm.
Oh, and no fail pets!
ozten 12:01 pm on September 21, 2009 Permalink |
This is going to be a really useful page!
Maybe give a place for someone to report an issue on a green site, that should be red? This would put them into bugzilla with the correct product/component?
If this site had optional LDAP authentication, it would be great to display links to our monitoring dashboards for logged in users, on a site by site basis. (so you can dig deeper and see DB health, etc)
Philipp 2:25 pm on September 21, 2009 Permalink |
developer.mozilla.org should be added, this seems to break quite often.
Stephen Donner 12:00 am on October 12, 2009 Permalink |
Jeremy, could it also include SVN revision #’s? Not all our pages/apps have it in the source (in fact, I think only AMO does), and this would really help QA and dev; thanks!
Stephen Donner 12:02 am on October 12, 2009 Permalink |
Also, can you add a Mozilla.com background? The white is a bit jarring