Last night, May 26th, the site was down for about 3 hours. This happened after midnight, European time, which means most of us were sleeping.
Our administrator, Mads Jørgensen, was attentive though, and had to endure long hours with coffee and disk resizing.
What happened: Late last night, we started getting errors saying “no space left on device.” True enough, the space had almost run out (well, the system reported 6gb free, but OK), and what we did to remedy this was to truncate a large logfile, and restart the services. Everything then went humming along, and we figured no one was going to push 8gb over night, and we’d deal with it early this morning.
Not so.
Just after I had gone to bed, allegedly things began acting up once more. Mads jumped to action, and replaced the front page with status updates. If you want to read just exactly what went wrong, and how he remedied it, you can read it on his blog:
http://swag.dk/bitbucket/downtime_27052009.html
Again, Mads, you saved the day, and we wouldn’t want to continue without you!