Power corrupts. Power failure corrupts absolutely.
We're currently recovering servers from a power failure in our colocation facility. This means backing up 170gb of database on several servers and running recovery. Back soon...
Where's my Wikipedia?
In a hurry to look something up? Try a google search for the page you need and view it from the google cache...
Wikimedia fundraising drive
Wikimedia Foundation fundraising page is available offsite for the time being.
What happened?
At about 14:15 PST some circuit breakers were tripped in the colocation facility where our servers are housed. Although the facility has a well-stocked generator, this took out power to places inside the facility, including the switch that connects us to the network and all our servers.
What's wrong?
After some minutes, the switch and most of our machines had rebooted. Some of our servers required additional work to get up, and a few may still be sitting there dead but can be worked around.
The sticky point is the database servers, where all the important stuff is. Although we use MySQL's transactional InnoDB tables, they can still sometimes be left in an unrecoverable state. Attempting to bring up the master database and one of the slaves immediately after the downtime showed corruption in parts of the database. We're currently running full backups of the raw data on two other database slave servers prior to attempting recovery on them (recovery alters the data).
If these machines also can't be recovered, we may have to restore from backup and replay log files which could take a whle.
