29 Feb Emergency Downtime

[Paul]

The box mysqld is on had its 3ware card fail. A replacement should be installed tonight Pacific time. Until then, www.castlecops.com is offline.

Our last full backup was about two days ago, so hopefully we won't have to restore that. The log files indicate only a couple index files are corrupt, which means we should be able to repair and walk away just fine.

The only error I'm finding is:

Got error 127 when reading table
aka
MySQL error code 127: Record-file is crashed

[Paul]

OK inspection of the error log file for mysqld, shows these further unique entries:

Got error 5 from storage engine = OS error code 5: Input/output error

So all in all, three key files and thirteen tables crashed and need repairing. Only one table has found a certain number of rows that need repairing.

Based on this I'm confident we'll be able to restore 99% of the site. The only question is that one table which has some rows that need repairing. It may be because of the failed 3ware card that the error is showing.

[Paul]

We're launching a server donation drive marathon:

http://www.castlecops.com/server_marathon.html

[MAPKOBKA]

Good luck with getting the site back up. It's always nice when a component decides today is the day to fry.


Made a small donation to the server drive... Wish it could have been more, but every little helps

[Paul]

Thank you. I hope to soon get a graphic online depicting our goal and where we stand in the marathon.

[Paul]

The card is in hand, and its being taken to the data center now.

[Paul]

OK the card is replaced, and rebuilds are underway.

[Paul]

OK we're no longer getting hardware errors showing up, but we do have a file system I/O failure. So we're improving.

We'll be down a bit longer, no ETA.

[Paul]

OK... some research reveals that we may end up with data loss. Won't know for certain until the current maintenance completes. If that is the case, we'll have roughly a day and a half of loss as I'll have to reload the last full backup from about 10:13PM two nights ago.

Can't wait for March.

[Paul]

My mistake, the loss goes back to:

Dump completed on 2008-02-27 19:01:22

UTC time.