White Wednesday

Jick announced via the KoL Forums:

Q: How many clicks does it take for Jick, when he's not paying enough attention, to destroy the contents of the primary database server.

A: Ask Mr. Owl.

O: One, Two, Three, Four. Four, because Jick is the biggest dumbass who has ever traversed this earth by shuffling his dumb, dumb buttocks in a grisly and pitiful parody of walking.

So, yeah. In an effort to drop a couple of tables from the database, I accidentally dropped the entire database. Which meant we had two options.

1) Get the host to try to undelete the files, since it was such that it required physical access to the box to attempt it, or

2) Restore from backups. This option is less desirable.

The first option proved impossible, partly because of technical issues, but mostly because of the host's antagonistic policies. Their techs are not allowed to interact with customer data in any way, due to "liability issues." We pressed the issue, but to no avail.

So, we have to restore from backup. We do two kinds of backups. Full backups, which are done during the times when rollover lasts a long time, and incremental "core data" backups that are done every night, about five hours after rollover.

The last full backup was done on September 6th. It had been a while, and I was getting nervous. That's why I announced a long rollover yesterday. Luckily, I managed to drop the database about 30 seconds before I was gonna back it up.

Ideally, this would've meant a one-day rollback. We'd have to manually process a day's worth of donations, and everybody would lose a day's progress, but not such a terrible thing, in the grand scheme of things.

In our less than ideal world, though, some stuff went wrong. There was an issue with the core backup that caused a couple of tables to not get backed up, and there were also a couple of other tables that had never been added to the core backup, since they were added since the last time I updated the backup script, which was towards the end of ascension testing. This is going to cause some data to revert to September 6th.

So, basically, here's what's gonna happen. As near as we can tell, inventory, player data, flags, quest status, all the core stuff, that'll swing back to what it was last night, five hours after rollover. Some people lose a day, some people don't. Mall store inventory is still up-to-date, since that's on the secondary server, along with kmail and a bunch of other stuff that wouldn't have been a big deal to lose.

However. This is where the groaning begins. There are a few important things that are gonna revert to September 6th, because they didn't, for one of the two reasons I described above, get saved during the nightly backup run. Notice how I use the passive voice to shift responsibility away from myself. I mean, notice how the passive voice is used by me.

Chief among them: Hagnk's. Yeah. I know. Ouch. Basically, your items in ascension storage will revert back to what they were 6 weeks ago. Which is gonna do, for a lot of people, one of two things: If somebody had stuff there yesterday that wasn't there six weeks ago, it'll be gone. And if somebody had stuff in there six weeks ago that is now in their inventory, it'll basically be duped.

Also, trade offers. This will also cause some duping and/or lost items, but it should be relatively minimal.

Also... and I know this is gonna hurt some people a lot, ascension records. We need to figure out what to do about this. A lot of the staff and dev team are in favor of a total reset on these, since they feel it'd be better than reversion. I'm open to suggestions from the community at large on how to handle it.

Last, but not least, some permanent skill data will be lost. We can reconstruct most of it. The worst case scenario will be the person who had a ton of softcore skills and was in Hardcore yesterday. Everybody else, we can reconstruct most, if not all of what was lost.

Right now (1PM my time on Wednesday,) what's going on is that we're actually restoring the data from last night's backup. It's been running all night, because apparently the scheme we used for backing stuff up is quick to dump it out, but incredibly slow to dump it back in. It's as aggravating for us as it is for you, because we can't really do ANYTHING until this is done.

I suppose this goes without saying, but I take full responsibility for this. It was a dumb mistake, and a series of prior dumb mistakes that screwed up the backup process. It wasn't Mr. Skullhead, it wasn't anybody else on the staff or the dev team, it was all me. And I will try to make it better.

We'll have some kind of special eventy type stuff after we see what kind of shape we're in, and at the very lest get everybody some sort of "I was there" bit of in-game goodies. Riff and I have a lot of time to sit around talking about stuff today while the restore runs, so I'm sure we'll come up with some kind of interesting narrative to provide an explanation for what happened and some nifty new content to soften the blow.

No idea how to end this. Thanks for your patience and support. We'll get through this together, us and y'all. You with your beverage or drug of choice, us with a mixture of equal parts coffee and Pepto-Bismol.

p.s. i love ween0rz

depending on the URL used, this appeared:

Warning: mysql_connect(): Can't connect to MySQL server on '10.0.0.1' (111) in /home/jick/htdocs/adventure.php on line 19 Could not connect to database server:0 -

---

This may so be that there are some that will lose many valuable items, and that some may gain duplicates of their valuable items. This could be in the range of Black Sunday, but not quite as bad.

For those of you who are witnessing this today, please speak your mind in discussion! Just type a line or so and leave some space between each comment.. leave your name if you wish, too.

Oh yay! No offense, but way to go Jick. Hey this might almost be good for me because I actually had some money then. ^_^ I wonder if we got a store in the last six weeks will we still have it? -Waterbou