Thank god that's over!
So we're back, with no loss of time on the site (no r-word), as promised.
You may have already read the explanation of what happened on our downtime message, but if you missed it (or want to know more), I'll go ahead and explain it here.
For the past few months, one of our biggest threats to stability has been that our database file is gradually becoming corrupted (think of someone pulling files out of a filing cabinet and tossing them all over the floor). We're currently talking to MySQL, the people who make our database, and trying to determine if this is a problem in our usage, or (more likely) a bug in the database program itself, in which case we will help them track it down and fix it.
The temporary solution to this is a complex procedure where we dump the database completely and "re-import" it, creating a totally new database file free of any corruption. There is a way we can do this transparently, with only minimal downtime. The procedure takes about a day.
Unfortunately, we were too late - the site crashed while we were in the middle of dumping the database. Both copies of the database were so corrupt that they would crash immediately upon being loaded.
Our first choice, after realizing that we had this problem, was to restore from a backup made earlier that day, and then "replay" all of the events on Subeta that had happened that day (from about 10:30 am that morning until 7:45 pm when the site crashed). This is possible because the database logs all changes, and we can use those logs to replay events.
However, this method of recovery proved ineffectual - replaying yesterday's events against the backup merely brought the database to the same corrupt state that it was in before.
So we had no other choice but to go ahead and finish dumping and reimporting data (from the backup made yesterday morning). Once that completed (around 1:15 this afternoon), we began replaying yesterday's events, until the site caught up completely, and we had a copy of the database free of corruption.
There are upsides, however. Dumping and reimporting the database optimizes it (it shrank by about 66% as a result). An optimized database is a faster database.
So yeah, that's the whole story. We're working with MySQL to fix the corruption issue, and if that is taking too long or not turning out the way we want it to, we are going to look into alternative database systems. One way or another, we'll find a solution to the problem.
--Alex
So we're back, with no loss of time on the site (no r-word), as promised.
You may have already read the explanation of what happened on our downtime message, but if you missed it (or want to know more), I'll go ahead and explain it here.
For the past few months, one of our biggest threats to stability has been that our database file is gradually becoming corrupted (think of someone pulling files out of a filing cabinet and tossing them all over the floor). We're currently talking to MySQL, the people who make our database, and trying to determine if this is a problem in our usage, or (more likely) a bug in the database program itself, in which case we will help them track it down and fix it.
The temporary solution to this is a complex procedure where we dump the database completely and "re-import" it, creating a totally new database file free of any corruption. There is a way we can do this transparently, with only minimal downtime. The procedure takes about a day.
Unfortunately, we were too late - the site crashed while we were in the middle of dumping the database. Both copies of the database were so corrupt that they would crash immediately upon being loaded.
Our first choice, after realizing that we had this problem, was to restore from a backup made earlier that day, and then "replay" all of the events on Subeta that had happened that day (from about 10:30 am that morning until 7:45 pm when the site crashed). This is possible because the database logs all changes, and we can use those logs to replay events.
However, this method of recovery proved ineffectual - replaying yesterday's events against the backup merely brought the database to the same corrupt state that it was in before.
So we had no other choice but to go ahead and finish dumping and reimporting data (from the backup made yesterday morning). Once that completed (around 1:15 this afternoon), we began replaying yesterday's events, until the site caught up completely, and we had a copy of the database free of corruption.
There are upsides, however. Dumping and reimporting the database optimizes it (it shrank by about 66% as a result). An optimized database is a faster database.
So yeah, that's the whole story. We're working with MySQL to fix the corruption issue, and if that is taking too long or not turning out the way we want it to, we are going to look into alternative database systems. One way or another, we'll find a solution to the problem.
--Alex
Posted by 
alx
User not found:
AHHHHHHHHHHHHH
0
Daighnaulte
All of your work is really appreciated ,and updates. Thank you so much for all your hard work and glad we're back<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
We love you so much darling =)
0
User not found:
OMG, that was hard for me kinda, I sleep in really late and it was like ten o clock at night when it was back up where I am.
0
User not found:
Alex...thanks for your hard work and for the explanation <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />...the page is better since the updated...
peace!
peace!
0
Coppy
Dear Alex,
thank you for all you do! the site is running so much faster now! sincerely, coppy
thank you for all you do! the site is running so much faster now! sincerely, coppy
0
Paddy
Thanks for the detailed info, and all your hard work yesterday! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Morticia
Still the site isn't working properly at the moment. I see an error line and somehow the bottom of the page does not know that it's supposed to be at the bottom. But I guess you already know that. I just do as I always do after a big bang on Subeta, wait 3 days and if the problem is still there ... make contact.
0
Morticia
Thank you for protecting the hard work every user has done for his or her account. Thanks for the update. Thank you for all the hard work.
Good luck in finding the problem.
Good luck in finding the problem.
0
User not found:
not mysql...php wonkyness i mean...sorry for double posting...just had to fix the error in what is going on with the site right now i experiencing...ya know the "Warning: invalid argument" php code messages that are on top pages or whatnot i experiencing now
0
User not found:
thankies for the update on what happened...now that i finally respond to this now there are mysql wonkyness with the site i guess at this very moment with coding...must be fixing something with the database...you all keep up the hard work...i loves coming to Subeta to unwind and have fun <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
Yay thanks programmers <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
So hopefully not much more downtime within the next few days - right?
So hopefully not much more downtime within the next few days - right?
0
User not found:
o.O the -r- word...lol
Well at least you guys got subeta back up and running and I agree with what onlysimplicity
is saying =D
Thanks for your hard work---
Well at least you guys got subeta back up and running and I agree with what onlysimplicity
is saying =D
Thanks for your hard work---
0
onlysimplicity
Thanks to all for their hard work. I very much appreciate it and, although a little disappointed with, also recognize the need for downtime, so as such don't become frustrated with them. Thank you!
0
Alleycat
Thanks again for the update Alex. It is so nice to be informed on what is happening, don't ever leave us!
0
gidgetron
Great work guys! Glad to see Subeta is back up and running <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
Okie! :3 Thanks for bringin it back up.
0
LadyJan
It was a rough day without my Subeta fix, but I want to thank you for the efforts and for the explanation. You rock!
0
Silvy
thank you SO MUCH alex! Thank you for the updates and for all your hard work <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> it's greatly appreciated *hugs*
0
Witch
Your efforts are much appreciated<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
AdamRose444
I wanna thank all the staff for the hard work you guys put in last night. So Thank You!!!!
0
TERRORYAKI
Thanks for your hard work Subeta! <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
User not found:
Oh, I'm sorry to hear that. I was kinda impatient when I kept coming back to see the downtime Feli, but now that I hear what you guys have been through I feel really bad >_<''. Sorry, and thanks so much for working hard for us all <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />.
0
Notion
Thank you for taking the time to let everyone know what was going on. We appreciate all the hard work you put into this site! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Loki_826
Thank you so much for this detailed explanation and all the hard work! We appreciate it! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
mister_ed
Thank you Alex!!! <img src="http://images.subeta.net/smilies/6398_emoticon_neko.gif" border="0" />
0
Weezy_897
Thank you very much for then information I am for ever grateful for all of everyone's efforts to make subeta the best place to be x-D
0
SethA989
Alex, words are not enough to show my gratitude to you for making this site running and up to date day after day. From the bottom of my black dead heart. I give my most humblest thanks to you for your hard work and determination.
0
User not found:
Thanks Alex! Subeta loves you<img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
User not found:
thanks for explaning what happend !!!
0
User not found:
thanks alex i am glad you were able to do what you did <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> and i hope that means we gat to dance longer so my daughter can play and get a few items she will be over tomorrow lol..... i will say a prayer for a quick and painless solution to subetas problems witht he database and mysql
0
User not found:
Sweet, no rollback!
0
SkyWolfAlpha
Heh, glad I missed the downtime.
Reducing it by 66%? That's AMAZING!
Here's a neat, orderly, and corruption-free database, and faster calls for everyone! *raises a glass in a toast*
Reducing it by 66%? That's AMAZING!
Here's a neat, orderly, and corruption-free database, and faster calls for everyone! *raises a glass in a toast*
0
SeriousGreen
Great work guys! I was just beginning to go into second stage withdrawal symptoms...the ttttwitching has just about stopped. Happy squirrel hunting. <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
User not found:
Thanks for avoiding the dreaded rollback! And for keeping us informed too! Keep up the good work :3
0
StarofEvening
Thank you for keeping us informed and for no rollback!! <3<3 You are the best, absolutely.
0
Kiarae
I dont' understand lots of that, but I do know that I love to be informed like this. Thank you so much for keeping everyone up to date and getting Subeta back up for everyone!
0
Tachi
And this is what i wov about ya, Alex, you take the time to explain all this to us, thank you so much. <3
0
Alkuna
I'm just grateful you guys are working as hard as you are for all of us. believe me, I can live with losing a day or two of subeta if it means you guys are actually making the effort to fix the problems. And we have proof that you are: the site gets fixed! So y'know what? who cares about the downtime! *huggles you all* you fixed it and that's all that matters! <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
GreenRowan
I can't tell you how much it means to have you communicate and explain to us. Your hard work is appreciated, as is your attention to your customers and players!
0
iamnotepad
Keep up the hard work guys - I'm grateful you guys keep us in the loop on this stuff <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> I'm cool with down time if it means less (eventually) later on <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Hopefully you guys can find a solution soon!
0
User not found:
Thanks for explaining everything to us, Alex. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> We appreciate your hard work! Keep it up! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" /> I hope things go well for you.
0
Tennyo
Well glad you could fix it! <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
Sirenia
thank you for all your hard work and for keeping us informed and i too realised how much i missed the site but i found out there is sun and grass and things outside wow it was awesome lol.your subeta is so great and you all get involved in making it a great fun place even with the bad .thanks again.
0
znogirl
Thanks for keeping the explanation simple, Alex. You guys work so hard! And your efforts are very much appreciated. <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
HighQueenSelundar
I have no idea what most that meant as I am technologically illiterate...all i want to say is I am so glad Subeta is back! I was starting to have withdraws!
0
User not found:
to**
0
User not found:
thanks alex and how do you ask someone do dance??
0
Aperture_168
Thanks for keeping us up to date! I know that we all really do appreciate it. And thank you to the awesome staff who've been working so hard on this for us.
0
User not found:
=o Did i loose gold account days? = Argh
0
YeranceK
Thanks for the explaination and that u all worked so hard to get it back up and with no loss to us users <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Great Job! I know we all appreciate it <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> (Although we did miss the site and were bored out of our minds lol) <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> All better now <img src="http://images.subeta.net/smilies/6398_emoticon_wink.gif" border="0" />
0
Rarre
thanks for putting in all the hard work <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Its so nice to get all the details <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
This over 12 hours of downtime got me realizing how much I love Subeta, I spent the entire day refreshing the page. XD
0
ricecake
It reminded me about dumplings <img src="http://images.subeta.net/smilies/smiley_xface.gif" border="0" />
Thanks for letting us know <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
Thanks for letting us know <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
LadyTerentia
Thanks for all your hard work, and letting us know what was going on! I'm glad to have the site back without a rollback being needed. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Palitato
BAD Database! NO MORE CORRUPTION! -smacks it-
Thanks to everyone whose worked so hard on fixing everything. =D No more corruption cookies for the database! Not even on the holidays! It is on a strict diet. :K
Thanks to everyone whose worked so hard on fixing everything. =D No more corruption cookies for the database! Not even on the holidays! It is on a strict diet. :K
0
Rainey
~*~Gives you a standing ovation for a job well done!!~*~ I love this site and ALL the people who make it possible for us to play here!!
0
SharpCrayon
Props for the lack of that scary r-word and another successful restore/replay/refixing <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
BritishMindslave
Wow, nice work. c: That's all pretty hard stuff. I do hope that the Wizard Exchange is back soon, though. <img src="http://images.subeta.net/smilies/smiley_sadface.gif" border="0" /> Don't work yourselves TOO hard.
0
Allura
I love how explanations are given on subeta. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
sindos
An optimized database is a happy database.
Relief! the items are were I left them! thanks for all the efforts guys! we appreciate!
Relief! the items are were I left them! thanks for all the efforts guys! we appreciate!
0
Julie
Wow, that's a lot of computer drama. Thanks for the explanation and I'm glad you guys are back up and running. *goes off to the cash shop to help with the expense* <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
Temper
good job you guys <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
User not found:
Welcome back Subeta and I hope we don't have to go through all this again as I know you all have worked so hard to keep the site up and running. I must say that you all have done a great job. The Information you gave us is greatly appreciated. Thank you.
0
Sathalan
thanks for all the hard work you've done <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
Doe
Thanks for the information. Always nice to know what's going on. Thank you, Alex.
Also, thanks to everyone who keeps the site running and gets the site running again after problems occur. ♥
Also, thanks to everyone who keeps the site running and gets the site running again after problems occur. ♥
0
Lisa
*clings to Subeta* I MISSED YOU! *bawls like an idiot* Oh...and thanks for all the hard work, info, and most of all, lack of the r-word.
0
Wabbit
One word: YAY! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
why
A big big thank you to everyone who worked so hard to fix this.
You are awesome! *sending some love thru the computer* <img src="http://images.subeta.net/smilies/6398_emoticon_blush.gif" border="0" />
You are awesome! *sending some love thru the computer* <img src="http://images.subeta.net/smilies/6398_emoticon_blush.gif" border="0" />
0
Sahi
Thanks for all the hard work.
Perhaps another database provider would give you better reliability?
Perhaps another database provider would give you better reliability?
0
User not found:
No doubt it drove you at Subeta even more nuts than us players wanting to get back. Glad you were able to fix it, though.
0
o0_CareBear_0o
Thanks Alex For The Update And Thanks For All The Hard Work You And Everyone Else Is That Is Trying To Keep Subeta Up For Us.. I Am Just Glad It's Back Up!! I Have Missed It!! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Keep Up The Awesome Work!!! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
Thank you so much for all your hard work and keeping us informed all the time!
You gguys all did a great job and I missed the site SO much!!!
So just thank you for the great job and that you managed everything so great without any lost of data <img src="http://images.subeta.net/smilies/6398_emoticon_wink.gif" border="0" />
Feel hugged!
You gguys all did a great job and I missed the site SO much!!!
So just thank you for the great job and that you managed everything so great without any lost of data <img src="http://images.subeta.net/smilies/6398_emoticon_wink.gif" border="0" />
Feel hugged!
0
Liz_885
I'm really glad there was no rollback.
last time was really discouraging.
so, thank you guys <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
last time was really discouraging.
so, thank you guys <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
OtakuOokami
Thank you so much! We all appreciate how hard you guys work and the constant status reports to keep us up to date.
0
User not found:
Thanks for letting us in on what's going on! Good luck on getting things fixed! :3
0
User not found:
Thank you for keeping us informed and keeping Subeta going strong <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
miz_gadget
I really appreciate the explanation! I still love Subeta and all the people that run it!
0
Ruby
Thanks for telling us what is going on and for all of your hard work. Everyone really appreciates it <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
mayla
Thank you to all of the wonderful staff for working so hard and keeping your users informed. I know that I (and probably many, many other users) appreciate everything that you do to keep this a great site!
0
draithypie
Glad to see the site up and running (after seeing just how addicted I'd become in its absence... XD...)
Thanks for keeping us informed and updating during the downtime so we know what's going on... We appreciate being in the loop!!
Thanks for keeping us informed and updating during the downtime so we know what's going on... We appreciate being in the loop!!
0
User not found:
what did you do when it was down?
0
Katlamos
Wow, the database was that bad o_O And like everyone else is saying, thanks for keeping us updated and for all the hard work <3
0
fotfoh
Thanks for working so hard to get Subeta back online <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> *feels back at home*
0
Elvira
When you say replaying yesterdays events, almost sounds time travelish hehe. Excellent job!!!
0
User not found:
im so glad it's back, thanks for the hard work, and telling your subeta citizens whats up <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Azriel
I'm so glad Subeta is back! I can't imagine how taxing it must have been to go through all that work for the site, but it's fantastic to have such dedication going into the work. Thank you so much!
0
User not found:
all you staffers who worked on this are fantastic, thank-you so much! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
TwistedMiss
Tough day for you... but thanks so much for your hard work. It sounds like it paid for it self in system stability. Way to go.
0
Friskysquirt
Thank you for the explanation Alex, and it was beautifully exlained as always.
And I think me and the rest of the users are certainly noticing how much better the site is running now.
We really do appreciate your hard work *squishes*
And I think me and the rest of the users are certainly noticing how much better the site is running now.
We really do appreciate your hard work *squishes*
0
taffyann
Thanks for explaining. You guys rock! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
xpandagirl
thanks for keeping us in the know. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Innerflame
Alex, I am thankful for what you and the rest do for this site. Didn't realize how addicted I was to this place lmol.
0
CARRINGTON
Well thank goodness for back up stuff..
The dedication around here is awesome and usually hard to come by on site like this.
You guys are great, keep it up!
^^
The dedication around here is awesome and usually hard to come by on site like this.
You guys are great, keep it up!
^^
0
Otter
Thank you for keeping us up to date on everything.All who worked on this has are thanks and appreciation for there hard work
0
Santa
Thanks for all the work and keeping us updated Alex. <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
User not found:
Keep working, dude, you're awesome
0
Volpe
Thank you for working so hard for us, Alex, and telling us all whats been going on. <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" /> AND NO R-WORD! HUZZAH~! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
KadoTheCat
Thank you Alex. I am liking this way of keeping us in the loop. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Marcus
Thank you for the explanation. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> We appreciate your hard work for the site!
0
Iason567
Thanks for taking the time to do all this! I'm alright with downtime, so long as things are done effectively, which it sounds like they were. So, thanks for all your hard work <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />.
0
hourglass
thanks so much for keeping us updated!!! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
User not found:
Thank you, Alex. You all work so hard for us! ♥
0
User not found:
Thanks for making sure there was no rollback! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
And for the detailed explanation.
♥
And for the detailed explanation.
♥
0