Thank god that's over!
So we're back, with no loss of time on the site (no r-word), as promised.
You may have already read the explanation of what happened on our downtime message, but if you missed it (or want to know more), I'll go ahead and explain it here.
For the past few months, one of our biggest threats to stability has been that our database file is gradually becoming corrupted (think of someone pulling files out of a filing cabinet and tossing them all over the floor). We're currently talking to MySQL, the people who make our database, and trying to determine if this is a problem in our usage, or (more likely) a bug in the database program itself, in which case we will help them track it down and fix it.
The temporary solution to this is a complex procedure where we dump the database completely and "re-import" it, creating a totally new database file free of any corruption. There is a way we can do this transparently, with only minimal downtime. The procedure takes about a day.
Unfortunately, we were too late - the site crashed while we were in the middle of dumping the database. Both copies of the database were so corrupt that they would crash immediately upon being loaded.
Our first choice, after realizing that we had this problem, was to restore from a backup made earlier that day, and then "replay" all of the events on Subeta that had happened that day (from about 10:30 am that morning until 7:45 pm when the site crashed). This is possible because the database logs all changes, and we can use those logs to replay events.
However, this method of recovery proved ineffectual - replaying yesterday's events against the backup merely brought the database to the same corrupt state that it was in before.
So we had no other choice but to go ahead and finish dumping and reimporting data (from the backup made yesterday morning). Once that completed (around 1:15 this afternoon), we began replaying yesterday's events, until the site caught up completely, and we had a copy of the database free of corruption.
There are upsides, however. Dumping and reimporting the database optimizes it (it shrank by about 66% as a result). An optimized database is a faster database.
So yeah, that's the whole story. We're working with MySQL to fix the corruption issue, and if that is taking too long or not turning out the way we want it to, we are going to look into alternative database systems. One way or another, we'll find a solution to the problem.
--Alex
So we're back, with no loss of time on the site (no r-word), as promised.
You may have already read the explanation of what happened on our downtime message, but if you missed it (or want to know more), I'll go ahead and explain it here.
For the past few months, one of our biggest threats to stability has been that our database file is gradually becoming corrupted (think of someone pulling files out of a filing cabinet and tossing them all over the floor). We're currently talking to MySQL, the people who make our database, and trying to determine if this is a problem in our usage, or (more likely) a bug in the database program itself, in which case we will help them track it down and fix it.
The temporary solution to this is a complex procedure where we dump the database completely and "re-import" it, creating a totally new database file free of any corruption. There is a way we can do this transparently, with only minimal downtime. The procedure takes about a day.
Unfortunately, we were too late - the site crashed while we were in the middle of dumping the database. Both copies of the database were so corrupt that they would crash immediately upon being loaded.
Our first choice, after realizing that we had this problem, was to restore from a backup made earlier that day, and then "replay" all of the events on Subeta that had happened that day (from about 10:30 am that morning until 7:45 pm when the site crashed). This is possible because the database logs all changes, and we can use those logs to replay events.
However, this method of recovery proved ineffectual - replaying yesterday's events against the backup merely brought the database to the same corrupt state that it was in before.
So we had no other choice but to go ahead and finish dumping and reimporting data (from the backup made yesterday morning). Once that completed (around 1:15 this afternoon), we began replaying yesterday's events, until the site caught up completely, and we had a copy of the database free of corruption.
There are upsides, however. Dumping and reimporting the database optimizes it (it shrank by about 66% as a result). An optimized database is a faster database.
So yeah, that's the whole story. We're working with MySQL to fix the corruption issue, and if that is taking too long or not turning out the way we want it to, we are going to look into alternative database systems. One way or another, we'll find a solution to the problem.
--Alex
Posted by alx
User not found:
AHHHHHHHHHHHHH
All of your work is really appreciated ,and updates. Thank you so much for all your hard work and glad we're back<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
User not found:
We love you so much darling =)
User not found:
OMG, that was hard for me kinda, I sleep in really late and it was like ten o clock at night when it was back up where I am.
User not found:
Alex...thanks for your hard work and for the explanation <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />...the page is better since the updated...
peace!
peace!
Thanks for the detailed info, and all your hard work yesterday! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
Still the site isn't working properly at the moment. I see an error line and somehow the bottom of the page does not know that it's supposed to be at the bottom. But I guess you already know that. I just do as I always do after a big bang on Subeta, wait 3 days and if the problem is still there ... make contact.
Thank you for protecting the hard work every user has done for his or her account. Thanks for the update. Thank you for all the hard work.
Good luck in finding the problem.
Good luck in finding the problem.
User not found:
not mysql...php wonkyness i mean...sorry for double posting...just had to fix the error in what is going on with the site right now i experiencing...ya know the "Warning: invalid argument" php code messages that are on top pages or whatnot i experiencing now
User not found:
thankies for the update on what happened...now that i finally respond to this now there are mysql wonkyness with the site i guess at this very moment with coding...must be fixing something with the database...you all keep up the hard work...i loves coming to Subeta to unwind and have fun <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
User not found:
Yay thanks programmers <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
So hopefully not much more downtime within the next few days - right?
So hopefully not much more downtime within the next few days - right?
User not found:
o.O the -r- word...lol
Well at least you guys got subeta back up and running and I agree with what onlysimplicity
is saying =D
Thanks for your hard work---
Well at least you guys got subeta back up and running and I agree with what onlysimplicity
is saying =D
Thanks for your hard work---
Thanks to all for their hard work. I very much appreciate it and, although a little disappointed with, also recognize the need for downtime, so as such don't become frustrated with them. Thank you!
Thanks again for the update Alex. It is so nice to be informed on what is happening, don't ever leave us!
Great work guys! Glad to see Subeta is back up and running <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
User not found:
Okie! :3 Thanks for bringin it back up.
It was a rough day without my Subeta fix, but I want to thank you for the efforts and for the explanation. You rock!
thank you SO MUCH alex! Thank you for the updates and for all your hard work <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> it's greatly appreciated hugs
Your efforts are much appreciated<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
I wanna thank all the staff for the hard work you guys put in last night. So Thank You!!!!
Thanks for your hard work Subeta! <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
User not found:
Oh, I'm sorry to hear that. I was kinda impatient when I kept coming back to see the downtime Feli, but now that I hear what you guys have been through I feel really bad >_<''. Sorry, and thanks so much for working hard for us all <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />.
Thank you for taking the time to let everyone know what was going on. We appreciate all the hard work you put into this site! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
Thank you so much for this detailed explanation and all the hard work! We appreciate it! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
Thank you Alex!!! <img src="http://images.subeta.net/smilies/6398_emoticon_neko.gif" border="0" />
Thank you very much for then information I am for ever grateful for all of everyone's efforts to make subeta the best place to be x-D
Alex, words are not enough to show my gratitude to you for making this site running and up to date day after day. From the bottom of my black dead heart. I give my most humblest thanks to you for your hard work and determination.
User not found:
Thanks Alex! Subeta loves you<img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" /> <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
User not found:
thanks for explaning what happend !!!
User not found:
thanks alex i am glad you were able to do what you did <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> and i hope that means we gat to dance longer so my daughter can play and get a few items she will be over tomorrow lol..... i will say a prayer for a quick and painless solution to subetas problems witht he database and mysql
User not found:
Sweet, no rollback!
Heh, glad I missed the downtime.
Reducing it by 66%? That's AMAZING!
Here's a neat, orderly, and corruption-free database, and faster calls for everyone! raises a glass in a toast
Reducing it by 66%? That's AMAZING!
Here's a neat, orderly, and corruption-free database, and faster calls for everyone! raises a glass in a toast
Great work guys! I was just beginning to go into second stage withdrawal symptoms...the ttttwitching has just about stopped. Happy squirrel hunting. <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
User not found:
Thanks for avoiding the dreaded rollback! And for keeping us informed too! Keep up the good work :3
Thank you for keeping us informed and for no rollback!! <3<3 You are the best, absolutely.
I dont' understand lots of that, but I do know that I love to be informed like this. Thank you so much for keeping everyone up to date and getting Subeta back up for everyone!
And this is what i wov about ya, Alex, you take the time to explain all this to us, thank you so much. <3
I'm just grateful you guys are working as hard as you are for all of us. believe me, I can live with losing a day or two of subeta if it means you guys are actually making the effort to fix the problems. And we have proof that you are: the site gets fixed! So y'know what? who cares about the downtime! huggles you all you fixed it and that's all that matters! <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
I can't tell you how much it means to have you communicate and explain to us. Your hard work is appreciated, as is your attention to your customers and players!