Hi everyone. Unfortunately, as many of you noticed we had 2 moderate (more than 15 minutes) periods of downtime today, one this morning and one this evening. I wanted to let you guys know what happened, and what needs to happen as a result. :)
This morning, our database server locked up (which happens to us occasionally, you could call it "growing pains.") This morning's lockup was particularly bad as all 3 web servers became overloaded with people trying to load the site, and so slow that it took forever for me to log into any of them (for example, to take the site down and stop people from hammering the web servers xD)
To further complicate matters, the database became corrupted as a result of the lockup, and refused to start. At that point we switched to our backup database, which is synchronized with the main one, and brought the site back up.
The downtime this evening was the result of db2 locking up, which was corrected with a restart of the database.
This left our primary database server offline though, so Subeta will be taken down over Monday/Tuesday (Tuesday morning) at 3 am in order to synchronize the two and return to operating on db1.
In case you're curious why the database locks up like this: Subeta uses caching to save the answers to frequently asked questions that would otherwise be asked of the database (for example: "give me a random avatar to show on the homepage?"). The cache expires after a certain time, however, and if there's lots of users on the site, hundreds of users suddenly ask the database the same question. You'd think this wouldn't cause a problem, but the database really doesn't know that lots of people are asking the same thing, so it dutifully tries and answer the same question hundreds of times. Meanwhile, all the normal Subeta queries get backed up while 200 database threads compete over access to the same information, and eventually it gets so bogged down that it only answers 100 or so questions/second, as opposed to the 5,000 we normally run.
There are ways around this, of course, and we've been implementing them whenever we come across this problem. For the moment, though, whenever we fix one of these problems (like the random HA on the home page), another one steps up to take its place. Rest assured that each time this happens, progress is being made!
Anyway, in summary, site will be down Tuesday at 3am - 4:30am to resync the database servers.
-Alex
This morning, our database server locked up (which happens to us occasionally, you could call it "growing pains.") This morning's lockup was particularly bad as all 3 web servers became overloaded with people trying to load the site, and so slow that it took forever for me to log into any of them (for example, to take the site down and stop people from hammering the web servers xD)
To further complicate matters, the database became corrupted as a result of the lockup, and refused to start. At that point we switched to our backup database, which is synchronized with the main one, and brought the site back up.
The downtime this evening was the result of db2 locking up, which was corrected with a restart of the database.
This left our primary database server offline though, so Subeta will be taken down over Monday/Tuesday (Tuesday morning) at 3 am in order to synchronize the two and return to operating on db1.
In case you're curious why the database locks up like this: Subeta uses caching to save the answers to frequently asked questions that would otherwise be asked of the database (for example: "give me a random avatar to show on the homepage?"). The cache expires after a certain time, however, and if there's lots of users on the site, hundreds of users suddenly ask the database the same question. You'd think this wouldn't cause a problem, but the database really doesn't know that lots of people are asking the same thing, so it dutifully tries and answer the same question hundreds of times. Meanwhile, all the normal Subeta queries get backed up while 200 database threads compete over access to the same information, and eventually it gets so bogged down that it only answers 100 or so questions/second, as opposed to the 5,000 we normally run.
There are ways around this, of course, and we've been implementing them whenever we come across this problem. For the moment, though, whenever we fix one of these problems (like the random HA on the home page), another one steps up to take its place. Rest assured that each time this happens, progress is being made!
Anyway, in summary, site will be down Tuesday at 3am - 4:30am to resync the database servers.
-Alex
Posted by 
alx
Fallenemiko
thank you for the explanation, you keep a lot of people from pulling out their hair ^^
0
User not found:
Thanks ;P <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
User not found:
and i deleted one blee, cause that was what i had twice oO
but: man, it's getting full here - guess i rarely have seen about 1200 people online.
i hope you guys can handle that =)
but: man, it's getting full here - guess i rarely have seen about 1200 people online.
i hope you guys can handle that =)
0
Rainey
nothing is working, makes me wonder if the DB should be synced now? I went to melody and it gave me 3 blees, but I didn't know until I went to the inventory cuz melody was a blank page
0
anthony57
thanks for the info sounds like a big headache to me but i'm glad your having it and not me
0
Jenni
For once, I actually sat down and paid attention, and read this through.
Thanks for explaining. I have a huge lack of database knowledge, so this cleared things up. XD
Thanks for explaining. I have a huge lack of database knowledge, so this cleared things up. XD
0
User not found:
It's really awesome to play on a site where the staff goes the extra mile to explain any problems and let users know the issues are being addressed. Thanks for the update! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
Thanks for letting us know! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /><img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
All technology can have its quirks at awkward times. Not to worry, we all understand how hard you all work at the site and love all the new improvements. Hope the rest of your tech time goes easier!
0
aiya
I love your update Alex, you put it in terms I can understand and I know that is not always easy. Thank you so very much for taking the time and effort to do this.
0
Pippa
Thanks for the update.
That seems like so much work. <img src="http://images.subeta.net/smilies/smiley_xface.gif" border="0" />
That seems like so much work. <img src="http://images.subeta.net/smilies/smiley_xface.gif" border="0" />
0
User not found:
Ack, I'm so sorry about the blank post. <img src="http://images.subeta.net/smilies/6398_emoticon_broken_heart.gif" border="0" /> But I understand now, thanks for explaining! =)
0
skrillex
Thanks for posting this and keeping us up to date! :3
Also thank you for having it early in the morning..I'll be fast asleep when the downtime is happening <img src="http://images.subeta.net/smilies/6398_emoticon_wink.gif" border="0" />
Also thank you for having it early in the morning..I'll be fast asleep when the downtime is happening <img src="http://images.subeta.net/smilies/6398_emoticon_wink.gif" border="0" />
0
alx
Senynith: yeah, usually i try to put an explanation on the downtime page. yesterday morning though, the sheer number of users locked up the web servers (which i would need to access in order to post a downtime message), so i decided to just fix the problem asap instead.
dachshundsr4me/AuntRainey: unfortunately a new database server would not fix the problem as subeta's code currently does not allow using more than 1 db server at once (it is a design goal of the recode to fix this). while a more powerful server might technically help, it is a better use of our resources to continue improving the code so this doesn't happen at all. it's a slower, but far more rewarding process.
PsYcOtIc: actually the most recent lockup was over an old piece of code called configurator which seems to be loading something from the database that really doesn't need to be in the database at all. the new achievement system really didn't play a role in this. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
dachshundsr4me/AuntRainey: unfortunately a new database server would not fix the problem as subeta's code currently does not allow using more than 1 db server at once (it is a design goal of the recode to fix this). while a more powerful server might technically help, it is a better use of our resources to continue improving the code so this doesn't happen at all. it's a slower, but far more rewarding process.
PsYcOtIc: actually the most recent lockup was over an old piece of code called configurator which seems to be loading something from the database that really doesn't need to be in the database at all. the new achievement system really didn't play a role in this. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
lovey
Very awesomely written !! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" /> I understand everything now <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
User not found:
yet again a problem caused by something new being introduced into the site and yet again it seems no foresight into what problems may result from this
it is good to get a proper explanation though so thank you for that
it is good to get a proper explanation though so thank you for that
0
satanael
Thanks for letting us know. I'm so glad subeta doesn't suffer from DDoS attacks - that's usually what I'm dealing with + angry customers LOL
0
Xianeia
Well glad know about it then thanks thats my main time for popping on here going to have to try get on before cause that timing is 8am till 9:30am for me.
0
Katt
Thanks Alex you are awesome and I don't know about everyone else but i appreciate you going out your way to explain why the site was having problems. Thanks for all your hard work hun
0
kudzu
That almost went over my head. Almost.
But I got it.
It would be nice to know what's going on on the "site is down" page, but whatever.
THANKS FOR LETTING US KNOW ANYWAYS! ♥
(I really am grateful)
But I got it.
It would be nice to know what's going on on the "site is down" page, but whatever.
THANKS FOR LETTING US KNOW ANYWAYS! ♥
(I really am grateful)
0
User not found:
have to also remember that alot of the overloading on the servers is 'cause alot of users on at once in one area of the site trying to get the achievements all at one time too...like when i was trying to get the dark matter game one the site all of a sudden when slowdown on me then said i claimed the points but i didn;'t and no achievement so had to wait 6 hrs...thankies for keeping us updated muchly...it is always appreciated...not many game sites do that
0
Meddy
Thank you for the update... it is like a foreign language to me but I totally understand the scheduled downtime part <img src="http://images.subeta.net/smilies/6398_emoticon_raspberry.gif" border="0" />
0
Fraggle
Big thanks to you for taking the time and having the patience to explain things to everyone in a way we can all understand. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Rainey
Posted by dachshundsr4me
Is there anyway with contributions that we can help possibly getting another server.
Great idea!
Is there anyway with contributions that we can help possibly getting another server.
Great idea!
0
Rainey
Alex, you rock! I think this site is growing so fast, the servers can't keep up . Thank you for explaining this to us, and I hope you get the dbs fixed soon. maybe our withdrawals are not a good thing when subeta is down. hehee!
0
Tachi
*Hugs* Alex, some days I really think that the site would be eaten by a squeal if it wasn't for you and are other programeys.
0
Hope
You are amazing, Alex! You deal with all this stuff AND manage to keep us all informed, explaining everything so that even people like me can understand what's going on! Thanks for that. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
User not found:
I thought the achievement system was causing the trouble XD Glad to know what's really up. Thanks for the update!
0
Alleycat
Alex, you are wonderful! No other site keeps us as informed as Subeta. I even heard about IE problems here before it hit our news stations!
0
dachshundsr4me
Is there anyway with contributions that we can help possibly getting another server...wouldn't that help as well to help with the growing population..I know most of us here love this site and donate regularly if not to get cash shop items or just to have that super gold account.
0
LadyTerentia
Thank you for explaining it so well! And for giving us a heads up about the time time. Keep up the great work. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Bagheera23608
I don't understand much of what you just said, but I appreciate the effort, lol. And a 15 minute down time is a biggie? Not in my experience. Maybe those multiple day downtimes are a thing of the past! Alx, you're great.
0
bracket
Thanks for the heads up! I'll be asleep then anyway, but it's nice to know we're being kept in the loop~
0
Twisted_Pretzel
I hope the Melody's Cottage items stack for those 2 days on the next day the site is up =3
0
Aztec
Thanks for the warning ahead of time ^^
It's good to know when this site is going to go down so I can find ways to occupy myself :p
Is this going to affect the advent calender at all though? Or will the down time be short enough for us to still complete it?
It's good to know when this site is going to go down so I can find ways to occupy myself :p
Is this going to affect the advent calender at all though? Or will the down time be short enough for us to still complete it?
0
KHERRERA2000
Thanks 4 telling me guys!<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
ache
Thanks for keeping all of us updated. I really appreciate the time the staff takes to do so, as well as keep the site in working order. It means a lot to me and the other users. ♥
0
Rarre
thanks for the upate <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Sirensong
Having played another site that not only didn't explain why the site went down, but didn't even acknowledge that it did crash, I sincerely appreciate being kept in the loop. It makes us feel good that you take the time to explain it to us and I, for one, very much appreciate that.
0
Monbohn
Thank you for taking the time to let us know what's going on and for always explaining things in term I can understand.
0
User not found:
Ok. Good. ^-^ thanks.
0
EMMA_778
Thanks, Alex! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
User not found:
gee...thanks for telling us. i thought one of thosepeople that can hack our computers got to subeta
0
Chou
Thanks for keeping us updated! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
MoogerMint
Thank you for letting everyone know what's going on! I was getting a little worried myself. :3 You guys are doing great!
0
Lillydivine
Wow, you guys are amazing! Thanks for doing such a great job! <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
kiruna
Thank you for keepin' us in the loop... <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
smilesnz
Thanks for the update on these downtimes Alex<img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> as I was kinda wondering why Subeta was going down and was hoping it wasn't due to the Christmas month.
0
Glitter
God. I LOVE being a part of a site that actually tells users what's going on! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
0
Inknote
Thanks for the heads up AND the details. i didn't understand all of it, but it makes me feel smart when i read it XD but really, it's cool to know a bit of the behind the scenes workings of my favorite pet site!
0
YeranceK
Thanks for the update <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Its always nice to know whats going on, even if we dont understand it all <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" /> Keep up the good work <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
Laurie
These updates and information posts are awesome. Thanks for taking the time to let us know. ^^
0
AlwaysLeelee
Thanks for not only an answer and a heads up as to what we should expect and when we should expect this, but thanks as well for details. Just one more reason why I basically live here hahaha!
0
KathyRunkle
Thank you, Alex. As always you helpful updates are appreciated. I've never actually been on a site where regular users were involved like this. I think that most of us are extremely loyal to Subeta. We love this game and we love the community here and we are happy to wait out any technical issues that happen. Thank you everyone for all your hard work you do for us regular folks. <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0
znogirl
Good luck! If anyone can fix it, the Subeta team can! <img src="http://images.subeta.net/smilies/6398_emoticon_happy.gif" border="0" />
0
Carol
Thank you as always, my dear. <img src="http://images.subeta.net/smilies/6398_emoticon_smile.gif" border="0" />
0
pharmacy
Thanks, Alex! <img src="http://images.subeta.net/smilies/6398_emoticon_heart.gif" border="0" />
c:
c:
0
Cinnatrim
Thanks so much for keeping us updated, Alex. <img src="http://images.subeta.net/smilies/6398_emoticon_grin.gif" border="0" />
0