We are working, I promise!
Hey guys, as our systems start to come back online, you might notice a few bumps here or there, as we start bringing everything back online.
What happened, is our memcached stack failed miserably, but in a really, incredibly sneaky way. From all of our tools and monitoring services, it appeared that memcache was still online and working, just slower than usual, which happens when we have a lot of users doing things that require caching (such as liking avatars on the last day of an event!)
However, what was secretly happening is memcache was "holding" connections, and a server can only have a certain number of open connections running at a time. This includes apache (our web server!), and other services that we run to make Subeta run as smoothly (cough) as possible.
This meant that our web server (and even memcache, trying to steal more connections!) couldn't make the connections to the database, or the other services, and serve pages to you guys, the end user. We couldn't see what was causing that problem, but now we've established that it is memcache and have (hopefully) fixed the problem, and are now in the process of bringing everything back online.
THANK YOU for being so patient with us today, I know that it can be stressful trying to get around the site while something like this is happening, but we really appreciate you sticking with us.
Update: For those of you interested, here is a link to what the graph looked like, when things got seriously bad yikes!
What happened, is our memcached stack failed miserably, but in a really, incredibly sneaky way. From all of our tools and monitoring services, it appeared that memcache was still online and working, just slower than usual, which happens when we have a lot of users doing things that require caching (such as liking avatars on the last day of an event!)
However, what was secretly happening is memcache was "holding" connections, and a server can only have a certain number of open connections running at a time. This includes apache (our web server!), and other services that we run to make Subeta run as smoothly (cough) as possible.
This meant that our web server (and even memcache, trying to steal more connections!) couldn't make the connections to the database, or the other services, and serve pages to you guys, the end user. We couldn't see what was causing that problem, but now we've established that it is memcache and have (hopefully) fixed the problem, and are now in the process of bringing everything back online.
THANK YOU for being so patient with us today, I know that it can be stressful trying to get around the site while something like this is happening, but we really appreciate you sticking with us.
Update: For those of you interested, here is a link to what the graph looked like, when things got seriously bad yikes!
Posted by Keith
Load this on 🆕 Kumos site
TestSubject991
It's still laging for me =/
Problems just do happen, so it's not a big deal to me. :)
However, I appreciate that fast fixing a lot, so thanks for your hard work and for keeping us updated. :heart:
However, I appreciate that fast fixing a lot, so thanks for your hard work and for keeping us updated. :heart:
I have no idea what any of the means, except that you respect your users enough to keep us informed and give us these kinds of details. I love it. And thanks for working so hard to get things back and running
O.o wtf picture o.o....epic photo. i know takes a long and its so -sigh- to do all, but i know still working on it. too much for do i guess but well, hope the works ends on good way :)
What I love most about Subeta, is that you guys have the dedication and respect of users so much that you go into detailed accounts of site problems. That makes me have like 10000000 respect points. GO SUBETA!
Thank you for working hard and being honest with us. Some of the sites leave users hanging for many months and in the worst case years.
I'm really gonna start taking this personally, guys! Every time I've tried to play I crash the site. Hey! Maybe it's me........
I love you guys for being so honest with us all, unlike other sites that just crash and don't tell you what's going on. :heart:
Oh gosh, over the past week when I was trying to browse on my phone the pages would constantly reload, reload, reload, reload, reload, reload, reload, reload. It was so annoying and impossible to browse, but it is fixed now! YAY! Thank you! :heart:
Wow, that's quite a graph. It really went crazy didn't it.
Thanks again for working so quickly to fix it. :)
Thanks again for working so quickly to fix it. :)
Thanks so much for the explanations!! It helps so much to have at least some idea of what's going on =)
Thank you for your hard work. A lot of us have stuck with Subeta through worse, so I'm sure a little lag and a few glitches won't scare anyone away. Thank you for keeping us updated. Its nice to know that the staff cares, unlike some other petsites *cough*neo*cough*
Thank you guys, Subeta staff, for working so hard to keep us running <3
And for being so awesome about it. Double <3
And for being so awesome about it. Double <3
Thank you very much for all the hard work to get Subeta back up and running! Thanks for all the updates too!!!
:) i love that you always explain the problems in detail, unlike other sites that just go down for ever and then when they eventually come back up you never get an explanation D:
xD thanks subeta-people.
xD thanks subeta-people.
Cheers, said it on FB i'll say it here. Thanks staff for fighting off Error's SQL army of doom.
~Kai
~Kai
Glad to see are you guys are trying to help us out. ^^ Best of luck fixing things! I figured it had to do with the event but never has it been that laggy. It's much better then it was tho. That's for sure.
It can't be as stressful as running a huge site like Subeta! We're cool, what happens happens and it's not the end of the world if we have to wait to have things fixed. Thanks again!
I'm glad you figured out the problem and got the site back up and running again.
Thanks for keeping us informed ^^
Thanks for keeping us informed ^^
Thank you for telling us. =) This whole updated this is nice! 8'D It's so new. I'm getting a tingly feeling. ♥️
Thanks for getting everything up and working again so quickly! Most sites take days to take care of lag like this.
Thanks for once again keeping us updated. It's refreshing to have a staff that actually cares about its users.
