Server Issues

Firsql
Sql
Firsql




I\'m sure everyone has noticed a few bugs around the site the past few days. Things aren\'t working properly in relation to time (like auto training, or gold accounts expiring, or quests resetting) or any other number of things.



What happened is our development server crashed. This is the server where all of our tools are kept (including all the files that run the cron which makes time tick on Subeta) as well as gives us (the programmers) the ability to edit the site while it\'s live.



The server essentially crashed and burned, one of the hard drives is completely unreachable and had to be replaced. Unfortunately, it took awhile to be replaced, and it meant that we have to do an entire OS reload on that server. We\'ve lost months of work (because our top secret projects are kept on that server, and off the other webservers) between all of the developers, as well as access to those tools I was talking about above.



We\'re going to be spending the weekend trying to get things working. There are two things that we are going to be doing.



1.



Priming our new hosting solution at Rackspace for a move soon. The type of hosting we have at Rackspace should prevent something like this from happening in the future. We\'re also now taking preventive measures to keep our code in multiple (safe!) places to ensure that we don\'t lose tons of work in the future.



We anticipate moving to Rackspace sometime in the next two weeks, which will mean (at most, hopefully!) a day of downtime. Once we are closer to that, I will fill you in on the exact date/time that we intend to switch.



2.



Getting our development server up and running on our existing hosting. Once we\'ve got this all worked out, we should have the cron running again, and be able to make changes to the site. Right now we can, it\'s just complicated and annoying.



SO FOR THE GOOD NEWS, RIGHT?!





We are going to be extending the Masquerade dancing until Monday. At that point, we should have firm control over the situation. And it gives you a few more days to collect those trinkets ;).



As always, thank you guys so much for being patient with us. This is just as frustrating (if not more) for us as it is for you. We :heart: you!
User Avatar: 1

Posted by Keith

Load this on 🆕 Kumos site
Thank you Keith for the extended Masquerade - thanks to those few extra days i reached all that i wanted :)
Thank you Keith for extending the Masquerade, I really enjoyed it, but I have a favor to ask. Would it be possible to just "lock" the Ballroom Forums and leave them in place so that we can go back and read them at the end of the ARM rather then just eliminating them. Things got so fast and furious and I was on several dancing boards and role-playing boards at the same time and haven't had a chance to catch up on reading them. Thanks, it was a great event.
This is what I love about Subeta -we get to know what's going on. :3 Thanks, you guys!
Source control? RAID?
Source control? RAID? >_>
Thanks about the update.
Thank you for the hard work. :heart:

And yay for more trinkets.
That sounds terrible. Thank you so much for keeping us updated, Keith. It really means a lot.
Awe Keith, I'm sry to hear that. D:
We appreciate you informing this incident to us! Bad things happens. (Me don't like. We don't like. D:)
And we :heart: you too!

Lastly yay for the Masquerade extension! :D
Thanks for the update and the hard work! :) Its nice to know all are working for make a perfect performance-more that at the moment O.O i mean godness hehehe- on the subeta site, its awesome to know it :P
We :heart: you too! :)
Thanks for keeping us all updated!

Hooray, thanks for extending the Masquerade!
It's my favorite event. :heart:

Rackspace is such a neat name.
I like names that start with the letter R and have an cks / x sound in them. xD
Sorry to hear about this, I hope everything will be fixed soon!
I know this is probably frustrating people who can't do any quests or other timer/daily things. :c
Ouuuuuch that sucks! thanks for the update about it though
You guys are so great at keeping us updated on what's happening with the site! Thank you for that! So sorry to hear about all the lost work... that really stinks! Hope the transition is smooth for you all! Love you Subeta and company! :heart:
Thank you for taking the time to make sure users are taken care of and informed!
RAID 5.
Spring on subeta, another year another host.
Thanks to everyone working so hard on fixing it :)
Wow. That's awful. D: Well, I wish you guys a lot of luck with getting everything fixed. I don't think any of your loyal Subeta lovers are too frustrated about the site not working properly and the site going down for a day. Sometimes accidents happen and you can't do anything about them, but start over. It sucks that all of your hard work is gone, but, hey, maybe something good will come out of it? Like perhaps you'll think of something new to add in that you hadn't thought of before.

It's nice of you guys to always keep us updated on the big things going on with the site. And we all <3 you too. :P
Wow, ouch. I hope everything looks up for you guys soon. :heart:
just wanted to join in saying thank you for keeping us informed and extenting masquerade. Also so sorry to hear about the lost work :( It must be so frustrating I wish there was something we could do to help but for now I guess it will just have to be good vibes ^_^
I'm so sorry about all the lost work. :( Thanks for the update, we love you keith, staff, and subeta!
Oh I am so sorry :( I wish we could help. Glad to hear your getting back on your feet, though!
I don't mind when these things happen because staff does a good job keeping us up-to-date. Things happen and that's understandable. Thanks for working so hard, and thanks also for the Masquerade extension.
Sorry to hear so much stuff was lost though, that really REALLY sucks.
Baw, we lurv you too.
My sympathies to y'all, possibly excluding whoever is responsible for you backup systems.
Thanks for keeping us updated! I was wondering why Novalee was still in the training center autotraining after so long.
Thank you as always for keeping us so incredibly informed on stuff like this... a comedian I was listening to just last night mentioned something about airlines never telling you the REASON behind the delay, and thus pissing more people off... Well, you guys are just AWESOME at explaining delays and stuff like that, so I for one do not get pissed. :D

Thanks for extending the Masq. again! Wow! I'm going to be trinket FILTHY RICH.
Second day of no autopricer D:
Oh wow, I'm so sorry to hear this! You guys must be heartbroken to lose all that hard work. Keeping my fingers crossed that the switch goes smoothly.
</3 Heart break =( Very sad. I hope you get things working well again!
This kinda stinks so gotta wait on my loyalty box and i'm just kinda in limbo
Thanks for explaining everything Keef =) Sorry this happened!!
"You've autopriced in the last 24 hours, but you can autoprice again for 25CSC!"

Is this because of the mess up? I haven't been able to autoprice for two days.
two days =/= 24hrs
Ouch! That really sucks =/
Sorry you lost your stuff ;(
Thanks so much for the update... this site Rocks.... go to another site and when things go wrong, you hear nothing from them...

Keith, you do an awesome job of keeping your members updated with site info... Thanks so much for being a respectful site owner.

Sorry everything turned topsy turvey on you and hope you do not end up with no sleep over the weekend... dont worry about us, we will be here for you !!!
Isn't this like the fourth time this has happened...
Not enough D: on the internet... I'm so sorry you lost so much work.
Rackspace???? Get some cloud space with Savvis, we beat the pants off those clowns ;)
Ouch! That really bites. Hope you guys don't stress too much. Thank you for the update and for all your hard work. You guys are awesome. hugs you guys
Sorry this happened but hope quests are working soon.
Months of work? I am so sorry you lost all of your hard work, especially the secret projects. I hope you can redo some of it. But thanks for extending the masquerade!
My Autopricer has not been working for a few days -_-

But its no big deal, I hope the move goes well.
Sorry for all the work you lost, and I hope everything is fixed soon and that you get it back :) hug
thanks for the heads up, it must be frustrating. Good luck :)
We love you too Subeta guys and gals <3

Thank you for the extended dancing time. So sorry to hear about all the top secret projects :( Losing work is highly frustrating.

Carrie
Sorry to hear about your work, it must be immensely frustrating!
Thanks for keeping us posted and for extended dancing time!
Mod
Oh geeze I probably would have started throwing things if I lost months of work like that. Thank-you for the update, and I hope the fix/move goes smoothly!
:heart:
Thanks for the update! Keep working as hard as you can do and we'll still love :3
So sorry to hear you lost the work. :(
Best of the luck fixing it!

Thank you for the update. We :heart: you too! :)
Does this mean there is going to be a cash shop sale because you love us so much? <3
Well, at least the weekend quest wasn't the Wizard or Saggi, because oh man, if it were....
Ah well, I guess this gives me a few more days to bring back the Running Man.
i think all gold account members should be reimbursed for this downtime. this is a service we pay for that we should be getting. im not saying pay us with cash but somehow reimburse us with items or something. i do realize this is a hard time for the subeta crew and i hope you work everything out to solve this. i hope you can get your work back and i hope the site is up and running smoothly shortly. i love subeta so much
I don't know about this stuff, but, I hope you will be able to get those months of work back? If not, that must be incredibly frustrating for you guys!

Thank you for letting us know what's going on. (:
I didn't notice anything either. Though I'm sorry to hear that lots of months of work was lost.

Thanks for extending the masquerade! :)
You know what would probably make everyone feel better? Another round of Potion of Loyalty! ^^
I actually haven't noticed anything, but good luck fixing it all up, guys.
So sorry about the all of that work you all lost. That's got to really upset all of you. :( Hopefully the move will take care of everything and you all won't have to worry about anything like this happening again. Good luck.

Thank you for the extra days for the masquerade. That's really nice of you. :)
I have been wondering what was happening, and have been bummed that I can't quest and have gotten behind on training my pet. That is nothing though to the work you guys lost! I am so sorry to hear about that! Can't wait in till you guys fix and recover from things. This is my favorite site. :heart:
We &#9829; you guys too!! :)

I appreciate to the depths of my soul how informative you all are of any errors. It makes me, and I'm sure others, feel like an equal to the staff around here.
Ouch D: Thank you so much for the update!
Good luck guys. Not sure what you said but good luck. X3 Sounds like a headache and a half.
-continues dancing-
does that server holds info only to Subeta or Valenth and Squiby as well?
Thanks for the update!
We know you're working hard, and hey, these things happen!
Thank you so much for keeping us updated! Like others have said, this is much better than being kept in the dark about glitches and problems, which is the way other sites do things. It's no fun having no clue what's going on. :P Sorry to hear about all that work lost. Good luck sorting everything out! :(
That's terrible! I hope the recovery goes well.

On a side note; I think you should be thanked for your honesty of the situation. It really shows how much you care about the members when you update instead of keeping us in the dark and lying like certain /other/ sites.
And this is why I love you guys ^^ Thanks for the update and I'm terribly sorry about all of that data lost :(
Aw, sorry about losing all your hard work. It must suck majorly. :/
We know that you're gonna do your best to get things back under control, though! ^^ Good luck!

Thank you for extending the dancing- that's great. :)
Thank you for the update. So sorry you lost months of work!!!
Maybe you could get a guru/recovery specialist to get the info off the dead drive?
Good luck getting things back up and running!
We :heart: you too!! {hugs}
Awww! Bless your hearts! :-/ Y'all work sooo hard to keep this place fun and exciting so I do understand how this has to be even more frustrating for y'all than for us as, on top of the problems with the server, you then have to see the complaining from members who just don't understand what's going on or why.

Awesome that you take the time to let us know what's wrong! Thank you for your hard work and taking the time to fill us in. xx <3 Just one of the myriad ways y'all keep Subeta the place to be! :)
Yes, thank you so much for the update. I was immensely irritated when I looked at my sidebar and saw the quests hadn't been reset, but knowing what's happened and why they weren't reset goes a long way toward soothing my irritation. We appreciate that you take the time to let us know what's going on, instead of assuming we are just stupid users that don't need to know. lol :) Hope all goes well and you are back up and running soon.
Ugh ouch :(, Thank you for letting us know. I hope everything goes well for you guys, and it sucks you guys lost all that hard work. Keep up those chins though you guys are awesome and know you'll get it working. And thank you for extending the event as well.
Ah, server crashes. Gotta love those. Hope the restoration goes well :O
I love being in the know about these things. It makes me feel like the site actually cares about its users! Which I know you guys do! Love you guys too!
No, no. I think we're cool. I feel worse for you guys! I can't even wrap my head around programming and such. e_e I am sorry that you lost months of work though. Good luck with your plans. :)
Terrible news that you all lost so much and all that hard work is down the drain. The way you all are handling this situation though is first class. Letting us all know what is going on and extending the dancing is something that not all sites would do. Good luck getting everything back in order :)
Thanks for the update
Hope all goes well fixing it
one question though what about this weekends special quest?
Thanks again best wishes about getting the server back up and running
Really glad for the update! Thanks for explaining what's going on :heart:

I'm so sorry that you lost all that work though :c But as someone else mentioned, I often find I do things better the second time around! Maybe this will prove the same? c:
Really glad for the update! Thanks for explaining what's going on :heart:

I'm so sorry that you lost all that work though :c But as someone else mentioned, I often find I do things better the second time around! Maybe this will prove the same? c:
I'm so sorry for all the stuff you lost, that must be so frustrating.
As far as user communication goes, your site is absolutely A+ which is just one more reason why we love you too.
Thanks for the update.
And good luck with getting it fixed.

And masquerade until monday? I LOVE YOU
Thanks for the update, we :heart: you too!
Oh man that's what I call bad luck oO... And it's awful losing so much important things about the site, I understand. Well, I just hope this bad situation doesn't happen anymore to you. Thank you so much for telling us that :heart: :)
We love you too Keith! ♥️

Thanks very much for the well explained update :)
Thanks for the update. ^^
Oh gosh, to think of all that hard work lost..really sucks. =/
Thank you for the extensive update, sorry about the loss but appreciate everyone that is around working on it this weekend!
The update is very much appreciated, as is the extended dancing time.

I'm really sorry you all lost so much of your work. Ugh. And to think I used to freak out if I lost a thesis paper before it auto-saved. Sadly, if it wasn't for things like this we wouldn't learn to save anything in the first place.

Good luck on restoring the server and your projects. I hope it all goes as smoothly as possible.
Ouch. :/
Thanks for the update, and good luck with getting things back in order again!
Woow, that sucks so much! >: It sucks that all your hardwork is gone, but look on the bright side, you've done it already, so once you do it again you won't be spending time trying to sort out little things because you've already figured them out. :)
Aw, the poor server.
I'm also kind of curious if pets in autotraining will receive the right stats or not =/ I don't want to have wasted CSC. Sucks about the servers guys.
Thank you for the update. News, even if bad, is better knowing than wondering what is going on. My heart goes out to you and staff for all of the information you must now retrieve or re-create. Continuing the Ball is a very nice thing to do to keep us entertained while you all have stressful overtime work to do. I love Subeta!!!! Group hug!!!!
Way depressing... Super crappy when something like that happens, makes it hard to start over with things. On the upside, when I lose work and have to start over, I always end up getting new ideas and doing things a bit different... so maybe this is an opportunity in disguise. Maybe someone's BEST IDEA EVER will come out of this. Anyway you guys rock, and you will not hear any complaining from me. This is my favorite site on the whole internets. <3
@Keith
That's unbelievably painful that the unthinkable would happen... if they are top secret, then the developers might not have a full working copy on their computers. I hope a copy of the tools are safe and ready to be restored quickly so the dailies can keep running at least.
Wow.... whats with my ____

I was just saying how that sucks i couldn't even imagine loosing THAT much work for everyone like that. I'm sorry about that. Hopefully you guys remember most of it so it wont take to long to put back together.
Sucks that happened >_< i could only imagine the hell you guys are going threw trying to get back all the work and everything. I know I've probably been a pain as well >_< Sorry.

Tho i have a question. With auto training or other things like the quests do you know once everything's settled ( since this isn't exactly a top priority) the system will give us the stats or if gold accounted the extra quests we lost? Or...?

its fine if you don't know since you guys have a lot on your plate just curious.

* loves on subeta staff and subeta itself *
I'm sorry for all the work you lost! Thank you for the update. I hope everything goes as planed for you.
yay extension, me likes it!
Thanks for extending the Masquerade. Keep up the good work. :)
Oh man. That shit sucks. Nothing worse than losing months of hard work D8
But thanks for the dance exchange. It's been making getting trinkets so much easier and more awesome.
end
That's terrible ffff D :
hugs I am so very very sorry to hear that you lost so much hard work. The work you put into this sight is awe-inspiring and I can only imagine your frustration at losing projects you had put so much effort into. The only upside to this is I think I can make it to 2500 dances now...
Thats just awful to hear about all of your work lost. You guys work so hard on making this site so much fun. To think I was getting a little frustrated cause I had a pet stuck in training. This is way worse than the just site clock getting stuck. I hope you get every thing as fixed as can be soon. The extra dances are nice but I wish we weren't getting them for this reason.
Erf, crashes are never fun - here's hoping to a speedier go-round when recoding stuff.
Appreciated the update, now I know why Zhuge's been at a more nonstop AT than normal. Lol My pet the gym rat.
Cheers for the dancing extension, I might just get 2500 trinkets by monday ...hahaha right. Lol
~Kai
Lea
I'm so sorry you guys lost all that work. =( That's awful. Thanks for explaining things, though, and extending dancing time!
Awwww huggles I hate it when things like that happen. Good luck and thank you so much for working so hard for us. The extended Masquerade more than makes up for the loss of autopricing and interest, as I am still prowling for trinkets anyway. :D Don't burn yourself out and remember to take breaks during your hard work! You love us, but we love you too!
Bummer :( Good luck with the switch!
Thanks for the update. I'm sorry to hear about the work that you lost, but hope that you can eventually retrieve it. Keep up the good work and we do appreciate all you do for us members.
Keith STAFF
@Rocketlauncher

Unfortunately we didn't really think to backup this deep. All of the "live" files are always running on the site. We've got multiple backups of it (as it's deployed across multiple servers) and it's really impossible to lose that information.

Our private developer "branches" of the code on the other hand weren't pushed across all the webservers, so it never really clicked to really put it anywhere else. Our mistake, and we're going to pay for it :(
Oh gosh, you guys... That's terrible, I'm so sorry to hear that you've lost all that hard work. I know how much time and effort you dedicate to the site, and it really is just heartbreaking to hear about your problems. I'm willing to wait as long as it takes for you to get things back into place. You've made Subeta a wonderful place to be; the least I can do is sit patiently while I wait for you to fix the site. Good luck, all, and thank you for the update~!
Best of luck fixing it over the weekend! I'm shocked about the amount of work lost, aren't they regularly backed up? :( Thank you also for letting us know about the move.
boooo. im tired of dancing >.< lol but i do like the trinkets you get from it.
Good luck, Keith/Team... We love you! This is an amazing place and thanks for keeping us in the loop.
Ouch. D: Sorry to hear that you guys lost your work. Good luck and keep up the good work! ♥️
Sorry to hear that things malfunctioned like that... Just out of curiosity, does this mean that the User Interaction Interface is going to be postponed again? Is it one of the projects that was wiped out? D: I sure hope not :/

Best of luck to you guys in fixing it all and getting the kinks worked out though. :heart:
Oh man; I know what it's like to lose all your hard work. I lost over 1200 pictures because my computer crashed and I hadn't had the chance to back them up. 3 years of work lost.

Good luck you guys! Just know we appreciate you!
I'm really sorry about the crash. Best wishes for getting things back together. And I say, the more Masquerade, the better!
aw man sorry about all the work you guys lost :(

Thanks so much though for getting everyone updated about the situation! We all know how much work is put into this site, so a little down time is no biggie if it means that you can keep all your hard work safe.

We all love you guys for everything that you do for us and put up with!
Big thanks for the extension for dancing :)
Thanks for keeping us updated! I hope you are able to recover the lost work :( sounds so complicated, good luck getting things back to normal!
Thank you for the update on the situation! I feel sorry for you guys... All that time and effort... Gone.

I appreciate everything you all are doing to get it back up and running. I will be waiting, patiently, for things to return to normal. You rock, Subeta staff!
Thanks so much for the updates, and for extending the masquerade. :D Collecting more trinkets is always exciting.
Thank you so much for giving us an update! So SO sorry you guys lost all that work though. :( That's never ever fun!

BEST WISHES IN FIXING STUFF!

We :heart: you too!
I'm so sorry you lost months of work.
Thank you for the update and your hard work.
We love you too!
There wont be a rollback will there? :( :(

I'm sorry for your lost Keith and the other programmers. There is no worst feeling then working on something for a long time and then losing it to a crash.
Oh no!! I feel so bad you guys have lost so much hard work that is awful :(
Thankyou for the update and everything you all do to make us happy :)
I was wondering when you'd make a public announcement about this, and I'm glad you have. Thanks for the update and especially for the masquerade extension. I'm sorry you guys lost so much work, but at least it's a lesson well learned and hopefully this won't happen again.
We love you guys too, and we're all thankful for your hard work <3
Good luck! You won't hear me complaining about it at all, I know you're doing what you have to. Thanks for letting us know.
Just as I went to the bugs forums to see what this was.
Thank you for informing us, I was terribly confused.
While I'm happy to see the Masq extending, I'm so sorry to hear about all the work that's been lost. I know how much goes into this site and that's just heartbreaking! Thanks for letting us know though, and good luck getting things together! We love you guys!
Aww, it sucks about losing all that work you guys did! D; I hope it was backed up somewhere at least...
Wow, good luck fixing those problems. You guys work so hard for this site.
You all work hard to keep this site up and running as smoothly as you can. I for one appreciate all the effort, work and time you all put in, to make this such a great site. Thank you
thank ypou so much for working on this site so hard!! im really greateful you guy do this! keep up the good work!
Oh man, I'm sorry you guys lost so much stuff D:
Thank you very much for the update ♥️
Oh, I'm so sorry about all your lost work! Hang in there. We'll keep dancing and hopefully raise your spirits.
Thanks for the update! Sorry for the stress and loss of work tho, thats not good :( But Once again Thank you ALL for what you do!!
I'm so sorry about losing all your hard work! thanks for the extended dance time, and good luck with the move and all that other stuff!
Thanks for the update^^ And we :heart: you too!
Oh my, thank you for the good news c:
We :heart: you too, guys.
Thanks for letting us know what is going on and that you are all actively working on it. It makes the little annoyances less annoying :D
No problem ... we all love being here and support your efforts 100% .. do what you have to do we will all wait until you get it squared away ... have a great weekend Keith and staff!
Thank you! :heart: you.
That's horrible! It's all good though. You're working on it and that's all that matters. Thanks for informing us about it though!
Thanks for the update and dance extention. More art for me to gank with dancing. :D

I must say I'm so sorry about you guys losing so much stuff... i can't even imagine how much of a headache and just heart breaking that would be for those that put the effort into it. Hopefully all the new safeguards will work out!
Thank goooooodness ♥️
Thanks for the update. Huge bummer you lost all of that work. Sorry you guys :(
Dill STAFF
Thank you for letting everyone know, Keith. And yay, an extra weekend of dancing!
Oh man month's of work lost? That really sucks :( Thank you for the updates, and for letting us know what is going on!
Thanks for all the hard work & for keeping us updated ! (:
Thanks for the update, Keith! :heart:
Hope it all goes as planned.
THANK YOU :)
Thanks for the update. Yay for extended time on the dance floor =D
This happens sometimes. Thanks for the update!
No worries, things happen. We love you too and know you work very hard to make such a beautiful and wonderful site! Thank you!

Leave Comment

Comments are currently on a short cache, meaning your comment may take a few minutes to show up after you post it.