Hi all! For those of you who don't know me, I'm Alex, Subeta programmer. This news post is kind of a big announcement, but it's also a long one, so bear with me. I promise, it's worth it.



It's always amazed me that Subeta's problems with crashes and downtime have become something of a running joke on the site, but I can see why they have. Back when this year started, we were suffering from database crashes every couple of days, sometimes going down for an hour or two before we were able to recover, sometimes much longer. And we had a couple times where we had no choice but to roll the site back for hours (or days) when backups didn't work or corruption was too bad.



Keith and I have been hard at work fixing this situation though. And things have gotten much better. The database now stays up for days without crashes or killer lag, and when we do crash it takes us, on average, less than 45 minutes to get back online with no rollbacks.



But still our core problem remains: both of our database servers somehow have their data corrupted, overwritten with nonsense, which eventually causes us to crash and have to restore from another copy. This has been a vexing problem for us, and it was one of my main focuses when I attended the MySQL User's Conference back in April. And I'm happy to say we now have a very solid theory of what's happening. The details are pretty technical, but the general theory is that the Solid State Disks (SSDs) that our database servers use, and the RAID cards they are connected to, are somehow corrupting the database. This is actually more likely than it sounds, because the disks our hosting company uses in our servers, we've discovered, are a cheaper brand of SSDs meant for things like storing your music on, not running high-concurrency, high-churn databases on.



So we're moving... to a different hosting company, one that has put together a much better setup for us. Our servers will have more memory, faster disks, no RAID cards (which end up slowing the SSDs down), and better support for when things go wrong. The new setup is currently being inspected by the new company, after which WE get to inspect it (likely tomorrow) and then we'll move the site. Even if it doesn't fix the corruption issue (which we are almost positive it will), this move represents a pretty nice improvement for us.



So, what should you expect? Downtime should be contained. I'm actually hoping to only have the site offline for a few minutes, but it may take longer than that. When copying gigabytes of data across the internet, you never really know how much time you'll need. The site will NOT be changing names again, we'll still be subeta.net.



With any luck, when Subeta comes up it will hopefully be much more stable and (crosses fingers) noticably faster.



We (the entire staff) also wanted to thank everyone, both for their patience during the next few days as we execute the move, but also for being patient with us over the past months as we've worked hard to hunt down problems and get to this point. You guys are the reason why I don't mind waking up to my phone at 3 in the morning to go recover the database :)



-Alex



P.S. Keith and I will be lurking in the comments, answering any questions you may have.