From March 23rd 7:15 AM to March 24th 10:50 PM Subeta time, there was an error where every time an item was removed through the new wardrobe, it would be deleted from other users' wardrobes as well. We fixed this error as soon as we could, but many items have been lost. Our daily backup service had also stopped working, so we were unable to roll back the database by a day to restore the items.
We own up to this; this was an egregious, unacceptable error, and there is no excuse for not better monitoring our backups.
This error has now been fixed, and we are taking steps both to resolve this in the immediate future and to prevent this from ever happening again. Here's what you need to know.
Immediate Actions
We understand many items are irreplaceable, and we're going to do everything we can to restore all individual items. While our daily backup service had failed, we've booted up an older backup from about 4 months ago which will help a lot. We will be using this backup, various logs, and existing avatar data to further determine which items need to be restored.
We've set up a report form here that should streamline the process of reporting items you know you're missing. From now on, please go ahead and use that report form instead of the ticket system to make your reports.
We've also anticipated some questions which we've put in the FAQ spoiler below. If you have further questions, please leave them in this thread; we want to make sure your concerns are heard and taken care of.
Reports FAQ
Do I need to use the report form in order to get my items back?
No; our goal is to restore every item possible, regardless of if the user has reported it or not. We understand that most users don't remember every item in their wardrobes, and we don't want to punish anyone for that. We do ask you use report what you can, however; this information will help us immensely in restoring the items both for you and anyone else who was affected.
How long will I have to report things?
The Report Form will be open indefinitely, so if you realize an item is still missing, say, a month from now, you can report it then.
I have already made a ticket about my items. Should I still use the report form?
Yes, please do. This will help us consolidate the information.
I am missing a lot of items and it will take forever to enter them into the form one at a time.
The Report Form will be open indefinitely, so you can take your time reporting what you've lost. If it is too much to list at once, you can start with the items you're missing most. We do encourage people to start by listing more expensive items and CWs so we can restore the most valuable items first.
There's an item that I'm only 90% sure if I had it. Should I still report it?
Yes; if you're mostly certain, go ahead and report.
Finally, we are giving
1000 CSC to every user as additional compensation. Please
click here to receive your CSC.
This link will also be available indefinitely for all accounts potentially affected by the glitch.Future Preventative Measures
While coding errors will sometimes slip through, the full scale and impact of this error could have been prevented. And while it is too late for this situation, we are taking actions to ensure this never happens again in the future.
1.
More reliable backups - We've improved our backup service and have added monitoring so that we will be alerted immediately if a problem with it occurs.
2.
Changing the way wardrobe data is deleted - We've changed the wardrobe to no longer delete item data immediately when you remove items; we'll now have recent delete data in our database itself as an extra layer of backup. This gives us more ability to respond to all wardrobe-related incidents moving forward, not just incidents like this one, and makes our system more robust in general.
3.
Further database reinforcements - Finally, we're investigating other changes we can make to our database infrastructure to further ensure resilient data and backups.
We want to make it clear that we understand how truly upsetting this is for everyone. We are so, so sorry. We are working hard to make this right, and are putting every effort in to restore your wardrobes and your faith in us.
