Loading errors around the site (resolved)



Subeta has a lot of different features, but by far the most intensives are the wardrobe and the forums. Those in particular are grouped together with their own servers on Subeta V2, with more memory and ability to spin up for events and so forth. The issues you're seeing right now are related to the Subeta V2 servers running out of memory and not tripping the notifications when that happens, and thus not clearing the data automatically (this kind of situation is also the reason why sometimes pings don't go through, or shop events don't show up for people). Keith cycled things manually the other day and was trying to figure out why it's not handling running out of memory correctly, but Subeta's infrastructure is bespoke and it's going to take a bit more pinning down. So this is in progress! I'm going to be updating the status page as well. Thank you!
EDIT 10/10 2:04pm Subeta time: Update:
Keith is going to rebuild the API servers entirely from a backup, which will take about 15 minutes but should improve things rather than just patch the issue.
EDIT: 10/10 8:06pm Subeta time
In terms of what was going on, there were issues with the servers' local cache directories filling up, so we disabled that while investigating. Because of that, it was making a request for every image on the avatar (separate piece of clothing) and Amazon started rate limiting the requests thinking they were fraudulent. Restarting the servers wasn't helping because Keith needed to go into an account setting and confirm that Subeta hadn't been hacked by a botnet or anything. However, it should be fixed now!
Posted by
Galaxia
Load this on 🆕 Kumos site