Hello caching, our old friend...
...we've come to talk about it again.
Y'all may have noticed site issues the last few days around the wardrobe and forums, basically the New Subeta side of things. This is because the local cache for the API servers is filling up in a way Keith hasn't really seen before and he still needs to figure out the cause.
The API servers are the servers that convey information between New Subeta and legacy Subeta and their caching is an example of a process meant to make the site faster going too far -- it caches all of the images & avatars as users make them (and also caches a version of each avatar loaded) instead of generating them from scratch. These servers have been up for a little bit, so they've taken on a lot of cached images, and once an event like Morostide hits those caches fill up extra fast.
Keith is spinning up a much bigger web server to handle the load, which as a pro should be taking over by the time this message posts and you should see some improvement. As a con, that much server space and traffic is expensive to upkeep so he's going to keep investigating to figure out something more efficient.
We know this is frustrating, this is a fun time of year and we still have so much planned! We're going to keep working on delivering more treats than tricks, promise.
Y'all may have noticed site issues the last few days around the wardrobe and forums, basically the New Subeta side of things. This is because the local cache for the API servers is filling up in a way Keith hasn't really seen before and he still needs to figure out the cause.
The API servers are the servers that convey information between New Subeta and legacy Subeta and their caching is an example of a process meant to make the site faster going too far -- it caches all of the images & avatars as users make them (and also caches a version of each avatar loaded) instead of generating them from scratch. These servers have been up for a little bit, so they've taken on a lot of cached images, and once an event like Morostide hits those caches fill up extra fast.
Keith is spinning up a much bigger web server to handle the load, which as a pro should be taking over by the time this message posts and you should see some improvement. As a con, that much server space and traffic is expensive to upkeep so he's going to keep investigating to figure out something more efficient.
We know this is frustrating, this is a fun time of year and we still have so much planned! We're going to keep working on delivering more treats than tricks, promise.
Posted by Galaxia
Load this on 🆕 Kumos site
catghost
Is the site still broken for anyone else? I've tried closing my browser (chrome) entirely multiple times, and clearing my cache, but nothing has worked. No images load, and the menu on the site is in a vertical list, instead of the normal horizontal. Wondering if it's just me at this point haha
I got a "Forbidden" error on trying to log into New Subeta, so I can't continue my giveaway going on in the forums. :(
Thank you for keeping us updated. I hope an efficient solution is found quickly. Happy Morostide!<3
Thank you for explaining why I couldn't view the Moro forums or change into a more seasonally appropriate outfit yesterday! I was wondering why it wasn't working when I was using New Subeta....
Galaxia
STAFF
@Stiles
Hmm.
So, for an analogy, let's say that you've been chosen as a fashion model. You're specifically changing clothing a lot, playing around with lots of different items and looks and you've told your friends to call or ping or videochat your agent to find out what I'm wearing because it might change at any minute! So exciting!
You are a great model and your agent is getting a lot of hits on 'what's he wearing now?' moment to moment. It's a lot to keep checking in on you, takes a lot of time and you also get delayed in your moment-to-moment scheduling and have to take minutes at a time telling them every time they ask or posing for the camera; frustrating, especially when you haven't actually changed recently. So your agent starts checking in at, say, ten-minute intervals and showing people the picture they took they last time they looked in, rather than checking down to the moment.
It can be more convenient both for you as the model and for your agent to just have that 'cached' picture, you don't have to slow down every time someone asks for an update and your agent can spread themself out more. On the other hand, it isn't always accurate if you changed five minutes after the last check-in. It also requires your agent to keep your last look on hand, rather than just being able to convey the information from a spot check and keep moving.
You are, well, you, or your avatar is you. The API servers are your agent, but they are also repping all the other users. The cached information about your avatar is that 'this is what he was wearing ten minutes ago' and, like that agent making the trade-off between slowing you down but also not being accurate, can take time to catch up (which goes into what you were saying, @LothlorienRain). In this case, with Morostide being an amazing event where a lot of Subetans really go all out with their aesthetics and styling speed, the agents (who are also acting as secretaries taking messages in this analogy, since they also handle the forums) are keeping a lot of those headshots and snapshots and fashion updates to be able to refer back to and eventually they just...run out of room in their print-folders or flash drives or whatever they're using to keep the clients' information because there is so much of it.
You can hire more agents to answer requests or store more information which would be spinning up more servers but, like agents' time, Subeta is paying for those extra servers (Keith has mentioned before that it can cost up to 500 dollars a day at peak traffic). As well, the agents/our servers aren't always telling us when they're full up on clients, er data, and need a reset. Or they're telling us through bad handwriting scribbled on a napkin rather than the actual form we need/not actually tripping the monitoring until it's way too late and y'all start noticing the downstream effects.
Does this help explain it all better?
Hmm.
So, for an analogy, let's say that you've been chosen as a fashion model. You're specifically changing clothing a lot, playing around with lots of different items and looks and you've told your friends to call or ping or videochat your agent to find out what I'm wearing because it might change at any minute! So exciting!
You are a great model and your agent is getting a lot of hits on 'what's he wearing now?' moment to moment. It's a lot to keep checking in on you, takes a lot of time and you also get delayed in your moment-to-moment scheduling and have to take minutes at a time telling them every time they ask or posing for the camera; frustrating, especially when you haven't actually changed recently. So your agent starts checking in at, say, ten-minute intervals and showing people the picture they took they last time they looked in, rather than checking down to the moment.
It can be more convenient both for you as the model and for your agent to just have that 'cached' picture, you don't have to slow down every time someone asks for an update and your agent can spread themself out more. On the other hand, it isn't always accurate if you changed five minutes after the last check-in. It also requires your agent to keep your last look on hand, rather than just being able to convey the information from a spot check and keep moving.
You are, well, you, or your avatar is you. The API servers are your agent, but they are also repping all the other users. The cached information about your avatar is that 'this is what he was wearing ten minutes ago' and, like that agent making the trade-off between slowing you down but also not being accurate, can take time to catch up (which goes into what you were saying, @LothlorienRain). In this case, with Morostide being an amazing event where a lot of Subetans really go all out with their aesthetics and styling speed, the agents (who are also acting as secretaries taking messages in this analogy, since they also handle the forums) are keeping a lot of those headshots and snapshots and fashion updates to be able to refer back to and eventually they just...run out of room in their print-folders or flash drives or whatever they're using to keep the clients' information because there is so much of it.
You can hire more agents to answer requests or store more information which would be spinning up more servers but, like agents' time, Subeta is paying for those extra servers (Keith has mentioned before that it can cost up to 500 dollars a day at peak traffic). As well, the agents/our servers aren't always telling us when they're full up on clients, er data, and need a reset. Or they're telling us through bad handwriting scribbled on a napkin rather than the actual form we need/not actually tripping the monitoring until it's way too late and y'all start noticing the downstream effects.
Does this help explain it all better?
I'm not upset. Heck, I prefer to experience an occasional hiccup to support a nice little site over giving my time to other online presences with predatory or exploitive practices.
I know how frustrating it is, but you guys are at least acknowledging it! I haven't seen too many issues, but I'm on the legacy site. Thank you for your guys' hard work!
Thanks for the update, and Happy Birthday! I don't know a lot of tech, but enough of it made sense to me that I found myself nodding in agreement with the findings.
"it caches all of the images & avatars as users make them (and also caches a version of each avatar loaded) instead of generating them from scratch"
This explains all the issues I have been experiencing and others have been noticing as well, some of the glitched results can be outright hilarious, and although I was worried at first that not being able to remove one zapped result might block the next, I did discover that is not the case, so I carry on along my merry trick o treating way. Fact is, that is a mess of data for a server to keep track of, we are blessed that it didn't blow up really. In thinking about it, when I first joined Subeta it was during the Luminar Festival, broken images everywhere, I actually couldn't decorate my tree till months after the event was over. Hopefully a workable solution is found, and I hope the resources to keep it up are there to keep it going!
"it caches all of the images & avatars as users make them (and also caches a version of each avatar loaded) instead of generating them from scratch"
This explains all the issues I have been experiencing and others have been noticing as well, some of the glitched results can be outright hilarious, and although I was worried at first that not being able to remove one zapped result might block the next, I did discover that is not the case, so I carry on along my merry trick o treating way. Fact is, that is a mess of data for a server to keep track of, we are blessed that it didn't blow up really. In thinking about it, when I first joined Subeta it was during the Luminar Festival, broken images everywhere, I actually couldn't decorate my tree till months after the event was over. Hopefully a workable solution is found, and I hope the resources to keep it up are there to keep it going!
Thank you very much for the update Galaxia. :) I figured it was being worked on (I trust ya'll), but it's nice to know what's going on.
Btw, Happy Birthday! ^_^
Btw, Happy Birthday! ^_^
Thanks for keeping us in the loop! Knew there was a reason why I love Subeta so much :wink: Also Happy Birthday :heart: :rose:
Thank you for keeping us updated! I think we are all still having a blast with morostide even with a few little technical mishaps :)
@Galaxia Those of us who have been here a minute, know how the cookie crumbles sometimes. But the cookie is still edible, and still oh so yummy! :grin: I know I won't be going anywhere, as Subeta is my home away from home. Sometimes I get scared tho, that something is going to happen, and it might be broken forever. So it's nice to hear that our leaders have our backs. hehe
Galaxia
STAFF
@NoCuteNameForMe
It's been something we've been playing whack-a-mole with for a bit, unfortunately. And I meant to post earlier, but got delayed during my drive. We appreciate y'all sticking with us through it all.
It's been something we've been playing whack-a-mole with for a bit, unfortunately. And I meant to post earlier, but got delayed during my drive. We appreciate y'all sticking with us through it all.
Thank you for posting about this, and especially on the news page where everyone can catch up on it!
Thanks for the update! I've just been glad that the majority of the site has still been running well. Good luck with the fixes! :)
@Galaxia, thank you SO much for keeping us informed. It is reassuring to actually know that issues are being heard, instead of feeling like we are all out here just floating on an island of the unknown, alone. ;) SO thank you, once again, for taking time to keep us in the loop! <3
Thank you for the update, and thanks Keith for working so hard on the issues c: I wish I was a millionaire, I'd sponsor the hecc out of Subeta.
As long as it's being worked on. -lol- I was trying to clear my events out and check forums and nothing was loading. Glad it's being looked into and something is being done in the mean time. :)
It's good to see you all know and is on top of the things happening with the wardrobe and forums. Thank you so much for the update.