Welcome to Gaia! :: View User's Journal | Gaia Journals

 
 

View User's Journal

Lanzer's Journal
The case of the mysterious site outage
After being back, I've often heard that new item release would cause the site to slow down, or the site to go down all together!

Some had attributed this to badly created CI or bundles, while others think it might have to do with large inventories.

Yesterday night the problem came back. The items database had went down, taking the whole site along with it! gonk We had to disable avatar inventory until the website came back.

But this time, our operations engineer helped us locate the slow database call which caused the slow down. (thanks James!)

Finally, we're able to track down the "bug" that had been causing the site to go down.

On other news, a few more inventory problems were tackled. Looks like the front end code need a lot of optimization, especially after global stacking is enabled. More about that later.

I wrote a report of the site outage bug to other developers, thought it might be
interesting to share with you guys too:

Site outage bug

Our inventory reader is responsible for bringing up everyone's inventories. Whether for marketplace, trading, avatar dress up, or item arranging. The reader has two modes:

- regular query join - which read from the user inventory table then read the gaia_items table for item properties such as item names. These queries are big and slow, but data is cached on a page by page basis
- mashup mode - this mode will read and store all of the user's inventory info in memory, then perform filtering and pagination on the web server instead of the database server. This mode is great for lowering the amount of queries to the database. It takes up a lot of web server memory, so we use it for inventories under 1000 items.

As it turns out, mashup mode need to store a ton of metadata so it can filter items, such as filter by housing items, tops, bottoms etc. To have this ability the script need to collect and cache item_ids for every item group type (we have 15), reading through the item database table with 250K rows for 15 times. As it turns out, creating this filtering data takes about 2 minutes and it's refreshed every hour.

Now here's the kicker - this 2 minute task can be triggered by any users who brings up the inventory. So while one wait, if 10 users brings up inventories, then we have 10 more update requests happening at the same time!

There are several fixes.

- First of all the refreshing could happen less often.
- The database could have a better indexing scheme to speed up the process
- The process itself can be optimized to not run through 15 times to get the desired result
- The actual update task need to be run by one scheduler, not by hundreds of users at the same time

But at the end of the day, I did some performance testing, and looks like the mashup mode does not give us that big of a speed saving. I ended up disabling it, and nobody should really notice the 0.1 second difference in load time.

I'm not 100% sure if that's the only thing that is causing servers to go down, but I'll be sure to keep an eye on things starting tomorrow. 3nodding

I have another item server optimization job to do. After that, we will no longer need to close the avatar builder from time to time! whee






User Comments: [10]
Knight Yoshi
Community Member





Thu Jan 26, 2017 @ 01:36pm


"- The actual update task need to be run by one scheduler, not by hundreds of users at the same time"
I mean... Come on, people. Cron jobs! 3nodding

These are things I tell people when they ask for some feature that would cause some issue like this. "Take X amount of time per-user and multiple it by X amount of users." Cycles' [lives] matter! lol


Jisen Meizuki
Community Member





Thu Jan 26, 2017 @ 02:35pm


"After that, we will no longer need to close the avatar builder from time to time!"

That's good to hear since there are those who do like to use that feature but unable to when the site is down. So I take it you and the developers find a way to keep new items from "leaking" in the avi builder. While it's good to keep the surprises from spoiling, I'm also kinda sad since I usually check there if new user items are coming in as well as what items are going to be sold directly from Cash Shop. Not to mention, helps find exclusive items in case no one finds it or listed it when RIG/CI/Bundle are released. Oh well... It was good while it lasted.

Well, whenever you or one of the developers decide to tackle the avi builder, can you have them "organize" the items to what shop? It's still scrambled when you try to find specific item when you use the shop list scroll in the avi builder (aka: If you try to find items listed under "Durem Depot," you get Salon Durem items instead.)


Hey Crazy Cat Lady
Community Member





Thu Jan 26, 2017 @ 03:17pm


I don't understand what all you did but thanks for doing it! heart


True Karma
Community Member





Thu Jan 26, 2017 @ 03:18pm


So you are saying the database is having a panic attack form all these requests at once and the user doesn't know it's doing it.
Well I'm sure it's more then one thing bringing down the servers but finding them and fixing them one at s time certainly helps...

*cheers* ohhh not having the avatar builder would be delightful OwO


Reve Casse
Community Member





Thu Jan 26, 2017 @ 09:26pm


I so enjoy reading this journal! But I have to admit, there are times I feel for you. You have your hands full with all these bugs. Please know that every fix is greatly appreciated. Thank you for all you are doing.


.Arriana.
Community Member





Thu Jan 26, 2017 @ 11:47pm


Thanks James!


Nemone
Community Member





Fri Jan 27, 2017 @ 12:14am


I just came back to Gaia after a break and was so so so glad to see you here. I'm so touched. I know you probably don't recognize me since I have always been a bit of a lurker, especially in the old days. ninja

Thanks for your hard work fixing things and also thanks for sharing the details. I got my degree in computer science but I lack experience so I find this sort of insight very interesting. heart heart heart


De La Luna IV
Community Member





Fri Jan 27, 2017 @ 03:24am


I am not sure you check the comments on here or not but I feel like things have gotten worse with the inventory management rather than better. Let me explain, today while organizing my inventory I had to go back 6 or 7 times and every time I moved 1 item had to save otherwise IT ALL REARRANGED. I couldn't get it to stay where I put the item, I don't have massive stacks of items however still kept it to only viewing 300 items per page and STILL had issues. I took one WHOLE page of already organized items and put it into my locker and I don't even KNOW what happened there, feel free to access my account and look because I am NOT going to organize it again until this is all done. its a pain in the tush!


Yumitoko II
Community Member





Fri Jan 27, 2017 @ 09:13am


I didn't even realise the site had this many problems, the management did a good job of ignoring or covering it up. I'm glad you're back to give it a go but give yourself a day off occasionally okay? x'D

captcha: just dance
(I think I will. )

Edit: With the avatar builder bugs being squashed, I wonder what you think of the fall of Tek Tek.
captcha: never quit


Sefyrion
Community Member





Fri Jan 27, 2017 @ 08:50pm


I ended up subscribing to your Journal after I read about your earlier work on the Inventory Arranger and I just want to say that I like being able to keep up through here and that being able to read your progress notes is a great incentive for patience (on my end). 3nodding

I'm really appreciative of all the work you and the rest of your team has done and is still doing and as someone who owns a tablet + laptop combo, I super appreciate the troubleshooting that has gone into the whole business of drag and drop not working for machines using touchscreens.

As for the Avatar Builder needing to be down, I never knew that was the cause! gonk

... Or maybe it's because I'm not always online at optimal times and so I miss the notices related to such things. sweatdrop

But I'm really glad it's getting a fix! 3nodding

Thanks for all your hard work and please thank your team for me!

whee


User Comments: [10]
 
 
Manage Your Items
Other Stuff
Get GCash
Offers
Get Items
More Items
Where Everyone Hangs Out
Other Community Areas
Virtual Spaces
Fun Stuff
Gaia's Games
Mini-Games
Play with GCash
Play with Platinum