Welcome to Gaia! :: View User's Journal | Gaia Journals

 
 

View User's Journal

Lanzer's Journal
The final stretch
Hello everyone,

Sorry it's been such a long time since the last update. For the past few weeks I've been living under a rock, with nothing but coding for this month's server migration. In case it wasn't mentioned before, we are moving all of Gaia's servers to cloud computing systems.

Today Gaia runs on more than 300 servers. We purchased and ran those servers in a "colocation facility" which is a giant air conditioned wear-house full of computers from different companies. We pay a hefty rental fee on top of power usage fee to the service provider, while every week we deal with maintaining the computers. Remember how old computers could have a noisy fan that died of old age, or a hard drive might start getting corrupted? With hundreds of machines, parts die and require replacement on a weekly basis. Maintaining these hardware ourselves is just too much.

Over the years we also wrote our own software for monitoring the servers, make sure they run smoothly and alert us if things go wrong.

But with the advent of cloud computing, companies such as Amazon had built a huge suite of tools to make it easy for companies like Gaia to setup and maintain a large number of servers. Whether it's setting up a system that will add or remove serves depending on usage, or adding more backup or security, the new system will save us a lot of time from having to do everything ourselves.

The idea of moving everything to the cloud is great. But the actual work is a massive undertaking. Mainly because we have so many different systems that were built since 2004. Gaia is much more than a bunch of web servers, we have:

- 30 database clusters, each with its own master/slave replication
- API servers for handling games such as Gaia Towns
- MMORPG servers for handling chatting and positioning info in Towns and zOMG (sushi, smartfox)
- memory cache servers for speeding up data retrieval
- search engines for searching through forums/guilds/avatars
- file system servers for managing terabytes of avatar data
- host of dynamic web servers
- load balancing servers to distribute load between all web and game servers
- static server for loading images
- avatar sprite server for avatar compilation
- internal wiki and bug reporting server
- intranet servers for remote developers/artists to access internal data
- versioning servers for backing up program source code and item sprites
- management servers for accessing databases and server health monitoring
- email server

Then there are softwares that works in conjunction with existing hardware that need to be changed for the new system

- system to deploy new code to all web servers when changes were made
- system to edit or upload new item sprites
- search related applications need to use new search engine system
- all applications involving file storage require modifications

My mind is blurry so I'm missing a lot of details in the migration. But we reserved 3 months of our time for the migration effort, and we're using up every single minute of it. After 2 and a half months, we're starting to come close with all our data uploaded, content delivery networks setup, database replicated, web server farm created, and most of the applications had been configured to work with new systems and their locations.

Of course, for such a large project, the last two weeks are going to be even more hectic as we patch up all the bugs from the many massive changes that were made. Some of you had already seen different bugs popping up here and there in the past few weeks, and unfortunately there might be more to come, especially for smaller features such as our flash games.

I can't wait to complete this migration so we can have a stable system, less downtime, and get more time to work on features you love. I very much look forward to features like image uploading which we should have had years ago.

Time to jump back to work! I hope that I'll be less stressed the next time I work on my journal. smile






User Comments: [47]
8OO813S
Community Member





Sat Oct 20, 2018 @ 02:54am


Thank you for all the hard work Lanzer. Cheers and hope the migration goes smoothly!


Dawnpainterz
Community Member





Sat Oct 20, 2018 @ 03:03am


Thank you for all the hard work!


Shiro Nishida
Community Member





Sat Oct 20, 2018 @ 03:05am


Journal Posts such as this one are some of my favorite things to read on Gaia. Precisely because this gives us an otherwise hidden view to what actually makes Gaia run on its day-to-day activities. it's also something most users take for granted, but reading about it here just makes me realize how far along Gaia has gone since then. Thank you, Lanzer, for giving us such an update of things to come. It may not be something us end-users will get to see besides more reliable uptime for the site, but it's great news for all the teams involved for gaia if the service can create new tools that will work better than the current set you've used so far.

All in all, thank you. I hope the site migration will go swimmingly too. 3nodding emotion_brofist


Knight Yoshi
Community Member





Sat Oct 20, 2018 @ 03:32am


Phew! Lots of work, and there was one user saying that it wouldn't take too long to migrate to cloud hosting and services and that it'll be less cost-effective than you think and that you'll have more difficulties.

Sounds more like everything is on the up and up with these migrations. Hopefully you'll do a post-migration follow up to let us know how well everything is running and other stuff. Like what's the next big project after the migration. biggrin


Purplesnoopy
Community Member





Sat Oct 20, 2018 @ 04:59am


As Shiro said,
thanks for giving us a peek into all the hard work you do to keep Gaia running.


Sianserais
Community Member





Sat Oct 20, 2018 @ 08:24am


Good to hear an update on this! I can imagine even just keeping track of what all needs to be done isn't easy. Best of luck with the final stretch, and thanks for all the hard work!


aetherArcanist
Community Member





Sat Oct 20, 2018 @ 02:15pm


This kind of transparency should be adopted by every other team, it's great to know Gaia is marching forward!


Vinified Victoria
Community Member





Sat Oct 20, 2018 @ 03:43pm


This was so interesting to read. I had no idea what it took to make Gaia run and really enjoyed this journal entry!

Good luck on your last two weeks!


Alyx Aphelia
Community Member





Sat Oct 20, 2018 @ 04:55pm


I love this! Thank you for everything! And I hope you get some well deserved rest soon!


Raenora
Community Member





Sat Oct 20, 2018 @ 09:15pm


Despite towns and avatar saves being fixed I'm still having issues with towns freezing along with my entire browser and my avi not showing in the dressing room. is there a solution to this despite having fixed everything?


BlueRainsong
Community Member





Tue Oct 23, 2018 @ 08:11am


Thank you for all you and the rest of the staff do Lanzer! I love these journal posts of yours, it feels like we get a peek behind the scenes of Gaia. These posts also give us a chance to see exactly how much work goes into maintaining the site.


Castiel Lover
Community Member





Tue Oct 23, 2018 @ 10:44pm


Thanks for the update and all the hard work! You guys should definitely throw a party when it's done to celebrate. Does this mean you no longer need the warehouse for physical servers? Or will you still need some physical space for the remainders? Is this saving Gaia money, or does the cloud service cost more?


Armored Interior
Community Member





Tue Oct 23, 2018 @ 11:28pm


Thank you lanzer for all your hard work and dedication to this site, you care about Gaia and it really shows. I don't know all too much about servers, but does this mean you'll be taking all of Gaia's data and moving it from a physical data storage area and putting it in the cloud? Will you bring over all of the old forums, guilds, posts, etc and will anything be lost in this process? Thank you very much.


Peachy Sol
Community Member





Wed Oct 24, 2018 @ 02:41am


So glad Lanzer is back!
With that being said, I hope all this money saving allows for more zomg features. Even though I have very little time to breathe let alone play games. Yay medical school! ... emo


L0FTY Au
Community Member





Wed Oct 24, 2018 @ 09:05am


so proud of the team for literally building and upgrading this site from the humble beginnings of a php forum to this massive coding behemoth.

On the behalf of the entire community, I fully look forward to the future plans cooking up behind the scenes. Particularly excited for the new thread/flash chat hybrid system.


Sagebomb
Community Member





Wed Oct 24, 2018 @ 10:51pm


Exciting times.

Hope it frees up a lot more time for staff to work on other projects. Especially devs.

Heres to burning it down, and making it better ^_^.


sweetkittycupcake
Community Member





Thu Oct 25, 2018 @ 02:39am


This makes me kinda happy for the what's to come but I do wish I could see this sort of open "hey this is what we're working on / this is when we expect to be done / then we want to work on xyz" but also be upfront about "abc isn't going to be worked on until this time or after we finish mno project" and make it where more people could see it (ex. ATS's or community announcements) and maybe it would stop some of the repeat "when are you going to fix ___ " type questions.


Milk2
Community Member





Wed Oct 31, 2018 @ 12:52pm


heart Good Luck Lanzer , we are supporting you From your Work heart


raggedy grrl
Community Member





Wed Oct 31, 2018 @ 02:59pm


Wow, this sounds like a massive effort!

A big thank you to you and the rest of the Gaia staff for all of your hard work!!!

emotion_hug

heart heart heart


User Image
emmacoolz
Community Member





Wed Oct 31, 2018 @ 03:06pm


This is awesome... go Lanzer go!


Crystal Coriander
Community Member





Wed Oct 31, 2018 @ 10:02pm


Both you and the staff have spent many hours working on the migration. Thank you for everything that each of you do smile Just don't over do it and get some sleep. smile
You have been online more than you have been offline.


CooperationIsKey
Community Member





Thu Nov 01, 2018 @ 03:40am


Just wanted to leave a quick note thanking you for all your hard work on this site, not just recently but from the very beginning. I've spent many hours on this site and still enjoy it over ten years after discovering it. I know you guys have had a major headache and a ton of work the past few days with your rush server migration, and while the outages and glitches were frustrating, I'm grateful you all are working so hard to avoid even bigger problems in the future. Here's hoping you can rest soon! heart


midosuji
Community Member





Thu Nov 01, 2018 @ 01:26pm


you guys are rockstars. you work so hard on this site, from making it run smoothly to taking time to tell your members what's going on. thank you for all your hard work and your effort! make sure to take time for yourselves, get lots of sleep, and stay healthy. <3


xXRed Moon VampiressXx
Community Member





Thu Nov 01, 2018 @ 09:57pm


Thank you to you and the team for doing some awesome. In an age where everyone expects instant satisfaction I know that it takes to time do these kinds of things. I have been apart of this sight for 13 years and I am rooting for you guys.


music-ecstasy
Community Member





Fri Nov 02, 2018 @ 02:58am


thanks so much to you and to your team for all of your hard work!


Nyadriel
Community Member





Fri Nov 02, 2018 @ 06:49am


I am also so glad you are back. I appreciate all the hard work you and Team are putting into all of this and I hope it all works out well. Lots of ups and downs, but I understand the difficulties and have been very patient. It is a miracle that any of this works in the first place in the beginning of it all.
Thank you. And I hope all this is done sooner than you think and can get some downtime so you can enjoy the upcomming Holidays.


Queentintin
Community Member





Sat Nov 03, 2018 @ 10:45pm


Hi, sorry to bother you, while you finalize the fixing of glitches xd , but would oyu happen to know when The Aquariums, will be back up, and viewable, and BG playable? =)

thanks for your time, tintin xd


Anamosa Valentine
Community Member





Sun Nov 04, 2018 @ 03:14am


Wow, that's a lot of work! o.o But as always, thank you for putting so much effort into Gaia! I never knew migrating servers would be stressful!

We like it when you post your journal entries here, it helps to keep the community updated on how you're doing in maintaining this site! whee


animoni
Community Member





Sun Nov 04, 2018 @ 11:40pm


Sending you love and appreciation to the work you are doing Lanzer. heart


Bonny Lass Minty
Community Member





Tue Nov 06, 2018 @ 02:33am


There's not much I can say that hasn't already been said...

Thank you for all your hard work and for keeping us all posted pirate


PizzaNyah
Community Member





Thu Nov 08, 2018 @ 09:10pm


Thank you for the great explanation, it was informative and interesting. I’m looking forward to Towns 3 and all the other awesome new features.


Lamecgod
Community Member





Sat Nov 10, 2018 @ 02:51pm


Still can't play zOMG but all I want to say is to never stress in the first place! As long as the imagination is there we can day dream of a perfect heaven from our mind into life at any time and bring that perfection, that is our personal thoughts, to life.


kdgalaxy
Community Member





Sat Nov 10, 2018 @ 10:04pm


Thank you! emotion_awesome


baby_kitty_258
Community Member





Mon Nov 12, 2018 @ 08:07am


heart heart Thank you for all the hard work you and your team are doing!! heart heart Don't forget to eat, sleep, and take breaks when necessary.


Oleviski
Community Member





Thu Nov 15, 2018 @ 09:08pm


Thank you for everything you have done and continue to do.


esdeath777
Community Member





Thu Nov 15, 2018 @ 10:16pm


Tl;dr Lanzer is working hard.

Thank you for your hard work. ! heart heart heart

Let us know of all the upcoming zOMG events planned.


Yeibichai
Community Member





Fri Nov 16, 2018 @ 07:39pm


Thank you, Lanzer! I hope Gaia is giving you much rewards, and big recognition for all the hard work you do. It only seems legit. R&R often.


Lillendandie
Community Member





Wed Nov 21, 2018 @ 09:25pm


I've heard of Amazon's Cloud service. I hope this is a cheaper solution for the site. Good luck and get some rest after it's over. 3nodding


commitsudoku
Community Member





Sun Dec 02, 2018 @ 05:31am


This was very enlightening for me, I had no idea how things work behind the scenes and it's fascinating. Thank you for your hard work Lanzer!


LotusCurtiss
Community Member





Thu Dec 06, 2018 @ 07:14am


Thank you o much for what you are continuing to do and for repairing the issue with Marketplace and those who still are on the gold currency. (I have yet to try it out yet.... crying )

So. I know that you are almost done and that there are a few wrinkles that will be ironed out along the way. However, I would like to deeply express an improvement on an existing feature that I dearly love and I believe would make other users even more pleased.

The Color Switcher in the tool bar. Please add more colors. More colors so we can match our moods or to match with the holidays. Why not go Bi-colored? Blue Pink, Blue Purple, red green, green yellow, etc... you get the point. Or even allow a feature in the Color Switcher that we can make custom.

Expand the color palette to 15 colors, black, white, Brown, gray, blue, red, yellow, purple/violet, orange, green, pink, and a few others. After the expansion of this, try to introduce Bi-colors. Just basic combinations or related to team colors and popular colors.

Upon the explosion of "wow this is epic," Bring out the customization. Ex, the tool bar is like red but upon hovering over the dropdown, something like pink or red or blue. And extends into the mini menu below.


Then if that takes off, why not introduce "Skins."

SKINS is basically popular color schemes for avatar items such, Wishful, Starry, Sherbet, CMYK, Painted, Etc... you know where I am getting at. Where once or twice a month you hold a poll to see which skin gets released.

Now these SKINS are set, they cannot be customized, perhaps, I recant that. SKINS are set but the person can chose to have one solid color for the selection of bi-color. The Solid color can only be for the main menu tool bar or for the drop down with they are doing a SKIN. Or they have to keep the SKIN as one entire selection. You cannot have two SKINS at the same time. One or the other with or with out a solid color.

I may have more epic and creative ways where we can make it awesome for Gaians, alike. Please contact me if you want to hear(read) more.


faeriebell
Community Member





Sat Dec 08, 2018 @ 02:33pm


Thank you for all the hard work you do. I worked in database dev a while ago with a huge cluster like that with replication and all the headaches that come with it, so I can imagine the scope of the migration you just accomplished. Bravo and I hope you take some time to congratulate yourself and your team for their hard work.


`kyu
Community Member





Tue Dec 18, 2018 @ 11:38am


I just came back to this site today to go on a nostalgia trip. I remember being on this website as a 12 year old, and loving it back then.

Now I'm in my late twenties, and also a fellow software developer. I can completely appreciate the technical challenges you are facing in migrating what must be a huge amount of legacy into the cloud.

My company is getting close to finishing its migration of all its services into the cloud, too. I'm sure it'll be incredibly fulfilling once you guys get this over the line. Good luck for the future, and please do keep posting - it was enlightening to look at things from the other side.

I don't know if I'll be back, but the next time I do visit, I'll be sure to look for updates here.


Mixed_Feelingz
Community Member





Wed Dec 19, 2018 @ 09:23am


Wow this all sounds very hopeful, there is so many solutions but the problem is maintaining it...hang in there, catch some breaks if you can and thank you all Gaia staff for your hard work!


Queen Guu
Community Member





Thu Dec 20, 2018 @ 03:14pm


when will the forum search engine(s) be fixed?
never?


p33p33s
Community Member





Tue Jan 08, 2019 @ 11:17pm


you can do it! thank you for sticking with it this far~ heart


pavidaustin
Community Member





Thu Jan 31, 2019 @ 05:08am


DID YOU SAY IMAGE UPLOADING OMG I WILL CRY TEARS OF JOY blaugh blaugh blaugh


OtakuKat
Community Member





Tue Aug 13, 2019 @ 10:49pm


It's amazing what all you guys do to keep Gaia up and working.


User Image
Lil'Scenery Adoptables

User Image
User Image - Blocked by "Display Image" Settings. Click to show.
User Comments: [47]
 
 
Manage Your Items
Other Stuff
Get GCash
Offers
Get Items
More Items
Where Everyone Hangs Out
Other Community Areas
Virtual Spaces
Fun Stuff
Gaia's Games
Mini-Games
Play with GCash
Play with Platinum