Welcome to Gaia! :: View User's Journal | Gaia Journals

 
 

View User's Journal

Lanzer's Journal
A long day
Feeling good yesterday as I was figuring out more about Groovy, the scripting language used by zOMG. I woke up and was ready to see people tackling the Gnome General in Barton Town, but then all the servers had the "[no event]" flag on...

Then came the bad news.

Our logging servers were severely lagged. Main problem being the replication log server who had trouble backing up data due to an excessive amount of database writes. The problem caused errors all over the sites, as most features ended up writing logs in one way or another. Many features, including the Easter event, was turned off due to the error.

"What's going on???"

Upon closer inspection, we found that there were 5 times the activity on this one database for handling transactions. We were getting a TON of gold grants for this one application labeled... 737, which is Mini Games!

We initially thought that there were some errors in the coding, so we kept testing and debugging, but nothing suspicious was found. Then we started parsing the log to further investigate...

80% of all gold grants were generated by the same group of users.

"Yup, they're bots."

"This is why we can't have nice things" - James

As the minigame was brand new, it didn't went through a full security audit and was prone to botting. What followed was a lot of repair work, mainly because a lagging server works even slower than normal, and we need to minimize the amount of work the systems are placing on it.

We ended up doing a big search on ALL the logging functions to find what are things that are unnecessary, and discovering functions that were written years ago but we've never used since then. I guess if we had to do house cleaning, this is just as good a reason as any to get things done.

The backup server was 11 hours behind schedule when we started. Hopefully it'll catch up by later this morning and everything can be turned back on.






User Comments: [27]
Nyadriel
Community Member





Thu Apr 20, 2017 @ 03:35pm


Echoing the "why we can't have nice things". evil
I could never understand why there are people who think they have the need to use those friggin bots! Lazy cheats!

Well at least you got to catch up on some long needed stuff at last.

I thank you guys yet again for all your hard work. It is much appreciated by many of us.

Captcha of the hour: circle of life

Indeed it is!


AnneLaurant
Community Member





Thu Apr 20, 2017 @ 03:52pm


That sounds really hectic. Thanks for looking into the errors and working on them, Lanzer. I wish you and the team good luck!


star dazl
Community Member





Thu Apr 20, 2017 @ 04:00pm


So not Z! but greedy gaians who use bots!
Stop breaking r toys people!
Thanks lanzer


Anamosa Valentine
Community Member





Thu Apr 20, 2017 @ 04:04pm


It's sad and true that there are people out there who don't play fair and cheat using bots. I've seen that in all MMORPGs I've played in the last 10 years, and even zOMG is no exception. There should be something to beef up the security and keep botters out. sad

Thank you for working hard on this site, and hope you are getting good rest!


Daniel Hyuuga
Community Member





Thu Apr 20, 2017 @ 04:05pm


I've kept saying that Gaia could use some maintenance, but everyone was like nope "as long as things doesn't break" neutral


Knight Yoshi
Community Member





Thu Apr 20, 2017 @ 04:16pm


Well. At least you cleaned house. lol

James is right though.


VKiera
Community Member





Thu Apr 20, 2017 @ 04:25pm


James is right, this is why we can't have nice things. Thanks to you and the rest of the staff for working so hard to fix things.


OMFG Shushu
Community Member





Thu Apr 20, 2017 @ 05:03pm


LOL I hope who ever bots they were got banned too!
Geez we never get have nice things without some d**k messing it up for every!
Thank yall so much for working so hard to fix everything!

@Lazer; Well we get more days for the zOMG! Easter Event?


Parodius-Daryl
Community Member





Thu Apr 20, 2017 @ 05:08pm


I recall seeing 19,000 guests..wow i thought something weird was going on but..bots?! That's crazy..glad things are under control


Meridot
Community Member





Thu Apr 20, 2017 @ 05:22pm


It's such a shame that people can be so greedy and ruin the fun things for everyone. It's sort of sad that you always have to be aware that if there is a way for people to cheat, then people will cheat. gonk

Thank you for all the hard work you and the team do. heart I really enjoy reading your journal and seeing what is going on behind the scenes.


baby_kitty_258
Community Member





Thu Apr 20, 2017 @ 05:49pm


Wow! Kudos... that's a lot of work. Thank you for taking the time to search through everything. You know what? I say the exact same thing when no one cares to take the time to appreciate the effort others make for them: "this is why we can't have nice things" yup, so true. Thank you again! heart heart


Just Tips For All
Community Member





Thu Apr 20, 2017 @ 05:53pm


"I apologize Lanzer, just a bit of fun. I've always enjoyed this site, dont want to break things, just enjoy the site a little differently" - what the botters probably would say, this site is awesome heart sweatdrop


DarkRybrin
Community Member





Thu Apr 20, 2017 @ 06:48pm


Thank you for your hard work in tackling these problems.
Getting rid of bots and going through old code and functions is a terrible pain. You're doing a good job.


demonic_angelkynd1
Community Member





Thu Apr 20, 2017 @ 06:59pm


i hope you were able to track down the accounts that were botting and take back the gold and then ban them from gaia.


OtakuKat
Community Member





Thu Apr 20, 2017 @ 08:23pm


I'm glad to hear you guys were able to fix everything.


Shiro Nishida
Community Member





Thu Apr 20, 2017 @ 08:32pm


Thank you for your hard work as always, Lanzer & Co.

Sad to see botting running rampant... Hell, to even THINK Botting was the cause everything went FUBAR yesterday... It's a sad thing to see even after being in this site for this many years. sad


Tsukiryu-Kogo
Community Member





Thu Apr 20, 2017 @ 08:51pm


There is always one idiot to ruin things for everyone due to their greed.


Sianserais
Community Member





Thu Apr 20, 2017 @ 08:58pm


That figures. I also hope they were banned. People like that will never stop if there aren't consequences.
Thank you again for all the effort you put into this!


MewMewSun
Community Member





Thu Apr 20, 2017 @ 09:48pm


You guys can never catch a break, huh? Hope the bots are banned. Why can't people just play fair?


Kailer_Raimonji
Community Member





Thu Apr 20, 2017 @ 11:35pm


"a big search on ALL the logging functions" That sounds like it was a long, tedious and boring thing to do, but thank you Lanzer and all the devs that put up with such task to fix the issue.

And specially THANK YOU ALL for extending the event 2 days, that's considered. I don't think we would've gotten an extension, if this had happened with the previous management. They seemed to care more about GCash, than making things right for the community.

Bottom line: Thanks again and your team for taking ownership and caring!


True Karma
Community Member





Fri Apr 21, 2017 @ 12:07am


I agree with James ><


`Hydro
Community Member





Fri Apr 21, 2017 @ 12:11am


Thank you so much for your hard work!

;u; I'm glad you caught the bots.


Terminator Thot
Community Member





Fri Apr 21, 2017 @ 03:07am


Relax, Lanzer
and just get stoned on this very day.


Yuekazehime
Community Member





Fri Apr 21, 2017 @ 05:24am


This reminds of the time where they had just released one of the Loyal's gates (hey that's been awhile) from a kickstarter. There was no botting, but the collective rush of everyone happily opening their 5/10/25 packs and hitting play again collectively crashed gaia's servers in a similar fashion. They had to lock up the RIGs for a few hours, then only unlocked the singles followed eventually by the 5/10/25 ones. And ever since, Gaia has been scared to grant us large piles of packaged RIGs :3

It's interesting that just a handful of botters could create such chaos though. Makes you wonder what strain the daily "lesser" botters on various games have on the site.


Ziro Officer
Community Member





Sat Apr 22, 2017 @ 02:47am


Hey Lanzer.

Thanks for the update, and good luck skill on future progress.
How do you like using Groovy? Never used it myself, but heard it's similar to Java.


Arashisora
Community Member





Sat Apr 22, 2017 @ 04:36am


Hmm, time for a task force of Gaians whose sole objective is to try and "bot the system", allowing potential leaks to be discovered and plugged before they're exploited? They could all have matching terminator eye patches, rocket packs, and Bildeau hoodies. ...no? lol sweatdrop

Anyway, thanks for fixing things! heart


roselessfaith
Community Member





Mon Apr 24, 2017 @ 12:22am


I hope some users met the ban hammer sad thank you for the hard work


Been here since 2004...
Agate Achievement Granted - 3/8/17
User Image - Blocked by "Display Image" Settings. Click to show.User Image - Blocked by "Display Image" Settings. Click to show.User Image - Blocked by "Display Image" Settings. Click to show.
Click My Dragons Please!

User Image - Blocked by "Display Image" Settings. Click to show.
User Comments: [27]
 
 
Manage Your Items
Other Stuff
Get GCash
Offers
Get Items
More Items
Where Everyone Hangs Out
Other Community Areas
Virtual Spaces
Fun Stuff
Gaia's Games
Play with GCash
Play with Platinum

Customize your own avatar now!

Join Now

Have an account? Login Now!