This forum is flooded with bots

Gaming-related threads & forum meta.
User avatar
Porcupine
Virtual-On Positive
Posts: 597
Joined: 19 Jul 2012, 19:07

This forum is flooded with bots

Post by Porcupine »

Starting around maybe 5 years ago, I noticed that unknown bot armies had flooded oratan.com forum. Prior to that time, the average number of guests and registered users at any given time would be 0 or 1 people (which matches the active population of the game). When the bots came, that number often jumped to 30~50, but it would dwindle back down to 9 or so most of the time, and it didn't really affect anything so it wasn't a problem.

These past few months the amount of bot activity seems to have increased by some multiple, and today it's suddenly multiplied yet again. The maximum number of simultaneous users is the same, but the swarms appear and disappear much more often. Now there seem to be leader bots with registered accounts who regularly summon these swarms of 20~40 guest bots. I'm not sure what they do, I think they just read posts, but they are doing it so often today that this site is slowing down. Opening posts and pages, which is usually fairly consistent taking half a second or so, is sluggish and will randomly vary from the typical half second up to 5 seconds.

I guess this isn't a big deal but I'm not so sure these bots help us in any way. The leader bot accounts seem to be Ahrefs [Bot], Bing [Bot], and Google [Bot]. I don't know if it is possible to ban them, or if that would do any good, since it's their unregistered guest bot swarms that are probably the real slowdown factor.

Maybe the guest swarms are not bots, but random people around the world who are clicking on oratan.com links having no idea what Virtual On is, but having come up in a search for some random reason? Hm, this wouldn't explain why the guest swarms typically rapidly increase then decrease in number, though.

edit: A day later, now I see Majestic-12 [Bot] for the first time. Internet search suggests that it's a known malicious attack bot that eats up bandwidth to take down sites.
User avatar
MentholMoose
Virtual-On Positive
Posts: 2048
Joined: 15 Dec 2008, 22:06

Re: This forum is flooded with bots

Post by MentholMoose »

Thanks for the heads up! The traffic causing the problem was from Alibaba Cloud and is now blocked.
MentholMoose
User avatar
Porcupine
Virtual-On Positive
Posts: 597
Joined: 19 Jul 2012, 19:07

Re: This forum is flooded with bots

Post by Porcupine »

Thanks, everything is working normally again. There are still the same 20~40 guest swarmers, with several named bots, but it's been this way for the past years and generally isn't a problem. Now I see a Baidu [Spider] account logged in. I guess the real problem though was essentially just one or a few unnamed guests causing mass mayhem?
User avatar
MentholMoose
Virtual-On Positive
Posts: 2048
Joined: 15 Dec 2008, 22:06

Re: This forum is flooded with bots

Post by MentholMoose »

I think it was a misconfigured or broken web scraper, although the user agent was randomized which isn't typical of legitimate scraping e.g. by search engines. Thanks for letting me know.
MentholMoose
User avatar
Porcupine
Virtual-On Positive
Posts: 597
Joined: 19 Jul 2012, 19:07

Re: This forum is flooded with bots

Post by Porcupine »

I'll just use this place to catalog all the registered bot users that I notice. Today I newly see Brandwatch [bot], and Bytespider [Bot]. We are really popular!

Also, after a day the total number of simultaneous unregistered users has dwindled back down to "normal" again, like 9 or so. Plus we typically have like 2 or 3 registered bots on now.

Just as I wrote this, the biggest group of registered bots online together that I have ever seen by far has suddenly appeared: Ahrefs [Bot], Brandwatch [bot], Bytespider [Bot], Petal [Crawler], Pinterest [Bot], Porcupine, Sogou [Crawler]

Plus SinisterSticks, a new regular user who is probably not a bot, but I'll list him for now because who knows what could be going on hehe. I noticed that the bot names are all in bold but people names aren't, not sure how this forum identifies them.

Now I see FriendlyCrawler [Crawler] too, what is this some kind of parade hehe.
User avatar
MentholMoose
Virtual-On Positive
Posts: 2048
Joined: 15 Dec 2008, 22:06

Re: This forum is flooded with bots

Post by MentholMoose »

Yes SinisterSticks is a new user, welcome!

Since I was looking at logs I went ahead and "registered" some of the legitimate bots that weren't previous registered.
Identifying bots is important so that phpBB can serve them content which is more appropriate for search engines - dead links to pages without content are omitted, e.g. posting pages, report post pages etc. Bots never receive a session ID in the URL, which should not appear in the search results.
MentholMoose
User avatar
Porcupine
Virtual-On Positive
Posts: 597
Joined: 19 Jul 2012, 19:07

Re: This forum is flooded with bots

Post by Porcupine »

No apparent problems with site usage or speed at the moment, but...

In total there are 185 users online :: 4 registered, 0 hidden and 181 guests (based on users active over the past 5 minutes)
Most users ever online was 375 on 02 Dec 2018, 01:16

Holy cow, 185 users? Most I've ever seen by an order of magnitude. I guess the record is 375, which was kind of recent as well (around when I last stopped playing; I did not buy Masterpiece for many years).
User avatar
MentholMoose
Virtual-On Positive
Posts: 2048
Joined: 15 Dec 2008, 22:06

Re: This forum is flooded with bots

Post by MentholMoose »

Thanks for the heads up. I'll check the logs but probably it is just a scraper gone wild or something.
MentholMoose
User avatar
Porcupine
Virtual-On Positive
Posts: 597
Joined: 19 Jul 2012, 19:07

Re: This forum is flooded with bots

Post by Porcupine »

In total there are 340 users online :: 2 registered, 0 hidden and 338 guests (based on users active over the past 5 minutes)
Most users ever online was 404 on 19 Apr 2024, 12:24

We just set a new record! I think the site is still functioning at normal speed, regardless.
User avatar
MentholMoose
Virtual-On Positive
Posts: 2048
Joined: 15 Dec 2008, 22:06

Re: This forum is flooded with bots

Post by MentholMoose »

That one was Anthropic. I registered their bot.
MentholMoose