so, I should provide some more context to that, I guess. my web server setup isn't "small" by most casual hosters definition. the total traffic usually is always above 5req/s, and this is not an issue for me.

also, crawlers are everywhere. not just those that I mentioned, but also search engine crawlers, and others. a big chunk of my traffic is actually from "bytespider", which is the LLM training bot from the TikTok company. It wasn't mentioned in this post, because although they make a lot of traffic (in terms of traffic size), that's primarily because they also ingest images, and their request volume is generally low.

some spiders are more aggressive than others. a long, long time ago - I've seen a crawler try to enumerate diaspora*s numeric post IDs to crawl everything, but cases like this are rare.

in this case, what made me angry was the fact that they were specifically crawling the edit history of the diaspora wiki. that's odd, because search engines don't care about old content generally. it was also odd, because the request volume was so high, it caused actual issues. MediaWiki isn't the best performance-wise, and especially the history pages are really, really slow. and if you have a crawler firing requests are multiple requests per second, this is bad - and noteworthy.

I've talked privately to others with affected web properties, and it indeed looks like some of those companies have "generic web crawlers", but also specific ones for certain types of software. MediaWiki is frequently affected, and so are phpBB/smf forums, apparently. those crawlers seem to be way more aggressive than their "generic web" counterparts - which might actually just be a bug, who knows.

a few people here, on mastodon, and on other places, have made "suggestions". I've ignored all of them, and I'll continue to ignore all of them. first, suggesting blocking user agent strings or IPs is not a feasible solution, which should be evident to everyone who read my initial post.

I'm also no a huge fan of the idea to feed them trash-content. while there are ways to make that work in a scalable and sustainable fashion, the majority of suggestions I got were along the lines of "use an LLM to generate trash content and feed it to them". this is, sorry for the phrase, quite stupid. I'm not going to engage in a pissing contest with LLM-companies about who can waste more electricity and effort. ultimately, all you do by feeding them trash content is to make stuff slightly more inconvenient - there are easy ways to detect that and to get around that.

for people who post stuff to the internet and who are concerned that their content will be used to train LLMs, I only have one suggestion: use platforms that allow you to distribute content non-publicly, and carefully pick who you share content with. I've gotten a lot of hate a few years ago for categorically rejecting a diaspora feature that would implement a "this post should be visible to every diaspora user, but not search engines" feature, and while that post was written before the age of shitty LLMs, the core remains true: if your stuff is publicly on the internet, there's little you can do. the best thing you can do is be politically engaged and push for clear legislative regulation.

for people who host their own stuff, I also can only offer generic advice. set up rate limits (although be careful, rate limits can easily hurt real users, which is why the wiki had super relaxed rate limits previously). and the biggest advice: don't host things. you'll always be exposed to some kind of abuse - if it's not LLM training bots, it's some chinese or russian botnet trying to DDoS or crawl for vulnerabilities, or some spammer network who want to post viagra ads on your services.