The amount of resources consumed by LLMs is staggering. And I don't just mean the "oh someone is generating a picture of a cat" application of those models.

I mean the whole way big tech currently operates and also sends a bunch of crawlers everywhere in a way that search engines *never* did.

Here's an amazing writeup about just how much of an impact that is having.

https://people.kernel.org/monsieuricon/creepy-crawlies

Creepy crawlies Konstantin Ryabitsev

Despite the huge scale difference compared to the Linux Kernel project, I experienced similar situations with some websites I take care of hosting (especially if they are 'content-rich', such as wikis). AI scrapping bots is another step in this kind of Red Queen's Race of keeping distributed web infrastructure alive.

RE: https://darmstadt.social/users/claudius/statuses/117184717002108837

Shares
@SignorMacchina@hessen.social @claudius@darmstadt.social