this post was submitted on 08 Jun 2025
209 points (98.2% liked)

Fuck AI

3036 readers
957 users here now

"We did it, Patrick! We made a technological breakthrough!"

A place for all those who loathe AI to discuss things, post articles, and ridicule the AI hype. Proud supporter of working people. And proud booer of SXSW 2024.

founded 1 year ago
MODERATORS
 

It's impossible, i got this instance to just see lemmy from my own instance, but no, it was slow as hell the whole week, i got new pods, put postgres on a different pod, pictrs on another, etc.

But it was slow as hell. I didn't know what it was until a few hours before now. 500 GETs in a MINUTE by ClaudeBot and GPTBot, wth is this? why? I blocked the user agents, etc, using a blocking extension on NGINX and now it works.

WHY? So google can say that you should eat glass?

Life is now hell, if before at least someone could upload a website, now even that is painfull.

Sorry for the rant.

you are viewing a single comment's thread
view the rest of the comments
[โ€“] carrylex@lemmy.world 7 points 12 hours ago (2 children)

So I just had a look at your robots.txt:

User-Agent: *
  Disallow: /login
  Disallow: /login_reset
  Disallow: /settings
  Disallow: /create_community
  Disallow: /create_post
  Disallow: /create_private_message
  Disallow: /inbox
  Disallow: /setup
  Disallow: /admin
  Disallow: /password_change
  Disallow: /search/
  Disallow: /modlog
  Crawl-delay: 60

You explicitly allow searching your content by bots... That's likely one of the reasons why you get bot traffic.

[โ€“] plz1@lemmy.world 1 points 6 hours ago

AI crawlers ignore robots.txt. The only way to get them to stop is with active counter measures.