@Clearwater@lemmy.world
member since 26 Jun 2023 04:02
comments
- on Google’s AI Is Destroying Search, the Internet, and Your Brain in c/technology@lemmy.world · 9h ago
-
on Google’s AI Is Destroying Search, the Internet, and Your Brain in c/technology@lemmy.world · 19h ago
I run a few personal sites with some being available to the wider internet, and I have found that they generally do respect it. All the bots which advertise themselves as OpenAI, Anthropic, or Google do obey. However, I do occasionally see a stealth bot appear which uses a normal browser’s user agent and those just do whatever they want.
I’ll have to double check my stats later. Haven’t looked at it for a while.
I certainly don’t recall them ignoring robots.txt, but I also don’t remember seeing Meta in my dash at all, so it’s entirely possible they, for wherever reason, never found my site.