An index of 150 web crawlers and the 74 operators behind them - with the one field the vendor lists leave out: what blocking each one actually costs you
www.pathwren.workers.devpreview
Every crawler list I could find answers the easy half of the question - which bots exist - and skips the half that decides the config: what you lose by blocking each one.
So that is a field on every one of the 150 records here, in plain language. Blocking an assistant fetcher that follows a person’s link costs you the referral, not the training use. Blocking a search crawler costs you the index entry. Blocking a training crawler costs you approximately nothing you can measure, which is a legitimate answer and is written down as one.
Each record carries the robots token as the operator documents it, the operator (74 of them, deduplicated across rebrands), the category - AI training, AI search, user-triggered fetch, classic search, SEO, archive, liveness check - and the verification method that operator actually publishes: reverse DNS where they document it, a published IP range list where they do not. That distinction matters, because a user-agent string is free to type: 1987 IPv4 and 1062 IPv6 prefixes from 15 operator endpoints are mirrored here for exactly that reason, re-fetched every six hours.
Also in it: which crawlers are documented as honouring robots.txt versus merely observed doing so. The index does not pretend the second group is the first.
Static files, no key, no rate limit, CORS open. Data CC0, tooling MIT.
www.pathwren.workers.dev/c/lemmy/crawler/
(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)