Claude-User
Anthropic · Answers right now
Claude-User fetches a page when someone asks Claude a question that needs it. Anthropic says its bots honour robots.txt, and that disallowing Claude-User stops Claude retrieving your content in response to those requests.
Fact sheet
- Operator
- Anthropic
- What it is for
- It fetches a page when someone asks the assistant a question, to answer with it.
- Assistant it feeds
- Claude
- Counts towards the Parsigo score
- Yes. It fetches pages for one of the assistants we measure.
- Official documentation
- support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Official IP ranges
- JSON list
If you block it
You ask the assistant (Claude) not to fetch your pages when someone asks it a question.
Allow or block
A group of its own makes Claude-User ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: Claude-User Allow: /
Block
User-agent: Claude-User Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether Claude-User can get in, with the robots.txt line that decides it. Free, no sign-up.