CCBot
Common Crawl · Collects for training
CCBot builds Common Crawl's open repository of web crawl data, which anyone can access and analyse. Common Crawl warns that other crawlers pretend to be CCBot; on IPv4, the real one's reverse DNS ends in crawl.commoncrawl.org.
Fact sheet
- Operator
- Common Crawl
- What it is for
- It collects text that is used to train AI models.
- Assistant it feeds
- Does not feed any assistant's answers
- Counts towards the Parsigo score
- No. What it collects is used to train models, and training does not decide whether you are cited.
- Official documentation
- commoncrawl.org/ccbot
- Official IP ranges
- JSON list
If you block it
You ask it to stop collecting your content. It changes neither whether you are cited nor your score.
Allow or block
A group of its own makes CCBot ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: CCBot Allow: /
Block
User-agent: CCBot Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether CCBot can get in, with the robots.txt line that decides it. Free, no sign-up.
Worth knowing
Common Crawl feeds a good part of the industry's training corpora.