Diffbot-User
Diffbot · Another search engine or assistant
Diffbot-User makes requests on behalf of a person browsing a URL with Diffbot software, in response to their input. Diffbot lists it apart from the Diffbot crawler and says Diffbot does not crawl pages to train generative AI foundation models.
Fact sheet
- Operator
- Diffbot
- What it is for
- It fetches a page when someone asks it to, for an assistant the score does not measure.
- Assistant it feeds
- None of the assistants we measure
- Counts towards the Parsigo score
- No. It crawls for search or answers, but not for any of the assistants we measure.
- Official documentation
- www.diffbot.com/docs/crawl/faq/robots-txt
If you block it
You ask it not to fetch your pages when someone asks it to. Your score does not change: it belongs to none of the assistants we measure.
Allow or block
A group of its own makes Diffbot-User ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: Diffbot-User Allow: /
Block
User-agent: Diffbot-User Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether Diffbot-User can get in, with the robots.txt line that decides it. Free, no sign-up.