AhrefsBot
Ahrefs · SEO tool
AhrefsBot builds the database behind Ahrefs and Yep, the search engine Ahrefs develops. It follows Crawl-delay for HTML pages but not when rendering JavaScript, so your logs may show more requests than the delay allows.
Fact sheet
- Operator
- Ahrefs
- What it is for
- It crawls for an SEO tool that agencies and competitors use to audit sites.
- Assistant it feeds
- Does not feed any assistant's answers
- Counts towards the Parsigo score
- No. It is an SEO tool, not an AI one.
- Official documentation
- ahrefs.com/robot
If you block it
You ask the tool to stop analysing your site. It is your call, and your score does not change.
Allow or block
A group of its own makes AhrefsBot ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: AhrefsBot Allow: /
Block
User-agent: AhrefsBot Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether AhrefsBot can get in, with the robots.txt line that decides it. Free, no sign-up.
Worth knowing
Covers only Ahrefs's index. Its site audit uses a separate token, AhrefsSiteAudit.