SemrushBot
Semrush · SEO tool
SemrushBot builds the link index behind Semrush's Backlink Analytics. It honours Crawl-delay up to 10 seconds and reads robots.txt per subdomain: a subdomain without its own file is treated as open to crawling.
Fact sheet
- Operator
- Semrush
- What it is for
- It crawls for an SEO tool that agencies and competitors use to audit sites.
- Assistant it feeds
- Does not feed any assistant's answers
- Counts towards the Parsigo score
- No. It is an SEO tool, not an AI one.
- Official documentation
- www.semrush.com/bot
If you block it
You ask the tool to stop analysing your site. It is your call, and your score does not change.
Allow or block
A group of its own makes SemrushBot ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: SemrushBot Allow: /
Block
User-agent: SemrushBot Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether SemrushBot can get in, with the robots.txt line that decides it. Free, no sign-up.
Worth knowing
Covers only the link index. Semrush's site audit uses a separate token, SiteAuditBot, which has to be blocked on its own.