Transparency

Our crawler

We analyse sites that other people ask us to look at. If yours is one of them, here is everything we do — and how to stop it in a minute.

Our user agent

ParsigoBot/0.1 (+https://parsigo.com/bot)

How to block us

Add this to your robots.txt. We obey it from the next request onwards, without you having to tell us.

User-agent: ParsigoBot
Disallow: /

To block only the test that uses bot names (see below) and still let us analyse the rest, use User-agent: ParsigoBot-teste with Disallow: /.

If you'd rather we only stayed out of one part, swap the slash for the path: Disallow: /private/. And if your robots.txt already has a Disallow: / for *, nothing else is needed: we already obey it.

Would you rather write to us? info@parsigo.com .

What we request

When someone analyses a domain, we request:

Up to 12 pages of the site
Chosen from your sitemap and the links on the home page, spread across different sections. The home page tends to be the least representative page of a site, and a score drawn from it alone would say very little.
Your entry page, 8 more times
Once for each AI crawler we test (GPTBot, ClaudeBot, PerplexityBot and their search and answer variants), with its name in the user agent and ours next to it: GPTBot/1.0 (compatible; ParsigoBot-teste; …). That shows whether your firewall treats AI bots differently. If your robots.txt blocks us, as ParsigoBot or ParsigoBot-teste, we skip this test.
Your robots.txt
To work out, agent by agent, who is allowed in. It is the check that weighs most in the result.
Your sitemap
Whichever one is declared in the robots.txt or, failing that, /sitemap.xml.
Six more requests, if your robots.txt allows them
The index for agents (llms.txt and llms-full.txt), the commerce manifest (/.well-known/ucp), the Web Bot Auth directory, the Markdown version of your entry page, and a made-up address — something like /parsigo-a1b2c3d4-no-existe — to see how your error page responds. None of them count towards the score. If your robots.txt disallows any of these paths for ParsigoBot, we don't request it.
Up to six pages again, if someone asks for it in the report
Only for the «What an assistant understands from your site» section, and only when someone asks for it from the report: we read your robots.txt again and request up to six pages it allows, to give them as text to an AI assistant (Anthropic's). If your robots.txt closes the site to us, we request nothing. It doesn't count towards the score.

The pace

A full analysis is up to about 28 requests over a few seconds, once (fewer if your robots.txt closes off a path), plus up to seven more — the robots.txt and six pages — if someone asks for the assistant section from the report. We don't come back of our own accord: we only turn up when someone asks. We also cap how often the same domain can be analysed in a row, so as not to become a nuisance to your server.

What we don't do

We don't run JavaScript
We read the HTML exactly as it leaves your server — precisely what an AI crawler sees. That is the object of the measurement, not a limitation.
We don't pass ourselves off as anyone else
When we use another bot's name, in the test above, the user agent still says who we are and links to this page. Your robots.txt is evaluated by reading it and applying its rules; the test only looks at what reading cannot see: a firewall that filters by name.
We don't keep your content
We analyse the HTML in memory and keep the result: the score, the findings and the affected paths. We don't store the text of your pages. In the assistant section, up to six pages go as text to Anthropic, which as a rule deletes it within 30 days.
We don't go where we shouldn't
No private areas, forms, baskets, accounts or admin sections — those paths are discarded even when we come across them. Nor do we try to get around any protection.

How do you know it's us?

The user agent carries this very address in brackets. Anyone can write that text into a request — which is why we don't ask you to trust it, but to treat it as what it is: a signature you can verify as far as this page, and no further.

If you see that agent in your logs and weren't expecting it, it's because someone asked for an analysis of your domain. Block us with the lines above, or write to us.

What we check, how much each item weighs and how the score is worked out →