Transparency

Our crawler

We analyse sites that other people ask us to look at. If yours is one of them, here is everything we do — and how to stop it in a minute.

Our user agent

ParsigoBot/0.1 (+https://parsigo.com/bot)

How to block us

Add this to your robots.txt. We obey it from the next request onwards, without you having to tell us.

User-agent: ParsigoBot
Disallow: /

If you'd rather we only stayed out of one part, swap the slash for the path: Disallow: /private/. And if your robots.txt already has a Disallow: / for *, nothing else is needed: we already obey it.

Would you rather write to us? hola@parsigo.com .

What we request

When someone analyses a domain, we request:

Up to 12 pages of the site
Chosen from your sitemap and the links on the home page, spread across different sections. The home page tends to be the least representative page of a site, and a score drawn from it alone would say very little.
Your robots.txt
To work out, agent by agent, who is allowed in. It is the check that weighs most in the result.
Your sitemap
Whichever one is declared in the robots.txt or, failing that, /sitemap.xml.

The pace

A full analysis is about 14 requests over a few seconds, once. We don't come back of our own accord: we only turn up when someone asks. We also cap how often the same domain can be analysed in a row, so as not to become a nuisance to your server.

What we don't do

We don't run JavaScript
We read the HTML exactly as it leaves your server — precisely what an AI crawler sees. That is the object of the measurement, not a limitation.
We don't pass ourselves off as anyone else
We don't send requests with GPTBot's user agent, or anyone else's, to see what happens. Your robots.txt is evaluated by reading it and applying its rules, which is just as rigorous without impersonating anybody.
We don't keep your content
We analyse the HTML in memory and keep the result: the score, the findings and the affected paths. The text of your pages is not stored.
We don't go where we shouldn't
No private areas, forms, baskets, accounts or admin sections — those paths are discarded even when we come across them. Nor do we try to get around any protection.

How do you know it's us?

The user agent carries this very address in brackets. Anyone can write that text into a request — which is why we don't ask you to trust it, but to treat it as what it is: a signature you can verify as far as this page, and no further.

If you see that agent in your logs and weren't expecting it, it's because someone asked for an analysis of your domain. Block us with the lines above, or write to us.

What we check, how much each item weighs and how the score is worked out →