Directory

AI crawlers: who they are and how to manage them

One page for each name you can use in robots.txt, from AI crawlers to social network and SEO bots: who runs it, what it is for, whether it counts towards the diagnostic's score and what happens if you block it.

Each page links to the operator's official documentation, and says when we could not find it. The robots.txt blocks to copy are checked by the same robots.txt parser the diagnostic uses.

To write a robots.txt with these names, use the robots.txt generator: choose who gets in, group by group, and take the finished file.

To find out which of these bots really visit you, use the log checker: it reads your site's access log in your browser and confirms the IP of every visit.