Meta-WebIndexer
Meta · Another search engine or assistant
Meta-WebIndexer crawls the web to improve the quality of Meta AI's search results. Meta says that allowing it in your robots.txt helps Meta AI cite and link to your content in its answers.
Fact sheet
- Operator
- Meta
- What it is for
- It crawls for a search engine or assistant the score does not measure.
- Assistant it feeds
- None of the assistants we measure
- Counts towards the Parsigo score
- No. It crawls for search or answers, but not for any of the assistants we measure.
- Official documentation
- developers.facebook.com/documentation/sharing/webmasters/web-crawlers
If you block it
You ask for your pages to be left out of the service it crawls for. Your score does not change: it belongs to none of the assistants we measure.
Allow or block
A group of its own makes Meta-WebIndexer ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: Meta-WebIndexer Allow: /
Block
User-agent: Meta-WebIndexer Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether Meta-WebIndexer can get in, with the robots.txt line that decides it. Free, no sign-up.