meta-externalagent
Meta · Collects for training
Meta uses meta-externalagent to train foundation AI models and to improve products by indexing content directly. Meta says a robots.txt change can take up to 24 hours to apply, because its crawlers cache the file.
Fact sheet
- Operator
- Meta
- What it is for
- It collects text that is used to train AI models.
- Assistant it feeds
- Does not feed any assistant's answers
- Counts towards the Parsigo score
- No. What it collects is used to train models, and training does not decide whether you are cited.
- Official documentation
- developers.facebook.com/documentation/sharing/webmasters/web-crawlers
If you block it
You ask it to stop collecting your content. It changes neither whether you are cited nor your score.
Allow or block
A group of its own makes meta-externalagent ignore the “User-agent: *” rules. If you have Disallow lines there that should still apply to it, repeat them inside its group. To allow it, writing nothing is usually enough.
Allow
User-agent: meta-externalagent Allow: /
Block
User-agent: meta-externalagent Disallow: /
Test it on your site
We run the diagnostic on your site and show you whether meta-externalagent can get in, with the robots.txt line that decides it. Free, no sign-up.