Andreas Reuter

Google's Gary Illyes: robots.txt is an access-preference protocol (RFC 9309). It says nothing about what happens to data after a crawl. To block an AI crawler, name its user-agent token. https://www.searchenginejournal.com/google-illyes-robots-txt-crawling-ai-training/593957/

robotcheck v1.19.0 is out. Two new llms.txt checks: machine-readable data references and dash-separated formats. Both are info-level, up to 1 point each. If you publish an llms.txt, review these two points. Changelog: https://robotcheck.coffee/changelog

robotcheck v1.18.0 is out. Two new checks look at how your llms.txt points to an MCP server: whether it references a server card and names the transport protocol. Both are info-level, up to 1 point each. 74 active checks now. https://robotcheck.coffee/changelog #llmstxt

Meta's new Commercial Terms let Meta and its agents crawl business sites from Oct 30, 2026. robots.txt isn't mentioned. Effect unclear. https://marketingnewsroom.com/seo-ai-search/ai-search/meta-commercial-terms-crawling-business-websites/

Rule update v1.17.0: two new checks for llms.txt. One flags a missing MCP server section (info, up to 1 point). One flags llms.txt blocked by robots.txt (critical, up to 3 points). 72 active checks now. Check your setup: https://robotcheck.coffee/changelog

Can ChatGPT, Perplexity & Gemini actually read your website? Most sites fail — not because of bad content, but missing structured data or broken robots.txt. RobotCheck.coffee checks 8 factors. Free. No signup. Instant. ☕ robotcheck.coffee