Post

Not sure how we went from "just provide a robots.txt if some content should not be indexed" to "all the traffic on the website is AI scraping".