New mechanism for publishers prevents data scraping

Publishers currently must apply multiple blocking orders for each data-collecting bot active on the internet individually, but the new proposed solution will allow them to enter unified instructions recognized and approved by any artificial intelligence company that adopts the protocol.

However, the solution will not enter full implementation for several months, with Microsoft set to begin applying it starting February 1, 2027, and Google at some point in 2027.

This announcement comes as publishers face a surge in bots copying their posts to feed and train AI models. Some entities, such as The New York Times, have turned to the courts on this issue.

In December 2023, The New York Times filed a lawsuit against OpenAI and Microsoft, accusing them of using millions of articles without permission or compensation to train the ChatGPT model.

Meanwhile, cloud services provider Cloudflare has automatically blocked AI-driven software on its new clients’ websites since July 2025, and there is also a program called “Very Simple License” that allows publishers to charge fees for bots that visit and scrape their pages.

Microsoft clarified in a blog post that the new setting will apply only to future model training, but otherwise will not prevent bots from crawling the internet, and content extraction will be permitted until the publisher changes this setting.

Meta spokesperson Andy Stone expressed support for this effort on the X platform, writing, “We believe that creating a shared signal that conveys these preferences to many providers simultaneously is an important step,” adding, “We look forward to working with other AI companies on these standards and implementing them across the sector.”