According to an announcement on Tuesday, the leading internet architecture provider Cloudflare will now block the AI ​​web crawler known by default AI web crawler to prevent “reaching material without permission or compensation”. With changes, Cloudflare will start asking the new domain owners whether they want to allow AI scrapers, and will even allow some publishers to apply the “per crawl pay” fee.

Paying per crawl program will determine publishers to determine a value for AI scrapers to reach their content. AI companies can then see pricing and choose whether to register or have been removed for “per crawl payment” fee. This is only available for “a group of some major publishers and material creators”, but Claudflair says that it will ensure that “AI companies can use quality materials correctly – with permission and compensation.”

Cloudflare Domain has been helping owners fight the AI ​​Craler for some time. The company started giving websites Block AI Craler In 2023, but it applies only to those who follow Robot of a site .txt fileThe unattainable agreement indicates that bots may reflect its content. Cloudflare started Allow to block websites “All” AI Bots last year – whether they respect a site robot. (The company identifies scrapers to block by comparing them to the list of their known AI bots.) Claudflaler also rolled out a feature in March. It sends web-claruling bots In a “AI Labbrolean” to prevent sites from scraping the sites without permission.

Many major publishers and online platforms, including The Associated Press, Atlantic, LuckStack overflow, and Quora, Cloudflare’s new AI Craler are on board with restrictions, as websites struggle with the future, where more people find information through AI chatbot instead of search engines. “People rely more on AI in the last six months, meaning they are not reading the original material,” CEO Matthew Prince of Claudflare Said during the Exios Live event last week,

Additionally, Cloudflare says it is working with AI companies so that they can help verify their craler and “clearly tell their purpose” such as they are using materials for training, estimates or search. The owners of the website can then review this information and determine which crawler to go inside.

Prince said in a press release, “The original material is one that makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it.” “AI Craler is scrapping materials without border. Our goal is to bring power back to the creators, while still AI helps in innovation.”