Millions of websites, including Sky News, The Associated Press and Buzzfeed, will now be able to block artificial intelligence (AI) bots from accessing information they publish on their pages, the BBC reports.
The new system is being offered by network infrastructure company Cloudflare, and will eventually allow websites to charge for content that AI companies extract from their pages.
Many well-known writers, artists, musicians and actors have accused AI companies of using their work to train systems without offering compensation.
Cloudflare’s technology works by identifying AI companies’ bots, so-called “crawlers” – programs whose main task is to scout the web, collecting and organizing data. These programs are a vital part of the process of building and training AI, as they provide the necessary information for AI to work with.
Roger Lynch, CEO of Condé Nast, which publishes GQ, Vogue and The New Yorker, said Cloudflare’s offering would be a game-changer for publishers and a major step towards a fairer internet exchange that
protects authors, supports quality journalism and holds AI companies accountable.
The system will initially be automatically applied to new Cloudflare customers, as well as websites that have already participated in attempts to block crawlers.
Many publishers have accused AI companies of using unauthorized content. The BBC has threatened legal action against US AI company Perplexity, demanding that it immediately stop using BBC content and pay for content it has already used.
Website owners are generally happy with search engines like Google crawling their pages because it allows search engines to direct people to their pages. Perplexity has accused the BBC of supporting Google’s “monopoly.”
Cloudflare, however, says AI companies are violating the unwritten agreement between authors and crawlers. That is, AI collects content – articles, text and images – to generate answers without taking the person to the source of the information, thereby depriving content creators of profits. Cloudflare CEO Matthew Prince said that if the internet is to survive the AI era, it must
give authors the control they deserve and create an economic model that benefits everyone.
According to Cloudflare, there is currently an “explosion” of AI bot activity. In March, the company reported that AI crawlers were making more than 50 billion requests per day on its network, and there are growing concerns that AI bots can bypass current protocols for protecting against bots. To combat infringers, Cloudflare previously developed a system that sent the most intrusive crawlers to a maze of websites filled with useless AI-generated works. The new system will seek to use technology that protects the content of websites while allowing AI companies to charge for the use of the content.
In the UK, there is currently a fierce debate between the government, the creative industry and AI companies about how much of artists’ work can be used without permission and compensation to train AI. On both sides of the Atlantic, owners have repeatedly gone to court to prevent AI companies from infringing on their creative rights.
Ed Newton-Rex, founder of Fairly Trained, said he was looking forward to the new system. His company certifies AI developers who use properly acquired data to train their programs. However, Newton-Rex said this is “just putting a band-aid on a broken bone” and
the only way to truly protect authors and artists from the theft of their intellectual property is through legislative change.
Director Beeban Kidron, a long-time advocate for protecting the creative industries, said Cloudflare had shown initiative, adding that AI companies should contribute to the community by paying taxes, making amends with artists whose work was stolen to train AI, and using technology in ways that ensure equality between the digital and human worlds.
Read also: Macron speaks with Putin; stresses support for Ukraine