Or Lenchner, the CEO of Brilliant Knowledge, one of many world’s largest web-scraping companies, says that his firm’s bots don’t accumulate nonpublic data. Brilliant Knowledge was beforehand sued by Meta and X for allegedly improperly scraping content material from their platforms. (Meta later dropped its suit, and a federal decide in California dismissed the case brought by X.)
Karolis Stasiulevičiu, a spokesperson for an additional cited firm, ScrapingBee, informed WIRED: “ScrapingBee operates on one of many Web’s core rules: that the open net is supposed to be accessible. Public net pages are, by design, readable by each people and machines.”
Oxylabs, one other scraping agency, stated in an unsigned assertion that its bots don’t have “entry to content material behind logins, paywalls, or authentication. We require prospects to make use of our companies just for accessing publicly accessible data, and we implement compliance requirements all through our platform.”
Oxylabs added that there are a lot of authentic causes for companies to scrape net content material, together with for cybersecurity functions and to conduct investigative journalism. The corporate additionally says that the countermeasures some web sites use don’t discriminate between completely different use instances. “The truth is that many fashionable anti-bot techniques don’t distinguish effectively between malicious visitors and legit automated entry,” Oxylabs says.
Along with inflicting complications for publishers, the web-scraping wars are creating new enterprise alternatives. TollBit’s report discovered greater than 40 corporations that at the moment are advertising bots that may accumulate net content material for AI coaching or different functions. The rise of AI-powered serps, in addition to instruments like OpenClaw, are possible serving to drive up demand for these companies.
Some companies promise to assist corporations floor content material for AI brokers relatively than attempt to block them, a technique generally known as generative engine optimization, or GEO. “We’re primarily seeing the rise of a brand new advertising channel,” says Uri Gafni, chief enterprise officer of Brandlight, an organization that optimizes content material in order that it seems prominently in AI instruments.
“This may solely intensify in 2026, and we’re going to see this rollout sort of as a full-on advertising channel, with search, adverts, media, and commerce converging,” Gafni says.
This story initially appeared on wired.com.
