Reddit, Yahoo and Medium launch new licensing standard for AI content

Major web publishers including Reddit, Yahoo, Medium, and People Inc. have adopted a new Really Simple Licensing (RSL) standard that allows them to set compensation terms for AI companies scraping their content. The initiative creates a structured approach for publishers to negotiate fair payment from AI firms, addressing the ongoing crisis in web publishing as artificial intelligence companies have historically used online content without compensation.

What you should know: The RSL standard integrates licensing terms directly into the robots.txt protocol, the basic file that provides instructions for web crawlers.

• Supported licensing options include free, attribution, subscription, pay-per-crawl, and pay-per-inference models.
• Pay-per-inference means AI companies only compensate publishers when their content is actually used to generate responses.
• The standard is managed by the new RSL Collective nonprofit, which models itself after music royalty organizations like ASCAP and BMI.

Who else is involved: The participating companies represent a mix of internet veterans and major content platforms.

• Beyond the headline names, participants include Internet Brands, Ziff Davis, wikiHow, O’Reilly Media, The Daily Beast, Miso.AI, Raptive, Ranker, and Evolve Media.
• Former Ask.com CEO Doug Leeds and RSS co-creator Eckart Walther lead the RSL Collective.
• Cloud company Fastly serves as a technical partner to help enforce the licensing terms.

The enforcement challenge: Whether AI companies will actually honor the new standard remains uncertain, given their history of ignoring existing robots.txt instructions.

• The group believes its terms will be legally enforceable, with Leeds pointing to Anthropic’s recent $1.5 billion settlement as evidence that “there’s real money at stake” for AI companies that don’t train “legitimately.”
• The collective structure could help spread legal costs among publishers, making challenges to violations more financially feasible.
• Fastly acts as “the bouncer at the door to the club,” serving as a technical gatekeeper for content access.

What they’re saying: Reddit CEO Steve Huffman emphasized the collaborative nature of the initiative in addressing AI-era challenges.

• “The RSL Standard gives publishers and platforms a clear, scalable way to set licensing terms in the AI era,” Huffman stated.
• “The RSL Collective offers a path to do it together,” he added, noting it represents “important steps toward protecting the open web and the communities that make it thrive.”
• Leeds suggested the standard addresses AI companies’ complaints about lacking effective means to license web-wide content: “We have listened to them, and what we’ve heard them say is… we need a new protocol.”

Why AI companies might participate: The standard potentially offers benefits beyond just avoiding legal troubles.

• It could simplify licensing by providing a unified system rather than requiring individual deals with each publisher.
• AI companies could use the best available sources without worrying about over-relying on any single publisher, potentially improving response quality and reducing hallucinations.
• The pay-per-inference model means companies only pay for content that actually contributes to generated responses.