A 1950s toy robot Photo: T-Bone Sandwich // CC-BY-NC-SA 2.0
You might have heard of Crawler bots before. In fact, they are how search engines populate and index their search results. But A.I. Crawlers are very different. After scraping data on mass, indiscriminately, then storing said data in its ever-growing knowledge base - the next time you ask an LLM (Large Language Model) a question, it doesn't matter where its summerised information comes from, in most cases there's no source attribution and websites lose out on valuable traffic.
However, Cloudflare's new private beta programme may see A.I. Crawlers hit a paywall of their own, and the two other giant Content Delivery Network providers may be allowing their customers to do the same thing soon: Fastly (they do this already) and Akamai. Hearing this for the first time made our own Corey Fuimaono think what ramifications this might have for the media, content creators, and A.I. companies?
To discuss more, they spoke with Kelly Shortridge, VP of Security at Fastly.
