Why some websites are blocking AI agents from accessing the web
AI-generated summary Cloudflare has made available new controls that let website owners distinguish between search crawlers, AI training bots, and agents acting on behalf of users. The new settings default to allowing search crawlers while blocking AI training bots and agents on pages displaying ads, though website owners can adjust those rules.