A sign without a lock: How the web decides which AI Agents to trust

AI agents no longer just answer questions. They browse, read and act on online services on our behalf. Every website now faces a new decision: trust this automated traffic, restrict it, or block it.

This talk presents the results of an empirical study on how websites actually treat AI-related User Agents. Combining large-scale measurement with controlled testing, we compared what websites declare in their policies, such as robots.txt, with what they really enforce. We also examined whether they identify AI agents, apply differentiated rules, and penalize some agents more than others.

The results show a fragmented and immature ecosystem. Some AI agents are widely accepted, while others face significantly more restrictions and blocking. In many cases, declared rules and real enforcement do not match. Together, these patterns point to new forms of trust and reputation between websites and autonomous systems.

Attendees will leave with a clear view of how the Internet is adapting to AI-driven traffic, which agents are penalized most often, and what this means for security, for businesses that publish content, and for the future of machine-to-web communication.