← All posts
· · 12 min read

The Web Was Not Built for Agents. We Spent Months Figuring Out What That Actually Means.

We built a tool that measures how ready any website is for AI agents. We are publishing it today. This post explains what we found along the way, what the industry signals are telling us, and why we think the window for getting ahead of this is narrower than most people realise.

We have not done any of this publicly until now. No announcements, no posts, no previews. Just building, testing against our own infrastructure, throwing out what was wrong, and rebuilding. That is the version you are reading about today.

Why agent readiness is not the same as SEO

For most of the past decade, "being findable online" meant one thing: ranking well in Google. SEO is a well-understood discipline. You optimise your title tags, build inbound links, ensure your pages load quickly, write content that matches search intent. The rules are imperfect but known.

AI agents work differently. When a user asks Claude, ChatGPT, or Perplexity to research a product, book a service, or complete a task that involves visiting your website, the agent is not a search crawler passively indexing your content. It is an autonomous actor trying to complete a goal. It needs to:

  • Understand what your site does and what it offers, without reading every page
  • Retrieve clean content without fighting through navigation HTML and cookie banners
  • Discover what tools or capabilities your site exposes -- APIs, payment rails, booking flows
  • Authenticate and act, in some cases without any human in the loop

The web was not built for this. Most websites are built for human eyes on a screen. That gap is what we set out to measure.

What the industry is already telling us

We are not the only ones who have noticed. The signals from infrastructure providers and market researchers in recent weeks have been unusually clear.

Cloudflare

Cloudflare launched their own Agent Readiness score on 17 April 2026, having scanned the 200,000 most-visited domains on the internet. Their headline finding: robots.txt is nearly universal (78% of sites have one), but the vast majority are written for traditional search crawlers, not AI agents. Only 4% of sites have Content Signal directives -- the emerging standard for declaring AI usage preferences. Markdown content negotiation, which allows an agent to request clean structured content instead of raw HTML, passes on just 3.9% of sites.

In other words: the web is almost entirely unprepared for the traffic that is already arriving.

Cloudflare's April 2026 Agent Cloud expansion makes their intentions clear. They are building the infrastructure layer for production-grade agents -- Dynamic Workers, Git-compatible storage, sandboxed Linux environments -- and they have declared the intent to make Cloudflare "the definitive platform for the agentic web." When the company running the network layer for a significant portion of the internet makes that bet, it is worth paying attention to.

Coinbase and the x402 protocol

The payment layer that agent commerce will run on is being built right now. Coinbase's x402 protocol -- which revives the dormant HTTP 402 status code and turns it into a machine-readable payment negotiation layer -- has already processed over 50 million machine-to-machine transactions. An agent requests a paid resource. The server responds with a 402 and payment instructions. The agent pays in USDC. The resource is released. No accounts. No API keys. No human approval per transaction. The entire cycle takes two to four seconds.

The x402 Foundation now includes Google, Visa, AWS, Circle, Anthropic, Vercel, Cloudflare, and Coinbase. When that coalition backs a protocol, it is not a niche experiment -- it is foundational infrastructure. The implication for web operators: if an AI agent can autonomously discover your API, authenticate via OAuth, request your content in markdown, and pay for gated resources via x402, it can do business with you without a human being involved at any step. Whether that is relevant to your business today or not, the infrastructure for it is being laid right now.

BrightEdge: the traffic data

BrightEdge published research in April 2026 showing that AI agent requests have reached 88% of human organic search activity in volume. Agent activity already accounts for approximately 15% of total web traffic -- but because it does not appear in traditional analytics platforms like Google Analytics, most companies have no visibility into it and therefore no strategy for it.

Only 19% of sites have specific directives for ChatGPT-related bots. Most companies are treating AI agents like traditional bots, applying obsolete or conflicting rules that limit how effectively their content can be surfaced by AI systems. BrightEdge projects that agent traffic will surpass human-driven search by the end of 2026.

The pattern is consistent across all three sources: agents are here, the infrastructure for them is being built at pace, and almost no websites are ready for either.

What we built and why we measure what we measure

Our scanner assesses websites across five dimensions. Here is the reasoning behind each.

Structure (25%)

Before an agent can do anything useful with a site, it needs to understand what it is looking at. JSON-LD structured data, semantic HTML, proper heading hierarchy, and schema markup are not just SEO niceties. They are how an agent determines whether a page describes a product, an article, a local business, or something else entirely. A site with poor structure forces the agent to guess -- and agents that guess return worse results for users.

Citability (25%)

For an agent to confidently surface your content in a response to a user, it needs to be able to attribute that content. Publication dates, authorship signals, canonical URLs, and outbound citations to primary sources all affect whether an AI system treats your content as a reliable source or a vague signal. The E-E-A-T signals that SEO practitioners have tracked for years translate directly into citability for agents.

Discoverability (15%)

Agents navigate via protocols, not curiosity. A well-formed robots.txt with Content Signal directives tells agents what they are allowed to do with your content. A sitemap helps them understand the scope of your site. Link headers (RFC 8288) expose important relationships between pages and resources. Without these, agents have to infer structure that should be declared.

Agent Interface (20%)

This is the dimension that has no analogue in traditional SEO. Does your site expose an MCP server card so agents can discover your tools? Do you publish an API catalog (RFC 9727)? Do you support OAuth server discovery (RFC 8414) so agents can authenticate programmatically? Do you serve markdown content when an agent requests it? These are the signals that separate a site an agent can interact with from one it can only read.

Agent Correlation (15%)

The final dimension is verification: are real agents actually visiting your site, and are the signals you emit being confirmed by real traffic? This dimension moves organically with usage and time. It is the one that cannot be manufactured by implementation alone -- which is why it is weighted as a confidence signal rather than a pure optimisation target.

What we found when we ran it on ourselves

Before we asked anyone else to use the tool, we ran it on covenai.io.

We scored 33 out of 100 on an independent agent readiness check -- Level 1 out of 5. Our own scanner put us at 64.7. That was humbling and clarifying in equal measure. If we were going to publish a methodology for this, we had to be willing to live by it.

We spent the following weeks implementing what we were measuring. OAuth server discovery at /.well-known/oauth-authorization-server. An MCP server card at /.well-known/mcp.json. An API catalog. Markdown content negotiation. Structured data covering our organisation, our methodology, and our content. Citability signals throughout. Content Signal directives in robots.txt for 16 AI crawlers.

After all of that, the independent check moved to 92 out of 100 -- Level 5, "Agent-Native". Our own scanner moved to 83 out of 100, grade: good. The remaining gap is agent correlation -- real agent traffic confirming the signals we emit. That one moves with time and genuine use. We are working on it.

What this means for web operators

The question we are asked most often in testing is: does this matter right now, for a business that is not in the AI space?

Our answer is: it depends on how quickly you want to act before the default behaviour of AI agents is set.

The analogy to SEO in 2004 is imperfect but instructive. In 2004, most businesses had websites. Most of those websites were not optimised for search engines. Over the following five years, search engine optimisation went from a niche technical practice to a core marketing function -- and the companies that understood it early captured positions that were difficult for later movers to dislodge.

The agent traffic data suggests we are at a similar inflection point. Agents are already visiting your site. They are already shaping recommendations and decisions on behalf of the users who sent them. Whether your site is legible, citable, and interoperable with those agents will increasingly determine whether you appear in agent-mediated experiences at all.

The good news is that the implementation work, at its core, is not new. Structured data, semantic HTML, clear authorship, well-maintained sitemaps -- these are things that good web practice has recommended for years. The agent-native layer on top of that (MCP, OAuth discovery, RFC 9727) is newer, but the standards are converging fast, not fragmenting.

Where to start

We built our scanner to give any site operator a clear picture of where they stand and what to fix first. It covers all five dimensions, produces a grade, and shows you exactly which signals are missing.

You can run a free scan at covenai.io/scan. No account required.

The methodology is published openly at aa.covenai.io/methodology. If you disagree with how we have weighted something, or think we are missing a dimension entirely, write to us at [email protected]. We are building this in public because the standard should not belong to any one company.

Sources

Want to see how your site scores for AI agent readiness?

Run a Free Scan