Agent-Friendly Websites: What Developers Should Know
A handful of new acronyms — llms.txt, MCP, agent-readable markup — have entered developer conversations over the past two years, with wildly different levels of actual evidence behind them. Here's a grounded look at what's proven, what's promising but unproven, and what's mostly hype.
Agent-friendliness is mostly good engineering practice with a new name, plus a few genuinely new standards still finding their footing.
Most of It Is Good Engineering With a New Name
The most effective steps for agent-friendliness in 2026 are largely the same as long-standing good web engineering practice — semantic HTML, content that doesn't require JavaScript execution to appear, clean documented APIs — while a couple of genuinely new, agent-specific standards (llms.txt, MCP) remain unevenly proven. Developers chasing agent compatibility get the most value from fundamentals, not exotic new file formats.
Where the Evidence Actually Stands
| Practice | Evidence Level | Notes |
|---|---|---|
| Server-rendered / crawlable HTML | Strong | Long-established that content requiring client-side JS is a common blind spot for crawlers and simpler agents. |
| Semantic HTML & accessible markup | Strong | Directly overlaps with web accessibility standards with decades of established benefit for assistive technology and machine parsers alike. |
| Documented APIs / MCP servers | Strong for capability, still early for adoption | MCP, introduced by Anthropic in late 2024, gives agents a structured way to interact beyond visual page parsing — genuinely useful where implemented, but not yet universal. |
| llms.txt | Weak / unproven | Google's John Mueller has stated no major AI system is confirmed to use it; studies found no measurable citation uplift. |
| robots.txt directives for AI crawlers | Moderate | Major AI companies generally respect it, but it's a voluntary standard with uneven enforcement across the ecosystem. |
A Real Standard Worth Understanding
The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI systems a consistent way to connect to external tools, data sources, and services. For a website or platform with meaningful functionality — booking, search, data lookup — exposing that functionality through an MCP server gives an AI agent a structured, documented way to interact with it, rather than forcing the agent to interpret a visual interface built for human eyes. This tends to be dramatically more reliable than screen-based interaction for well-defined tasks, since there's no ambiguity about what a given action does.
MCP adoption has grown quickly since its introduction but remains far from universal — most of the web is still interacted with by agents the same way a human browser would, reading rendered pages and simulating clicks, rather than through a dedicated protocol.
Where to Actually Start
1. Fix crawlability first
Ensure core content and functionality don't require JavaScript execution to appear — the highest-impact, lowest-effort change for most sites.
2. Clean up markup
Semantic HTML and proper form labels serve accessibility and agent-readability at the same time.
3. Document any API you already have
Clear API documentation is useful to human developers and dramatically more reliable for agents than screen-based interaction.
4. Consider llms.txt as low-cost, low-certainty
Cheap to add, but treat it as forward-looking infrastructure rather than an expected ranking or citation lever.
Frequently Asked Questions
Is llms.txt worth implementing?
The evidence for direct citation benefit is weak, though it's low-cost to add and may help agentic tools navigate a site's structure.
What is an MCP server?
An implementation of the Model Context Protocol, an open standard from Anthropic that lets AI systems connect to external tools and data in a structured way.
Should developers build a separate API for agents?
For sites with meaningful functionality, a documented API is far more reliable for agent interaction than a visual interface built for humans.
Do robots.txt rules apply to AI agents?
Major AI companies generally respect robots.txt for their crawlers, but it's a voluntary standard with uneven enforcement.
What's the highest-impact change developers can make?
Ensuring core content works without requiring JavaScript execution, since many crawlers and simpler agents rely on the initial HTML response.
More AI Guides, Written Clearly
Browse our full library of AI explainers and how-tos, updated as the tools change.
More from the creator
Explore two other projects from the same creator.