How to Create Machine-Readable Policies for AI Agents

What Does "Machine-Readable Policy" Mean?

A machine-readable policy is a rule about your site stated in a format software can parse, such as a robots.txt directive, an HTTP header, a meta tag or schema.org markup, alongside a plain-language page for people. It lets an agent find out what it may do without guessing from legal prose.

Two cautions belong at the start. First, most of these signals are requests, not locks. A crawler that respects them will behave, and one that does not will not. Second, the standards are unfinished. The IETF has a working group on AI preferences, and vendors have introduced their own conventions in the meantime. You are publishing clear statements in the best available formats, not enforcing law.

Four Kinds of Rules Worth Publishing

Each one has different formats and different owners inside a business. Marketing may own the return policy, legal owns the terms and engineering owns robots.txt. Getting them to agree on one statement is often harder than the technical part.

Telling Bots How Content May Be Used

Start with the file every crawler already looks for. Robots.txt controls which paths a named bot may fetch. Well-known AI bots publish user agents you can address separately. OpenAI documents GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-triggered access, and Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Google uses the Google-Extended token to let sites control whether crawled content is used for certain Gemini purposes, and Google says that token does not affect ranking in Search.

Content signals add a description of use on top of access. A file using them can look like this:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: GPTBot Disallow: / ```

The Content-Signal line says content may be used for search and as input to AI answers but not for training. It is a preference statement supported by some crawlers and ignored by others. Related tools include the X-Robots-Tag HTTP header and robots meta tags for page-level rules, and Google documents nosnippet, max-snippet and data-nosnippet for controlling how much of your text appears in Search features, including AI answers. The Text and Data Mining Reservation Protocol offers another way to express rights reservations, and a non-standard "noai" meta tag circulates but has no recognized authority. For load management, see AI crawler rate limiting explained.

Return, Shipping and Pricing Rules in Structured Data

Commerce policies are where machine readability pays off fastest, because agents comparing products care about returns and delivery. Schema.org defines MerchantReturnPolicy for returns and OfferShippingDetails for shipping, and Google documents support for both in Product markup. A compact return policy in JSON-LD looks like this:

{
  "@context": "https://schema.org",
  "@type": "MerchantReturnPolicy",
  "applicableCountry": "US",
  "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
  "merchantReturnDays": 30,
  "returnMethod": "https://schema.org/ReturnByMail",
  "returnFees": "https://schema.org/FreeReturn"
}

Attach it to your organization or product offers as Google's documentation describes, and keep the numbers identical to your visible policy page. If the markup says thirty days and the page says fourteen, a careful agent will notice and trust neither. The same discipline applies to price and availability in product feeds.

Saying What Automated Clients May Do

There is no universal file for "agents may log in but not check out." Until one exists, combine three things.

1. A plain-language agent policy page. State whether automated access is permitted, which actions are allowed, what rate limits apply and how to contact you. Link to it from your footer and terms.
2. Technical enforcement. Authentication, scoped tokens, rate limits and confirmation steps do the actual controlling. Read AI agent permissions for websites for the design.
3. Identity signals. Where agents sign their requests or publish IP ranges, allow the verified ones more freedom. The article on how autonomous systems prove who they are explains the direction of travel.

A policy that only exists as text cannot stop a misbehaving client, and enforcement with no explanation frustrates well-behaved ones. Do both.

Optional Extras

An llms.txt file is a proposed Markdown file at your site root that points language models to your most useful pages, with short descriptions. It can be a handy index, and it is cheap to generate. But as of this writing, no major AI search provider has committed to reading it, so do not expect a visibility boost. If you publish one, keep it short, accurate and in sync with your site. Treat it as documentation, not policy. A /.well-known/ path is a common convention for machine-readable site information, such as security.txt, and could host future agent-related files, but no widely adopted standard exists for AI policies there yet.

A Step-by-Step Approach

1. Inventory what you already say. Collect crawling rules, return and shipping terms, terms of service and privacy statements, and find contradictions.
2. Decide your positions. For each bot category, choose allow, limit or block. For each commerce rule, confirm the exact numbers.
3. Write plain-language pages first. Short, dated, with headings a reader can scan. Include the effective date and a contact.
4. Add structured versions. Robots.txt and content signals for crawling, JSON-LD for returns and shipping, headers or meta tags for page-level control.
5. Keep them in sync. Generate structured data from the same source as the visible text. A single source of truth prevents drift.
6. Publish a stable location. Use permanent URLs and link them from the footer and sitemap.
7. Test and monitor. Validate robots.txt in Search Console's tools, test structured data with Google's Rich Results Test and check logs to see which bots respect your rules.
8. Review quarterly. Vendors add and rename bots, and the standards are moving.

A Policy Set for a Small Online Store

Take a store that sells handmade candles and ships within one country. Its policy set could be small and still clear.

For crawling, the robots.txt file allows search bots everywhere, blocks a training crawler from the whole site because the owner prefers not to contribute product photos to training sets, and disallows the cart and account paths for all bots. A content-signal line states that search and AI input are welcome and training is not.

For commerce, the returns page says thirty days, unused items, return shipping paid by the customer, and refunds within ten days of receipt. The same page carries MerchantReturnPolicy JSON-LD with matching values, and the product pages include shipping details markup with the flat delivery fee and typical handling time.

For agents, a short "Automated access" page explains that browsing and product lookup are fine, that account creation and purchases require a signed-in human to confirm the final step, and that requests above a modest rate will receive a 429 response. A contact address is included.

Nothing here required special software, and the whole set could be reviewed in an hour. It would answer most of the questions an agent would otherwise have to guess at.

Mistakes to Avoid

• Treating a signal as a lock. Do not protect paid or private content with robots.txt alone.
• Contradictory statements. A page that says one thing, markup that says another and a PDF that says a third undermine trust.
• Blocking search bots by accident. A broad Disallow aimed at training crawlers can catch bots that build AI search indexes.
• Undated policies. Without a date, neither a person nor an agent can tell if a rule is current.
• Overpromising in markup. Do not claim free returns in schema if they apply only to certain products.

Review the result the way an outsider would. Fetch the robots.txt file, open the policy pages without logging in and read the structured data with a validator. If a stranger, or an agent, could not tell what you permit within a few minutes, simplify until they can.

Frequently Asked Questions