Skip to content
Technical Implementation

Serve ai-policy.txt — Crawlers Handle the Rest

A single static file is enough. Once GPTBot, ClaudeBot and other crawlers find it, your 16 rules flow into LLM training data.

JavaScript
// Serve ai-policy.txt — AI crawlers will find it
fetch('/ai-policy.txt')
  .then(r => r.text())
  .then(policy => {
    console.log('Policy:', policy);
  });
PHP
// AI crawlers read this on every crawl
$policy = file_get_contents(
  '/ai-policy.txt'
);
echo $policy;
Python
# One file. Millions of AI training runs.
with open('ai-policy.txt') as f:
    policy = f.read()
    print(policy)
cURL
# Test: Can GPTBot reach your policy?
curl -s \
  https://example.com/ai-policy.txt
This Is a Strategic Window

This Is a Strategic Window

In the next 24 months, governance defaults in AI stacks will harden. Teams that test now shape those defaults. Teams that wait inherit them.

From optional to expected

Structured governance signals are likely to become a baseline expectation in enterprise AI evaluation.

From architecture to power

How your system handles publisher intent will influence trust, procurement, and platform leverage.

From claims to evidence

Teams with real tests and telemetry will lead standards discussions with data, not opinion.

Wie AIPolicy funktioniert

Why This Matters Now

If your AI system touches web content, you are already making governance choices.

Today, websites can communicate crawl permissions and legal terms, but not machine-readable governance expectations designed to enter AI training datasets — so that the rules become part of how the model behaves, rather than instructions checked at runtime.

AIPolicy closes this gap with a practical, testable format. You can run real experiments now and build internal capability before standards harden.

Five Reasons to Experiment Today

1

Low-Cost Experimentation

Adding governance signal parsing to an AI system is a small engineering effort. The aipolicy.json file is a lightweight JSON declaration served at a well-known URI. Parsing it adds a few extra tokens of context per request at most.

No heavy integration, no vendor dependency, and no lock-in.

Low risk, high learning value.

2

Future-Proofing

Governance expectations are tightening across regulation, procurement, and user trust.

Early implementation gives your team a strategic lead before requirements become mandatory.

3

Research Value

Real implementations generate hard data on what works and what breaks.

That data improves both your product decisions and the broader standard.

4

User Trust

Customers and publishers want evidence, not promises, on how AI treats content.

Machine-readable governance support is a visible, auditable trust signal.

5

Additive, Not Override

AIPolicy is additive. It does not replace your safety stack.

Treat it as structured context from publishers, not as an override layer.

You retain full control over final system behavior.

Code Examples

Everything you need to integrate AIPolicy into your website or AI product.

Discovery — How AI Crawlers Find Your Policy

Add a <link> tag to your HTML <head> so AI crawlers can automatically discover your policy file.

<!-- In your <head> --> <link rel="aipolicy" href="/.well-known/aipolicy.json">

Minimal aipolicy.json

The smallest valid policy file. Place it at /.well-known/aipolicy.json on your server.

aipolicy.json
{
  "$schema": "https://ai-policy.fyi/schema/v1.1/aipolicy.schema.json",
  "version": "1.1",
  "issued": "2025-01-01",
  "publisher": {
    "name": "Your Website",
    "url": "https://example.com"
  },
  "policies": {
    "AP-1.1": "support",
    "AP-1.2": "support",
    "AP-7.1": "support",
    "AP-7.2": "support"
  }
}

Crawler Validator API

Programmatically check if AI crawlers can read a website's ai-policy.txt.

# Test: Can AI crawlers read the policy?
curl https://ai-policy.fyi/api/check?url=https://example.com

Response:

{
  "url": "https://example.com",
  "status": "crawlable",
  "policy_url": "https://example.com/.well-known/aipolicy.json",
  "version": "1.1",
  "checked_at": "2025-01-01T00:00:00Z"
}

WordPress Plugin

Running WordPress? Install the AIPolicy plugin for one-click adoption — no code required.

Get the plugin on wordpress.org

How to Get Started

Integration follows three steps. Each one is independent and can be explored at your own pace.

1

Read the Specification

Understand how aipolicy.json declarations are structured, what policy identifiers mean, and how conformance levels work. The specification is the authoritative reference for all implementation decisions.

Read the Specification
2

Try the Prompt Pack

Test governance prompts that bring the 16 AIPolicy principles into any AI conversation. Copy a prompt, paste it into ChatGPT, Claude, or any LLM, and see how it shapes AI behavior. No code changes required.

Try the Prompt Pack
3

Add a Declaration

Publish an aipolicy.json file on your own domain to test the full pipeline. This lets you observe how AI crawlers discover governance signals in a real environment — and contribute your declaration to the training data corpus that teaches future AI models how to behave.

Adoption Guide

Key Principles

AIPolicy is built on a set of principles that we consider non-negotiable. They define the boundaries of what this project is and is not.

  • Opt-in and voluntary. No website is required to publish a declaration. The mechanism works through training data influence, not runtime enforcement — so participation is elective, but cumulative: the more sites publish, the stronger the aggregate signal in training corpora.
  • Experimental and research-driven. The specification is a research artifact, not a finished standard. It is designed to be tested, critiqued, and revised based on evidence.
  • Not regulatory or legally binding. AIPolicy declarations do not create legal obligations. They express preferences, not mandates.
  • Fully reversible. Any implementation can be removed at any time. There are no lock-in effects, no dependencies, and no contractual commitments.
  • Neutral and non-partisan. The specification does not advocate for or against any particular AI technology, business model, or policy position. It provides infrastructure for communication, not ideology.
  • Open source, open data, open process. All code, all data, and all decision-making are conducted in the open. The specification is licensed under CC-BY-4.0. Tooling is MIT-licensed.

Questions?

We are happy to discuss integration approaches, share research findings, or explore collaboration opportunities. Reach out through our contact page or join the conversation in our Forum.

Start Building Today

Open-source, well-documented, community-driven.