]> Independent

hello@arsentev.ai https://arsentev.ai/

large language model crawler discovery well-known uri link relation robots.txt Publishers have begun to serve a curated, plain-text summary of a web origin intended for consumption by large language models and by the crawlers that feed them, most visibly under the de facto file name "llms.txt". The practice has no specification, no media type, and — of direct operational consequence — no discovery mechanism: a consumer that does not already guess the path cannot learn that such a file exists. This document specifies discovery and retrieval for publisher-curated context files. It defines the well-known URI "llm-context", the link relation type "llm-context", and an extension record for the robots exclusion protocol, so that a publisher may advertise a context file by three independent paths and a consumer may find it without guessing. It specifies a two-tier arrangement of an index resource and optional detail resources, states conditional-request and size requirements that keep retrieval affordable for both parties, and describes the relationship of this mechanism to the robots exclusion protocol, to sitemaps, and to work in progress on expressing AI usage preferences. This document also reports measurements from an operational deployment in which twenty crawlers operated by search and language-model providers issued 44,005 requests to a host over fifteen days without once retrieving the context file the host was serving, while the same crawlers retrieved that host's robots.txt 577 times in the three days after the context file was deployed. The absence of a discovery mechanism, rather than the absence of interest, is the hypothesis this document acts upon.

Introduction A web origin that wishes to be represented accurately to a large language model (LLM) faces a problem that ordinary search engine optimisation does not solve. The origin's HTML is written for a browser: it carries navigation, boilerplate, advertising, and markup whose only purpose is presentation. A consumer that ingests that HTML in order to build a representation of the origin must strip it, and the stripping is lossy and unsupervised. The publisher, who knows which pages matter and how the origin should be described, has no channel through which to say so. Beginning in 2024 a convention emerged in which publishers serve a plain-text, Markdown-formatted summary of the origin at the fixed path "/llms.txt", frequently accompanied by a larger concatenation at "/llms-full.txt". The convention is widely deployed and widely discussed. It is also entirely unspecified: there is no media type, no statement of what a conforming consumer is required to do, no size discipline, and no way for a consumer to learn that the file exists other than to request a path it has guessed. Guessing is the weak point. The web already has two standardised mechanisms by which an origin tells an automated consumer what it may and should retrieve — the robots exclusion protocol and the Sitemap protocol — and automated consumers use both routinely. A third convention that is not reachable through either of them is invisible to any consumer that has not been specifically programmed for it. reports what that invisibility looks like in an operational deployment. In summary: on a host serving a context file, crawlers operated by search and language-model providers issued 13,917 requests over the three days following deployment, of which 577 were for robots.txt and 249 for sitemap.xml, and none at all were for the context file. This document does not attempt to standardise the content of a context file, which is a matter of editorial judgement and is likely to remain so. It specifies the three things that must be common for the mechanism to work at all:

• how a publisher advertises that a context file exists ();
• how a consumer retrieves it without imposing unbounded cost on either party ();
• how the mechanism relates to the robots exclusion protocol, to sitemaps, and to the expression of AI usage preferences ().

The registrations required to make the mechanism usable are requested in .

Scope and Non-Goals This document is Informational. It describes a mechanism and requests the IANA registrations that the mechanism needs. It does not claim consensus of the IETF. The following are explicitly out of scope:

• Editorial content. What a publisher should say about its origin is not specified. This document constrains the container, not the prose.
• Permission. A context file is an offer of material, not a grant of rights over it. Whether a consumer may use retrieved content for training, for retrieval-augmented generation, or for any other purpose is governed by mechanisms outside this document; see .
• Model behaviour. Nothing here constrains what a model does with retrieved text, and explains why a consumer must not assume the text is trustworthy.
• Ranking. Serving a context file confers no entitlement to be retrieved, cited, or ranked.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

context file

A resource, served by an origin, whose content is a publisher-curated description of that origin intended for consumption by a large language model or by software acting on behalf of one.

index resource

The context file that a discovery mechanism points to. It is expected to be small and to link onward. See .

detail resource

A larger resource linked from the index resource, containing full text rather than description.

consumer

Software that retrieves a context file. This includes crawlers that build a corpus offline and agents that fetch on demand in response to a user's request; distinguishes them where the distinction matters.

publisher

The party that controls what an origin serves.

origin

As defined in .

Observed Behaviour of an Undiscoverable Convention The design in this document is a response to a measurement rather than to an intuition. This section reports the measurement, its method, and its limitations. It is presented here, before the specification, because the specification is difficult to justify without it.

Method A single publisher operates two hosts, "arsentev.ai" and "ru.arsentev.ai", serving the same content in two languages. Both hosts began serving a context file at "/llms.txt" and a detail resource at "/llms-full.txt" on 2026-09-09. Both files are generated from the same source as the HTML pages and are rebuilt on deployment. The access logs of the origin server were parsed for the period 2026-08-28 to 2026-09-11 inclusive, a window of fifteen days containing 331,758 logged requests. Requests were attributed to a crawler when the request's User-Agent header contained one of twenty-two tokens published by search and language-model operators as identifying their crawlers. Attribution by User-Agent is self-asserted and unverified; returns to this. The sizes of the files as served, in octets, were: Context file sizes as served, 2026-09-11

Host	/llms.txt	/llms-full.txt
arsentev.ai	29,146	367,587
ru.arsentev.ai	38,982	614,939

Results Over the full fifteen-day window, twenty distinct crawler tokens were observed, accounting for 44,005 requests. The number of those requests that retrieved "/llms.txt" or "/llms-full.txt" was zero. Restricting attention to the three days after the context files were deployed, 2026-09-09 to 2026-09-11, so that the crawlers had something to retrieve: Crawler requests, 2026-09-09 to 2026-09-11

Quantity	Value
Requests attributed to crawlers	13,917
Distinct crawler tokens	18
Retrievals of /robots.txt	577
Distinct crawlers retrieving /robots.txt	12
Retrievals of /sitemap.xml	249
Retrievals of /llms.txt or /llms-full.txt	0

The comparison that matters is internal to the table. The same crawlers, in the same window, on the same host, retrieved the two resources that a standard tells them to retrieve — robots.txt, defined by , and sitemap.xml — several hundred times, and retrieved the resource that no standard mentions zero times. The crawlers were present, were active, and were willing to fetch metadata about the origin. What they had no way to learn was that a third piece of metadata was on offer.

Limitations The following bound what may be concluded. They are stated in full because the measurement is a single observation and the temptation to over-read it is real.

• Three days. The post-deployment window is short. A crawler that probes unfamiliar paths on a long cycle would not appear in it. The fifteen-day figure of zero retrievals is consistent with the three-day figure but covers a period during which, for most of its length, there was nothing at the path to retrieve.
• One publisher. Two hosts under one operator, with one content generator, in two languages. The hosts are modest in traffic and authority, and crawler behaviour is known to vary with both.
• Self-asserted identity. Crawlers were identified by User-Agent token and not by reverse DNS or by published address range. A consumer that retrieved the context file while presenting a browser User-Agent would not have been counted as a crawler. In the full window, 40 of the 56 requests to the context files came from a command-line HTTP client operated by the publisher during verification, and the remainder from ordinary browsers; none presented a crawler token.
• Absence of evidence. Zero retrievals is consistent with the hypothesis of this document — that consumers cannot find what they are not told about — and also with other hypotheses, including deliberate disregard of an unspecified convention. The measurement distinguishes these poorly. It establishes that the convention is not working on this deployment, not why.

The design response to this uncertainty is to make the mechanism cheap for a consumer to ignore: every discovery path defined in is additive, none changes the meaning of an existing record, and a consumer that implements none of them sees the origin exactly as it does today.

The Context File

Format A context file MUST be a UTF-8 encoded text document. It SHOULD be formatted as Markdown, and a consumer MUST NOT fail to process a context file that is not valid Markdown; the format is a convenience for human maintainers and a hint for consumers, not a validation gate. A context file MUST NOT be served with a "Content-Type" of "text/html". Publishers that generate the file through an HTML template routinely misconfigure this, and a consumer that receives HTML cannot distinguish a context file from an error page. A context file SHOULD be served as "text/markdown;charset=UTF-8" or as "text/plain;charset=UTF-8". The registration of a distinct media type is discussed in and is deliberately not requested by this document.

Index and Detail Resources Deployed practice separates a short summary from a long concatenation of full text, and the separation is load-bearing. shows the two differing by a factor of twelve on one host and sixteen on the other; the detail resource on one host exceeds 600 kilobytes, and it grows with the site without bound. Accordingly:

• A discovery mechanism defined in MUST point at an index resource, never directly at a detail resource.
• An index resource SHOULD NOT exceed 100,000 octets. A publisher whose description does not fit SHOULD move material into detail resources rather than grow the index.
• An index resource MAY link to any number of detail resources, using ordinary Markdown or plain-text URI references. Relative references are resolved against the URI of the index resource as specified in .
• A detail resource SHOULD be divided when it exceeds 1,000,000 octets, so that a consumer may retrieve part of the material without retrieving all of it.

The size guidance is a floor for interoperability, not a recommendation to fill it. A consumer MAY impose its own limits and MUST fail safely — by treating the origin as having no context file — rather than truncate mid-document in a way that changes meaning.

Language and Variants A publisher serving an origin in several languages SHOULD serve a separate context file per origin where languages occupy separate origins, as in the deployment of , and MAY use proactive content negotiation on a single origin. Where negotiation is used, the response MUST carry an appropriate "Content-Language" and MUST include "Accept-Language" in "Vary".

Discovery Three mechanisms are defined. They are independent: a publisher MAY deploy any subset, and a consumer MAY implement any subset. A publisher SHOULD deploy at least the well-known URI of , because it is the only one of the three that a consumer can use without first retrieving another resource. Where more than one mechanism is present and they disagree, a consumer MUST resolve the conflict as specified in .

The "llm-context" Well-Known URI This document registers the well-known URI suffix "llm-context" (), following . A publisher advertising a context file through this mechanism MUST arrange that a GET request for "/.well-known/llm-context" on the origin returns either:

• the index resource itself, with a 2xx status and a "Content-Type" consistent with ; or
• a redirect (301, 302, 307, or 308) to the index resource.

A consumer SHOULD follow at most five redirects and MUST treat a redirect chain that leaves the origin as requiring the same scrutiny as any other cross-origin reference; see . A 404 response MUST be interpreted as "this origin does not advertise a context file by this mechanism" and MUST NOT be interpreted as an error to be retried aggressively. A consumer SHOULD NOT re-probe a 404-returning origin more often than once per 24 hours. The well-known path is preferred over the deployed convention of a root-level "/llms.txt" for the reason gives generally: the root namespace belongs to the publisher, and a specification that claims a root path claims something it does not own. A publisher that already serves "/llms.txt" can satisfy this document with a single redirect and need not move the file.

The "llm-context" Link Relation This document registers the link relation type "llm-context" (). A publisher MAY advertise a context file by including a "Link" header field in responses from the origin: ; rel="llm-context"; type="text/markdown" ]]> or, in an HTML document, by a link element in the document head: ]]> A publisher using the header form SHOULD emit it on responses for the origin's home document and SHOULD NOT emit it on every response, since the field costs bytes on each response and carries no per-response information. The "type" parameter, when present, is a hint. A consumer MUST rely on the "Content-Type" of the retrieved representation and not on the hint. This mechanism exists because it is the only one of the three that works for a consumer which has already retrieved a page and has no reason to make a second request speculatively. An agent fetching a single document on a user's behalf () is in exactly that position.

The "LLM-Context" Record in robots.txt A publisher MAY advertise a context file by including a record in the origin's robots.txt: The record's name is "LLM-Context", compared case-insensitively as requires of record names. Its value MUST be an absolute URI. A relative reference MUST be ignored rather than resolved, so that a parser error cannot cause a consumer to request an unintended path. Section 2.2.4 of permits a crawler to interpret records that are not part of the robots exclusion protocol, giving "Sitemaps" as its example, and requires that the parsing of such records "MUST NOT interfere with the parsing of explicitly defined records". This record is therefore safe to deploy: a crawler that has never heard of it parses the rest of the file exactly as before. That property is the reason for defining it despite the well-known URI already existing — robots.txt is retrieved by consumers today, as shows, and a publisher can therefore reach them through it without waiting for new code to be deployed on the consumer side. The record is nevertheless the weakest of the three mechanisms, because it depends on parsers being as forgiving in practice as the specification requires. A record appearing outside any group applies to the origin as a whole. A publisher SHOULD place it outside all groups. A consumer encountering the record inside a group MUST treat it as applying to the origin as a whole, matching the handling Section 2.2.4 of gives to the "Sitemaps" record, which "MUST NOT terminate a group". A publisher MUST NOT use this record to advertise a context file whose retrieval the same robots.txt disallows. A consumer encountering that contradiction MUST honour the exclusion and MUST NOT retrieve the file: a permission mechanism and a discovery mechanism disagreeing is resolved in favour of the permission mechanism, always.

Conflict Resolution and Precedence Where several mechanisms advertise different URIs, a consumer MUST apply the following precedence, highest first:

1. the "Link" relation from a response the consumer has already received, because it is the most specific to what the consumer is currently doing;
2. the well-known URI;
3. the robots.txt record.

A consumer MUST NOT retrieve more than one index resource per origin per retrieval cycle merely because several mechanisms are present. All three mechanisms are subordinate to the robots exclusion protocol as a permission mechanism: a consumer MUST evaluate the exclusion rules of against the index resource's URI before retrieving it, regardless of which mechanism advertised it.

Retrieval

Conditional Requests and Caching A publisher SHOULD serve context files with a strong validator — an "ETag", or a "Last-Modified" where an entity tag is not available — as specified in . A consumer that has previously retrieved a context file MUST issue a conditional request on subsequent retrievals, using "If-None-Match" where it holds an entity tag and "If-Modified-Since" otherwise. This requirement is not a nicety. A detail resource of the size reported in , retrieved unconditionally by eighteen crawlers on the cadence at which those crawlers retrieved robots.txt in , would transfer more data per day than the rest of the origin combined. A mechanism that is expensive to support will not be supported. A publisher SHOULD set an explicit "Cache-Control" "max-age" on context files. A consumer SHOULD NOT retrieve an index resource more often than once per hour, or a detail resource more often than once per day, in the absence of an explicit shorter freshness lifetime.

Compression A consumer SHOULD offer content coding in "Accept-Encoding", and a publisher SHOULD serve context files compressed where the consumer offers it. Context files are highly compressible prose and the saving is substantial at the sizes involved.

On-Demand Consumers Two kinds of consumer are in view and they have opposite cost profiles. A crawler retrieves on its own schedule, in bulk, and can afford a detail resource. An agent acting on a user's immediate request retrieves one document, once, under a latency budget measured in seconds, and cannot. The asymmetry is not merely one of politeness. In an agent that retransmits its accumulated context to a model on every step, material admitted once is paid for on every subsequent step of the run, so the cost of retrieving a 600-kilobyte detail resource is not the transfer but its persistence in the context window; specifies an interchange format for accounting of exactly that effect. Accordingly an on-demand consumer SHOULD retrieve only the index resource, and SHOULD NOT retrieve a detail resource unless the user's request cannot be served from the index. A publisher SHOULD therefore write the index resource so that it is useful alone, rather than as a table of contents that is meaningless without its targets.

Relationship to Other Mechanisms

Robots Exclusion Protocol answers the question "may this consumer retrieve this URI". This document answers the question "which URI should this consumer retrieve". The questions are disjoint and this document does not alter the answer to the first. A context file is subject to exclusion like any other resource, and and both resolve contradictions in favour of exclusion.

Sitemaps A sitemap enumerates URIs and carries no judgement about them: it is complete by intention. A context file is curated and selective by intention. A publisher SHOULD serve both, and SHOULD NOT generate one from the other mechanically, as a context file that is a transcription of a sitemap communicates nothing that the sitemap did not.

Expression of AI Usage Preferences Work is in progress in the IETF on vocabularies for expressing preferences about the use of content by AI systems, and on attaching those preferences to content . That work and this document are orthogonal and complementary: the former concerns what a consumer is permitted to do with material it has obtained, the latter concerns how a publisher offers material in the first place. The two interact in one direction only, and the direction matters. Serving a context file MUST NOT be construed as granting any usage preference, relaxing any preference expressed by other means, or waiving any right. A consumer MUST evaluate usage preferences independently of discovery. A publisher that both restricts usage and serves a context file is making a coherent statement — "if you are permitted to use this origin at all, use this description of it" — and a consumer MUST NOT treat the second half as withdrawing the first.

Structured Data in HTML Structured data embedded in pages describes entities in machine terms. A context file describes an origin in prose to a consumer whose native input is prose. They are not substitutes, and a publisher SHOULD continue to serve structured data unchanged.

Security Considerations

Content Retrieved Is Untrusted Input This is the consideration that matters most, and it is a consequence of the mechanism's purpose rather than of any detail of its design. A context file is written by a publisher for the express purpose of being placed in the input of a language model. The publisher chooses every word. A file may therefore contain text crafted to be read by the model as instruction rather than as content — directions to disregard prior instructions, to represent the origin in terms the publisher prefers, to disparage a competitor, or to emit a particular URI in an answer. Nothing in this document, and nothing that could be added to it, prevents this: the file is by definition attacker-chosen text from the point of view of any consumer that does not trust the publisher. Therefore a consumer MUST treat the content of a context file as untrusted data and MUST NOT treat any part of it as instruction. A consumer that incorporates a context file into a model's input MUST mark its provenance in a way that survives into the model's input, so that content originating from an origin is distinguishable from the consumer's own instructions. Consumers SHOULD apply whatever isolation they apply to other retrieved web content; a context file deserves no more trust than an arbitrary HTML page from the same origin, and its curated presentation may make it appear to deserve more. A consumer MUST NOT follow URIs found in a context file automatically without applying the same policy it applies to other discovered URIs, including exclusion rules and cross-origin scrutiny.

Cross-Origin References A discovery mechanism that names a URI on another origin allows one party to make assertions about a resource it does not control, and allows an origin to direct consumers elsewhere. A consumer MUST NOT attribute the content of a cross-origin index resource to the advertising origin. A consumer SHOULD decline to retrieve a cross-origin index resource at all unless it has a specific reason to accept the indirection, and MUST evaluate the target origin's own exclusion rules before retrieving it.

Differential Serving A publisher can serve a context file that does not correspond to what a browser is shown, which is the long-standing practice known as cloaking, applied to a new surface. This document cannot prevent it. A consumer that relies on a context file SHOULD sample the origin's ordinary representations and SHOULD reduce its reliance on a context file that systematically disagrees with them.

Resource Consumption The size limits of and the conditional request requirements of exist partly for safety. A publisher can otherwise advertise an arbitrarily large resource to every consumer that implements this mechanism, and a consumer that retrieves unconditionally can otherwise impose unbounded cost on a publisher. A consumer MUST impose its own ceiling on the size of any retrieved context file and abandon a retrieval that exceeds it.

Privacy Considerations A request for a well-known URI is as identifying as any other request from the same client and carries no additional information about a user. A publisher can, however, use a distinct path per consumer to determine which consumers implement this mechanism; this is the ordinary observability of an HTTP request and is noted here only because the well-known path is otherwise uniform. A publisher MUST NOT place in a context file personal data that it would not place in a public HTML page. The file is world-readable, and its purpose is to encourage copying of its contents into corpora from which withdrawal is not practical.

IANA Considerations

Well-Known URI Registration IANA is requested to register the following entry in the "Well-Known URIs" registry established by :

URI suffix:

llm-context

Change controller:

IETF

Reference:

This document

Status:

permanent

Related information:

Returns, or redirects to, a publisher-curated context file describing the origin for consumption by large language models.

Link Relation Type Registration IANA is requested to register the following entry in the "Link Relation Types" registry established by :

Relation Name:

llm-context

Description:

Refers to a publisher-curated context file describing the origin of the link context, intended for consumption by large language models.

Reference:

This document

Robots Exclusion Protocol Record does not establish an IANA registry of record names, so no registration is requested. The record name "LLM-Context" is documented in of this document. Should a registry be established, this document requests that "LLM-Context" be entered in it with this document as reference.

Media Type No media type registration is requested. A context file is Markdown or plain text and the registrations of and are adequate. A distinct media type would allow a consumer to recognise a context file retrieved without discovery, but at the cost of a registration whose only distinguishing property is the intent of its author. The trade-off is noted here so that it can be revisited if operational experience favours the other side.

Implementation Status This section records the status of known implementation in accordance with . It is to be removed before publication as an RFC, should that occur. Publisher-side, arsentev.ai and ru.arsentev.ai. Two production hosts serve an index resource at "/llms.txt" and a detail resource at "/llms-full.txt", generated from the same source as the HTML pages, since 2026-09-09. Sizes are given in . Coverage of this document at the time of writing is partial: the two-tier arrangement of is implemented and the index resource is within the size guidance on both hosts; the detail resource on ru.arsentev.ai is within the division threshold but approaching it; the discovery mechanisms of are not yet deployed, since this document defines them. The operator is the author, and this is therefore a report of one deployment rather than of independent interoperation. Consumer-side. No consumer implementation of the discovery mechanisms defined here is known. reports that eighteen crawlers observed over the measurement window implemented none of them, which is expected, since they did not exist. Interoperation. None to report. The mechanism has not been exercised between independent implementations, and the author invites reports from any publisher or consumer willing to attempt it.

Open Questions The following are known to be unresolved and are listed to invite comment rather than to promise resolution.

• Authenticity. Nothing binds a context file to the publisher beyond the origin's own transport security. Whether a signature mechanism is warranted, and what it would defend against given , is open.
• Freshness signalling. Deployed context files are generated at deployment time and can silently diverge from the pages they describe. Whether the format should carry a generation timestamp, and whether a consumer should act on staleness, is open.
• Per-consumer variants. Whether a publisher should be able to offer different context files to different classes of consumer, and whether that is useful curation or an invitation to the cloaking of , is open.
• Measurement. The observation of is one publisher over three days. A multi-publisher measurement, ideally one in which discovery is deployed on some origins and withheld on others, would test this document's central hypothesis properly. The author intends to conduct one and will report it in a revision.

References Normative References Informative References IETF AI Preferences Working Group IETF AI Preferences Working Group

Acknowledgements The convention that this document attempts to make discoverable was proposed and popularised by others, and this document takes no credit for it. The measurement of was made on the author's own hosts, with the limitations that implies and which states.