# Search, Training, and User Fetch: Three Different Access Decisions

> Search crawlers, training crawlers, and user-triggered fetchers perform different jobs. Learn how to make three separate access decisions without misreading robots.txt.

- Canonical: https://www.aixindar.com/news/search-training-and-user-fetch-three-different-access-decisions
- Markdown: https://www.aixindar.com/news/search-training-and-user-fetch-three-different-access-decisions.md
- Author: Daoyu Guan — https://www.aixindar.com/experts/daoyu-guan
- Published: 2026-09-07T10:56:36.548Z
- Last updated: 2026-09-07T10:56:36.620Z
- Evidence checked: Not separately recorded in CMS
- Editorial status: Published
- Corrections: No correction record supplied by CMS.

*An AI company may send several agents to the same website, but those agents do not necessarily perform the same job. A useful access policy begins by deciding which job the publisher intends to allow.*

Search discovery, model training, and user-initiated page retrieval are separate uses of web content. Current documentation from OpenAI, Anthropic, and Perplexity assigns these uses to different user agents and, in some cases, different robots.txt behavior. A single rule for “AI bots” can therefore produce a policy the publisher never intended: a site may block search discovery while permitting training, or allow automated search while a firewall rejects the user fetch needed to answer a live question. The practical solution is a three-decision access matrix, backed by official identity data, delivery-layer rules, and dated verification rather than a copied list of bot names.

A publisher opens a crawler report and sees four unfamiliar names. The security team wants to block them all. The content team wants every page visible in AI answers. Legal wants assurance that public articles will not enter model-training datasets. Each request sounds like a yes-or-no decision about “AI access.” It is actually a set of decisions about different activities.

Collapsing those activities is costly because the controls are not interchangeable. Allowing a search crawler does not necessarily authorize a training crawler. Blocking a training crawler does not necessarily remove a page from search. A fetch initiated by a user may use another agent and may be treated differently from continuous crawling. The exact behavior is product-specific, so the policy must name the product, the agent, the purpose, and the page scope.

## One website request can represent three different jobs

An automatic search crawler discovers and refreshes pages so a product can surface or cite them in search results. A training crawler collects material that may be used to improve a foundation model. A user-triggered fetcher visits a URL because a person has asked the product to inspect, summarize, or use that page at that moment.

These jobs can produce similar-looking server log entries. All may request HTML over HTTPS. All may identify themselves in a User-Agent header. That superficial similarity does not make their purposes or controls equivalent.

OpenAI's current crawler documentation is unusually explicit. It describes OAI-SearchBot as the agent for search, GPTBot as the crawler for content that may be used in training generative AI foundation models, and ChatGPT-User as an agent used for certain user actions. OpenAI says the OAI-SearchBot and GPTBot settings are independent. It also says ChatGPT-User is not the agent that determines Search inclusion. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

Anthropic documents a comparable functional split among Claude-SearchBot, ClaudeBot, and Claude-User. Claude-SearchBot supports search result quality, ClaudeBot concerns potential model-training collection, and Claude-User supports retrieval following a Claude user's request. Anthropic says these bots honor robots.txt directives, including a supported non-standard Crawl-delay extension. [Anthropic's crawler guidance](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

Perplexity distinguishes PerplexityBot, which is intended to surface and link websites in search results, from Perplexity-User, which supports a user's question. Its documentation says PerplexityBot is not used to crawl content for foundation-model training, while Perplexity-User is neither a general web crawler nor a training collector. It also states that Perplexity-User generally ignores robots.txt because the request was initiated by a user. [Perplexity's crawler documentation](https://docs.perplexity.ai/guides/bots)

Those descriptions establish documented roles for named products as of the review date. They do not prove that every vendor uses the same taxonomy or that a published user-agent label reveals every downstream data practice. The defensible conclusion is narrower: publishers should stop treating all automated access as one policy object.

## A current role map for three major services

The table below translates official product descriptions into policy questions. It is a review aid, not a permanent registry; agent names, IP ranges, and vendor behavior can change.

| Provider and documented agent | Primary documented role | Question the publisher should answer |
| --- | --- | --- |
| OpenAI OAI-SearchBot | Automatic search discovery for ChatGPT search features | Do we want eligible public pages available for this search experience? |
| OpenAI GPTBot | Crawl that may supply content for foundation-model training | Do we permit this potential training use on the specified paths? |
| OpenAI ChatGPT-User | Fetch associated with certain user actions | Should a user be able to ask ChatGPT to retrieve this public page? |
| Anthropic Claude-SearchBot | Search indexing and result-quality support | Do we want these pages discoverable for Claude search? |
| Anthropic ClaudeBot | Collection that may contribute to model training | Do we allow future material on these paths to be collected for that purpose? |
| Anthropic Claude-User | Retrieval following a Claude user's request | Should user-directed retrieval reach this content? |
| PerplexityBot | Search discovery and linking, not foundation-model training | Do we want the page surfaced and linked in Perplexity search? |
| Perplexity-User | Retrieval to answer a user's question | Should a user-directed request be able to reach the page? |

The right-hand column matters more than the list of names. It converts a technical rule into an accountable content decision. The answer may differ by path. A public help center might permit all search agents and user fetchers while a licensed research archive permits neither. A company may allow search discovery for product documentation and decline a separate training use. That is a coherent policy when the controls are implemented and verified separately.

The table also exposes a missing category. Perplexity says its search crawler is not a training crawler, but it does not list a training crawler on the cited page. A publisher should record that fact as “no training agent documented on this page,” not turn absence from one document into a universal claim about all data sources or contractual uses.

## Robots.txt is a scoped protocol, not a universal consent ledger

The Robots Exclusion Protocol lets a service owner publish rules that crawlers can retrieve and apply to paths. RFC 9309 defines how groups, user-agent lines, allow rules, and disallow rules are parsed. It also makes an essential security point: these rules are instructions to crawlers, not access authorization. A blocked URL can still be requested by a client that does not follow the protocol, and the file itself is publicly visible. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)

This distinction has two practical consequences. First, robots.txt should not protect confidential material. Authentication and authorization must do that work. Second, a robots.txt decision is effective only for an agent that reads and honors the applicable rule. Product documentation must therefore be part of the implementation.

OpenAI states that robots.txt rules may not apply to ChatGPT-User because those actions are user-initiated. Perplexity says Perplexity-User generally ignores the file for the same reason. Anthropic, by contrast, says its documented bots honor robots.txt, and describes Claude-User as controllable through those rules. These are materially different contracts. Copying a rule from one provider to another without reading its documentation can reverse the intended result.

Google's crawler documentation offers another useful model of purpose-specific control. It separates common crawlers used for products such as Search from special-case crawlers and user-triggered fetchers. Google-Extended is described as a standalone product token that publishers can use to manage whether crawled content may help improve Gemini Apps and Vertex AI generative APIs; it does not affect Google Search inclusion or ranking. [Google's common crawler reference](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) The broader taxonomy shows why the product role, rather than the word “bot,” should drive the rule. [Google's crawler overview](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)

## Make the search-access decision on its own merits

Search access answers a discovery question: may this product automatically crawl the public material needed to consider it for search results or supporting links?

For OpenAI, disallowing OAI-SearchBot removes the site from appearing in ChatGPT search answers, although OpenAI notes that navigational links may still appear. Allowing it is a route to search discovery, not a promise of inclusion or citation. OpenAI recommends both permitting the agent in robots.txt and allowing requests from its published IP ranges. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

For Perplexity, the documented equivalent is PerplexityBot. Its page recommends allowing the agent in robots.txt and permitting the published IP ranges. It also warns that a web application firewall may need an explicit rule. [Perplexity's crawler documentation](https://docs.perplexity.ai/guides/bots)

A publisher making this decision should define the public corpus. It may include product pages, support articles, company policies, and original research while excluding account areas, internal search results, filter combinations, and other low-value URL spaces. The purpose is not to maximize the number of crawlable URLs. It is to expose the stable, intended sources while reducing redundant or operationally expensive paths.

Then separate eligibility from selection. Search crawl permission can make retrieval possible. It cannot establish that the page will be indexed, understood, selected for a question, or cited. Content quality, source competition, query context, and product-specific serving decisions remain outside what a robots.txt rule proves.

## Make the training decision explicitly

Training access is a data-use choice. It should not be inferred from a marketing team's desire to appear in search, and it should not be delegated accidentally to a broad wildcard rule.

OpenAI's example makes the separation concrete: a webmaster can allow OAI-SearchBot while disallowing GPTBot. The company also notes that when both are allowed, it may use one crawl for both purposes to avoid duplicate fetching. That operational optimization does not merge the settings; OpenAI says the controls remain independent. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

Anthropic describes ClaudeBot as collecting web content that could potentially contribute to model training. Restricting that bot signals that future material should be excluded from its training datasets. Anthropic also warns that blocking only its crawler IPs may fail as a durable opt-out because doing so can prevent the bot from reading robots.txt. Its stated mechanism is the applicable robots.txt rule. [Anthropic's crawler guidance](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

The policy owner should document more than “training: no.” Record which vendor and agent the decision covers, the paths covered, the date and version of the reviewed documentation, and any separate contractual or platform setting. Robots.txt expresses a crawler instruction; it does not retroactively describe historic datasets, third-party archives, licensed material, or copies held elsewhere.

The same precision is useful when the answer is yes. A permissive rule does not prove that content was collected, accepted into a dataset, or used to train a released model. The log can establish a request. Vendor documentation can establish the stated purpose. Neither alone establishes the complete downstream lifecycle of a specific page.

## Treat user-triggered fetching as a product capability

User fetch is easiest to misunderstand because it can resemble a crawler request while behaving more like a browser action. A person asks an assistant to open a product manual, compare a pricing page, summarize an article, or follow a URL. The service then retrieves the page in response to that request.

The publisher's question is practical: should that public page be retrievable when a user explicitly asks for it? A support site may say yes because live retrieval reduces outdated answers. A private workspace must rely on authentication, not on a crawler directive. A metered publication may expose previews while enforcing entitlements at the application layer.

Product behavior differs. OpenAI says robots.txt rules may not apply to ChatGPT-User and directs Search opt-outs to OAI-SearchBot. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots honor robots.txt and describes blocking Claude-User as a way to prevent user-query retrieval. These statements should be implemented as vendor-specific rules, not averaged into a single assumption.

Even when user-triggered access is intended, the delivery stack may block it. A WAF can reject a valid agent because its IP list is stale, a JavaScript challenge cannot be completed by the fetcher, or a rate limit groups the service with abusive automation. The result is a product capability failure despite a permissive policy file.

## Build the access matrix in five steps

The following method produces a reviewable policy and a test plan. It does not claim to predict search citations.

1. **Inventory information by path and sensitivity.** Group URLs by what they contain and how they are meant to be used. Public technical pages, licensed reports, account data, search-result pages, and administrative routes should not inherit one undifferentiated rule.

2. **Name the product purpose before the agent.** For every target service, decide separately on automatic search discovery, potential training collection, and user-directed retrieval. If a provider does not document one category, record it as unknown or not documented rather than guessing.

3. **Map the decision to the current official control.** Capture the documented user agent, the applicable robots.txt behavior, and the official identity source such as a published IP file or verification procedure. Store the retrieval date because these references change.

4. **Align every delivery layer.** Check robots.txt, CDN bot controls, WAF custom rules, rate limits, authentication, redirects, and origin behavior. The intended result must survive the complete request path. A broad firewall allow rule should require the vendor's documented identity evidence, not a User-Agent string alone.

5. **Verify each allowed and denied case.** Test representative URLs, inspect edge and origin logs, and confirm the effective HTTP response and content. Preserve both positive and negative cases: the search crawler can reach a public article, the training crawler receives the intended exclusion, and protected content still requires authorization.

The completed decision table can be compact:

| Content group | Search discovery | Training crawl | User-triggered fetch | Enforcement and owner |
| --- | --- | --- | --- | --- |
| Public product documentation | Allow named services | Policy decision by provider | Allow where supported | robots.txt plus verified edge rules; web platform owner |
| Public newsroom | Allow named services | Policy decision by provider | Allow where supported | robots.txt, rate limits, canonical URLs; editorial operations |
| Customer portal | Do not expose as public search content | Disallow | Require user authorization | application authentication; product security |
| Internal search and filters | Usually restrict low-value URL spaces | Usually restrict | Permit only through intended UI | robots.txt plus routing controls; web engineering |

This is a template, not a universal recommendation. A publisher should replace every cell with its own approved decision.

## Coordinate robots.txt with the edge

Robots.txt describes the requested crawl policy, while the edge decides whether a request reaches the origin. Both need to agree.

Perplexity publishes separate IP endpoints for PerplexityBot and Perplexity-User and tells WAF users to combine User-Agent checks with source IP checks. OpenAI also publishes IP ranges for its named agents. Those ranges can change, so hard-coded copies require an update process. [Perplexity's crawler documentation](https://docs.perplexity.ai/guides/bots) [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

A User-Agent header by itself is not identity proof because a requester can copy the string. At the same time, an IP address without the expected agent and request behavior may be too broad for a narrowly scoped allow rule. Use the provider's documented verification method, limit the rule to the intended hosts and paths, and keep ordinary authorization in place.

Avoid making the policy file unreadable to the very crawler that needs it. RFC 9309 specifies how a crawler should handle successful, redirecting, unavailable, and unreachable robots.txt responses. Anthropic's warning about IP-only blocking illustrates the operational issue: if the agent cannot retrieve the file, the publisher cannot assume the desired rule was read. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) [Anthropic's crawler guidance](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

## Measure policy execution without overclaiming

A crawler audit should report evidence at three levels. Configuration evidence records what the files and controls say. Request evidence records what identified agents requested and what the stack returned. Product evidence records what happened in the named search or user experience.

Keep those levels distinct. A robots.txt entry proves that a rule was published at the time it was fetched. A server log proves that a request with certain observed properties reached a layer of the stack. A successful fetch proves that a response was delivered. None of these alone proves index inclusion, training use, answer citation, or commercial impact.

Likewise, a missing request is ambiguous. The service may not have attempted the URL, the log source may be incomplete, the request may have stopped at another layer, or the agent may use a documented path the team did not monitor. State the visibility of the evidence before drawing a conclusion.

A useful quarterly review compares the policy matrix with current vendor documentation, current IP data, representative edge tests, and a small set of relevant product observations. Update changed identifiers, remove obsolete rules, and investigate differences between intended and observed behavior. The goal is a policy that remains understandable as products evolve.

## Questions publishers should be able to answer

**If we block a training crawler, will we disappear from AI search?** Not necessarily. OpenAI explicitly documents independent search and training controls. Check the named product and agent rather than assuming one provider's implementation applies everywhere.

**Does allowing a crawler guarantee citations?** No. It enables a documented access path. Indexing, selection, citation, and recommendation are separate outcomes.

**Can robots.txt protect confidential information?** No. RFC 9309 describes crawler rules, not access authorization. Use authentication and application security for restricted material.

**Should we allow every request carrying a recognized bot name?** No. User-Agent strings can be imitated. Use the provider's current identity guidance and narrowly scoped delivery rules.

**What belongs in the policy record?** The purpose decision, provider, agent, page scope, official source, review date, enforcement layers, owner, and verification evidence.

---

*Sources checked on September 7, 2026. This article interprets current public documentation from OpenAI, Anthropic, Perplexity, Google, and the IETF. It proposes an access-governance method; it does not report a crawler experiment, a legal conclusion, or a guaranteed effect on search visibility or model training.*

## Editorial references

- [Editorial policy](https://www.aixindar.com/editorial-policy)
- [Research methodology](https://www.aixindar.com/research-methodology)
- [Corrections policy](https://www.aixindar.com/corrections)