# Allowed in Robots, Blocked at the Edge: Why AI Crawlers Still Cannot Reach Your Site

> A permissive robots.txt file cannot override a CDN challenge, WAF rule, or rate limit. Learn how to trace crawler failures across the delivery path.

- Canonical: https://www.aixindar.com/news/allowed-in-robots-blocked-at-the-edge-why-ai-crawlers-still-cannot-reach-your-site
- Markdown: https://www.aixindar.com/news/allowed-in-robots-blocked-at-the-edge-why-ai-crawlers-still-cannot-reach-your-site.md
- Author: Daoyu Guan — https://www.aixindar.com/experts/daoyu-guan
- Published: 2026-09-07T10:57:16.863Z
- Last updated: 2026-09-07T10:57:16.929Z
- Evidence checked: Not separately recorded in CMS
- Editorial status: Published
- Corrections: No correction record supplied by CMS.

*A permissive robots.txt file can express the right policy while the network delivers the wrong result. Diagnosing that mismatch requires following the request through every layer that can answer it.*

Robots.txt governs crawler behavior only after a crawler can retrieve and interpret the file. It cannot override DNS failure, TLS rejection, a CDN bot rule, a WAF challenge, a rate limit, authentication, or an unhealthy origin. This is why an AI search crawler can be explicitly allowed yet receive a 403, 429, 503, challenge page, redirect loop, or empty rendered shell. The reliable method is to verify the requester's identity using the provider's current guidance, trace one representative URL from edge to origin, locate the first layer that changes the response, and preserve the evidence. An allow rule should be narrow, testable, and consistent with the site's security model.

The incident often begins with two facts that seem incompatible. The content team reads a robots.txt file that permits the named crawler. The platform dashboard reports that the page cannot be retrieved. One team concludes that the search product is ignoring the rule. Another concludes that the dashboard is wrong.

Both conclusions skip the delivery path. The crawler does not teleport from robots.txt to the article. Its request passes through name resolution, network routing, TLS termination, a CDN, bot classification, firewall rules, rate limiting, application routing, and the origin. Any of those layers can respond before the content system sees the request.

The useful question is therefore not “Is the bot allowed?” It is “Which component returned what response to a verified request, for which URL, at which time?”

## Robots permission and network access are different control planes

The Robots Exclusion Protocol is a set of instructions that a crawler retrieves from `/robots.txt`. RFC 9309 defines how matching user-agent groups and path rules work. It does not create network access, authenticate a requester, or replace application authorization. The RFC explicitly treats robots rules as crawler instructions rather than a security control. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)

A crawler needs network access before it can read those instructions. If the CDN blocks the request for `/robots.txt`, the published policy may never reach the agent. RFC 9309 also defines distinct handling for successful retrieval, redirects, server unavailability, and network unreachability. That behavior is more nuanced than the folk rule that “missing robots means allowed.” A correct diagnosis must record the actual response state.

The same separation appears in vendor guidance. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. Those are two actions because the policy layer and the delivery layer solve different problems. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

Perplexity gives WAF configuration guidance in addition to its robots.txt recommendation. It publishes IP sources for PerplexityBot and Perplexity-User and says those endpoints should be the source of truth for WAF updates. A permissive text file is therefore only one part of the provider's documented access path. [Perplexity's crawler documentation](https://docs.perplexity.ai/guides/bots)

## Trace the whole request path

A practical model is a sequence of gates. Each gate can pass the request, modify it, redirect it, challenge it, rate-limit it, or terminate it.

| Layer | Typical failure | Evidence that locates the layer |
| --- | --- | --- |
| DNS and routing | Resolution failure, unreachable network, wrong endpoint | DNS records, resolver result, connection telemetry |
| TLS and hostname handling | Certificate or protocol failure, wrong virtual host | TLS handshake data, SNI and certificate logs |
| CDN and bot management | Block, managed challenge, cached denial | Edge event ID, bot category, rule and action |
| WAF and rate limiting | 403, 429, JavaScript challenge, request throttling | Matching rule ID, source identity, rate-limit counter |
| Application gateway | Authentication redirect, host/path rejection | Gateway access log and upstream selection |
| Origin application | 5xx, empty body, crawler-specific template | Origin log, application trace, response body |
| Rendering and content | HTML shell without useful text, blocked dependencies | Retrieved HTML, resource waterfall, rendered output |

The first layer that returns a terminal response owns the immediate failure. That does not mean its rule is wrong. A WAF may be correctly blocking an unverified client that merely copied a crawler name. The repair must preserve the purpose of the control while allowing the intended, verified traffic.

This model also explains why an origin log can show nothing. If the CDN responds with a challenge, the request never reaches the origin. Searching application logs more aggressively will not reveal it. Conversely, an edge log showing a 200 may describe a cached response that differs from what the origin would generate. The response body and headers matter along with the status.

## Read the response code as a clue, not a verdict

HTTP semantics narrow the investigation, but the same status can be generated by several layers. RFC 9110 defines 403 as a server understanding the request and refusing to fulfill it. It defines 503 as temporary inability to handle the request, commonly due to overload or maintenance, and permits Retry-After to communicate an expected delay. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html)

Google's crawler guidance is useful because it describes how a major crawling infrastructure treats broad response classes. Google says 2xx responses permit content consideration, redirects are followed subject to limits, 4xx responses other than 429 generally cause the content to be ignored, and 5xx or 429 responses slow crawling. Persistent server errors can eventually affect indexed URLs. These rules describe Google crawlers, not a universal contract for every AI agent, but they show why repeated edge failures matter. [Google's HTTP status guidance](https://developers.google.com/search/docs/crawling-indexing/http-network-errors)

Use the status to choose the next evidence source:

| Observed result | Likely classes of cause | Next verification |
| --- | --- | --- |
| DNS, connect, or TLS failure | Network, certificate, protocol, hostname | Resolve and connect from an external environment; inspect TLS and edge telemetry |
| 301/302 loop or long chain | Redirect configuration, geo routing, authentication | Record each Location header and the final response |
| 403 | WAF, bot rule, access control, origin rejection | Find the request in edge and origin logs; identify the exact rule or handler |
| 429 | Rate limit or shared traffic bucket | Inspect counters, window, scope, and Retry-After behavior |
| 503/502/504 | Origin health, gateway timeout, overload, maintenance | Correlate edge status with upstream status and origin health |
| 200 with challenge HTML | Soft block or bot challenge | Compare body, content type, page title, and challenge markers |
| 200 with empty or partial page | client rendering, resource blocking, variant response | Compare raw HTML, rendered text, and dependent resource access |

Do not label every 403 a robots problem or every 200 a successful content fetch. The body may contain “Access denied,” and a redirect may lead to a login page. Effective access means the verified client received the intended public representation.

## Verify the crawler before creating an exception

A User-Agent string is a claim supplied by the requester. It can be copied. An allow rule that trusts only a familiar name can let impersonators bypass a control.

Google documents a two-part DNS method for verifying its crawlers: perform a reverse DNS lookup on the source IP, confirm the resulting hostname belongs to an approved Google domain, then perform a forward lookup and confirm that the original IP appears. Google also publishes IP range files for relevant crawler categories. [Google's crawler verification guidance](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot)

OpenAI publishes JSON IP-range endpoints for OAI-SearchBot, GPTBot, and ChatGPT-User on its crawler page. Cloudflare's verified-bot system uses validation methods that can include IP validation and, for some bots, web-bot authentication. Cloudflare also warns that an IP range should be owned exclusively by the bot owner to serve as appropriate identity evidence. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) [Cloudflare's verified-bot documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/)

The implementation principle is straightforward: use the provider's current verification method, and bind the exception to the minimum hostnames, paths, methods, and actions needed. Preserve ordinary authentication. A verified crawler identity does not grant entitlement to private content.

IP lists are operational data, not text to copy into a ticket and forget. Retrieve them over an authenticated official endpoint, validate the format, update rules safely, monitor failures, and retain the previous known-good set for rollback. The provider may add or remove ranges. A stale allowlist can recreate the outage months later.

## Find the first blocking layer in five steps

The following runbook turns a vague crawler complaint into a bounded incident investigation.

1. **Freeze one failing example.** Record the exact URL, hostname, time window, claimed user agent, source IP if available, observed status, response headers, and response body. Avoid starting with aggregate traffic because several failure modes can be mixed together.

2. **Verify the requester's identity.** Apply the provider's current official method. If identity cannot be established, treat the request as unverified and continue investigating without creating a privileged allow rule. Save the source document and review date.

3. **Walk backward from the response.** Search the CDN or edge event store first, then the WAF, gateway, and origin. At each point record whether the request appeared, which rule matched, the action, and any upstream status. The first layer that produces the terminal response defines the immediate remediation owner.

4. **Create the narrowest correct change.** If the crawler is intended and verified, adjust the specific bot category, custom rule, rate-limit bucket, or path condition responsible. Keep authentication and unrelated abuse controls intact. Document the policy reason and an expiry or review date.

5. **Retest the complete representation.** Confirm `/robots.txt` and the target URL, follow redirects, inspect headers and body, and check that useful public content is present. Test a disallowed or protected URL too, so the change proves both access and containment.

The result should include evidence before and after the change. A statement such as “whitelisted bot” is too weak for later review because it does not identify the verified identity, matched rule, scope, or actual response.

## Common edge configurations that create false contradictions

Managed bot products can classify a request independently of robots.txt. Cloudflare's Verified Bots category contains bots judged to perform a recognized service, while AI Crawl Control offers visibility and controls specifically for AI crawler traffic. A custom WAF rule can still take precedence or apply another action depending on the ruleset and configuration. [Cloudflare's verified-bot documentation](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) [Cloudflare AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)

A broad “challenge automated traffic” rule is a frequent source of failure. Cloudflare's own example for challenging bad bots combines a bot-score condition with an exclusion for known bots. The exact fields depend on the product and plan, but the architectural lesson is stable: the exclusion and the blocking condition must be evaluated together. [Cloudflare's WAF bot-challenge example](https://developers.cloudflare.com/waf/custom-rules/use-cases/challenge-bad-bots/)

Other common mismatches include a security rule deployed to the apex domain but not the `www` host, separate policies for HTML and asset subdomains, a country block that captures the provider's egress region, and a rate limit that groups search crawlers with user-triggered fetches. Redirects can also cross into a host with a different policy. Testing only the original URL misses the final destination.

Caching adds another complication. An edge can cache a challenge or error, so a corrected origin continues to appear unavailable. The investigation should identify cache status and vary only what is necessary to test the effective policy. Purging content without understanding the matched security rule can hide the symptom briefly while leaving the cause intact.

## Do not forget robots.txt itself

The policy file is a production endpoint. It can fail, redirect, be cached incorrectly, vary by hostname, or contain a content-type and encoding problem. A site may serve one file to a browser and another through a CDN rule.

RFC 9309 expects the file at the top-level `/robots.txt` path for the relevant service. Rules are scoped to the protocol, host, and port where the file is served. If a business publishes content across `example.com`, `www.example.com`, `docs.example.com`, and a separate asset host, each relevant origin needs intentional handling. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)

Test the file as the verified crawler would reach it. Record redirect hops, status, content, and cache headers. Then parse the applicable group using the crawler's documented token. A wildcard group may be overridden by a more specific matching group under the protocol's group-selection rules. Visual inspection can miss that interaction.

Changes also take time to propagate through caches and crawler refresh cycles. OpenAI says its search systems may take roughly 24 hours to adjust after a robots.txt change. That statement applies to OpenAI's search process; it should not be generalized into a universal propagation window. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots)

## Rate limits need a purpose-aware design

Rate limiting protects availability, but a single anonymous bucket can punish legitimate crawlers for unrelated abuse. The right response is not unlimited access. It is a rate policy that recognizes verified service traffic, protects expensive routes, and communicates temporary limits predictably.

Separate inexpensive public HTML from costly endpoints such as internal search, exports, and personalized rendering. Apply tighter controls to the expensive paths regardless of crawler identity. Where a provider supplies distinct identities for automatic search and user-triggered requests, consider whether they need different burst behavior. A live user fetch may be latency-sensitive, while an automatic crawler can often retry later.

When temporary overload is the cause, return a response that accurately describes it. RFC 9110 allows Retry-After with 503 and defines 429 separately as Too Many Requests. A silent connection drop or challenge page can be harder for both the crawler and the operator to interpret. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html)

Measure the result at the service boundary: successful public-page responses, error rate by verified agent and path, response time, and origin load. Do not optimize solely for crawler request volume. More requests can mean redundant URL discovery, redirect loops, or a misconfigured calendar space rather than better search visibility.

## Build an evidence pack that another team can reproduce

The minimum useful incident record contains the URL, timestamp with timezone, source identity evidence, claimed user agent, edge request ID, matched rule, edge response, upstream response, relevant response headers, and a safely stored body sample or hash. Add the configuration change and the retest result.

This record prevents three recurring disputes. Security can see that the exception is tied to verified traffic. Engineering can see which layer generated the result. Editorial can see whether the intended content was actually delivered. None has to infer the answer from a screenshot of robots.txt.

The record also limits claims. It may establish that a verified crawler received the correct page after a rule change. It does not establish that the page was indexed, cited, or preferred. Those outcomes require product-specific observations after the delivery issue is resolved.

A healthy monitoring rule should alert on persistent changes in response class, sudden disappearance of verified traffic, repeated challenge bodies, or failure to retrieve robots.txt. Keep thresholds high enough to avoid treating every transient retry as an incident. The purpose is to detect policy drift and availability failures that survive beyond normal network variation.

## Questions to use during an incident review

**Can robots.txt override a WAF block?** No. The crawler must pass the network and edge controls to retrieve both robots.txt and the target content.

**Does a 200 response prove the crawl succeeded?** No. Inspect redirects, content type, body, and useful rendered content. A challenge or empty shell can arrive with a 200 status.

**Should we allowlist by User-Agent?** A User-Agent alone is not reliable identity evidence. Use the provider's official verification method and narrow the exception.

**Why is there no origin log entry?** The request may have been answered or dropped at DNS, network, CDN, or WAF layers before the origin.

**What does a successful repair prove?** It proves the tested, verified request received the intended response under the recorded conditions. It does not prove indexing or citation.

---

*Sources checked on September 7, 2026. This article combines public specifications and official crawler and edge documentation with an author-proposed diagnostic runbook. It does not report access to a customer's infrastructure, disclose a platform ranking rule, or claim that crawler availability guarantees search inclusion.*

## Editorial references

- [Editorial policy](https://www.aixindar.com/editorial-policy)
- [Research methodology](https://www.aixindar.com/research-methodology)
- [Corrections policy](https://www.aixindar.com/corrections)