Study date: 13 October 2026. Sample: Tranco top 10,000 domains, n=6,480 valid robots.txt files. Method: robots.txt parse per RFC 9309.
Key findings
- 18.6% of top sites block at least one AI training crawler site-wide (1,203 of 6,480 valid robots.txt files, Tranco top 10,000, October 2026).
- 11.0% of top sites block at least one AI search crawler site-wide (716 of 6,480).
- Only 2.4% of top sites block the main search engine crawler site-wide (157 of 6,480), the control for every AI share.
- 35.7% of the top 100 sites with a valid robots.txt block at least one AI training crawler site-wide (56 valid files), compared with 17.8% for sites ranked 1,001 to 10,000.
- 9.7% of top sites block an operator's training crawler while leaving the same operator's search crawler unblocked (631 of 6,480).
- 13.2% of top sites name at least one user-triggered AI fetcher in robots.txt (856 of 6,480).
- 22.8% of top sites name at least one AI crawler in robots.txt (1,475 of 6,480).
- 64.8% of the Tranco top 10,000 served a valid robots.txt (6,480 of 10,000).
Data: CSV download.
How we measured it
On 13 October 2026 we fetched /robots.txt for the top 10,000 domains of the Tranco standard list (list ID 64VKX, downloaded 2026-10-13). We used the Desearch Extract endpoint (GET /web/extract) and fell back to a direct HTTP fetch when Extract returned nothing or an HTML page. 6,480 domains returned a valid robots.txt; that is the denominator for every share on this page.
We parsed each file under RFC 9309 (Robots Exclusion Protocol). A crawler is "blocked site-wide" when the path / resolves to disallowed for it: the group that names it is used if one exists, otherwise the User-agent: * group; the longest matching rule wins, and Allow wins a tie. "Partial" means / is allowed but at least one Disallow path applies. We report three groups of AI crawlers: training crawlers (five user-agents that collect pages for model training, including usage-control tokens), search crawlers (three user-agents that index pages for AI search answers) and user-triggered fetchers (three user-agents that fetch a page when a user asks). The main search engine crawler is the control and comparison point: its 2.4% site-wide block rate (157 of 6,480) is the baseline for every AI share. Group-level data is in the CSV: one row per site with its result for each of the three groups. A group counts as blocked when at least one user-agent in the group is blocked site-wide (training, search); for user-triggered fetchers the CSV shows whether at least one is named in the file.
| Fetch outcome (n=10,000) | Count |
|---|---|
| Valid robots.txt | 6,480 |
| Unreachable (no HTTPS host or DNS) | 2,080 |
| No file (4xx) | 772 |
| HTML page instead of robots.txt (Extract) | 462 |
| HTML or non-robots body (HTTP check) | 109 |
| Redirect not resolved | 54 |
| Server error (5xx) | 31 |
| Other 2xx (204/202) | 12 |
How many top sites block AI crawlers in robots.txt
As of 13 October 2026, 18.6% of the Tranco top 10,000 sites with a valid robots.txt (1,203 of 6,480) block at least one AI training crawler site-wide, and 11.0% (716) block at least one AI search crawler. By comparison, 2.4% (157) block the main search engine crawler. Method: robots.txt parse per RFC 9309, files fetched the same day.
Blocking rises with popularity:
| Metric (share of valid files in band) | Rank 1-100 | Rank 101-1,000 | Rank 1,001-10,000 |
|---|---|---|---|
| Valid files | 56 of 100 | 512 of 900 | 5,912 of 9,000 |
| Training crawlers: any blocked site-wide | 35.7% | 25.8% | 17.8% |
| Search crawlers: any blocked site-wide | 32.1% | 16.6% | 10.4% |
| User-triggered fetchers: any named in file | 16.1% | 21.7% | 12.4% |
| Training blocked, same operator's search not | 8.9% | 11.1% | 9.6% |
| Any AI crawler named in file | 37.5% | 31.1% | 21.9% |
| Control: main search engine crawler blocked site-wide | 5.4% | 3.9% | 2.3% |
Note: the top 100 band has only 56 valid files, so its shares move by 1.8 points per site.
Are AI training crawlers blocked more often than search crawlers?
On 13 October 2026, training crawlers were blocked more often than search crawlers: 18.6% of 6,480 top sites block at least one training crawler site-wide, against 11.0% for search crawlers. 9.7% (631) block an operator's training crawler but not the same operator's search crawler.
| Group (n=6,480 valid files) | Measure | Sites | Share |
|---|---|---|---|
| Training crawlers | At least one blocked site-wide | 1,203 | 18.6% |
| Search crawlers | At least one blocked site-wide | 716 | 11.0% |
| User-triggered fetchers | At least one named in file | 856 | 13.2% |
| Training blocked, same operator's search not | Site-wide | 631 | 9.7% |
| Any AI crawler | Named in file | 1,475 | 22.8% |
| Control: main search engine crawler | Blocked site-wide | 157 | 2.4% |
Some user-triggered fetchers state that robots.txt rules may not apply to them.
Do sites block AI crawlers by name or with wildcard rules?
Most site-wide blocks of AI crawlers are explicit: they come from a robots.txt group that names the crawler, not from a User-agent: * rule. The main search engine crawler is the reverse: almost all of its site-wide blocks come from wildcard rules in allowlist-style files. The CSV holds group-level results only, not per-crawler named or wildcard results.
Next: JavaScript barriers
A follow-up run will measure how much page text is readable without JavaScript on the same domain list. Results will be added here when measured.
What this means for teams building scrapers and AI agents
- Check robots.txt for your own user-agent, not just the wildcard group: most AI blocks are written by name.
- Expect stricter rules on the most popular sites: 35.7% of the top 100 sites with a valid robots.txt block at least one AI training crawler site-wide, against 17.8% for ranks 1,001 to 10,000.
- Treat search and training access separately: 9.7% of sites block a training crawler while allowing the same operator's search crawler.
- Plan for fetch failures: 35.2% of the top 10,000 domains (3,520) returned no valid robots.txt, mostly unreachable hosts.
To fetch robots.txt files and pages the way this study did, see the Extract API guide and the web crawler overview. For turning crawled pages into embeddings, see crawl and embed.
Limitations
- robots.txt is a request, not enforcement. This study measures stated policy, not crawler behaviour or firewall blocking.
- CDN or firewall rules can block a fetch with no robots.txt rule; these appear as unreachable or HTML responses, not as blocks.
- Two of the five training user-agents are usage-control tokens that do not crawl; a Disallow for them is a usage signal.
- Tranco ranks popular domains; results describe popular sites, not the whole web. Many top domains are infrastructure hosts (CDNs, DNS) with no website.
- Our fetcher's user agent and IP may receive different answers than the named crawlers would.
- Only the registered domain and www were checked, not other subdomains.
- The top 100 band has 56 valid files; treat its shares as indicative.
- Single snapshot. Monthly re-runs will use the same method version for comparisons.
Download the data and how to cite this study
- CSV (group-level data): one row per site for all 10,000 Tranco domains. Columns: rank, domain, band, training, search, user_triggered. training and search: blocked (at least one user-agent in the group blocked site-wide) or not_blocked; user_triggered: named (at least one user-agent in the group named in the file) or not_named; no_valid_robots_txt when the site served no valid robots.txt. No per-crawler columns. License: CC BY 4.0. File URL: /data/ai-crawlers-robots-txt-study.csv.
- How to cite: Desearch, "AI Crawler robots.txt Study", 13 October 2026, Tranco top 10,000 (list ID 64VKX), n = 6,480 valid robots.txt files, https://www.desearch.ai/blog/ai-crawlers-robots-txt-study
FAQ
How many websites block AI crawlers?
In our October 2026 study of the Tranco top 10,000 (robots.txt parse per RFC 9309), 18.6% of sites with a valid robots.txt (1,203 of 6,480) block at least one AI training crawler site-wide.
Do sites block AI search crawlers too?
Less often. 11.0% of sites (716 of 6,480) block at least one AI search crawler site-wide, and 9.7% (631) block an operator's training crawler while leaving the same operator's search crawler unblocked.
Do sites block AI crawlers more than the main search engine crawler?
Yes. The main search engine crawler is blocked site-wide by 2.4% of sites (157 of 6,480), against 18.6% (1,203 of 6,480) that block at least one AI training crawler.
What does "blocked site-wide" mean?
In a robots.txt parse per RFC 9309, the root path / is disallowed for that user-agent, either through a group naming it or through the User-agent: * group when no named group exists.
Does blocking in robots.txt stop AI crawlers?
robots.txt is a voluntary standard. It states a site's policy; it does not technically enforce it.