DesearchBot

DesearchBot is the web crawler operated by Desearch. It fetches public web pages so that people and AI agents searching through Desearch can find them, and cites the source back in every answer. This page documents how to identify it, how to verify that a request really came from us, and how to control what it crawls.

Control what it crawls Verify a request

Identity

robots.txt token	DesearchBot
User-Agent	Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler)
Signature-Agent	https://www.desearch.ai
Key directory	https://www.desearch.ai/.well-known/http-message-signatures-directory
Contact	support@desearch.ai

There is no IP list to allowlist. Identify us by the signature on the request, described below.

Verifying a request

Anyone can put DesearchBot in a User-Agent header, so do not allowlist on it. Every request we send is signed with an Ed25519 key using HTTP Message Signatures (RFC 9421). To check a request:

Fetch our public keys from the key directory.
Find the key whose kid matches the keyid in Signature-Input.
Verify the signature. A request that fails is not from us.

Crawl behaviour

robots.txt	Fetched before we crawl a host and cached for up to 24 hours. DesearchBot respects robots.txt. We honor Crawl-delay for our token, falling back to the * group.
Request rate	At most 1 request per second per host, and slower if you set Crawl-delay or the host answers 429 or 503.
Revisits	A page is re-fetched when its sitemap lastmod changes.

What it never does

Bypass paywalls, logins, or metered access.
Solve or evade CAPTCHAs and bot challenges.
Submit forms or perform any other write action.
Ignore robots.txt.
Rotate residential IPs to disguise where it comes from.

Controlling what it crawls

Add any of the following to /robots.txt at the root of your host. You do not need to contact us, and you do not need an entry to allow us: with no rule for our token we follow your * group.

Block us completely

User-agent: DesearchBot
Disallow: /

Slow us down

Seconds to wait between our requests to your host.

User-agent: DesearchBot
Crawl-delay: 10

Removing a page that is already indexed

robots.txt controls crawling, not what is already stored. To drop a page, email us the URL and we remove it.

Contact

Wrong crawl rate, unexpected traffic, a page that should not be indexed, or a crawler claiming to be us that fails signature verification: support@desearch.ai. We reply within two business days.