Turn Web Pages into Markdown with Python

Page to Markdown is a small Python CLI and library. It calls GET /web/extract with js=false, converts the HTML to Markdown locally, and retries once when a page comes back with no readable text. Every call is logged with status, latency and the cost from X-Desearch-Cost-Usd.

It reads pages whose text is in the HTML the server sends, such as articles, docs and reference pages. Pages that build their content in the browser come back with little or no text in this mode. The tool marks those empty and writes no file for them.

The problem

Saving a few articles as clean text means opening each URL, skipping the menu and footer, and keeping the source link. Doing that by hand does not scale, and a file without the URL is hard to cite later.

What the app does

• Takes URLs from the command line or from a text file, one URL per line.
• Calls Extract with format=html by default, so headings, lists, links and code blocks can be kept.
• Converts the HTML locally and resolves relative links against the page URL.
• Retries once when a 200 response has no readable text.
• Writes one Markdown file per page that has text, plus summary.md and a call log.

How it works

1. Skip bad URLs locally. Blank or malformed URLs never become a call.
2. Extract. GET /web/extract with js=false and a custom User-Agent.
3. Convert. HTML becomes Markdown. Navigation, footers, forms and scripts are dropped.
4. Retry once after 2 seconds when the first 200 is empty.
5. Write the Markdown, the summary and redacted raw responses.

The code

def page_to_markdown(client, url, tag="page", retry=True):
    """Fetch one URL. Retries once (after 2 s) when a 200 comes back empty. Returns a result dict."""
    if not core.valid_url(url):
        return {"url": url, "status": "skipped", "attempts": 0, "title": "", "markdown": "", "links": [], "words": 0,
                "empty": False, "empty_first": False, "latency_s": 0.0, "latency_each": [], "cost_usd": 0.0,
                "error": "not an http(s) URL, skipped locally (no call made)", "fetched": now()}
    attempts, lats, cost = [], [], 0.0
    for i in range(2 if retry else 1):
        st, body, rec = client.extract(url, f"{tag}-a{i + 1}")
        title, md, links = to_markdown(body, client.fmt, url) if st == 200 else ("", "", [])
        attempts.append(rec); lats.append(rec["latency_s"]); cost += rec["cost_usd"]
        empty = st == 200 and core.is_empty(md)
        if not (empty and retry and i == 0):
            break
        time.sleep(2)
    return {"url": url, "status": st, "attempts": len(attempts), "title": title, "markdown": md, "links": links,
            "words": core.word_count(md), "empty": st == 200 and core.is_empty(md),
            "empty_first": attempts[0]["status"] == 200 and len(attempts) > 1,
            "latency_s": round(sum(lats), 2), "latency_each": lats, "cost_usd": round(cost, 6),
            "error": None if st == 200 else body[:300], "fetched": attempts[-1]["ts"]}

The folder has page2md.py, core.py, offline tests, a sample URL list and sample output.

Endpoints

• GET /web/extract

Example output

The sample sent 10 URLs and made 11 calls. The header total was $0.0045. Pages with readable text became Markdown files with a short front matter block: source, title, time, status, attempts and word count. One page came back empty after a retry and was not written.

Run it yourself

The public examples folder linked above includes the script and a README. Install its requirements, set DESEARCH_API_KEY from console.desearch.ai, and use the run command in that README.

You can also import page_to_markdown and pass a client that does not write to disk.

What it costs

In the sample, a page with text cost $0.0005 per call. The empty page was retried, so that URL cost $0.0010. Two error responses cost $0.0000. See pricing.

Ideas to extend it

• Feed the Markdown into a corpus builder.
• Skip domains you already have on disk.
• Write a CSV of word count and cost next to the files.

{
  "mcpServers": {
    "desearch": {
      "url": "https://mcp.desearch.ai/mcp",
      "headers": {
        "Authorization": "Bearer <DESEARCH_API_KEY>"
      }
    }
  }
}

Use Desearch Extract to read the URLs I list with JavaScript rendering off. Convert the HTML to Markdown, keep the source URL, and skip pages that return no readable text. Do not invent page text.