Search arXiv Papers by Topic with an arXiv API Script in Python

Made by Desearch (desearch.ai).

arXiv Paper Search to CSV is a small Python script for analysts and researchers who want an arXiv API workflow without writing a parser. Give it a few topics in plain words and it writes one spreadsheet of arXiv papers, one row per paper, ready to sort, share and annotate.

The problem

A literature review, a market scan or a reading list starts with the same chore: searching arXiv topic by topic and copying papers out of twenty browser tabs. The same paper also shows up more than once, as its abstract page, its HTML page or an older version. What helps is one sheet with each paper once and the topic that found it.

What the app does

• Takes two or more topics in plain language.
• Runs one Desearch AI Search call per topic, limited to arXiv.
• Pulls the arXiv id out of every link and merges repeats of the same paper into one row.
• Takes the month from the arXiv id itself, so every row has a month.
• Keeps the abstract text as a short snippet when one is returned.
• Writes a CSV with arXiv id, title, authors, abstract link, month, date, snippet, the topics that found it and how many links pointed to it.

How it works

1. Search. For each topic, POST /desearch/ai/search with tools: ["arxiv"], result_type: "ONLY_LINKS", streaming: false and count: 10.
2. Merge. Read the arXiv id from each link and keep one row per id.
3. Fill in. Take the month from the id, and the abstract snippet, date and authors where the results carry them.
4. Write. Save the CSV, newest month first, and the raw responses as JSON so you can rebuild the CSV offline.

The code

The key is read in run_live from the DESEARCH_API_KEY environment variable, or from a file named by DESEARCH_KEY_FILE:

    key = os.environ.get("DESEARCH_API_KEY") or open(os.path.expanduser(os.environ["DESEARCH_KEY_FILE"])).read().strip()

The core of the search call, copied from arxiv_papers.py on main (lines 22 to 35):

    body = {"prompt": query, "tools": ["arxiv"], "streaming": False, "result_type": "ONLY_LINKS", "count": count}
    at, t0 = now(), time.time()
    try:
        r = s.post(f"{BASE}/desearch/ai/search", json=body, timeout=180)
        status, hdr = r.status_code, {k.lower(): v for k, v in r.headers.items()}
        try: data = r.json()
        except ValueError: data = {"_text": r.text[:5000]}
    except Exception as e:
        status, hdr, data = None, {}, {"_error": repr(e)}
    rec = {"label": label, "query": query, "request": body, "at": at, "seconds": round(time.time() - t0, 2),
           "status": status, "cost_usd": hdr.get("x-desearch-cost-usd"), "usage_count": hdr.get("x-desearch-usage-count"),
           "body": data}
    print(f"{at} {label:20s} {status} {rec['seconds']:6.2f}s n={len(results(rec))} cost={rec['cost_usd']}", flush=True)
    return rec

The folder has one script, arxiv_papers.py, a requirements.txt and samples/raw-sample.json, a saved run you can rebuild with no key.

View the code

Endpoints

• POST /desearch/ai/search

Example output

Saved sample from a run on 2026-10-16. This page shows that stored response. It does not run a live demo.

This run used two topics on 2026-10-16: "retrieval augmented generation evaluation" and "small language models on device inference". The 20 links returned were all arXiv papers, all on topic by title, and merged into 16 unique papers.

Papers in the sheet included "On-Device Language Models: A Comprehensive Review" (2024-09), "Evaluation of Retrieval-Augmented Generation: A Survey" (2024-05) and "Edge-First Language Model Inference: Models, Metrics, and Tradeoffs" (2025-05), with months running from 2023-09 to 2026-09.

Run it yourself

git clone https://github.com/Desearch-ai/desearch-py-examples && cd desearch-py-examples/arxiv-paper-search-csv
pip install -r requirements.txt
export DESEARCH_API_KEY=...   # from console.desearch.ai
python arxiv_papers.py --queries "retrieval augmented generation evaluation" "small language models on device inference"

Flags that matter: --queries (your topics), --count (links per topic, default 10). To try it with no key and no cost: python arxiv_papers.py --from-raw samples/raw-sample.json.

What it costs

Each topic is one call, and each call cost $0.004 on 2026-10-16, read from the X-Desearch-Cost-Usd header, so the two-topic run cost $0.008. See pricing.

Ideas to extend it

• Run it weekly on your topics and compare new arXiv ids with last week's CSV.
• Load the CSV into a notebook and group papers by month and topic.
• Add more narrow topics rather than one very broad one.