infini-news API Reference

Public API and web interface for searching 1.36 billion news articles (CC-News, Aug 2016 -- Apr 2026) with sub-second full-text search powered by an infini-gram-mini FM-index. The corpus spans 117 monthly shards across 11 calendar years.

Base URL: https://infini-news.uni-graz.at (currently running from a work computer)

The curl and Python examples below use https://infini-news.uni-graz.at; replace the host if you run your own gateway.

The OpenAPI schema is served at /openapi.json for tooling. Interactive Swagger UI is disabled by default because its assets would otherwise load from a third-party CDN that the gateway's Content-Security-Policy forbids; this reference and the schema cover the API surface.

Table of Contents

1. Quick Start
2. REST API
   • GET /api/v1/info
   • POST /api/v1/count
   • POST /api/v1/find
   • POST /api/v1/get_doc
3. Web Interface
4. Full-Text Retrieval via HuggingFace
5. Typical Workflow
6. Rate Limits
7. Error Handling

Quick Start

Count how many times a phrase appears in the corpus:

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "climate change", "index": "ccnews"}'

{"count": 2345678, "latency_ms": 45.2}

REST API

All endpoints accept and return JSON (Content-Type: application/json). CORS is enabled for all origins.

Authentication: none. The API is public and keyless — no API key, token, or login is required. (The HuggingFace dataset holding the full article text is separately gated; see Full-Text Retrieval via HuggingFace.)

Versioning: every endpoint lives under the /api/v1/ prefix. A backward-incompatible change would ship under a new prefix (/api/v2/) rather than altering v1 in place.

Index and year scoping

"index": "ccnews" selects the corpus (Aug 2016 -- Apr 2026, 117 monthly shards). Behind the gateway it is one FM-index engine per year. A request may carry an optional inclusive year range — year_min, year_max — that scopes the search to those years' shards:

Each shard is searched on its own thread, so a narrow range is proportionally faster, and count/find reflect only the selected years. Results are cached (the corpus is static), so a repeat is instant. find returns shard_years — the year of each returned shard, index-aligned to segment_by_shard.

get_doc must repeat the same year_min/year_max it used for find, so the shard index s resolves to the same shard. The web UI defaults the range to the newest year (fast) and re-runs the search when you change it.

The authoritative index list is returned by GET /api/v1/info.

GET /api/v1/info

Returns corpus metadata, available indexes, and HuggingFace retrieval information.

Request:

curl https://infini-news.uni-graz.at/api/v1/info

Response:

{
  "name": "infini-news",
  "description": "1.36B-article news corpus (CC-News, Aug 2016-Apr 2026) with sub-second full-text search powered by the infini-gram-mini FM-index.",
  "article_count": "1,357,027,742",
  "date_range": "2016-08 to 2026-04",
  "available_indexes": ["ccnews"],
  "hf_repo_url": "https://huggingface.co/datasets/ruggsea/infini-news-corpus",
  "api_docs": "/openapi.json",
  "hf_retrieval": {
    "repo": "ruggsea/infini-news-corpus",
    "repo_url": "https://huggingface.co/datasets/ruggsea/infini-news-corpus",
    "viewer_url_template": "https://huggingface.co/datasets/ruggsea/infini-news-corpus/viewer/year_{year}/train?p={page}&row={offset}",
    "python_example": "...",
    "note": "Dataset access may require approval. See repo page for details."
  }
}

POST /api/v1/count

Count exact occurrences of a string in the index.

Request body:

Example -- count phrase:

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "neural network", "index": "ccnews"}'

{"count": 891234, "latency_ms": 12.5}

Example -- total tokens in index (empty query):

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "", "index": "ccnews"}'

{"count": 98000000000, "latency_ms": 0.1}

Response:

POST /api/v1/find

Find all occurrence segments across shards. Returns ranges that can be used to retrieve individual documents with get_doc.

Request body:

Example:

curl -X POST https://infini-news.uni-graz.at/api/v1/find \
  -H 'Content-Type: application/json' \
  -d '{"query": "climate change", "index": "ccnews"}'

{
  "count": 2345678,
  "segment_by_shard": [
    [0, 50000],
    [0, 48000],
    [0, 62000]
  ],
  "shard_years": ["2016", "2016", "2016"],
  "latency_ms": 125.5
}

segment_by_shard and shard_years have one entry per shard (117 for the shipped corpus; truncated to three above).

Response:

Each entry in segment_by_shard is a two-element array [start, end). The array index is the shard index s, and shard_years[s] is that shard's year. Any rank in [start, end) can be passed to get_doc to retrieve a document containing the match.

POST /api/v1/get_doc

Retrieve a document snippet by shard index and rank. Returns the text around the matched occurrence, structured metadata, and highlighted spans.

Request body:

Example:

curl -X POST https://infini-news.uni-graz.at/api/v1/get_doc \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "climate change",
    "index": "ccnews",
    "s": 0,
    "rank": 100,
    "max_ctx_len": 1000
  }'

{
  "doc_ix": 42,
  "doc_len": 15000,
  "disp_len": 1000,
  "needle_offset": 500,
  "text": "...extensive measures to combat climate change across the region...",
  "spans": [
    ["...extensive measures to combat ", null],
    ["climate change", "0"],
    [" across the region...", null]
  ],
  "metadata": {
    "url": "https://www.example.com/article/climate-2022",
    "date": "2022-01-15",
    "warc_source": "CC-NEWS-20220115...",
    "language": "eng",
    "country_tld": null,
    "title": "Climate Measures Announced",
    "author": "Jane Doe",
    "sitename": "Example News",
    "hostname": "www.example.com"
  },
  "latency_ms": 200.1
}

Response:

Spans format: Each element is [text, label]. If label is null, the text is plain context. If label is "0", the text is a highlighted match. Concatenating all text values reconstructs the full snippet.

Metadata fields (all nullable):

Note on country_tld: This is the article URL's country-code top-level domain (ccTLD), lowercased, when it has one — e.g. .de → de, .at → at. Generic TLDs (.com, .org) and anything that is not exactly two ASCII letters yield null. There is no TLD-to-country mapping: .uk stays uk, it is not converted to gb.

Note on escaping: Metadata string values (title, url, sitename, …) are returned verbatim from the source document and are not sanitized. A client that renders them as HTML must escape them — and validate url against an http/https allowlist — to avoid injection. The bundled web UI does both.

Note on decoding errors: Some documents may fail to decode at certain max_ctx_len values. If you get a 502 with "Document decoding failed", try a different max_ctx_len.

Web Interface

The web UI is served at the root URL (/). It provides all search functionality in a single page with no dependencies or login required.

Search

1. Type a query in the search field.
2. Press Enter or click Search to find documents.
3. Press Shift+Enter or click Count to count occurrences without loading documents.

Search runs across the whole corpus (all years) — there is no index to pick.

After searching, the interface shows:

• Occurrence count across all years.
• Years range selector — narrows the displayed and collected documents to a year range; the count line then also reports how many occurrences fall in that range.
• Document cards with highlighted text, metadata (title, date, language, site), and links.
• Load more documents button to fetch additional random samples.

Document Cards

Each card displays:

• Title (or URL if no title is available).
• Metadata: date, site name, language, country TLD.
• Text snippet with the search query highlighted in context.
• Source link to the original article URL.
• View on HF link that opens the HuggingFace dataset viewer at the article's exact row (its doc_ix offset within the year config).

Get the Full Articles Panel

After a search returns results, a collapsible "Get the full articles" panel opens in the count bar. It leads with a ready-to-run Python | curl snippet (pre-filled with the current query and year range) that fetches every matching article from the dataset by doc_ix offset — the Python version writes them to a CSV — plus a link to open the corpus on HuggingFace. A Copy button copies the active snippet.

Preview a sample (optional) -- expand this disclosure to sample matching articles in the browser:

1. Set Sample size (default 50) -- how many evenly-spaced samples are taken per shard.
2. Click Load preview. A progress bar tracks fetching; results are deduplicated by URL and shown in a table.
3. Optionally filter the table by Language, then Download CSV of this sample.

Full-Text Retrieval via HuggingFace

The API returns document snippets (up to 10,000 characters of context). Full article texts are stored in the HuggingFace dataset ruggsea/infini-news-corpus.

The dataset may be gated (require access approval). See the repo page for details.

Finding a Single Article

get_doc returns doc_ix — the article's row offset within its crawl year's HuggingFace config (year_{year}, where year is find's shard_years[s]). That addresses the exact row, so you fetch it directly from the datasets-server with no parquet scan:

import os
import requests

# year from find()'s shard_years[s]; offset from get_doc()'s doc_ix
year, offset = 2022, 12345
row = requests.get(
    "https://datasets-server.huggingface.co/rows",
    params={"dataset": "ruggsea/infini-news-corpus", "config": f"year_{year}",
            "split": "train", "offset": offset, "length": 1},
    headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"},  # the corpus is gated
).json()["rows"][0]["row"]
print(row["title"], "—", row["url"])
print(row["text"][:500])

Or open it directly in the dataset viewer in your browser (p is offset // 100):

https://huggingface.co/datasets/ruggsea/infini-news-corpus/viewer/year_2022/train?p=123&row=12345

Bulk Retrieval (Python)

import os
import sys
import time

import requests

API = "https://infini-news.uni-graz.at"
QUERY = "climate change"
INDEX = "ccnews"  # the whole corpus, all years

ROWS = "https://datasets-server.huggingface.co/rows"
DATASET = "ruggsea/infini-news-corpus"
HF_TOKEN = os.environ.get("HF_TOKEN")  # required: the corpus is gated
HF_HEADERS = {"Authorization": f"Bearer {HF_TOKEN}"} if HF_TOKEN else {}

# 1. Find all occurrence segments; shard_years[s] is shard s's crawl year (its HF config).
resp = requests.post(f"{API}/api/v1/find", json={"query": QUERY, "index": INDEX})
resp.raise_for_status()
find = resp.json()
print(f"Found {find['count']:,} occurrences")
years = find.get("shard_years", [])

# 2. For each match, get_doc -> doc_ix (the row offset), then fetch the row directly by
#    offset. No parquet scan. Rate limit: 120 get_doc requests/minute — back off on 429.
max_per_shard = 50
seen = set()
for s, seg in enumerate(find["segment_by_shard"]):
    if len(seg) != 2 or seg[1] <= seg[0] or s >= len(years):
        continue
    year = years[s]
    step = max(1, (seg[1] - seg[0]) // max_per_shard)
    for rank in range(seg[0], seg[1], step):
        try:
            r = requests.post(f"{API}/api/v1/get_doc", json={
                "query": QUERY, "index": INDEX, "s": s, "rank": rank, "max_ctx_len": 100,
            })
            if r.status_code == 429:
                time.sleep(int(r.headers.get("Retry-After", 5)))
                continue
            r.raise_for_status()
            offset = r.json()["doc_ix"]
        except requests.RequestException as e:
            print(f"skipping: {e}", file=sys.stderr)
            continue
        if (year, offset) in seen:
            continue
        seen.add((year, offset))
        rr = requests.get(ROWS, params={
            "dataset": DATASET, "config": f"year_{year}", "split": "train",
            "offset": offset, "length": 1,
        }, headers=HF_HEADERS)
        rr.raise_for_status()
        row = rr.json()["rows"][0]["row"]
        print(f"\n--- {row.get('url')} ---")
        print((row.get("text") or "")[:500])

print(f"\nFetched {len(seen)} unique articles")

Typical Workflow

1. Count (optional)

Check how common a phrase is before committing to a full search:

curl -s -X POST $API/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews"}' | python3 -m json.tool

2. Find

Get shard segments:

FIND=$(curl -s -X POST $API/api/v1/find \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews"}')
echo "$FIND" | python3 -m json.tool

3. Retrieve Documents

Pick a shard and rank from the segment_by_shard ranges:

curl -s -X POST $API/api/v1/get_doc \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews", "s": 0, "rank": 500, "max_ctx_len": 2000}' \
  | python3 -m json.tool

4. Get Full Article from HuggingFace

Use the url and date from the metadata to fetch the complete article text from the HuggingFace dataset. See Full-Text Retrieval via HuggingFace.

Rate Limits

Rate limiting is per-IP (using the X-Real-IP header behind nginx, falling back to the connection IP).

Exceeding the limit returns HTTP 429 with a Retry-After header.

Error Handling

All errors return a JSON object with a detail field (for HTTP errors) or an error field (for validation errors).

Example error responses:

{"detail": "Invalid index. Available: [...full list returned at runtime...]"}

{"detail": "Backend unavailable"}

{"detail": "Document decoding failed. Try a different max_ctx_len value."}

For 422 errors, the response body contains a detail array with per-field validation messages:

{
  "detail": [
    {
      "type": "string_too_short",
      "loc": ["body", "query"],
      "msg": "String should have at least 1 character",
      "input": ""
    }
  ]
}