infini-news API Reference

Public API and web interface for searching 1.36 billion news articles (CC-News, Aug 2016 -- Apr 2026) with sub-second full-text search powered by an infini-gram-mini FM-index. The corpus spans 117 monthly shards across 11 calendar years.

Base URL: https://infini-news.uni-graz.at (currently running from a work computer)

The curl and Python examples below use https://infini-news.uni-graz.at; replace the host if you run your own gateway.

The OpenAPI schema is served at /openapi.json for tooling. Interactive Swagger UI is disabled by default because its assets would otherwise load from a third-party CDN that the gateway's Content-Security-Policy forbids; this reference and the schema cover the API surface.

Table of Contents

1. Quick Start
2. REST API
   • GET /api/v1/info
   • POST /api/v1/count
   • POST /api/v1/find
   • POST /api/v1/get_doc
3. Web Interface
4. Full-Text Retrieval via HuggingFace
5. Typical Workflow
6. Rate Limits
7. Error Handling

Quick Start

Count how many times a phrase appears in the corpus:

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "climate change", "index": "ccnews"}'

{"count": 2345678, "latency_ms": 45.2}

REST API

All endpoints accept and return JSON (Content-Type: application/json). CORS is enabled for all origins.

Authentication: none. The API is public and keyless — no API key, token, or login is required. (The HuggingFace dataset holding the full article text is separately gated; see Full-Text Retrieval via HuggingFace.)

Versioning: every endpoint lives under the /api/v1/ prefix. A backward-incompatible change would ship under a new prefix (/api/v2/) rather than altering v1 in place.

Index and year scoping

"index": "ccnews" selects the corpus (Aug 2016 -- Apr 2026, 117 monthly shards). Behind the gateway it is one FM-index engine per year. A request may carry an optional inclusive year range — year_min, year_max — that scopes the search to those years' shards:

Each shard is searched on its own thread, so a narrow range is proportionally faster, and count/find reflect only the selected years. Results are cached (the corpus is static), so a repeat is instant. find returns shard_years — the year of each returned shard, index-aligned to segment_by_shard.

get_doc must repeat the same year_min/year_max it used for find, so the shard index s resolves to the same shard. The web UI defaults the range to the newest year (fast) and re-runs the search when you change it.

The authoritative index list is returned by GET /api/v1/info.

GET /api/v1/info

Returns corpus metadata, available indexes, and HuggingFace retrieval information.

Request:

curl https://infini-news.uni-graz.at/api/v1/info

Response:

{
  "name": "infini-news",
  "description": "1.36B-article news corpus (CC-News, Aug 2016-Apr 2026) with sub-second full-text search powered by the infini-gram-mini FM-index.",
  "article_count": "1,357,027,742",
  "date_range": "2016-08 to 2026-04",
  "available_indexes": ["ccnews"],
  "hf_repo_url": "https://huggingface.co/datasets/ruggsea/infini-news-corpus",
  "api_docs": "/openapi.json",
  "hf_retrieval": {
    "repo": "ruggsea/infini-news-corpus",
    "repo_url": "https://huggingface.co/datasets/ruggsea/infini-news-corpus",
    "viewer_url_template": "https://huggingface.co/datasets/ruggsea/infini-news-corpus/viewer/year_{year}/train?p={page}&row={offset}",
    "python_example": "...",
    "note": "Dataset access may require approval. See repo page for details."
  }
}

POST /api/v1/count

Count exact occurrences of a string in the index.

Request body:

Example -- count phrase:

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "neural network", "index": "ccnews"}'

{"count": 891234, "latency_ms": 12.5}

Example -- total tokens in index (empty query):

curl -X POST https://infini-news.uni-graz.at/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "", "index": "ccnews"}'

{"count": 98000000000, "latency_ms": 0.1}

Response:

POST /api/v1/find

Find all occurrence segments across shards. Returns ranges that can be used to retrieve individual documents with get_doc.

Request body:

Example:

curl -X POST https://infini-news.uni-graz.at/api/v1/find \
  -H 'Content-Type: application/json' \
  -d '{"query": "climate change", "index": "ccnews"}'

{
  "count": 2345678,
  "segment_by_shard": [
    [0, 50000],
    [0, 48000],
    [0, 62000]
  ],
  "shard_years": ["2016", "2016", "2016"],
  "latency_ms": 125.5
}

segment_by_shard and shard_years have one entry per shard (117 for the shipped corpus; truncated to three above).

Response:

Each entry in segment_by_shard is a two-element array [start, end). The array index is the shard index s, and shard_years[s] is that shard's year. Any rank in [start, end) can be passed to get_doc to retrieve a document containing the match.

POST /api/v1/get_doc

Retrieve a document snippet by shard index and rank. Returns the text around the matched occurrence, structured metadata, and highlighted spans.

Request body:

Example:

curl -X POST https://infini-news.uni-graz.at/api/v1/get_doc \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "climate change",
    "index": "ccnews",
    "s": 0,
    "rank": 100,
    "max_ctx_len": 1000
  }'

{
  "doc_ix": 42,
  "doc_len": 15000,
  "disp_len": 1000,
  "needle_offset": 500,
  "text": "...extensive measures to combat climate change across the region...",
  "spans": [
    ["...extensive measures to combat ", null],
    ["climate change", "0"],
    [" across the region...", null]
  ],
  "metadata": {
    "url": "https://www.example.com/article/climate-2022",
    "date": "2022-01-15",
    "warc_source": "CC-NEWS-20220115...",
    "language": "eng",
    "country_tld": null,
    "title": "Climate Measures Announced",
    "author": "Jane Doe",
    "sitename": "Example News",
    "hostname": "www.example.com"
  },
  "latency_ms": 200.1
}

Response:

Spans format: Each element is [text, label]. If label is null, the text is plain context. If label is "0", the text is a highlighted match. Concatenating all text values reconstructs the full snippet.

Metadata fields (all nullable):

Note on country_tld: This is the article URL's country-code top-level domain (ccTLD), lowercased, when it has one — e.g. .de → de, .at → at. Generic TLDs (.com, .org) and anything that is not exactly two ASCII letters yield null. There is no TLD-to-country mapping: .uk stays uk, it is not converted to gb.

Note on escaping: Metadata string values (title, url, sitename, …) are returned verbatim from the source document and are not sanitized. A client that renders them as HTML must escape them — and validate url against an http/https allowlist — to avoid injection. The bundled web UI does both.

Note on decoding errors: Some documents may fail to decode at certain max_ctx_len values. If you get a 502 with "Document decoding failed", try a different max_ctx_len.

Web Interface

The web UI is served at the root URL (/). It provides all search functionality in a single page with no dependencies or login required.

Search

1. Type a query in the search field.
2. Press Enter or click Search to find documents.
3. Press Shift+Enter or click Count to count occurrences without loading documents.

Search runs across the whole corpus (all years) — there is no index to pick.

After searching, the interface shows:

• Occurrence count across all years.
• Years range selector — narrows the displayed and collected documents to a year range; the count line then also reports how many occurrences fall in that range.
• Document cards with highlighted text, metadata (title, date, language, site), and links.
• Load more documents button to fetch additional random samples.

Document Cards

Each card displays:

• Title (or URL if no title is available).
• Metadata: date, site name, language, country TLD.
• Text snippet with the search query highlighted in context.
• Source link to the original article URL.
• View on HF link that opens the HuggingFace dataset viewer at the article's exact row (its doc_ix offset within the year config).

Get the Full Articles Panel

After a search returns results, a collapsible "Get the full articles" panel opens in the count bar. It leads with a ready-to-run Python | curl snippet (pre-filled with the current query and year range) that fetches every matching article from the dataset by doc_ix offset — the Python version writes them to a CSV — plus a link to open the corpus on HuggingFace. A Copy button copies the active snippet.

Preview a sample (optional) -- expand this disclosure to sample matching articles in the browser:

1. Set Sample size (default 50) -- how many evenly-spaced samples are taken per shard.
2. Click Load preview. A progress bar tracks fetching; results are deduplicated by URL and shown in a table.
3. Optionally filter the table by Language, then Download CSV of this sample.

Full-Text Retrieval via HuggingFace

The API returns document snippets (up to 10,000 characters of context). Full article texts are stored in the HuggingFace dataset ruggsea/infini-news-corpus.

The dataset may be gated (require access approval). See the repo page for details.

Finding a Single Article

get_doc returns doc_ix — the article's row offset within its crawl year's HuggingFace config (year_{year}, where year is find's shard_years[s]). That addresses the exact row, so you fetch it directly from the datasets-server with no parquet scan:

import os
import requests

# year from find()'s shard_years[s]; offset from get_doc()'s doc_ix
year, offset = 2022, 12345
row = requests.get(
    "https://datasets-server.huggingface.co/rows",
    params={"dataset": "ruggsea/infini-news-corpus", "config": f"year_{year}",
            "split": "train", "offset": offset, "length": 1},
    headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"},  # the corpus is gated
).json()["rows"][0]["row"]
print(row["title"], "—", row["url"])
print(row["text"][:500])

Or open it directly in the dataset viewer in your browser (p is offset // 100):

https://huggingface.co/datasets/ruggsea/infini-news-corpus/viewer/year_2022/train?p=123&row=12345

Bulk Retrieval (Python)

import os
import sys
import time

import requests

API = "https://infini-news.uni-graz.at"
QUERY = "climate change"
INDEX = "ccnews"  # the whole corpus, all years

ROWS = "https://datasets-server.huggingface.co/rows"
DATASET = "ruggsea/infini-news-corpus"
HF_TOKEN = os.environ.get("HF_TOKEN")  # required: the corpus is gated
HF_HEADERS = {"Authorization": f"Bearer {HF_TOKEN}"} if HF_TOKEN else {}

# 1. Find all occurrence segments; shard_years[s] is shard s's crawl year (its HF config).
resp = requests.post(f"{API}/api/v1/find", json={"query": QUERY, "index": INDEX})
resp.raise_for_status()
find = resp.json()
print(f"Found {find['count']:,} occurrences")
years = find.get("shard_years", [])

# 2. For each match, get_doc -> doc_ix (the row offset), then fetch the row directly by
#    offset. No parquet scan. Rate limit: 120 get_doc requests/minute — back off on 429.
max_per_shard = 50
seen = set()
for s, seg in enumerate(find["segment_by_shard"]):
    if len(seg) != 2 or seg[1] <= seg[0] or s >= len(years):
        continue
    year = years[s]
    step = max(1, (seg[1] - seg[0]) // max_per_shard)
    for rank in range(seg[0], seg[1], step):
        try:
            r = requests.post(f"{API}/api/v1/get_doc", json={
                "query": QUERY, "index": INDEX, "s": s, "rank": rank, "max_ctx_len": 100,
            })
            if r.status_code == 429:
                time.sleep(int(r.headers.get("Retry-After", 5)))
                continue
            r.raise_for_status()
            offset = r.json()["doc_ix"]
        except requests.RequestException as e:
            print(f"skipping: {e}", file=sys.stderr)
            continue
        if (year, offset) in seen:
            continue
        seen.add((year, offset))
        rr = requests.get(ROWS, params={
            "dataset": DATASET, "config": f"year_{year}", "split": "train",
            "offset": offset, "length": 1,
        }, headers=HF_HEADERS)
        rr.raise_for_status()
        row = rr.json()["rows"][0]["row"]
        print(f"\n--- {row.get('url')} ---")
        print((row.get("text") or "")[:500])

print(f"\nFetched {len(seen)} unique articles")

Typical Workflow

1. Count (optional)

Check how common a phrase is before committing to a full search:

curl -s -X POST $API/api/v1/count \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews"}' | python3 -m json.tool

2. Find

Get shard segments:

FIND=$(curl -s -X POST $API/api/v1/find \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews"}')
echo "$FIND" | python3 -m json.tool

3. Retrieve Documents

Pick a shard and rank from the segment_by_shard ranges:

curl -s -X POST $API/api/v1/get_doc \
  -H 'Content-Type: application/json' \
  -d '{"query": "Graz", "index": "ccnews", "s": 0, "rank": 500, "max_ctx_len": 2000}' \
  | python3 -m json.tool

4. Get Full Article from HuggingFace

Use the url and date from the metadata to fetch the complete article text from the HuggingFace dataset. See Full-Text Retrieval via HuggingFace.

Rate Limits

Rate limiting is per-IP (using the X-Real-IP header behind nginx, falling back to the connection IP).

Exceeding either limit returns HTTP 429 with a Retry-After header. The concurrency cap exists because the per-minute limits do not bound parallelism: a client with many workers can stay under them and still occupy every backend slot, so everyone else waits and gets 503s. Serial clients never hit it.

Error Handling

All errors return a JSON object with a detail field (for HTTP errors) or an error field (for validation errors).

Example error responses:

{"detail": "Invalid index. Available: [...full list returned at runtime...]"}

{"detail": "Backend unavailable"}

{"detail": "Document decoding failed. Try a different max_ctx_len value."}

For 422 errors, the response body contains a detail array with per-field validation messages:

{
  "detail": [
    {
      "type": "string_too_short",
      "loc": ["body", "query"],
      "msg": "String should have at least 1 character",
      "input": ""
    }
  ]
}