Infini-News API v2

API v2 binds search continuations to one validated corpus release. The gateway must set GATEWAY_INDEX_VERSION; the worker must start with the same --index-version. The worker starts only when the FM-index and metadata-store manifests identify the same accepted corpus release and exact shard set.

POST /api/v2/find accepts a query, an index name and an optional inclusive crawl-year range:

{
  "query": "climate change",
  "index": "ccnews",
  "year_min": 2026,
  "year_max": 2026
}

Omit both year fields to search every indexed year. find returns total_occurrences, segment_by_shard, shard_years, shard_ids, and index_version. The intervals in segment_by_shard use inclusive start and exclusive end ranks, aligned with shard_ids. An occurrence is a matched text position; an article can contain several occurrences.

POST /api/v2/count accepts the same fields and returns count, index_version and latency_ms. Metadata filters do not apply to either endpoint. Use documents to filter resolved articles.

All continuation bodies require expected_index_version with the exact value returned by find. A stale value returns HTTP 409 before rank resolution. A missing gateway or worker release identity returns HTTP 503.

POST /api/v2/get_doc retrieves the article at one occurrence rank. Use a shard_id and a rank from the corresponding find interval, and repeat the exact query and returned release identity:

{
  "query": "climate change",
  "index": "ccnews",
  "shard_id": "2026-03",
  "rank": 123,
  "expected_index_version": "release-id",
  "max_ctx_len": 1000,
  "full_text": false
}

Replace the illustrative shard, rank and release identity with values from your search. The response includes doc_ix, text, highlighted spans, metadata, index_version and length information. max_ctx_len controls context bytes on each side of the match, from 1 to 10,000. Set full_text to true to read the whole article locally. Articles over 1 MiB return HTTP 422 instead of truncated full text. This operation does not require Hugging Face.

Metadata includes the persistent warc_record_id and string-valued text_xxhash64 identity fields. Preserve the hash as a string to avoid numeric precision loss. country is the attributed publisher country; country_source and country_confidence describe that attribution. country_tld records a country-code domain suffix and is a separate field. publish_date, warc_date and date_source distinguish publication dates from crawl dates. iptc_topic, language (GlotLID) and lang (CommonLID) contain the article's classification labels where available.

POST /api/v2/occurrences accepts index, shard_id, rank_from, rank_to, and expected_index_version. It resolves at most 50,000 ranks. The returned page_occurrences applies only to [rank_from, rank_to).

POST /api/v2/documents accepts the same fields and resolves at most 1,000 ranks because every distinct document includes metadata. Optional filter lists are languages (GlotLID), commonlingua_languages, topics, sources (hostname or site name), and countries (publisher country). Matching is exact after Unicode normalization and case folding. Values within one list use OR; filter dimensions use AND. Publisher-country filtering returns HTTP 503 when the attribution table is not configured.

The response reports page_occurrences, page_distinct_documents, page_matching_documents, and page_matching_occurrences. These counts are page-local. A document can occur on adjacent pages. For query-wide totals, examine every page and merge equal (year, doc_ix) pairs. Count the merged records for the article total; sum their occurrences for the occurrence total.

POST /api/v2/metadata accepts at most 1,000 document references:

{
  "index": "ccnews",
  "expected_index_version": "release-id",
  "documents": [{"year": "2026", "doc_ix": 123}]
}

It preserves request order and returns complete structured metadata. A missing year store returns HTTP 503. An invalid doc_ix returns HTTP 422. Oversize responses return HTTP 413 and must be retried with a smaller page.

GET /api/v2/health passes only when the worker serves the gateway's configured release. GET /api/v2/info reports the loaded shard date range and document count from the worker instead of hard-coded corpus totals.

Local search works independently of Hugging Face. External article retrieval uses the matching published corpus. By default, the Python recipe resolves the latest repository commit once and pins every download to that commit. An explicit GATEWAY_HF_CORPUS_REVISION overrides this selection.

The Python example returned by /api/v2/info checks the publication's release identity and shard-manifest row count before retrieving selected Parquet rows. It maps (year, doc_ix) to partition rows, downloads each needed partition once at the selected commit and verifies each row's warc_record_id and string-normalized text_xxhash64 against API metadata. It stops when either value is absent or differs, or when the API release changes. The dataset viewer does not support immutable row retrieval; v2 omits its row links.

In the web interface, Filters accepts semicolon-separated values. Each search or continuation examines at most 1,000 occurrence ranks. The displayed phrase total remains unfiltered; article counts describe the examined pages. Article details shows country attribution, classification scores and article identifiers. Read full article retrieves text locally within the 1 MiB limit.

Under Export, Download Python script retrieves all articles matching the query, crawl-year range and selected metadata filters. Run the downloaded script after installing the dependencies listed at its top. It examines every matching occurrence page and writes the selected articles to infini-news-articles.jsonl. This operation is independent of the pages or samples loaded in the browser.

CSV and JSON exports include article identifiers, release, query, crawl-year range and export scope. Exports of examined matches also include filters, examined rank intervals and occurrence counts accumulated across those pages. Sample exports contain the loaded occurrence sample. JSON preserves metadata values; CSV quotes fields and prefixes potentially executable spreadsheet cells with a tab. Import hash columns as text to preserve their exact digits.