Skip to main content

Backfill Status Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-17
  • Slice/version reference: S34 prep + S57 + S58 + S59 + S61

Purpose and scope​

  • User-facing reference for what Origo means by historical availability while full canonical backfill is in progress.
  • Scope covers raw query, raw export, and historical HTTP/Python surfaces.
  • Unified cross-surface success and failure semantics are anchored by docs/failure-semantics-reference.md.
  • Grafana is derived-only. It must render authoritative Dagster and ClickHouse truth rather than define availability itself.

Derived backfill surfaces​

  • Canonical Whole System Truth
    • authoritative partition-completion surface
    • latest outcome per backfill unit and partition
  • Canonical Pipeline Residue
    • supporting residue surface for manifests, canonical current truth, proof/blockers, and serving delivery residue
  • Origo Backfill Residue By Dataset
    • supporting dataset-level residue summary

Inputs and outputs with contract shape​

  • This document is a reference artifact; it does not define a standalone endpoint.
  • It applies to:
    • POST /v1/raw/query
    • POST /v1/raw/export
    • POST /v1/historical/*
    • HistoricalData Python methods

Data definitions​

  • authoritative partition completion:
    • the latest ClickHouse-backed completion outcome for (source_id, stream_id, partition_id)
    • complete, failed, or blocked
  • historical availability:
    • the portion of dataset history that is currently queryable because the required serving surfaces are gated by authoritative partition completion
  • provisional residue:
    • manifests, canonical rows, proof rows, checkpoints, or serving rows that may exist before final completion promotion
  • source-specific availability boundary:
    • a dataset-specific lower bound determined by the real source contract rather than by already-written residue
  • logical reset boundary:
    • an audited partition-local cutover used during explicit reconcile so stale canonical rows stop counting as live truth without deleting append-only canonical evidence

Source/provenance and freshness semantics​

  • Origo serves supported historical truth only from partitions whose latest authoritative completion outcome is complete.
  • Terminal proof remains required supporting evidence, but proof alone does not make a partition complete.
  • Canonical Whole System Truth is the human-facing completion dashboard:
    • it answers whether a partition is complete, failed, or blocked
    • it includes the run that last changed authoritative partition truth
  • Canonical Pipeline Residue is supporting only:
    • Source Manifest Residue
    • Canonical Current-Truth Residue
    • Terminal Proof and Blockers Residue
    • Serving Delivery Residue
    • it explains residue beneath the current completion outcome; it does not replace completion authority
  • ETF history remains issuer-specific.
  • FRED history remains series-specific.
  • Source existence alone does not make data queryable.
  • Proof existence alone does not make data queryable.
  • Native or aligned rows alone do not make data queryable.
  • No-selector requests resolve to the full currently complete supported history for that dataset, not to speculative vendor history beyond the completion boundary.

Failure modes, warnings, and error codes​

  • Authoritative-surface disagreement is itself a hard failure:
    • Dagster green with missing or non-complete authoritative completion
    • Dagster red with authoritative complete
    • query/export/historical success claims while latest completion is not complete
  • 404:
    • no complete rows exist in the requested window
  • 409:
    • request contract failure
    • auth/rights failure
    • strict-mode rejection when warnings are present
  • failed and blocked are non-success completion states.
  • quarantined and reconcile_required remain blocking residue states.
  • 503:
    • runtime/backend failure
  • Origo does not silently serve beyond the completion boundary.

Determinism/replay notes​

  • Historical surfaces are replayable only on the portion of history whose latest authoritative completion outcome is complete.
  • As backfill advances, later windows become available without changing the replay contract for already-complete windows.
  • Slice 34 closeout is still the point where full-history availability claims become complete for each dataset, but S61 hardens what complete is allowed to mean.

Environment variables and required config​

  • HTTP/API access:
    • X-API-Key
  • Python access:
    • working Origo runtime config for the target environment

Minimal examples​

  • No-selector historical request:
    • returns the full currently complete history for that dataset
  • Bounded request beyond the completion boundary:
    • returns 404 or a fail-loud runtime error rather than partial silent fill