Skip to main content

Observability Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-17
  • Slice/version reference: S40 + S57 + S58 + S59 + S61

Purpose and scope​

  • Operator-facing reference for Origo observability after the one-truth partition completion cutover.
  • Grafana remains derived-only.
  • Human-facing green/red truth must now come from Dagster plus the ClickHouse-backed partition completion authority, never from proof or provisional residue alone.

Derived vs authoritative truth​

  • Grafana is never authoritative.
  • Authoritative surfaces are:
    • Dagster/Dagit run and partition truth
    • ClickHouse partition completion truth
    • ClickHouse proof/quarantine residue truth
    • runtime audit
    • immutable audit
    • runtime bootstrap and deploy truth
  • Proof, canonical current truth, native rows, aligned rows, checkpoints, and watermarks may all exist before final completion. They are supporting residue until completion authority says otherwise.

Day-zero operator questions​

  • Is Dagster green and does authoritative completion agree?
  • What is the latest authoritative partition outcome?
  • What reason keeps this partition non-complete?
  • What run last changed authoritative partition truth?
  • What provisional residue exists beneath the current completion outcome?

Live dashboard layout​

  • Canonical Whole System Truth
    • Authoritative Partition Completion
  • Canonical Pipeline Residue
    • Source Manifest Residue
    • Canonical Current-Truth Residue
    • Terminal Proof and Blockers Residue
    • Serving Delivery Residue
  • Origo Backfill Residue By Dataset
    • Per-Dataset Completion Residue
    • Open Unresolved Or Quarantine-Touched Residue Partitions
    • Latest Source Manifest Residue Activity
  • Origo Dagster Authority / Operator Truth
    • Runs That Last Changed Authoritative Partition Truth
    • Latest Projector Checkpoints
    • Latest Projector Watermarks
  • Origo Ingestion Guarantees / Source Integrity
    • Latest Source Completeness Checkpoints
    • Current Ingest Cursor State
    • Latest Manifest Activity
  • Origo Quarantine / Failure Residue Matrix
    • Dataset Failure Residue Summary
    • Latest Non-Terminal Proof Residue
    • Latest Quarantine Residue State Changes
  • Origo Runtime Recovery / Freshness / Alerting
    • Ingest Lag By Stream
    • Projector Watermark Lag
    • Latest Partition Reset Boundaries
  • Origo Store Truth / Event Sourcing
    • Canonical Table Row Counts
    • Envelope Schema Status
    • Latest Partition Reset Boundaries

Canonical whole-system truth dashboard fields​

  • backfill_unit_id
  • dataset_id
  • partition_id
  • completion_state
  • completion_reason
  • projection_mode
  • proof_state_at_decision
  • native_delivery_satisfied
  • aligned_delivery_satisfied
  • run_that_last_changed_truth
  • recorded_at_utc
  • dagster_job_link
  • The canonical inventory anchor remains docs/data-taxonomy.md.
  • ETF and FRED remain source-split at the unit level.
  • The visible Dagster navigation column renders as literal link.

Canonical pipeline residue dashboard rules​

  • Canonical Pipeline Residue is supporting/debug-only.
  • It is allowed to show provisional manifests, canonical current truth, proof residue, and delivery residue.
  • It may never be interpreted as final completion truth.
  • Empty units must render honest no-data state in residue coverage panels.

Inspection order​

  • Start with Canonical Whole System Truth.
  • Use it to answer whether a partition is authoritatively complete, failed, or blocked.
  • If the partition is not complete, move to Canonical Pipeline Residue to see where provisional work exists.
  • Then use Origo Backfill Residue By Dataset and Origo Quarantine / Failure Residue Matrix to inspect proof/quarantine residue.
  • Cross-check Origo Dagster Authority / Operator Truth for the run that last changed authoritative partition truth.
  • Use Origo Store Truth / Event Sourcing for store-level sanity checks and reset-boundary visibility.

Runtime-audit pivots​

  • adapter_boundary_rejection
    • authoritative source-boundary hard-stop before canonical write
  • adapter_boundary_drift
    • declared drift observation; alert-only unless a source-local contract says otherwise
  • late_arrival_rejection
    • authoritative outside-window rejection truth
  • late_arrival_supersession
    • authoritative aligned in-window supersession truth