Developer Observability Reference

Metadata

• Owner: Origo Engineering
• Last updated: 2026-04-17
• Slice/version reference: S40 + S57 + S58 + S59 + S61

Purpose

• Developer reference for the shared observability bootstrap, bridge contract, and the S61 authority/residue split.

Shared instrumentation path

• bootstrap module: origo.observability.bootstrap
• process bootstrap entrypoint: bootstrap_process_observability(service_name=...)
• API middleware: FastAPIObservabilityMiddleware
• work-unit span helper: start_work_unit_span(...)

Bridge contract

• authoritative audit logs remain file-backed and immutable
• runtime audit remains file-backed and authoritative
• best-effort Loki emission is derived-only and must never block the authoritative audit append path
• Grafana dashboards must render authoritative Dagster, completion, proof/quarantine, and audit inputs rather than invent dashboard-local truth
• authoritative completion truth must come from canonical_partition_completion_outcomes
• proof/quarantine, current-truth, and serving rows are residue unless the completion authority says complete
• the old exception that let Canonical Whole System Truth read current-truth coverage directly is gone; current-truth coverage now lives only on Canonical Pipeline Residue

Required pivot fields

• source_id
• stream_id
• partition_id
• run_id
• failure_class
• proof_state
• quarantine_status
• quarantine_reason
• correlation_id
• causation_id
• producer_id
• producer_version
• event_id
• payload_sha256_raw

Dashboard families

• store truth / event sourcing
• Dagster authority / operator truth
• ingestion guarantees / source integrity
• canonical whole system truth
• canonical pipeline residue
• backfill status by dataset residue
• runtime recovery / freshness / alerting
• quarantine / failure residue matrix

Concrete panel layout

• Canonical Whole System Truth
  • Authoritative Partition Completion
• Canonical Pipeline Residue
  • Source Manifest Residue
  • Canonical Current-Truth Residue
  • Terminal Proof and Blockers Residue
  • Serving Delivery Residue
• Origo Backfill Residue By Dataset
  • Per-Dataset Completion Residue
  • Open Unresolved Or Quarantine-Touched Residue Partitions
  • Latest Source Manifest Residue Activity
• Origo Dagster Authority / Operator Truth
  • Runs That Last Changed Authoritative Partition Truth
  • Latest Projector Checkpoints
  • Latest Projector Watermarks
• Origo Quarantine / Failure Residue Matrix
  • Dataset Failure Residue Summary
  • Latest Non-Terminal Proof Residue
  • Latest Quarantine Residue State Changes

Dashboard rules

• Canonical Whole System Truth is the only Grafana surface allowed to mean partition completion.
• Canonical Pipeline Residue is the only Grafana surface allowed to aggregate provisional manifest/current-truth/proof/delivery residue into one supporting dashboard.
• ETF and FRED remain source-split in both dashboards.
• Exchange and Bitcoin families remain one row per canonical backfill unit on residue coverage surfaces.
• The visible Dagster navigation column renders as literal link.
• A future dashboard that claims completion/failure/run-last-changed truth must either read completion authority directly or be explicitly labeled residue/debug-only.