Skip to main content

Unified Failure Semantics Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-17
  • Slice/version reference: S38, S43, S47, S48, S49, S61

Purpose and scope​

  • User-facing and operator-facing reference for how Origo classifies success, failure, disagreement, and quarantine across live runtime surfaces.
  • Scope covers:
    • Dagster/Dagit run and partition truth
    • partition completion truth
    • partition proof and quarantine residue truth
    • write-run and ingest status truth
    • raw query, export, and historical HTTP/Python surfaces
    • runtime bootstrap and deploy success semantics
  • Grafana and the rest of the observability stack are derived-only surfaces; they must render these semantics from authoritative Dagster, ClickHouse, and audit inputs rather than define them independently.

Authoritative surface families​

  • dagster_dagit_run_and_partition_truth
  • partition_completion_truth
  • partition_proof_and_partition_quarantine_truth
  • write_run_and_ingest_state_truth
  • raw_query_export_and_historical_http_python_truth
  • runtime_bootstrap_and_deploy_truth

Canonical failure classes​

  • contract_input_violation
  • runtime_dependency_unavailable
  • reconcile_required
  • already_complete_rerun_refused
  • quarantined_or_blocked
  • strict_warning_escalation
  • authoritative_surface_disagreement

Success semantics​

  • Dagster partition success:
    • Dagster green is success only when the same partition also has authoritative completion state complete.
    • proof terminal states proved_complete and empty_proved remain supporting evidence, not final success by themselves.
    • Dagster green without required completion is failure.
    • Dagster red with authoritative complete is failure.
  • Partition completion:
    • completion states are complete, failed, and blocked
    • the only terminal success state is complete
    • non-success completion states are failed and blocked
    • complete requires terminal proof at decision time and no active split-brain
  • Write run success:
    • completeness success is ok
    • completeness gap_detected is failure even if orchestration otherwise looks green
    • cursor/checkpoint duplicate outcomes are still successful idempotent outcomes, not failures
  • Query/export/historical request success:
    • success statuses are 200 and 202
    • non-success statuses are 404, 409, and 503
    • strict=false may serve with warnings
    • strict=true escalates warnings to 409
  • Runtime bootstrap/deploy success:
    • required quality gates must be green
    • required runtime dependencies must be healthy
    • missing either side is a hard failure
  • Canonical append success:
    • exact duplicate source-event delivery is successful idempotent duplicate handling, not conflict
    • stale-head conflict is a hard write failure for non-present append intent
    • caller-visible automatic retries on stale-head conflict are forbidden

Split-brain and disagreement​

  • Disagreement between authoritative surfaces is itself a first-class hard failure.
  • Operator-visible examples:
    • Dagster green while authoritative completion is missing or non-complete
    • Dagster red while authoritative completion is complete
    • proof terminal while authoritative completion is not complete
    • API/runtime success claims while completion obligations remain unsatisfied

Quarantine meaning​

  • The authoritative failure-hold surface is canonical_quarantine_incidents in ClickHouse.
  • Incident lifecycle statuses are:
    • active
    • cleared
    • reopened
  • Blocking incident statuses are active and reopened.
  • quarantined and reconcile_required remain the partition-level blocking residue states rendered by backfill proof/runtime compatibility surfaces.
  • Stream-level runtime gating still remains mandatory, but it now reads the same ClickHouse incident history rather than a JSON file.
  • File-backed stream quarantine remains available only for proof/legacy non-runtime use.
  • Dagster-visible remediation and replay stay the required operator surface for clearing and re-running quarantined work.

Canonical write conflict meaning​

  • Canonical write concurrency is optimistic and ClickHouse-only.
  • The authoritative head components are canonical_stream_sequence_heads, canonical_event_log_active_v1, and canonical_partition_reset_boundaries.
  • One authoritative append may contain multiple rows only for one canonical stream key.
  • Mixed-stream authoritative append is rejected.
  • Exact duplicate identity short-circuits before stale-head conflict evaluation.
  • Stale-head conflict must surface through writer error codes and runtime-audit/operator truth; it is not silently repaired.
  • Source-authority rejection is also pre-append failure.
  • Adapter-boundary rejection is also pre-append failure.
  • Payload-schema rejection is also pre-append failure.
  • Late-arrival rejection is also authoritative runtime truth.
  • In-window late-arrival supersession is a declared required truth surface.
  • Adapter-boundary drift is operator-visible runtime truth, but alert-only by default.

Representative scenarios​

  • already-complete rerun refusal
  • reconcile-required partition
  • quarantined partition
  • successful Dagster run with authoritative completion
  • strict-warning escalation
  • adapter-boundary rejection
  • runtime bootstrap contract failure
  • canonical stale-head write conflict