Skip to main content

Performance Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-16
  • Slice/version reference: S60

Purpose and scope​

  • User-facing and operator-facing reference for where numeric performance truth lives and how recurring performance evidence is refreshed.
  • Numeric performance authority is singular:
    • contracts/governance/performance-sla.json
  • contracts/governance/ingest-throughput.json remains a subordinate scoped source only. It may feed the registry mechanically, but it may not compete with it.

Current authority model​

  • Performance truth is keyed by governed runtime paths, not free-form prose.
  • Runtime-unit vocabulary stays aligned to the observability contract:
    • adapter_batch_or_file_ingest
    • projector_batch
    • rebuild_batch
    • query_execution
  • Measurement-shape vocabulary is closed:
    • latency_budget
    • throughput_budget
    • freshness_budget

Budget and threshold semantics​

  • latency_budget and freshness_budget are max_allowed budgets.
    • operator approach = 80% of budget
    • operator breach = 95% of budget
  • throughput_budget is min_required.
    • operator approach = 95% of budget
    • operator breach = 80% of budget
  • Operator thresholds are derived-only. They do not redefine user-visible freshness semantics.

Registry status​

  • Entries may be either:
    • hard_sla
    • evidence_building
  • evidence_building means the path is governed and measured, but it is not yet allowed to claim a hard numeric SLA.
  • The initial Slice 55 registry is intentionally conservative:
    • registry structure is authoritative now
    • unsupported or not-yet-repeatable paths remain evidence_building

Shared exchange hot path​

  • Binance, OKX, and Bybit are one governed exchange family for daily-file ingest performance.
  • The governed path ids are:
    • binance_daily_file_hot_path
    • okx_daily_file_hot_path
    • bybit_daily_file_hot_path
  • They share one performance-critical stage transport boundary after source-specific parse, integrity, normalization, and proof-input construction.
  • Exchange-specific source-proof identity inputs remain source-specific. The shared path does not flatten those differences away.

Hot-path transport rules​

  • Python .to_list() stage handoff is forbidden on the steady-state exchange hot path.
  • Hidden HTTP or raw fallback stage transport is forbidden on the steady-state exchange hot path.
  • The shared exchange transport must stay ClickHouse-native and columnar.
  • Transport regression and materialization regression are separate failure classes and are both guarded.

Exchange concurrency rules​

  • Source-safe ceilings and operator-performance ceilings are distinct.
  • Source-safe controls still live under:
    • contracts/source-pressure-v1.json
  • Operator-performance exchange backfill ceilings now live under:
    • control-plane/origo_control_plane/backfill/s34_contract.py
  • User-facing Dagster exchange runs must honor the lower of:
    • source_safe_concurrency_ceiling
    • performance_safe_concurrency_ceiling

Monthly load proof​

  • Authoritative execution home:
    • Dagster job origo_monthly_load_proof_job
    • Dagster schedule origo_monthly_load_proof_schedule
  • Artifacts are written under the root from:
    • ORIGO_MONTHLY_LOAD_PROOF_ARTIFACT_ROOT_DIR
  • The initial recurring proof is honest about probe coverage:
    • supported repeatable probes run and record observed measurements
    • unsupported paths are recorded as registry_audit_only
    • unsupported paths stay visible as evidence_building

Current repeatable probes​

  • binance_daily_file_hot_path
  • okx_daily_file_hot_path
  • bybit_daily_file_hot_path
  • Each probe is a synthetic fixture benchmark that measures:
    • parse
    • stage transport
    • staged source proof
    • staged canonical write
  • These are internal hot-path benchmarks, not live upstream network proofs.

Path-scoped performance gate​

  • CI gate:
    • .github/workflows/performance-gate.yml
  • Scope resolver:
    • scripts/validate_performance_scope.py
  • Touched implementation files and touched env/deploy/runtime-capacity surfaces both count.
  • The gate is not a universal benchmark run on every PR. It is triggered only when a governed performance path is touched.