Skip to main content

Canonical Dedup Strategy Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-14
  • Slice/version reference: S46

Purpose​

  • Freeze one authoritative canonical dedup contract for every current source family.
  • This reference explains the machine-readable registry at contracts/canonical-dedup-strategy-v1.json and how it relates to live write behavior.

What is authoritative​

  • Canonical dedup is authoritative only at the canonical write boundary.
  • The dedup lookup scope is current-truth lineage:
    • canonical_event_log_active_v1
    • plus logical reset-boundary lineage
  • Raw append history remains forensic truth and does not override logical reset-and-rewrite intent.

Strategy classes​

  • natural_key: stable source-native identity or stable composite.
  • deterministic_synthetic_key: deterministic key over stable logical-observation fields and never over value fields.
  • hybrid_key: durable source-native entity identity plus stable snapshot or revision boundary.

Family-level notes​

  • Binance / OKX / Bybit: trade identity remains exchange-native.
  • ETF: canonical identity is record_source_id:observed_date:metric_name:sha256(sorted_dimensions_json) and excludes value fields.
  • FRED: canonical identity is record_source_id:observed_date:metric_name:realtime_start:realtime_end and excludes value fields while keeping revision-window identity explicit.
  • Bitcoin block transactions: canonical identity is block_height:transaction_index:txid.
  • Bitcoin mempool: canonical identity is snapshot_at_unix_ms:txid.
  • Bitcoin blockchain-derived metrics: canonical identity is block-height keyed per stream.

What is not authoritative​

  • adapter-level duplicate filtering
  • file-level exact-duplicate cleanup
  • proof-layer duplicate allowances

Those behaviors may remain visible in proof outputs, but they are hygiene only and must not redefine canonical correctness.

Transitional exception​

  • Exchange fast or staged write paths may still operate as empty-partition initial-load exceptions where declared in the registry.
  • They are fenced exceptions, not a second idempotency regime.