Canonical Dedup Strategy Reference
Metadata
- Owner: Origo Engineering
- Last updated: 2026-04-14
- Slice/version reference: S46
Purpose
- Freeze one authoritative canonical dedup contract for every current source family.
- This reference explains the machine-readable registry at
contracts/canonical-dedup-strategy-v1.jsonand how it relates to live write behavior.
What is authoritative
- Canonical dedup is authoritative only at the canonical write boundary.
- The dedup lookup scope is current-truth lineage:
canonical_event_log_active_v1- plus logical reset-boundary lineage
- Raw append history remains forensic truth and does not override logical reset-and-rewrite intent.
Strategy classes
natural_key: stable source-native identity or stable composite.deterministic_synthetic_key: deterministic key over stable logical-observation fields and never over value fields.hybrid_key: durable source-native entity identity plus stable snapshot or revision boundary.
Family-level notes
- Binance / OKX / Bybit: trade identity remains exchange-native.
- ETF: canonical identity is
record_source_id:observed_date:metric_name:sha256(sorted_dimensions_json)and excludes value fields. - FRED: canonical identity is
record_source_id:observed_date:metric_name:realtime_start:realtime_endand excludes value fields while keeping revision-window identity explicit. - Bitcoin block transactions: canonical identity is
block_height:transaction_index:txid. - Bitcoin mempool: canonical identity is
snapshot_at_unix_ms:txid. - Bitcoin blockchain-derived metrics: canonical identity is block-height keyed per stream.
What is not authoritative
- adapter-level duplicate filtering
- file-level exact-duplicate cleanup
- proof-layer duplicate allowances
Those behaviors may remain visible in proof outputs, but they are hygiene only and must not redefine canonical correctness.
Transitional exception
- Exchange fast or staged write paths may still operate as empty-partition initial-load exceptions where declared in the registry.
- They are fenced exceptions, not a second idempotency regime.