Skip to main content

Retention And Capacity Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-04-15
  • Slice/version reference: S56

Purpose and scope​

  • User-facing reference for Origo's current retention, rebuild-substrate, and fixed-capacity doctrine.
  • Scope covers forever-hot canonical truth, forever-hot serving truth, retained-file rebuild substrate, and the operator-visible rebuild-complete boundary.

Current storage and rebuild model​

  • Origo keeps canonical truth hot in ClickHouse:
    • canonical_event_log
    • canonical_event_log_active_v1
  • Origo also keeps first-class serving truth hot in ClickHouse:
    • native historical serving tables
    • aligned_1s serving tables
  • Origo does not implement hot/warm/cold tiering in the current design.
  • Retained source files are rebuild substrate only where a governed substrate exists; they are not a user query tier.

Retained rebuild substrate by dataset family​

  • etf_daily_metrics: governed forever-retained object-store substrate exists
  • fred_series_metrics: governed forever-retained object-store substrate exists
  • binance_spot_trades: declared substrate gap
  • okx_spot_trades: declared substrate gap
  • bybit_spot_trades: declared substrate gap
  • all current Bitcoin families: declared substrate gap

Rebuild chain​

  • canonical-event rebuild authority:
    • retained source files where a governed substrate exists
  • projection rebuild authority:
    • canonical_event_log_active_v1
  • normal rebuild from upstream re-fetch is not the declared target model

Fixed internal objectives​

  • infrastructure budget:
    • $150/mo
  • historical read objective:
    • 10M rows/s Arrow-to-Polars
  • full-system rebuild objective:
    • 4h
  • priority order:
    • Binance
    • Bitcoin
    • ETF

Rebuild-complete boundary​

  • rebuild complete means all required surfaces are complete in the declared priority order:
    • canonical_event_log_active_v1
    • native historical serving tables
    • aligned_1s serving tables
    • required proof and promotion surfaces
  • canonical-only replay is not enough to claim rebuild completion.