Skip to main content

Raw Export API Reference

Metadata​

  • Owner: Origo Engineering
  • Last updated: 2026-03-26
  • Slice/version reference: S2, S5, S6, S8, S11, S13, S14, S15, S16, S17, S18, S19, S20, S29, S34 prep (API v0.1.28 branch state)

Purpose and scope​

  • This is the user-facing reference for asynchronous raw exports.
  • Scope includes native and aligned_1s export flows with parquet and csv.

Inputs and outputs with contract shape​

  • Endpoints:
    • POST /v1/raw/export
    • GET /v1/raw/export/{export_id}
  • Required header: X-API-Key
  • Submit request contract:
    • mode: native | aligned_1s
    • format: parquet | csv
    • dataset: binance_spot_trades | okx_spot_trades | bybit_spot_trades | etf_daily_metrics | fred_series_metrics | bitcoin_block_headers | bitcoin_block_transactions | bitcoin_mempool_state | bitcoin_block_fee_totals | bitcoin_block_subsidy_schedule | bitcoin_network_hashrate_estimate | bitcoin_circulating_supply
    • view_id: optional view identifier (must be paired with view_version)
    • view_version: optional positive integer (must be paired with view_id)
    • fields: optional field projection
    • Exactly one window selector:
      • month_year
      • time_range
      • n_rows
      • n_random
    • strict: boolean
    • auth_token: currently rejected for export dispatch in this slice
  • Submit response (202):
    • export_id, status, submitted_at, status_path
    • rights_state, rights_provisional, view_id, view_version
  • Status response (200):
    • export_id, status, mode, format, dataset, source, submitted_at, updated_at
    • rights_state, rights_provisional, view_id, view_version
    • optional artifact with {format, uri, row_count, checksum_sha256}
    • optional error_code, error_message

Data definitions (fields, types, units, timezone, nullability)​

  • Export artifact metadata:
    • uri: absolute path string (non-null on success)
    • row_count: integer >= 0 (non-null on success)
    • checksum_sha256: hex digest string (non-null on success)
  • Status lifecycle:
    • queued, running, succeeded, failed
  • Exported row field definitions follow dataset taxonomy in docs/data-taxonomy.md.

Source/provenance and freshness semantics​

  • Export payload comes from canonical ClickHouse data queried via Origo query core.
  • Dagster run tags preserve mode/format/dataset and request hash context.
  • Export freshness follows the latest successful ingestion for requested sources.
  • OKX native/aligned exports are served from canonical OKX projection paths as of Slice 18.
  • Bybit native/aligned exports are served from canonical Bybit projection paths as of Slice 19.
  • Bitcoin native/aligned exports are served from canonical Bitcoin projection paths as of Slice 20 with stream aligned completion in Slice 29.
  • During Slice 34 full-history backfill, exportable history is bounded by terminal proof coverage; export does not silently include unproved partitions.
  • User-facing backfill-boundary meaning is defined in docs/backfill-status-reference.md.
  • mode=aligned_1s supports only aligned-capable datasets:
    • binance_spot_trades
    • okx_spot_trades
    • bybit_spot_trades
    • etf_daily_metrics
    • fred_series_metrics
    • bitcoin_block_headers
    • bitcoin_block_transactions
    • bitcoin_mempool_state
    • bitcoin_block_fee_totals
    • bitcoin_block_subsidy_schedule
    • bitcoin_network_hashrate_estimate
    • bitcoin_circulating_supply

Failure modes, warnings, and error codes​

  • 404: export ID not found
  • 409: auth/rights/contract failure
  • 503: dispatch/backpressure/runtime/audit/metadata failure
  • Common failure codes:
    • EXPORT_QUEUE_LIMIT_REACHED
    • EXPORT_DISPATCH_ERROR
    • EXPORT_RIGHTS_*
    • EXPORT_AUDIT_WRITE_ERROR
    • EXPORT_ARTIFACT_METADATA_ERROR
    • EXPORT_STATUS_METADATA_ERROR
    • EXPORT_RUN_FAILED / EXPORT_RUN_CANCELED

Determinism/replay notes​

  • Export determinism and query/export parity are anchored by these slice records and proof execution references:
    • spec/slices/slice-6-fred-integration.md
    • spec/slices/slice-2-raw-export-native.md
    • spec/slices/slice-5-raw-query-aligned-1s.md
    • spec/slices/slice-15-binance-event-sourcing-port.md
    • spec/slices/slice-16-etf-event-sourcing-port.md
    • spec/slices/slice-17-fred-event-sourcing-port.md
    • spec/slices/slice-18-okx-event-sourcing-port.md
    • spec/slices/slice-19-bybit-event-sourcing-port.md
    • spec/slices/slice-20-bitcoin-event-sourcing-port.md
    • spec/slices/slice-8-okx-spot-trades-aligned.md
    • spec/slices/slice-11-bybit-spot-trades-aligned.md
    • spec/slices/slice-13-bitcoin-core-signals.md
  • Generated proof evidence for the migrated completed slices is written under spec/slices-generated/<slice-slug>/ and is intentionally not committed.

Environment variables and required config​

  • ORIGO_INTERNAL_API_KEY
  • ORIGO_EXPORT_MAX_CONCURRENCY
  • ORIGO_EXPORT_MAX_QUEUE
  • ORIGO_DAGSTER_GRAPHQL_URL
  • ORIGO_DAGSTER_REPOSITORY_NAME
  • ORIGO_DAGSTER_LOCATION_NAME
  • ORIGO_DAGSTER_EXPORT_JOB_NAME
  • ORIGO_EXPORT_ROOT_DIR
  • ORIGO_SOURCE_RIGHTS_MATRIX_PATH
  • ORIGO_EXPORT_AUDIT_LOG_PATH
  • ORIGO_AUDIT_LOG_RETENTION_DAYS (must be >=365)

Minimal examples​

  • Submit native export:
    • { "mode":"native", "format":"parquet", "dataset":"binance_spot_trades", "month_year":[8,2017], "strict":false }
  • Submit OKX native export:
    • { "mode":"native", "format":"csv", "dataset":"okx_spot_trades", "time_range":["2024-01-01T16:00:00Z","2024-01-02T16:00:00Z"], "strict":false }
  • Submit Bybit native export:
    • { "mode":"native", "format":"csv", "dataset":"bybit_spot_trades", "time_range":["2024-01-02T00:00:00Z","2024-01-03T00:00:00Z"], "strict":false }
  • Submit Bitcoin native export:
    • { "mode":"native", "format":"parquet", "dataset":"bitcoin_block_headers", "time_range":["2024-04-20T00:00:00Z","2024-04-22T00:00:00Z"], "strict":false }
  • Submit Bitcoin aligned export:
    • { "mode":"aligned_1s", "format":"csv", "dataset":"bitcoin_block_fee_totals", "time_range":["2024-04-20T00:00:00Z","2024-04-22T00:00:00Z"], "strict":false }
  • Submit aligned export:
    • { "mode":"aligned_1s", "format":"csv", "dataset":"etf_daily_metrics", "time_range":["2026-03-05T00:00:00Z","2026-03-07T00:00:00Z"], "strict":false }
  • Poll status:
    • GET /v1/raw/export/<export_id>