Historical Data

Decide which observations remain final

Before researching a protected final window, choose the Compiled project's separate final/ or final-event/ branch and reserve its declared dates. The complete branch starts with limen reserve, runs its YAML and finishes with limen assess --reservation; follow final assessment for those exact commands. Do not run exploratory cases against that branch's reserved observations first. The project ledger records local known exposure; downloading a file or starting a new Git directory does not establish a global never-observed claim.

Python Research excludes final observations in its caller-owned preparation and selection code. It does not acquire a native reservation merely by supplying a cutoff. Both projects state the population each result measures.

Prerequisites

• Python 3.10-3.13
• pip install "vaquum-limen[data]" for every built-in hosted or Arrow data path on this page
• network access for Hugging Face or remote URL examples; local files need no network

Source choice determines origin and resolution

HistoricalData provides hosted BTCUSDT spot candles and dollar bars, Binance archives, local or HTTP(S) Parquet/CSV/ZIP files, and a specialized local Arrow IPC path. Ordinary Python Research can also supply observations acquired through caller-owned code.

Hosted acquisition resolves and caches source snapshots, with retried requests and atomic cache writes. A changing hosted source therefore needs a retained snapshot identity for a repeated research comparison. expected_sha256 pins local Parquet bytes under its specific loader contract. The source reference defines acquisition arguments, aggregation, filtering, and cache behavior.

Population selection preserves an observation policy

Date bounds, latest-row limits, and elapsed-day lookbacks answer different population questions. Acquisition includes the entire date-only end day. Experiment date partitions use half-open UTC intervals. The distinction matters when the same date appears in acquisition and split declarations.

Sequential and date-based partitions retain chronology. Seeded random partitions provide disjoint samples under their declared design. Walk-forward adds anchored or rolling training, purge, and embargo. Pre-split selectors can retain a trailing elapsed-time window or one complete contiguous block of a specified regime. Their utility contracts define selection and unavailable cases.

The selection policy belongs beside the result. A latest-row limit depends on the source endpoint, while a fixed date window names its endpoints directly. A regime-selected block changes the assessed population and requires the recorded selection identity.

Fix the observations before research

The Python project's scenarios verify the complete recorded file before selecting a row prefix. Python Research can use ordinary file I/O; it does not require a Manifest or HistoricalData.

1def recorded(data_dir, name, rows):2    source = data_dir / name3    if hashlib.sha256(source.read_bytes()).hexdigest() != INPUTS[name]:4        raise ValueError('Recorded input digest differs: ' + name)5    return pl.read_parquet(source, n_rows=rows)

The Compiled project names the recorded quarter-hour source in its selection Manifest:

1    data_source:2      method: limen.data.HistoricalData.get_any_file3      params:4        file_path_or_url: inputs/spot_15m_20250101_20250531.parquet

The bundle pins the source by SHA-256. The YAML reads its literal inputs/ path from the bundle root. Both projects use the matching recorded-input bundle; research proof requires no live dataset download.

HistoricalData remains stateful for Manifest compatibility. Each retrieval method returns the loaded frame and stores it on the instance as data, with column names in data_columns.

The file-format diagnostics preserve the same observations

The Python format case reads the bundle's equivalent CSV, ZIP, and Arrow files through HistoricalData. It checks that a 12-row limit returns the final 12 observations and that the stateful loader retains that returned frame:

1def read_source_formats(data_dir):2    report = {}3    for suffix in ('csv', 'zip', 'arrow'):4        path = data_dir / ('recorded-hourly.' + suffix)5        historical = HistoricalData()6        frame = (historical.get_arrow_file(str(path)) if suffix == 'arrow' else historical.get_any_file(str(path)))7        latest = (historical.get_arrow_file(str(path), row_count_limit=12) if suffix == 'arrow' else historical.get_any_file(str(path), row_count_limit=12))8        if not latest.equals(frame.tail(12)) or historical.data is not latest:9            raise ValueError('Local source changed row-limit or retained-frame semantics')10        report[suffix] = {'rows': frame.height, 'columns': frame.columns, 'source_sha256': hashlib.sha256(path.read_bytes()).hexdigest(),11                         'timestamps_ns': frame['datetime'].dt.epoch('ns').to_list(),12                         'values': frame.select('open', 'high', 'low', 'close', 'volume').to_dicts()}13    return report

The Compiled format cases execute the corresponding literal file declarations:

1limen run --no-progress-bar \2  compiled-research/source_csv.yaml \3  compiled-research/source_zip.yaml \4  compiled-research/source_arrow.yaml

Independent checks reconcile timestamps and OHLCV values across formats. The native Compiled checks also require matching candidate identities and measured accuracy. File equivalence preserves the recorded observations; feature preparation and research partitions still follow their own declarations.

Carry the declared population forward

The selection examples use quarter-hour BTCUSDT observations. Their datetime is the start of the interval in UTC; prices are USDT per BTC and volume is BTC. A close can be used only after the interval completes. The Python inference adapter records that 900-second availability delay explicitly. Native execution declares its source interval with the trading policy.

Python selects 1,920 recorded rows: the first 960 may fit state, the next 480 may select it and the final 480 supply later decisions. Feature warm-up and forward labels remove rows within each partition. Compiled selection instead uses its literal half-open calendar dates; it is a different population. Later variants name their changed windows explicitly.

Keep the acquired inputs.json, raw files and publication identity together. Funding and retrospective observed-cost inputs carry separate source clocks and provenance; their reported availability must not be inferred from a bar timestamp. The three supplied target-label files are deterministic software labels derived within chronological partitions, rather than observed venue trades.

Historical data API describes local Parquet/CSV/ZIP, hosted sources, Binance archives and Arrow IPC, including filtering and cache behavior. Those acquisition methods do not make a changing hosted snapshot equivalent to the pinned project input. Continue to observation and label preparation with the unchanged file or frame.