Data bars

Use bars when time-based candles do not match the studied behavior. Skip bars when fixed-interval klines already match the research rhythm.

Prerequisites

• a kline-like Polars frame with the columns required by the chosen bar type
• vaquum-limen[data] when the source frame comes from HistoricalData
• a declared research reason for replacing fixed-time rows with activity-threshold rows

Current scope

The implemented bar surface today is:

• volume_bars
• trade_bars
• liquidity_bars
• imbalance_bars
• run_bars

imbalance_bars and run_bars additionally support individual native trade points. They reject aggregated candles: their direction and stopping rules require the original ordered trade observations. Fixed trade-count bars can also aggregate individual points.

Adaptive trade-point bars

The input uses the shared native OHLCV schema, with one actual trade per row: open = high = low = close, no_of_trades = 1, positive price/volume, and the observed quote turnover, maker quantities and timestamp. The recorded Binance-trade proof in tests/test_adaptive_bars.py shows the conversion from the retained archive. Typed Parquet datetimes retain their original precision through HistoricalData.get_any_file; this keeps trade-point timestamps aligned with their native clocks. Event execution requires unique source clocks and retains the full constituent availability support.

Both functions use the carried price tick rule: upward/downward price changes set the sign, unchanged prices retain it, and the first direction is unknown (zero). An online EWMA with alpha = 2 / (ema_span + 1) tracks signed, positive and negative volume, initialized at zero. Before the first bar, estimates update through the warmup of expected_num_ticks. Thereafter the stopping threshold is frozen from the previous completed bar. Imbalance bars compare absolute cumulative signed volume with expected length times absolute expected signed volume; run bars compare the larger directional cumulative volume with expected length times the larger expected directional volume. Expected length updates by the same EWMA at each completed bar. Accumulators reset on completion and the trailing partial bar is omitted.

This is an explicit recursive-EWMA variant of the information-driven bar construction, with no claim of identical initialization to other packages. Trade points have base_interval = 0; their event clocks govern availability. Parameters can be swept through compute_data_bars(bar_type="imbalance" | "run", expected_num_ticks=..., ema_span=...) using existing manifest references.

When bars help

Bars help when each row should represent a comparable amount of market activity instead of a fixed amount of clock time.

Suitable fits:

• high-volatility periods where fixed-time candles contain very uneven activity
• strategies that depend more on activity intensity than wall-clock spacing
• research where volume or liquidity concentration matters more than elapsed time

Unsuitable fits:

• experiments where explicit time-of-day structure matters
• workflows that depend on regular calendar spacing
• cases where the added aggregation step makes the experiment harder to interpret than the base klines

Shared output schema

All supported bar functions return a pl.DataFrame with this shared schema:

The source dataframe must already contain the columns needed to compute the chosen bar type. Kline-style input must include fields such as volume, no_of_trades, and liquidity_sum.

Supported functions

volume_bars(data, volume_threshold)

Aggregate rows until cumulative volume reaches volume_threshold.

trade_bars(data, trade_threshold)

Aggregate rows until cumulative trade count reaches trade_threshold.

liquidity_bars(data, liquidity_threshold)

Aggregate rows until cumulative liquidity reaches liquidity_threshold.

imbalance_bars callable contract

1def imbalance_bars(data: pl.DataFrame, expected_num_ticks: int=100, ema_span: int=20) -> pl.DataFrame:2    ...

run_bars callable contract

1def run_bars(data: pl.DataFrame, expected_num_ticks: int=100, ema_span: int=20) -> pl.DataFrame:2    ...

Form bars in either project

The Python project's bars scenario forms each of the three supported bar types from 384 recorded hourly rows. It derives a diagnostic threshold from that bounded population, writes the complete bars, and records the omitted incomplete tail. This is a bar-formation diagnostic; a model comparison must still choose fitting and assessment populations separately.

1def bars(data_dir, output):2    raw = recorded(data_dir, 'spot_1h_20240101_20241231.parquet', 384)3    results = []4    for kind, parameter, column in [('volume', 'volume_threshold', 'volume'), ('trade', 'trade_threshold', 'no_of_trades'), ('liquidity', 'liquidity_threshold', 'liquidity_sum')]:5        median = raw[column].median()6        if not isinstance(median, (int, float)):7            raise ValueError('Bar formation requires a numeric recorded threshold source')8        threshold = float(median) * 49        if kind == 'trade':10            threshold = int(threshold)11        formed = compute_data_bars(raw, bar_type=kind, **{parameter: threshold})12        formed.write_parquet(output / (kind + '.parquet'))13        results.append({'kind': kind, 'threshold': threshold, 'rows': formed.height, 'source_rows_in_complete_bars': int(formed['bar_count'].sum()), 'volume': float(formed['volume'].sum()), 'first_open': formed['open'][0], 'last_close': formed['close'][-1], 'first_start': formed['datetime'][0].isoformat(), 'last_start': formed['datetime'][-1].isoformat(), 'last_available': (raw['datetime'][int(formed['bar_count'].sum()) - 1] + timedelta(hours=1)).isoformat()})14    return {'source_rows': raw.height, 'formed': results, 'incomplete_tail': 'omitted; do not treat it as a completed bar', 'trade_profile': trade_profile(data_dir, output),15            'adaptive': adaptive_trade_bars(data_dir, output / 'adaptive')}

The Compiled project's bars scenario runs three declared parameter choices. Its Manifest forms bars separately after each chronological split:

1    bar_formation:2      func: limen.data.utils.compute_data_bars3      params:4        bar_type: '{bar_type}'5        volume_threshold: '{volume_threshold}'6        trade_threshold: '{trade_threshold}'7        liquidity_threshold: '{liquidity_threshold}'

The complete configuration declares bar_type as volume, trade or liquidity, with volume threshold 1000.0, trade threshold 100000, and liquidity threshold 10000000.0. The scenario tests verify all three completed rounds; the Python diagnostic independently checks complete-bar boundaries and conserved volume. The Python diagnostic also processes the short recorded trade slice for CFR; that diagnostic is separate from the Compiled ML population.

For component authors, the optional API Manifest.set_bar_formation() attaches the bar step. set_required_bar_columns() asserts that the resulting frame retains the columns required downstream. Parameter references in the bar declaration must resolve from the current round.

compute_data_bars()

limen.data.utils.compute_data_bars() is Limen's convenience router for manifest-driven bar selection.

It currently supports these bar_type values:

• base
• trade
• volume
• liquidity
• imbalance
• run

base returns the input data unchanged. Fixed threshold types require their matching threshold parameter. Adaptive types use expected_num_ticks and ema_span. Unknown bar_type values raise ValueError; they are not silently treated as base.

How bars fit into the manifest pipeline

Inside a manifest-driven experiment, the order is:

1. fetch raw input data
2. optionally apply a pre-split selector
3. split into train, validation, and test
4. apply bar formation inside each split
5. run indicators, features, targets, and scaling on the resulting bars

This keeps train-only fitting and test-only evaluation aligned with the actual post-bar data seen by each fold.

Recorded trade points exercise adaptive formation

The adaptive cases use the archive's first recorded trade at each unique microsecond timestamp. This declared view contains 817 actual observations. The source derivation preserves price, quantity, quote turnover, maker fields, and exact clocks. The bundle pins the resulting Parquet file and checks its equality with the archive-derived frame.

The Python diagnostic forms imbalance and run bars over that complete declared view, with expected_num_ticks=20 and ema_span=10:

1def adaptive_trade_bars(data_dir, output):2    output.mkdir(parents=True)3    points = trade_points(data_dir)4    rows = []5    for kind in ('imbalance', 'run'):6        formed = compute_data_bars(points, bar_type=kind, expected_num_ticks=20, ema_span=10)7        path = output / (kind + '.parquet')8        formed.write_parquet(path)9        consumed = int(formed['bar_count'].sum())10        rows.append({'kind': kind, 'rows': formed.height, 'consumed_trade_rows': consumed,11                     'omitted_trade_rows': points.height - consumed,12                     'volume': float(formed['volume'].sum()),13                     'last_available_at_ns': int(points['available_at_ns'][consumed - 1]),14                     'sha256': hashlib.sha256(path.read_bytes()).hexdigest()})15    return {'trade_rows': points.height, 'expected_num_ticks': 20, 'ema_span': 10,16            'population': 'first recorded trade per unique microsecond timestamp', 'formed': rows}

Independent checks reconstruct tick direction, exponentially weighted estimates, stopping boundaries, and each completed bar's constituents. The result also records the omitted trailing observations and final consumed availability clock:

The Compiled case uses its own declared controls: five expected ticks, span 10, and two chronological folds with 225 raw test observations each. Bar formation occurs separately inside the native partitions. DLinear predicts the declared next formed-bar return.

1schema_version: '1.0'2metadata:3  name: recorded_adaptive_bars4  mode: development5sfd:6  manifest:7    type: ml8    data_source:9      method: limen.data.HistoricalData.get_any_file10      params:11        file_path_or_url: inputs/recorded-trade-points.parquet12        expected_sha256: '873faec863bbcc6656f902b17daf4d29a043b4f9d38b2872670920a3dac4bec5'13    split_walk_forward:14      n_folds: 215      test_bars: 22516      purge_bars: 117      embargo_bars: 118      anchored: true19    bar_formation:20      func: limen.data.utils.compute_data_bars21      params:22        bar_type: '{bar_type}'23        expected_num_ticks: '{expected_num_ticks}'24        ema_span: '{ema_span}'25    indicators:26    - func: limen.indicators.window_return27      params:28        period: 129    features:30    - func: limen.features.lag_range31      params:32        col: ret_133        start: 034        end: 235    target:36      name: next_return37      class: limen.targets.NextReturnTarget38    reference_architecture: limen.sfd.reference_architecture.dlinear_regressor39    backtest:40      fee_bps: 4.041      slip_bps: 0.042      notional_rate: 0.543      product:44        kind: linear_perpetual45        instrument: BTCUSDT46        base_currency: BTC47        quote_currency: USDT48        quantity_step: 1.0e-949        min_notional: 0.050  params:51    bar_type: [imbalance, run]52    expected_num_ticks: [5]53    ema_span: [10]54uel:55  output_path: compiled-bars-adaptive56  n_permutations: 257  prep_each_round: true58  search_strategy:59    type: grid60  output_format: csv61  recording:62    predictions: true63    alignment: true64    trade_ledger: true65    fold_details: true

1limen run --no-progress-bar compiled-research/bars.yaml2limen run --no-progress-bar compiled-research/bars_adaptive.yaml

The native checks reconstruct observed labels on the formed test bars and reconcile their prediction counts and reported errors. These folds retain an explicit clock promotion-unavailable reason: the irregular trade source has no admitted native interval for that fold-promotion route. Research execution and promotion eligibility remain separate contracts.

The 90-second archive supplies a bounded formation and pipeline diagnostic. A research assessment of a market hypothesis requires its own declared population, selection rule, and retained evidence.

Inspect the completed bars

The Python diagnostic reports each threshold and the number of source rows admitted into completed bars. Its incomplete tail remains excluded. The Compiled table counts completed bars inside each declared split, before model preparation. The fitting split loses its ROC warm-up; later splits carry the preceding completed bar as context. The forward label removes the final row in each split. The proof checks the test rows against the saved prediction alignment. last_available is the end of the final source interval consumed by a completed bar; datetime alone is its start, not its availability.

Read next

Continue to feature design to define the information available on the chosen bar clock. Helper details remain in the bar reference.