Fractional Differentiation

The Python project calls the feature explicitly inside each partition and owns its fitted preprocessing. The Compiled project declares the same feature inside a native Manifest, with split preparation, CCO, ablation and compression. Their independent tests check the actual prepared values and training fits.

What This Guide Covers

• why stationarity matters for a model and what memory you lose with plain differencing
• how to find the minimum differencing order with an ADF test
• how to add a fractionally differentiated feature to a manifest or a YAML experiment
• the exact output columns produced and the edge cases to know

Prerequisites

• A working manifest or YAML experiment. Start with Experiment Manifest if you do not have one.
• The stats extra installed for the ADF helper: pip install vaquum-limen[stats]. find_min_d and adf_test depend on statsmodels.

Why Stationarity, And Why Not Just Take Returns

Stationarity concerns a series' distribution over time. Stable mean and variance are useful diagnostics, but do not establish strict stationarity. Differencing price levels and taking returns are distinct transformations; neither is universally stationary or memoryless.

At d=0, this helper copies the price column. At d=1, it subtracts the previous price. Intermediate orders combine prior observations with recursively derived weights. Fixed-width truncation discards sufficiently small weights, so the implemented history is finite. Choosing a smaller order can retain more level information; it does not prove that information is useful for the model.

Finding The Minimum d

find_min_d sweeps d from d_start to d_end in steps of step, fractionally differencing the column at each value and testing the result with an Augmented Dickey-Fuller (ADF) test. It returns the first d whose transformed training window passes its ADF criterion.

1def find_min_d(data: pl.DataFrame, col: str, d_start: float=0.0, d_end: float=1.0, step: float=0.1, significance_level: float=0.05, threshold: float=1e-05) -> float:2    ...

The return value is a single float: the first tested d that passed. If no tested d passes, it returns d_end. Very short series cannot be tested and also return d_end — the ADF test needs enough observations to be meaningful.

The stationarity call itself comes from adf_test, which wraps statsmodels' adfuller (with autolag='AIC') and reports a small result object:

1def adf_test(series: pl.Series, significance_level: float=0.05) -> AdfResult:2    ...

adf_test returns a non-stationary, inconclusive result (stationary=False, p_value=1.0) for empty or constant series rather than raising, so a sweep over a degenerate column degrades gracefully. See Utilities for the full adf_test and AdfResult reference.

Choose the order and window

An order chosen on one series and window is not a transferable stationarity claim. Keep diagnostic selection inside training and retain the test result as well as the returned order: d_end also represents the no-passing-order case.

Choose a finite order grid and weight threshold before inspecting assessment results. The Python project diagnoses the training-only grid 0.0, 0.2, 0.4; both projects separately use fixed order 0.4 and truncation threshold 0.05 for their prepared-matrix comparison.

Applying the helper to log prices is a separate modeling choice. Declare that transformation explicitly rather than treating the raw-close project case as a reproduction of every AFML recommendation.

Adding It As A Feature

fractional_diff is a feature-library function. It adds one new column per input column and preserves the originals, so indicators that depend on the raw price still work.

1def fractional_diff(data: pl.DataFrame, d: float=0.0, cols: list[str] | None=None, threshold: float=1e-05) -> pl.DataFrame:2    ...

It raises ValueError if cols is None, empty, a scalar string, or names missing input columns; if d is non-finite or negative; if threshold is non-finite or nonpositive; or if a recurrence weight becomes non-finite.

Python Research

This is the complete Python preparation case used by its search scenario. Its training-only ADF diagnostic and fixed-order comparison are separate decisions. Execute it through the complete project, including its imports and recorded-data helpers:

1def preparation(data_dir, output):2    from sklearn.decomposition import PCA3    from sklearn.linear_model import Ridge4    from limen.features import fractional_diff, find_min_d5    from limen.indicators import roc6    from limen.scalers import RobustScaler7    from limen.targets import TripleBarrierTarget8    from limen.utils import adf_test9 10    raw = hourly(data_dir, 1440)11    partitions = [raw[:864], raw[864:1152], raw[1152:1440]]12    d = find_min_d(partitions[0], 'close', d_end=.4, step=.2, threshold=.05)13    stationarity = adf_test(fractional_diff(partitions[0], d=d, cols=['close'], threshold=.05)['close_fracdiff'])14    cases = []15    for compressed in (False, True):16        name = 'compressed' if compressed else 'baseline'17        directory = output / name18        directory.mkdir(parents=True)19        target = TripleBarrierTarget(partitions[0], 'label', upper_multiple=1.5,20                                     lower_multiple=1., max_horizon=3, span=8, min_periods=8)21        columns = ['close', 'roc_3', 'roc_12', 'close_fracdiff']22        dropped = ['roc_12'] if compressed else []23        features = [column for column in columns if column not in dropped]24        prepared = []25        for partition in partitions:26            frame = fractional_diff(roc(roc(partition, period=3), period=12),27                                    d=.4, cols=['close'], threshold=.05)28            prepared.append(target.transform(frame).select('datetime', *features, 'label').drop_nulls())29        scaler = RobustScaler(prepared[0].select(features))30        scaled = [scaler.transform(frame.select(features)) for frame in prepared]31        pca = PCA(n_components=2, svd_solver='full') if compressed else None32        matrices = ([pca.fit_transform(scaled[0].to_numpy()),33                     *[pca.transform(frame.to_numpy()) for frame in scaled[1:]]]34                    if pca is not None else [frame.to_numpy() for frame in scaled])35        names = ['pc_0', 'pc_1'] if compressed else features36        model = Ridge(alpha=1., solver='svd').fit(matrices[0], prepared[0]['label'].to_numpy())37        predictions = model.predict(matrices[2])38        for split, frame, matrix in zip(('train', 'val', 'test'), prepared, matrices, strict=True):39            pl.DataFrame(matrix, schema=names, orient='row').with_columns(40                frame['datetime'], frame['label']).write_parquet(directory / (split + '.parquet'))41        cases.append({'name': name, 'features': features, 'dropped': dropped,42                      'rows': [frame.height for frame in prepared],43                      'medians': scaler.medians, 'iqrs': scaler.iqrs,44                      'pca_mean': None if pca is None else pca.mean_.tolist(),45                      'components': None if pca is None else pca.components_.tolist(),46                      'predictions': predictions.tolist(),47                      'rmse': float(np.sqrt(np.mean((predictions-prepared[2]['label'].to_numpy())**2)))})48    return {'cases': cases, 'partition_rows': [864, 288, 288], 'carried_context': False,49            'fixed_d': .4, 'weight_threshold': .05, 'adf_grid_result': d,50            'adf_stationary': bool(stationarity.stationary), 'adf_p_value': float(stationarity.p_value)}

Compiled Research

The Compiled YAML declares fixed-order fractional differentiation as a feature. The pipeline carries its admitted prior context into later partitions and fits robust scaling and optional PCA only on training. The complete project executes this declaration through the CLI:

1    features:2      - func: limen.features.lag_range3        params:4          col: roc_15          start: 16          end: 37          group: momentum8      - func: limen.features.fractional_diff9        params:10          d: 0.411          cols: [close]12          threshold: 0.0513          group: price

YAML resolves the same feature through the fields above; a fixed numeric order requires no search reference.

Output Columns

For each column in cols, fractional_diff adds {col}_fracdiff — so close produces close_fracdiff. The suffix is fixed (FRACDIFF_SUFFIX = '_fracdiff'). The original column is always kept.

When d = 0.0, the new column is a Float64 copy of the original. When d = 1.0, it equals the standard first difference (np.diff with a leading null). For 0 < d < 1 it is the fixed-width fractionally differenced series, with leading rows null until enough history exists to fill the weight window.

NOTE: Because the leading rows are null until the weight window fills, a short split can end up without a valid _fracdiff column while a longer split has one. During split alignment the manifest drops the extra column so the feature schema stays consistent across train, validation, and test. See Experiment Manifest for the alignment behavior.

Expected Artifacts

The project cases save baseline/ and compressed/ prepared Parquet partitions under their search scenario's preparation/ directory. The baseline retains the fractional feature; the compressed case drops roc_12 and replaces the surviving matrix with two fitted components. Ordinary round logs do not promise to serialize prepared feature matrices. A UEL sweep records its order parameter and metrics in the ordinary results; its payloads follow the recording policy.

Where To Look

• limen/features/fractional_diff.py — fractional_diff, find_min_d, and the fixed-width weight helpers
• limen/utils/adf_test.py — adf_test, AdfResult
• limen.features.fractional_diff — the resolvable path for YAML func: and manifest add_feature

Read Next

• Features for the full feature-library reference, including the stationarity helpers.
• Utilities for the adf_test and AdfResult reference.
• Experiment Manifest for add_feature, parameter references, and split alignment.
• Perturbation Strategies to search feature and model choices robustly once the feature is in place.