This page covers scaler selection, set_scaler() and set_scaler_from_params() behavior, and the custom scaler interface.
How Scalers Fit In Limen
The manifest pipeline is split-first:
- fetch and prepare raw data
- split into train, validation, and test
- build indicators, features, and target columns
- fit the configured scaler on
x_train - apply that exact fitted scaler to
x_valandx_test
That is why scalers live separately from the stateless helpers in Transforms.
Choosing A Scaler
| Scaler | Fit | Inverse support | Notes |
|---|---|---|---|
LogRegScaler |
the reference logistic-regression style feature sets used in foundational flows | yes | Uses a fixed per-column rule map. Columns outside the rule map are left alone. |
LinearScaler |
mixed feature sets needing regex-based scaling rules | yes | Broadest built-in scaler rule surface. Supports standard, log_standard, divide_100, and none. |
RobustScaler |
outlier-heavy numeric features | yes | Uses median and IQR instead of mean and standard deviation. |
CausalRollingRobustScaler |
non-stationary features whose scale drifts over time | no | Median and IQR from a strictly trailing rolling window, so no look-ahead. Not row-wise invertible from the fitted scaler. |
RankGaussScaler |
numeric features that benefit from a Gaussianized shape | approximate | The inverse is only approximate because rank-based transforms are lossy. |
Fit once, then reuse
The Python project fits a scikit-learn StandardScaler and Ridge model together. run_research retains that evaluated pipeline, and its Sensor uses the same fitted objects without fitting them again.
def fit(train, round_params): return make_pipeline(StandardScaler(), Ridge(alpha=round_params["alpha"])).fit( train.select("momentum"), train["next_return"].to_numpy() )The Compiled project resolves its scaler from the current round's scaler_type. Its selection configuration declares scaler_type: [logreg]; the Manifest fits LogRegScaler on x_train and reuses it on validation and test. Trainer later reconstructs the selected recorded definition with its original splits.
scaler: from_params: scaler_typeA fixed scaler uses Manifest.set_scaler(transform_class, param_name, extra_params). A searched scaler uses Manifest.set_scaler_from_params(param_name, extra_params), with a declared parameter whose values are registry keys. The current registry is:
SCALER_REGISTRY: dict[str, type] = {'linear': LinearScaler, 'logreg': LogRegScaler, 'robust': RobustScaler, 'rank_gauss': RankGaussScaler, 'causal_rolling_robust': CausalRollingRobustScaler}The class definitions below describe alternative scaling policies. The two selection runs prove the fitted pipelines shown above; they do not claim to search every registered scaler.
Built-In Scalers
LogRegScaler
LogRegScaler(x_train) uses a fixed column-to-rule mapping tailored to the classic Limen logistic-regression workflow.
- standardizes columns such as
open,close,atr,macd,roc, andreturns - log-standardizes columns such as
volume,no_of_trades, and liquidity fields - divides
wilder_rsiby100 - leaves columns such as
maker_ratiounchanged
This is the logistic-regression-oriented scaler in the package. It remains the legacy default for old-style foundational SFDs and exposes fewer rules than LinearScaler.
LinearScaler
LinearScaler(x_train, rules=None, default='standard') applies regex-driven scaling rules.
It supports:
standardlog_standarddivide_100none
Use set_scaler_from_params() for explicit scaling-policy control or search-time scaler choice.
RobustScaler
RobustScaler(x_train, quantile_range=(0.25, 0.75)) applies (x - median) / IQR.
It skips datetime and non-numeric columns automatically and fits heavy-tail or outlier-distorted standardization cases.
RankGaussScaler
RankGaussScaler(x_train, n_quantiles=1000) maps numeric columns to a quantile-Gaussian distribution through quantiles and the inverse normal CDF.
Use it when relative ordering matters more than preserving original spacing. Its inverse transform is approximate, not exact.
CausalRollingRobustScaler
CausalRollingRobustScaler(x_train, window=1000, quantile_range=(0.25, 0.75), clip=8.0, min_samples=50) applies (x - rolling_median) / rolling_IQR.
The median and IQR for each row are taken from the window rows strictly before it (.shift(1)), so the transform never reads the current row or any future row. That makes it suited to non-stationary series whose scale drifts over time.
Rows with fewer than min_samples of trailing history fall back to the median and IQR fitted on x_train, so during warmup it degrades to plain RobustScaler behavior. The scaled output is clipped to +/- clip.
It skips datetime and non-numeric columns automatically. It provides no inverse transform: each row's scaling factors are derived from the data itself, so the transform is not row-wise invertible from the fitted scaler alone.
The fitted instance exposes context_rows = window. The experiment pipeline reads this property to size the scaler warm-up portion of the Context Carry-Over (CCO) block prepended to val and test splits, ensuring the scaler sees enough preceding rows to be fully warm from the very first split row.
Scalers without a context_rows property (such as RobustScaler and StandardScaler) contribute zero to the scaler warm-up. CCO may still run for those scalers when indicators produce leading warm-up rows — those rows are prepended as indicator context regardless of scaler type. The total CCO block is indicator_warm_up_rows + scaler_context_rows.
Custom Scaler Contract
A Manifest scaler accepts training features at construction and exposes transform(df) -> pl.DataFrame. Store learned statistics during construction; transform later splits with those stored statistics. There is no required base class. RobustScaler exposes the concrete package signatures:
def __init__(self, x_train: pl.DataFrame, quantile_range: tuple[float, float]=(0.25, 0.75)) -> None: ...def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...The package's inverse helper is a separate callable accepting the frame and fitted scaler:
def inverse_transform(df: pl.DataFrame, scaler: RobustScaler) -> pl.DataFrame: ...An inverse is optional and must reflect the information the transform retains. A rolling scaler cannot promise a row-wise inverse from fitted state alone. Python Research may instead use any caller-owned preprocessing object, including the pipeline above; it does not require adopting the Manifest scaler convention.
Practical Notes
RobustScaler,CausalRollingRobustScaler, andRankGaussScalerautomatically skip datetime and non-numeric columns.LogRegScalerandLinearScalerare rule-driven, so only columns matched by their rule sets are transformed.LinearScalerfits cases where new feature names are expected to appear frequently.- If a prediction post-processing step needs original scale values, prefer a scaler with a meaningful inverse path.
Read Next
- Transforms for stateless preprocessing and post-model helpers
- Experiment Manifest for where scaling happens inside the split-first pipeline
- Trainer for the replay path that reconstructs manifests and preserves fitted preprocessing behavior
Callable contracts
These entries give the exact public signatures and the source-owned argument, return and failure contracts. Use the project examples above for the research context; a signature is reference evidence, not a claim that every option ran in the base experiment.
limen.scalers.CausalRollingRobustScaler
class CausalRollingRobustScaler: ...Causal trailing-window median and IQR scaling, with no look-ahead.
limen.scalers.CausalRollingRobustScaler.__init__
def __init__(self, x_train: pl.DataFrame, window: int=1000, quantile_range: tuple[float, float]=(0.25, 0.75), clip: float=8.0, min_samples: int=50) -> None: ...Fit fallback median and IQR statistics on training data.
The fitted statistics are not applied directly. They are the warmup
fallback used for rows whose trailing window holds fewer than
min_samples observations, where this scaler degrades gracefully to
plain median and IQR scaling.
| Argument | Contract |
|---|---|
x_train (pl.DataFrame) |
Training data |
window (int) |
Trailing window length, in rows, for rolling statistics |
quantile_range (tuple[float, float]) |
Lower and upper quantiles for IQR |
clip (float) |
Symmetric bound applied to the scaled output |
min_samples (int) |
Minimum trailing observations before a rolling statistic is used |
limen.scalers.CausalRollingRobustScaler.context_rows
@propertydef context_rows(self) -> int: ...Number of raw preceding rows needed to produce a fully warm scaled output.
limen.scalers.CausalRollingRobustScaler.transform
def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...Apply causal rolling robust scaling: (x - rolling_median) / rolling_IQR.
For each row the median and IQR are taken from the trailing window
rows strictly before it (.shift(1)), so the transform never reads
the current row or any future row. Rows with fewer than min_samples
trailing observations fall back to the train-fitted median and IQR.
The scaled output is clipped to +/- clip.
| Argument | Contract |
|---|---|
df (pl.DataFrame) |
Data to transform |
Returns: pl.DataFrame: Transformed data
limen.scalers.LinearScaler
class LinearScaler: ...limen.scalers.LinearScaler.__init__
def __init__(self, x_train: pl.DataFrame, rules: dict[str, str] | None=None, default: str='standard') -> None: ...Linear transformation utility for scaling features.
| Argument | Contract |
|---|---|
| x_train | Training DataFrame. |
| rules | Regex-to-rule mapping. |
| default | Fallback scaling rule. |
limen.scalers.LinearScaler.transform
def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...Apply linear scaling transformation.
| Argument | Contract |
|---|---|
| df | DataFrame to transform. |
Returns: Transformed DataFrame.
limen.scalers.LogRegScaler
class LogRegScaler: ...LogRegScaler class for scaling and inverse scaling data.
limen.scalers.LogRegScaler.__init__
def __init__(self, x_train: pl.DataFrame) -> None: ...Initialize the LogRegScaler object.
| Argument | Contract |
|---|---|
x_train (pl.DataFrame) |
The training data |
limen.scalers.LogRegScaler.transform
def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...Transform the data using the scaling rules.
| Argument | Contract |
|---|---|
df (pl.DataFrame) |
The input DataFrame |
Returns: pl.DataFrame: The transformed DataFrame
limen.scalers.RankGaussScaler
class RankGaussScaler: ...Rank transformation to Gaussian distribution via inverse normal CDF.
limen.scalers.RankGaussScaler.__init__
def __init__(self, x_train: pl.DataFrame, n_quantiles: int=1000) -> None: ...Fit scaler by storing training quantile boundaries per column.
| Argument | Contract |
|---|---|
x_train (pl.DataFrame) |
Training data |
n_quantiles (int) |
Number of quantile bins for rank interpolation |
limen.scalers.RankGaussScaler.transform
def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...Compute rank-based Gaussian transformation.
For each column: rank values against training quantiles,
convert to uniform [eps, 1-eps], apply inverse normal CDF.
NaN values are preserved as NaN in the output.
| Argument | Contract |
|---|---|
df (pl.DataFrame) |
Data to transform |
Returns: pl.DataFrame: Transformed data with approximately normal distributions
limen.scalers.RobustScaler
class RobustScaler: ...Median and IQR scaling, resilient to outliers.
limen.scalers.RobustScaler.__init__
def __init__(self, x_train: pl.DataFrame, quantile_range: tuple[float, float]=(0.25, 0.75)) -> None: ...Fit scaler on training data using median and interquartile range.
| Argument | Contract |
|---|---|
x_train (pl.DataFrame) |
Training data |
quantile_range (tuple[float, float]) |
Lower and upper quantiles for IQR |
limen.scalers.RobustScaler.transform
def transform(self, df: pl.DataFrame) -> pl.DataFrame: ...Apply robust scaling: (x - median) / IQR.
| Argument | Contract |
|---|---|
df (pl.DataFrame) |
Data to transform |
Returns: pl.DataFrame: Transformed data