A foundational SFD implements pipeline components used by the Compiled YAML/CLI project. The Python project shows the caller-owned alternative; a new template must not become a whitelist for ordinary Python research.
Foundational SFDs are Limen's reference-grade experiment templates. They are not one-off research scripts. A foundational SFD should be reusable, reviewable, and analytically justified inside a large Limen scan.
Prerequisites
- a Limen development environment installed with
python -m pip install -e ".[dev]" - the optional model and data extras required by the proposed SFD
- an approved capability scope and reproducible benchmark plan
What a foundational SFD owns
A foundational SFD packages:
- a
params()search space - a manifest-driven experiment pipeline
- a reference-architecture model function
Canonical example:
- foundational SFD:
limen/sfd/foundational_sfd/logreg_binary.py - reference architecture:
limen/sfd/reference_architecture/logreg_binary.py
The foundational SFD owns experiment design and parameter exposure. The reference architecture owns the train, predict, and evaluate logic for the model family itself.
Before work starts
Do the research work first. A foundational SFD proposal should answer:
- what problem or modeling thesis this SFD is meant to capture
- why this reference architecture fits the thesis
- which indicators and features are justified
- how the target should be constructed
- which scaler or transform choices belong in the default design
- which parameters materially affect the search space
If these answers remain unsupported, the design is not ready.
Design rules
- Keep the SFD manifest-driven.
- Put experiment intelligence in
params()andmanifest(), not inside bespoke hidden code paths. - Put model-specific fitting logic in the reference architecture.
- If a workflow improvement is reusable, contribute it as a shared Limen building block instead of hiding it inside one SFD.
- Expose only meaningful search dimensions. A parameter that does not materially change the experiment should not be in
params().
Contribution surface
A foundational SFD may compose these existing Limen building blocks:
- data access and selection
- optional bar formation
- indicators
- features
- target construction
- scalers
- transforms
- reference-architecture models
If a needed building block does not exist yet, add it in the right package first:
limen.datafor retrieval or bar-prep logiclimen.indicatorsfor low-level signal primitiveslimen.featuresfor derived signals and target helperslimen.transformsfor stateless transform helperslimen.scalersfor train-fitted preprocessing
Reference shape
A foundational SFD exposes the two entry points below. These are current package signatures; they describe the contribution boundary rather than an untested replacement template.
def params() -> dict[str, list[Any]]: ...def manifest() -> Manifest: ...The Compiled project's selection commands validate, profile and execute unchanged YAML, then reconstruct validation-selected Sensors through CLI assessment. Component APIs remain optional integration reference:
limen validate compiled-research/selection.yamllimen profile compiled-research/selection.yamllimen run --dry-run compiled-research/selection.yamllimen run --no-progress-bar compiled-research/selection.yamllimen assess compiled-research/selection.yaml --results results/dev/compiled-cliThe foundational factory declares Manifest (the base class) even though the body constructs MLManifest. This keeps the interface uniform across all foundational SFDs. For rule-based SFDs, use RuleBasedManifest instead — it provides with_strategy() and does not expose scalers or ablation.
That shape captures the required properties:
- split-first manifest execution
- explicit parameter exposure
- reusable shared building blocks
- no hidden manual glue
Review checklist
Before review, check all of the following:
- the thesis is explicit and literature-backed or otherwise justified
params()exposes real search dimensions- the manifest is readable and uses shared Limen primitives where possible
- target construction is split-safe
- scaler choice is appropriate for the feature surface
- the reference architecture is the right ownership boundary for model logic
- docs and docstrings are updated for any new public helpers added along the way
- the SFD can run inside Limen without custom manual intervention
Anti-patterns
Avoid these:
- baking reusable workflow logic into one SFD only
- turning every implementation detail into a parameter
- mixing model-family logic into the foundational SFD when it belongs in the reference architecture
- introducing a new helper in an arbitrary location just because the SFD needs it once
- writing the SFD for a one-off dataset or private workflow assumption
Deliverables
A foundational SFD contribution includes:
- a short research thesis or rationale
- the foundational SFD file
- any new shared helpers the design genuinely requires
- updated docs for any new public surfaces
- a template and registry/reference entry with declared task, units and optional extras
- a real recorded-data case through every admitted user interface
- independent assertions for training-only preparation and returned predictions
- an explicit reconstruction disposition: validated replay, stochastic tolerance or a named unsupported boundary
- updated source provenance, exact excerpts and publication evidence