Skip to main content

Advanced Search

Advanced search is Limen's artifact-backed extension path. It adds a mutable parameter domain, a search strategy abstraction, checkpointing, resumability, and mid-run feedback on top of the same UEL engine used by limen run.

This page covers:

  • a search strategy that is not the legacy ParamSpace sweep
  • a durable experiment_dir with resumable artifacts
  • mid-run interventions that can mutate the remaining search space
  • promotion-ready runs for Trainer

For a first experiment, stay on the YAML/CLI path. Advanced search is real and supported, but it is the extension-oriented layer of Limen.

Prerequisites​

  • a working SFD and bounded data source
  • an artifact experiment_dir
  • a ParamDomain and registered SearchStrategy
  • explicit proof that adaptive feedback is appropriate for the research design

The core pieces​

PieceWhat it owns
ParamDomainthe mutable legal values for each parameter
SearchStrategyhow new combinations are generated from the domain
MSQthe mutable search queue that applies interventions, filters, injections, and trims
FeedbackControllerreducer, callback, and file-driven feedback cycles
CheckpointManagerdurable save and restore of advanced-run state

The important mental model is:

  1. SearchStrategy proposes combinations
  2. MSQ decides what is still legal
  3. feedback mutates MSQ and sometimes the underlying ParamDomain
  4. checkpoints preserve enough state to continue later with resume=True

Minimal custom SearchStrategy​

Limen ships two built-in strategies (GridStrategy for exhaustive search, RandomStrategy for lazy sampling) and the SearchStrategy abstraction for custom strategies. A strategy must at minimum:

  • hold a reference to a shared ParamDomain
  • yield dictionaries of round parameters
  • expose get_state() and set_state() for checkpointing
  • react to domain changes if its internal state depends on the current domain

This is the smallest practical exhaustive strategy shape:

import itertools

from limen.experiment.param_domain import ParamDomain
from limen.experiment.param_search import SearchStrategy


class MiniGrid(SearchStrategy):
    @property
    def is_finite(self):
        return True

    def __init__(self, domain: ParamDomain, *, seed: int | None = None):
        super().__init__(domain, seed=seed)
        self._combos = self._build_combos()
        self._index = 0

    def _build_combos(self):
        params = self._domain.params
        keys = sorted(params.keys())
        return [
            dict(zip(keys, vals, strict=True))
            for vals in itertools.product(*(params[k] for k in keys))
        ]

    def __next__(self):
        if self._index >= len(self._combos):
            raise StopIteration
        combo = self._combos[self._index]
        self._index += 1
        self._generated_count += 1
        return combo

    def on_domain_changed(self, _domain, _changed_params):
        self._combos = self._build_combos()
        self._index = 0

    def get_state(self):
        return {'index': self._index, 'generated_count': self._generated_count}

    def set_state(self, state):
        self._index = state['index']
        self._generated_count = state['generated_count']

First artifact-backed run​

import limen

from limen.experiment.param_domain import ParamDomain
from limen.experiment.reducer import BudgetReducer

domain = ParamDomain(limen.sfd.random_binary.params())
strategy = MiniGrid(domain)

uel = limen.UniversalExperimentLoop(
    sfd=limen.sfd.random_binary,
    search_strategy=strategy,
    pruning_strategies=[
        BudgetReducer(max_permutations=4, check_after_pct=0.25),
    ],
    feedback_interval=2,
    checkpoint_interval=3,
    experiment_dir='advanced-budget',
)

uel.run(
    experiment_name='advanced-budget',
    n_permutations=6,
)

In the bundled budget-reducer example, this requested 6 permutations but finished with:

  • 4 rows in results.csv
  • 4 entries in round_data.jsonl
  • 1 feedback-cycle entry in audit.jsonl
  • a checkpoint saved after round 3

The reducer intervention that caused the trim was:

{'op': 'trim', 'target_count': 4, 'reason': 'budget trim to 4 total permutations'}

What experiment_dir stores​

When experiment_dir is set, advanced search writes:

FileMeaning
results.csvstreaming round log
round_data.jsonlround params, stored predictions, and alignment metadata
checkpoint.jsonsearch state for resume
audit.jsonlfeedback-cycle audit trail
metadata.jsonexperiment metadata used by Trainer
interventions.jsonoptional input file polled by the feedback controller when the file exists

Unlike the other files above, interventions.json is not produced by the run. It is an optional external control file that the feedback controller watches when experiment_dir is present.

Resume flow​

Resumption belongs to the advanced path only.

uel.run(
    experiment_name='advanced-budget',
    n_permutations=6,
    resume=True,
)

In the shutdown-and-resume example:

  • the first phase stopped after 2 completed rounds
  • round_data.jsonl and results.csv each contained 2 entries
  • the resumed phase finished the remaining work
  • the final files contained 4 rows with round ids 0, 1, 2, 3

For resume to work cleanly, keep these stable:

  • the same experiment_dir
  • the same SearchStrategy type
  • the same parameter content
  • the same configured reducer stack

What advanced search adds beyond the normal run path​

Compared to a normal YAML CLI run or direct standard UEL sweep, advanced search adds:

  • mutable pruning and focus during a run
  • stored round-level predictions and alignment data
  • auditability of feedback cycles
  • resumability after interruption
  • the artifact contract that Trainer expects

What it does not change:

  • SFDs still define the research unit
  • manifests still govern split-first prep
  • post-run analysis still flows through Log