ictonyx.runners

ExperimentRunner, VariabilityStudyResults, GridStudyResults.

Experiment runners for variability studies with memory management. Supports both standard and process-isolated execution modes.

class ictonyx.runners.ExperimentRunner(model_builder, data_handler, model_config, tracker=None, use_process_isolation=False, gpu_memory_limit=None, seed=None, verbose=True)[source]

Bases: object

Core engine for running variability studies with memory management.

Trains a model multiple times with different random seeds, collecting per-epoch metrics from each run. Supports standard in-process execution and subprocess isolation for GPU memory cleanup.

This class is the low-level interface. Most users should prefer variability_study(), which handles data resolution and model wrapping automatically.

Parameters:

model_builder (Callable[[ModelConfig], BaseModelWrapper]) – A callable f(ModelConfig) -> BaseModelWrapper that constructs a fresh model for each run.
data_handler (DataHandler) – A DataHandler that provides train/val/test splits via its load() method.
model_config (ModelConfig) – A ModelConfig containing training hyperparameters (epochs, batch_size, learning_rate, etc.).
tracker (BaseLogger | None) – An optional BaseLogger for experiment tracking. Defaults to a basic in-memory logger.
use_process_isolation (bool) – If True, each run executes in a child process, ensuring GPU memory is fully released between runs. Required for long Keras/TF studies that would otherwise OOM. Default False.
gpu_memory_limit (int | None) – Optional GPU memory cap in MB, applied per run when using TensorFlow. Default None (no limit).
seed (int | None) – Base random seed for reproducibility. Each run receives an independent child seed derived via np.random.SeedSequence.spawn(), which guarantees statistically uncorrelated RNG streams. If None, a random seed is generated and stored.
verbose (bool) – If True, log study progress to stdout. If tqdm is installed, a progress bar replaces per-run log messages. Default True.

__init__(model_builder, data_handler, model_config, tracker=None, use_process_isolation=False, gpu_memory_limit=None, seed=None, verbose=True)[source]

Initialize the experiment runner.

See class docstring for full parameter descriptions.

Parameters:

model_builder (Callable[[ModelConfig], BaseModelWrapper])
data_handler (DataHandler)
model_config (ModelConfig)
tracker (BaseLogger | None)
use_process_isolation (bool)
gpu_memory_limit (int | None)
seed (int | None)
verbose (bool)

run_study(num_runs=20, epochs_per_run=None, stop_on_failure_rate=0.8, checkpoint_dir=None, use_parallel=False, n_jobs=-1)[source]

Execute the complete variability study.

Trains the model num_runs times, collecting per-epoch training metrics from each run. Resets all internal accumulators before starting, so calling run_study() twice on the same runner is safe.

If tqdm is installed and verbose=True, a progress bar is displayed with a live postfix showing the most informative validation metric from the latest completed run.

Parameters:

num_runs (int) – Number of independent training runs. Default 20.
epochs_per_run (int | None) – Epochs per run. If None, uses the epochs value from model_config. Default None.
stop_on_failure_rate (float) – If the fraction of failed runs exceeds this threshold, the study halts early. Set to 1.0 to never stop early. Default 0.8.
checkpoint_dir (str | None) – Optional path to a directory for saving progress after each completed run. If the directory contains a checkpoint.pkl file from a previous interrupted run, execution resumes from where it left off. Default None (no checkpointing).
use_parallel (bool) – If True, fan training runs across multiple processes using joblib.Parallel. Safe for sklearn models. Not recommended for Keras/TF. Mutually exclusive with use_process_isolation. Default False.
n_jobs (int) – Number of parallel workers. -1 uses all available CPUs. Ignored when use_parallel=False. Default -1.

Returns:

VariabilityStudyResults with per-run DataFrames, final metric distributions, and optional test-set metrics.

Return type:

VariabilityStudyResults

get_summary_stats()[source]

Get summary statistics for the completed study.

Returns:

total_runs, successful_runs, failed_runs, failure_rate — run accounting.
<metric>_mean, <metric>_std, <metric>_min, <metric>_max — descriptive statistics for each tracked metric (e.g. val_accuracy_mean).

Return type:

Dict with keys

class ictonyx.runners.VariabilityStudyResults(all_runs_metrics, final_metrics, final_test_metrics, seed=None, run_seeds=<factory>)[source]

Bases: object

Container for variability study results with analysis methods.

Returned by variability_study() and ExperimentRunner.run_study(). Holds per-run training history DataFrames, aggregated final-epoch metrics, and optional test-set evaluation results.

Parameters:

all_runs_metrics (List[DataFrame])
final_metrics (Dict[str, List[float]])
final_test_metrics (List[Dict[str, Any]])
seed (int | None)
run_seeds (List[int])

all_runs_metrics

List of DataFrames, one per successful run. Each DataFrame has one row per epoch and columns for every tracked metric (e.g. train_accuracy, val_loss).

Type:: List[pandas.DataFrame]

final_metrics

Dict mapping metric names to lists of final-epoch values across runs. Example: {'val_accuracy': [0.82, 0.85, 0.83]}.

Type:: Dict[str, List[float]]

final_test_metrics

List of dicts, one per run, containing test-set evaluation results (empty if no test set was used).

Type:: List[Dict[str, Any]]

seed

The base random seed used for the study, for reproducibility.

Type:: int | None

run_seeds

List of per-run child seeds generated from the base seed via SeedSequence.spawn(). Empty when results are reconstructed from MLflow or loaded from JSON.

Type:: List[int]

all_runs_metrics: List[DataFrame]

final_metrics: Dict[str, List[float]]

final_test_metrics: List[Dict[str, Any]]

seed: int | None = None

run_seeds: List[int]

property n_runs: int: Number of successful runs.

property has_test_data: bool: True if at least one run produced test-set metrics.

get_metric_values(metric_name)[source]

Get collected final values for a specific metric.

Parameters:: metric_name (str) – Metric key, e.g. ‘val_accuracy’, ‘train_loss’, ‘val_f1’
Returns:: List of final-epoch values, one per run.
Raises:: KeyError – If metric was not tracked.
Return type:: List[float]

get_final_metrics(metric_name='val_accuracy')[source]

Extract final metric values for each run (labeled run_1, run_2, …).

Deprecated since version Emits: UserWarning. Use get_metric_values() instead. get_final_metrics() will be removed in v0.5.0.

Parameters:: metric_name (str)
Return type:: Dict[str, float]

get_available_metrics()[source]

Get list of all available metrics across all runs.

Return type:: List[str]

to_dataframe()[source]

Convert results to a summary DataFrame with one row per run.

Test metrics are aligned by run_id, not list position, so a failed test evaluation for one run does not corrupt the alignment of subsequent runs.

Return type:: DataFrame

preferred_metric(base='accuracy', context='scalar')[source]

Return the preferred metric name for this study.

The preferred metric depends on the context in which it will be used:

scalar (default): Terminal metric reporting. Returns f"test_{base}" when test data is present and the base metric was tracked, f"val_{base}" otherwise.
epoch: Per-epoch plotting and analysis. Always returns f"val_{base}" because test metrics are scalar (single final value) and per-epoch plotters cannot resolve them.

Before returning, verifies the resolved key exists in the study’s tracked metrics. If neither the primary nor alternate prefix resolves, raises a helpful KeyError naming the attempted keys, what metrics are tracked, and how to pass base explicitly.

Note

final_test_metrics stores metrics under bare keys (e.g. "accuracy", not "test_accuracy"). This method compares against the bare key and returns the prefixed form.

Parameters:

base (str) – Base metric name without prefix. Default "accuracy".
context (Literal['scalar', 'epoch']) – "scalar" for terminal reporting (prefers test_* when available), "epoch" for per-epoch plotters (always val_*). Default "scalar" for backward compatibility.

Returns:

The most appropriate metric key for this study’s results.

Raises:

KeyError – If neither the primary candidate nor the alternate prefix resolves to a tracked metric on this study. The error message names the attempted keys, the available val/test metrics, and suggests how to fix the call.

Return type:

str

summarize()[source]

Return type:: str

get_epoch_statistics(metric, confidence=0.95)[source]

Compute per-epoch mean, SD, SE, and confidence band across all runs.

Aligns all runs at each epoch position. Runs with fewer epochs than the maximum are excluded from epochs beyond their last.

Parameters:

metric (str) – Column name in per-run DataFrames, e.g. 'val_accuracy', 'train_loss'.
confidence (float) – Confidence level for the t-based confidence interval on the per-epoch mean across runs. The interval uses a Student t-distribution with n_runs - 1 degrees of freedom. Default 0.95.

Returns:

epoch, mean, sd, se, ci_lower, ci_upper, n_runs.

Return type:

DataFrame with columns

Raises:

ValueError – If no runs are available.
KeyError – If metric is not found in any run’s DataFrame.

Example:

stats = results.get_epoch_statistics("val_accuracy")
import matplotlib.pyplot as plt
plt.fill_between(
    stats["epoch"], stats["ci_lower"], stats["ci_upper"],
    alpha=0.2, label="95% CI"
)
plt.plot(stats["epoch"], stats["mean"], label="Mean")
plt.legend()

get_test_metric_values(metric)[source]

Get per-run final values for a test-set metric.

Accepts either bare key ('accuracy') or prefixed form ('test_accuracy') — both are resolved against the bare key, which is how final_test_metrics actually stores values.

Parameters:: metric (str) – Metric key, e.g. 'accuracy' or 'test_accuracy'.
Returns:: List of values, one per run, in run order.
Raises:: KeyError – If the metric was not tracked or no test data was provided.
Return type:: List[float]

test_against_null(null_value=0.5, metric=None, alpha=0.05, alternative='two-sided')[source]

Test whether a metric’s distribution differs from a null value.

Applies a one-sample Wilcoxon signed-rank test to the per-run final values of metric.

Parameters:

null_value (float) – The null hypothesis value. Use 0.5 for chance-level binary classification accuracy.
metric (str | None) – Metric name. Must be a key in final_metrics.
alpha (float) – Significance threshold (default 0.05).
alternative (str) – 'two-sided' (default), 'greater', or 'less'. Use 'greater' for one-sided tests of the form “is metric above the null” — or call test_above_chance() which is a shortcut for that.

Returns:

A StatisticalTestResult with is_significant, p_value, statistic, and conclusion.

Return type:

StatisticalTestResult

Note

The Wilcoxon signed-rank test requires at least 6 non-zero differences for reliable results. Use num_runs >= 20 for reliable inference — consistent with the library-wide minimum for Mann-Whitney U tests.

test_above_chance(metric=None, alpha=0.05, chance_level=0.5)[source]

Test whether a model performs significantly above chance.

One-sided variant of test_against_null() that tests the specific hypothesis “this model’s median performance is greater than chance.” Null hypothesis: median = chance_level. Alternative: median > chance_level.

Uses a one-sided Wilcoxon signed-rank test (equivalent to calling test_against_null(null_value=chance_level, alternative='greater')).

Parameters:

metric (str | None) – Metric name. Must be a key in final_metrics or the bare key in a final_test_metrics entry. Defaults to the study’s preferred accuracy metric.
alpha (float) – Significance threshold. Default 0.05.
chance_level (float) – The chance-level baseline to test against. Default 0.5 (binary classification with balanced classes). For multi-class tasks with k balanced classes, pass chance_level = 1.0 / k.

Returns:

A StatisticalTestResult with is_significant, p_value, statistic, and conclusion. The test is one-sided, so p_value reflects P(median <= chance_level | observed data).

Return type:

StatisticalTestResult

Note

Recommended sample size: num_runs >= 20 for reliable inference, consistent with the library-wide minimum for rank-based tests.

Example

>>> results = ix.variability_study(model=..., runs=20)
>>> outcome = results.test_above_chance()
>>> if outcome.is_significant():
...     print(f"Model above chance (p={outcome.p_value:.4f})")

compare_models_statistically(*args, **kwargs)[source]

Removed in v0.3.10.

This method applied Kruskal-Wallis to groups of one observation each (one per run), which is statistically incoherent — within-group variance is undefined with a single observation.

To compare a single model’s run distribution against a null value, use:

results.test_against_null(null_value=0.5, metric="val_accuracy")

To compare multiple distinct models against each other, use:

ix.compare_models(models=[model_a, model_b], data=..., runs=20)

save(path)[source]

Persist results to disk as a plain dict via pickle.

Preserves all_runs_metrics, final_metrics, final_test_metrics, seed, and run_seeds. Restore with load().

Parameters:: path (str) – File path. Recommended extension: .pkl.
Return type:: None

classmethod load(path)[source]

Restore results previously saved with save().

Parameters:: path (str) – File path written by save().
Returns:: Reconstructed VariabilityStudyResults.
Return type:: VariabilityStudyResults

to_json()[source]

Serialise final metrics to a compact JSON string.

Does not include all_runs_metrics (per-epoch DataFrames). Use save() / load() for full round-trip fidelity.

Returns:: JSON string containing final_metrics, final_test_metrics, seed, and n_runs.
Return type:: str

classmethod from_json(json_str)[source]

Reconstruct a VariabilityStudyResults from a JSON string.

Symmetric with to_json(). Note that to_json does not serialize all_runs_metrics (per-epoch DataFrames), so the reconstructed object has an empty all_runs_metrics and empty run_seeds. Use save() / load() for full round-trip fidelity including per-epoch history.

Parameters:: json_str (str) – JSON string produced by to_json().
Returns:: VariabilityStudyResults with final_metrics, final_test_metrics, and seed populated from the JSON. all_runs_metrics is empty; run_seeds is empty.
Raises:: ValueError – If the JSON is malformed or missing required keys.
Return type:: VariabilityStudyResults

Example

>>> serialized = results.to_json()
>>> restored = VariabilityStudyResults.from_json(serialized)
>>> assert restored.final_metrics == results.final_metrics

classmethod from_mlflow_experiment(experiment_name, metric, run_filter=None, tracking_uri=None)[source]

Reconstruct results from a completed MLflow experiment.

Allows running ictonyx statistical analysis on existing MLflow experiments without re-running training.

Parameters:

experiment_name (str) – Name of the MLflow experiment.
metric (str) – Metric to extract per run (e.g. ‘val_accuracy’).
run_filter (str | None) – Optional MLflow filter string (e.g. “params.seed = ‘42’”).
tracking_uri (str | None) – MLflow tracking URI. Uses the environment’s MLFLOW_TRACKING_URI if not set.

Returns:

VariabilityStudyResults with final_metrics populated from the experiment’s runs. all_runs_metrics is empty — per-epoch history is not reconstructed from MLflow.

Raises:

ImportError – If mlflow is not installed.
ValueError – If no matching runs are found, or if the requested metric is not present in the runs.

Return type:

VariabilityStudyResults

class ictonyx.runners.GridStudyResults(results, param_grid, base_config, metric='val_accuracy')[source]

Bases: object

Container for grid study results across multiple parameter configurations.

Returned by run_grid_study(). Holds one VariabilityStudyResults per parameter configuration, the param grid that generated them, and the base config used as the template for all runs.

Parameters:

results (Dict[Tuple, VariabilityStudyResults])
param_grid (Dict[str, List[Any]])
base_config (ModelConfig)
metric (str)

results

Dict mapping frozen parameter tuples to VariabilityStudyResults. Keys are sorted tuples of (param_name, value) pairs for hashability and deterministic ordering.

Type:: Dict[Tuple, ictonyx.runners.VariabilityStudyResults]

param_grid

The original param_grid dict passed by the caller.

Type:: Dict[str, List[Any]]

base_config

The ModelConfig used as the template for all runs.

Type:: ictonyx.config.ModelConfig

metric

The primary metric used for summary statistics.

Type:: str

results: Dict[Tuple, VariabilityStudyResults]

param_grid: Dict[str, List[Any]]

base_config: ModelConfig

metric: str = 'val_accuracy'

property n_configurations: int: Total number of parameter configurations run.

property n_runs_per_config: int: Number of runs per configuration (from first result).

get_results_for_config(param_combo)[source]

Retrieve VariabilityStudyResults for a specific parameter combination.

Parameters:: param_combo (Dict[str, Any]) – Dict of parameter names to values, e.g. {‘learning_rate’: 0.001, ‘batch_size’: 8}
Returns:: VariabilityStudyResults for that configuration.
Raises:: KeyError – If the configuration was not found in results.
Return type:: VariabilityStudyResults

list_configurations()[source]

Return list of all parameter combinations as dicts.

Return type:: List[Dict[str, Any]]

to_dataframe(metric=None)[source]

Summary DataFrame with one row per configuration.

Columns: all parameter names, mean, sd, se, min, max, n.

Parameters:: metric (str | None) – Metric to summarize. Defaults to self.metric.
Returns:: DataFrame sorted by first parameter in param_grid.
Return type:: DataFrame

summarize(metric=None)[source]

Generate text summary of grid study results.

For executed studies, returns a summary of configurations and their metric statistics. For dry-run results (empty self.results), returns a preview showing the parameter grid and the planned configurations that would be executed.

Parameters:: metric (str | None) – Metric to summarize. Defaults to self.metric.
Return type:: str

ictonyx.runners.run_variability_study(model_builder, data_handler, model_config, num_runs=20, epochs_per_run=None, tracker=None, use_process_isolation=False, gpu_memory_limit=None, seed=None, verbose=True, use_parallel=False, n_jobs=-1)[source]

Run a complete variability study (convenience function).

Creates an ExperimentRunner and calls run_study(). This is the lower-level entry point; most users should prefer variability_study().

Parameters:

model_builder (Callable[[ModelConfig], BaseModelWrapper]) – Callable f(ModelConfig) -> BaseModelWrapper.
data_handler (DataHandler) – A DataHandler instance.
model_config (ModelConfig) – A ModelConfig instance.
num_runs (int) – Number of training runs. Default 20.
epochs_per_run (int | None) – Epochs per run. Default None (uses config).
tracker (BaseLogger | None) – Optional experiment logger. Default None.
use_process_isolation (bool) – Run each iteration in a subprocess. Default False.
gpu_memory_limit (int | None) – Optional GPU memory cap in MB.
seed (int | None) – Base random seed. If None, one is generated.
verbose (bool) – Print progress to stdout. Default True.
use_parallel (bool)
n_jobs (int)

Returns:

VariabilityStudyResults.

Return type:

VariabilityStudyResults

ictonyx.runners.run_grid_study(model_builder, data_handler, base_config, param_grid, num_runs=20, metric='val_accuracy', use_process_isolation=True, dry_run=False, verbose=True, seed=None)[source]

Run a variability study across a grid of parameter configurations.

For each combination in the Cartesian product of param_grid, creates a modified copy of base_config, runs a full variability study, and collects results. All runs use process isolation by default to prevent CUDA memory accumulation across configurations.

Parameters:

model_builder (Callable) – Callable accepting a ModelConfig, returning a compiled model.
data_handler (DataHandler) – DataHandler instance providing train/val data.
base_config (ModelConfig) – ModelConfig used as template. Values are overridden per configuration.
param_grid (Dict[str, List[Any]]) –
Dict mapping parameter names to lists of values to sweep. Example:
```
{
    'learning_rate': [0.001, 0.0001],
    'batch_size':    [8, 16]
}
```
Generates the full Cartesian product: 4 configurations × num_runs each.
num_runs (int) – Number of training runs per configuration. Default 20.
metric (str) – Primary metric for summary statistics.
use_process_isolation (bool) – If True, each run executes in a subprocess. Strongly recommended to prevent GPU memory accumulation across configurations.
dry_run (bool) – If True, print execution plan and return without training. Use to verify configuration before committing to a long run.
verbose (bool) – If True, log study progress. Default True.
seed (int | None) – Base random seed for reproducibility. Each configuration receives an independent child seed derived via SeedSequence.spawn(). If None, each configuration uses a random seed.

Returns:

GridStudyResults

Raises:

ValueError – If param_grid is empty or contains no values.

Return type:

GridStudyResults

Example:

results = run_grid_study(
    model_builder=build_cnn,
    data_handler=data_handler,
    base_config=config,
    param_grid={'learning_rate': [0.001, 0.0001, 0.00001]},
    num_runs=20,
    use_process_isolation=True
)
print(results.summarize())
df = results.to_dataframe()