ictonyx.api

High-level study functions: variability_study, compare_models, compare_results, test_against_null.

The High-Level API for Ictonyx.

This module provides a interface for running variability studies and model comparisons. It abstracts away the complexity of DataHandlers, ModelConfigs, and ExperimentRunners into single function calls.

ictonyx.api.variability_study(model, data, target_column=None, runs=20, epochs=10, batch_size=32, tracker=None, use_process_isolation=False, seed=None, verbose=True, use_parallel=False, n_jobs=-1, **kwargs)[source]

Run a variability study: train a model N times and collect distributions.

Trains the same model architecture on the same data runs times, each with a different random seed, and returns the full distribution of training metrics. This is the primary entry point for most users.

The function handles data resolution, model wrapping, configuration, and execution automatically. For finer control, use ExperimentRunner directly.

Parameters:

  • model (Any) –

    What to train. Accepted forms:

    • A class with fit/predict (e.g. RandomForestClassifier) — a fresh instance is created for each run.

    • A callable f(config) -> BaseModelWrapper — called once per run.

    • A Keras Model or PyTorch nn.Module — auto-wrapped.

    • An instance with fit/predict — works, but a warning is emitted because fitted state persists between runs.

  • data (str | DataFrame | Tuple[ndarray, ndarray]) –

    The dataset. Accepted forms:

    • pd.DataFrame — requires target_column.

    • (X, y) tuple of array-likes.

    • str path to a CSV file (requires target_column) or an image directory (requires image_size in kwargs).

    • An existing DataHandler instance.

  • target_column (str | None) – Column name containing labels. Required when data is a DataFrame or CSV path.

  • runs (int) – Number of independent training runs. Default 20.

  • epochs (int) – Training epochs per run. Ignored by scikit-learn models. Default 10.

  • batch_size (int) – Batch size per run. Ignored by scikit-learn models. Default 32.

  • tracker (BaseLogger | None) – Optional BaseLogger (or subclass such as MLflowLogger) for experiment tracking. If None, a basic in-memory logger is used.

  • use_process_isolation (bool) – If True, each run executes in a subprocess to guarantee GPU memory cleanup. Useful for Keras/TF models that leak memory across runs. Default False.

  • seed (int | None) – Base random seed. Each run receives an independent child seed derived via np.random.SeedSequence.spawn(), guaranteeing statistically uncorrelated RNG streams. If None, a random seed is generated and stored in the results.

  • verbose (bool) – If False, suppress all training output. Default True.

  • use_parallel (bool) – If True, fan training runs across multiple processes using joblib. Safe for sklearn models. Not recommended for Keras/TF models. Mutually exclusive with use_process_isolation. Default False.

  • n_jobs (int) – Number of parallel workers. -1 uses all CPUs. Ignored when use_parallel=False. Default -1.

  • **kwargs – Additional arguments forwarded to both the ModelConfig and the data handler (e.g. image_size, test_split, val_split).

Returns:

VariabilityStudyResults containing per-run metric DataFrames, final metric distributions, and convenience methods for summarization and statistical analysis.

Return type:

VariabilityStudyResults

Example:

import ictonyx as ix
from sklearn.ensemble import RandomForestClassifier

results = ix.variability_study(
    model=RandomForestClassifier,
    data=df,
    target_column='target',
    runs=20,
)
print(results.summarize())
ictonyx.api.compare_models(models, data, target_column=None, runs=20, epochs=10, metric=None, seed=None, verbose=True, paired=True, **kwargs)[source]

Run variability studies on multiple models and compare them statistically.

Each model is trained runs times on the same data, producing a distribution of the chosen metric. The distributions are then compared using non-parametric statistical tests.

Seeding and pairing

All models receive the same base seed. Internally, np.random.SeedSequence(seed).spawn(runs) generates per-run child seeds that are identical across models: model A’s run i and model B’s run i always share the same child seed. The runs are therefore genuinely paired at the RNG level, regardless of framework.

For this reason paired defaults to True:

  • Two models (default, paired): Paired Wilcoxon signed-rank test on the per-run differences. More powerful than the independent-samples alternative because it removes run-to-run noise that is common to both models.

  • Two models (unpaired): Kruskal-Wallis omnibus + Mann-Whitney U with Holm correction. Valid but does not exploit the RNG pairing.

  • Three or more models: Kruskal-Wallis omnibus + pairwise Mann-Whitney U with Holm correction, regardless of paired. (Paired multi-group analysis requires a different design not yet implemented.)

Seeding guarantee by framework

  • scikit-learn: exact (random_state injected at wrapper construction).

  • PyTorch: approximately deterministic under cudnn.deterministic=True; rare non-deterministic CUDA ops may introduce small deviations.

  • TF/Keras + GPU: not fully controllable; pairing is approximate.

If exact pairing cannot be guaranteed (e.g. Keras with GPU), pass paired=False to fall back to the independent-samples test, which is always valid regardless of seeding.

Parameters:

  • models (List[Any]) – List of models in any form accepted by variability_study() (classes, callables, instances, etc.).

  • data (str | DataFrame | Tuple[ndarray, ndarray]) – The dataset, in any form accepted by variability_study().

  • target_column (str | None) – Column name containing labels, if applicable.

  • runs (int) – Number of independent training runs per model. Default 20.

  • epochs (int) – Training epochs per run. Default 10.

  • metric (str | None) – Metric name to compare across models. Must be a key in the training history (e.g. 'val_accuracy', 'val_loss', 'val_f1'). Default None (auto-resolved from results).

  • seed (int | None) – Base random seed for reproducibility. All models use the same seed so the comparison can be reproduced exactly. If None, a random seed is generated and stored in the result.

  • verbose (bool) – If False, suppress all output. Default True.

  • paired (bool) – If True (default), use the paired Wilcoxon signed-rank test for two-model comparisons, exploiting the fact that all models receive identical per-run seeds by construction. Ignored when comparing three or more models (KW + MW is used regardless). Pass False to use the independent-samples test instead — appropriate when seeds are not shared or when comparing against externally produced results.

  • **kwargs – Forwarded to each variability_study() call.

Returns:

ModelComparisonResults containing the omnibus test, pairwise comparisons, raw metric distributions, and summary methods.

Return type:

ModelComparisonResults

Example (two models, default paired analysis):

results = ix.compare_models(
    models=[RandomForestClassifier, GradientBoostingClassifier],
    data=df,
    target_column='target',
    runs=20,
    metric='val_accuracy',
    seed=42,
)
print(results.get_summary())

Example (three models, or unpaired two-model comparison):

results = ix.compare_models(
    models=[ModelA, ModelB, ModelC],
    data=df,
    target_column='target',
    runs=20,
    seed=42,
)
# Unpaired two-model:
results = ix.compare_models(
    models=[ModelA, ModelB],
    data=df,
    target_column='target',
    runs=20,
    seed=42,
    paired=False,   # fall back to KW + Mann-Whitney
)
ictonyx.api.compare_results(results_a, results_b, metric=None, paired=True, seed=None)[source]

Compare two pre-computed VariabilityStudyResults without re-running training.

Extracts metric values from each results object and compares them statistically. Use this when you already have results from variability_study() and want to compare them without retraining.

Pairing: If both results were produced with the same seed, the runs are paired by construction. Pass paired=True (default) to exploit this with the more powerful paired Wilcoxon signed-rank test. Pass paired=False to use the independent-samples Mann-Whitney U test instead.

Parameters:

  • results_a (VariabilityStudyResults) – First model’s results.

  • results_b (VariabilityStudyResults) – Second model’s results.

  • metric (str | None) – Metric to compare. If None, resolves via results_a.preferred_metric().

  • paired (bool) – If True (default) and run counts are equal, use the paired Wilcoxon signed-rank test. Falls back to Mann-Whitney U with a UserWarning when run counts differ.

  • seed (int | None) – Random state for bootstrap CI computation. Defaults to None (non-deterministic CIs).

Returns:

ModelComparisonResults.

Raises:

KeyError – If the resolved metric is not present in both results.

Return type:

ModelComparisonResults