ictonyx.plotting

Visualization functions for variability studies and model comparisons.

ictonyx.plotting.plot_confusion_matrix(cm_df, title='', show=None)[source]

Plot a confusion matrix as an annotated heatmap.

Parameters:

  • cm_df (DataFrame) – A square pd.DataFrame where rows are true labels and columns are predicted labels. Values are integer counts. Typically produced by get_confusion_matrix_df().

  • title (str) – Plot title. Default: "Confusion Matrix".

  • show (bool | None) – If True, call plt.show(). If False, return the figure without displaying. If None (default), defer to the global set_display_plots() setting.

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_training_history(history, title=None, metrics=None, show=None)[source]

Plot training and validation metrics over epochs.

Creates one subplot per metric, each showing train and validation curves. Accepts DataFrames, dicts, lists of dicts, or Keras History objects.

Parameters:

  • history (Any) – Training history in any of the supported formats. Column/key naming convention: 'train_accuracy' and 'val_accuracy', or 'accuracy' and 'val_accuracy'.

  • title (str | None) – Plot title. If None, auto-generated from epoch count.

  • metrics (List[str] | None) – List of metric base names to plot (e.g. ['accuracy', 'loss']). If None, auto-detected from available columns.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_roc_curve(model_wrapper, X_test, y_test, title='', show=None)[source]

Plot per-class ROC curves with AUC values.

Binarizes labels using one-vs-rest and plots one ROC curve per class. Requires the model wrapper to have a predict_proba method.

Parameters:
Returns:

The matplotlib.figure.Figure, or None if the model does not support probability predictions.

Return type:

Figure | None

ictonyx.plotting.plot_precision_recall_curve(model_wrapper, X_test, y_test, title='', show=None)[source]

Plot per-class precision-recall curves with AUC values.

Binarizes labels using one-vs-rest and plots one PR curve per class. Requires the model wrapper to have a predict_proba method.

Parameters:
Returns:

The matplotlib.figure.Figure, or None if the model does not support probability predictions.

Return type:

Figure | None

ictonyx.plotting.plot_variability_summary(all_runs_metrics_list=None, final_metrics_series=None, final_test_series=None, metric='accuracy', kind=None, show_histogram=True, show_boxplot=False, show_mean_lines=True, show_train=True, show_val=True, histogram_orientation='vertical', alpha=None, figsize=None, dpi=300, show=None, results=None)[source]

Plot a multi-panel variability summary for a completed study.

Generates a figure with:

  • Overlaid training curves — all runs plotted together, showing epoch-by-epoch convergence and spread.

  • Final metric distribution — histogram and/or boxplot of final-epoch values, showing the spread of outcomes across runs.

Parameters:

  • all_runs_metrics_list (List[DataFrame] | None) – List of per-run DataFrames (one per successful run), as stored in VariabilityStudyResults.all_runs_metrics.

  • final_metrics_series (Series | List | None) – pd.Series of final-epoch validation metric values across runs.

  • final_test_series (Series | List | None) – Optional pd.Series of test-set metric values. If provided, a second distribution is added.

  • metric (str) – Base metric name for labeling (default 'accuracy').

  • show_histogram (bool) – Show histogram panel (default True).

  • show_boxplot (bool) – Show boxplot panel (default False).

  • show_mean_lines (bool) – Overlay mean training curves on the trajectory panel (default True). Set to False when runs diverge dramatically and the mean is misleading.

  • show_train (bool) – Plot training metric curves (default True).

  • show_val (bool) – Plot validation metric curves (default True).

  • histogram_orientation (str) – 'vertical' (default) or 'horizontal'. Horizontal orientation aligns the distribution with the y-axis of the trajectory panel.

  • alpha (float | None) – Opacity of individual run curves. None (default) auto-calculates based on the number of runs.

  • figsize (Tuple[float, float] | None) – Figure size as (width, height) in inches. None (default) auto-calculates based on the number of panels.

  • dpi (int) – Figure resolution in dots per inch (default 150).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

  • kind (str | None)

  • results (Any | None)

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_run_trajectories(results, metric='val_accuracy', ax=None, show=None)[source]

Plot per-run epoch trajectories overlaid with the mean curve.

Each run is drawn as a thin semi-transparent line; the mean trajectory is drawn as a solid thicker line. High spread at convergence signals seed-dependent behaviour.

Parameters:
Returns:

The figure, or None if the metric is not found or display is active.

Return type:

Figure | None

ictonyx.plotting.plot_run_distribution(results, metric=None, ax=None, show=None)[source]

Violin plot of final metric values with individual run points overlaid.

Combines a KDE violin with individual run dots. Useful for understanding the shape of the performance distribution.

Parameters:
Returns:

The figure, or None on error or when display is active.

Return type:

Figure | None

ictonyx.plotting.plot_run_strip(results, metric=None, ax=None, show=None)[source]

Strip plot of final metric values with mean and SD markers.

Best for small n (< 20) where a violin would be oversmoothed. Each run is one dot; the mean and ±1 SD are marked with lines.

Parameters:
Returns:

The figure, or None on error or when display is active.

Return type:

Figure | None

ictonyx.plotting.plot_rank_correlation_over_epoch(results, metric=None, threshold=0.8, window=5, ax=None, show=None)[source]

Spearman rank correlation between epoch-k and final-epoch rankings.

For each epoch k, computes the Spearman correlation between the per-run metric ranking at epoch k and the per-run ranking at the final epoch. Plotted over epochs with a rolling mean smoother.

The first epoch where the smoothed correlation reaches threshold is annotated. This answers: at what point in training can you predict which seed will produce the best final model?

Parameters:

  • results (VariabilityStudyResults) – A completed VariabilityStudyResults with per-epoch data in all_runs_metrics.

  • metric (str | None) – Metric column to rank runs by. If None, resolved via preferred_metric(). Should be a validation metric.

  • threshold (float) – Smoothed correlation value at which to draw the annotation line. Default 0.8.

  • window (int) – Rolling mean window size for smoothing. Default 5.

  • ax (Axes | None) – Optional existing Axes.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The figure, or None on error or when display is active.

Raises:

ValueError – If n_runs < 15 — rank correlation is unreliable with fewer runs.

Return type:

Figure | None

ictonyx.plotting.plot_comparison_boxplots(comparison_results, metric='Accuracy', ax=None, show=None)[source]

Side-by-side boxplots comparing metric distributions across models.

Each model’s runs are shown as a boxplot with individual data points overlaid. If pairwise statistical comparisons are present in the input, significance annotations are added to the title.

Parameters:

  • comparison_results (Dict[str, Any]) – Either a dict returned by compare_models() (must contain a 'raw_data' key), or a plain dict mapping model names to lists/arrays of metric values.

  • metric (str) – Label for the y-axis (default 'Accuracy').

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

  • ax (Axes | None)

Returns:

The matplotlib.figure.Figure, or None if the raw data cannot be extracted.

Return type:

Figure | None

ictonyx.plotting.plot_comparison_forest(comparison_results, baseline_model, metric='Accuracy', ax=None, show=None)[source]

Forest plot of effect sizes relative to a baseline model.

For each non-baseline model, shows the mean difference from the baseline with a 95% confidence interval, plotted as a horizontal error bar. A vertical dashed line at zero marks no difference.

Parameters:

  • comparison_results (Any) – Dict with a 'raw_data' key mapping model names to arrays of metric values.

  • baseline_model (str) – Name of the model to use as the reference. Must be a key in raw_data.

  • metric (str) – Label for the x-axis (default 'Accuracy').

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

  • ax (Axes | None)

Returns:

The matplotlib.figure.Figure, or None if the baseline model is not found in the data.

Return type:

Figure | None

ictonyx.plotting.plot_pairwise_comparison_matrix(comparison_results, figsize=(12, 10), show_effect_sizes=True, annotate_significance=True, show=None)[source]

Heatmap matrix of pairwise p-values, significance, and effect sizes.

Creates 2–3 side-by-side heatmaps showing p-values, binary significance indicators, and (optionally) effect sizes for every pair of models.

Parameters:

  • comparison_results (Dict[str, Any]) – Dict returned by compare_models(), containing a 'pairwise_comparisons' key.

  • figsize (Tuple[int, int]) – Figure dimensions. Default (12, 10).

  • show_effect_sizes (bool) – If True, include an effect-size heatmap. Default True.

  • annotate_significance (bool) – If True, mark significant cells with asterisks. Default True.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if no pairwise comparisons are present.

Return type:

Figure | None

ictonyx.plotting.plot_grid_study_heatmap(grid_results, x_param, y_param, metric=None, stat='mean', show=None)[source]

Heatmap of a grid study metric across two swept parameters.

Parameters:

  • grid_results (Any) – A GridStudyResults object.

  • x_param (str) – Parameter name for the x-axis (columns of heatmap).

  • y_param (str) – Parameter name for the y-axis (rows of heatmap).

  • metric (str | None) – Metric to display. Defaults to grid_results.metric.

  • stat (str) – Statistic to display — 'mean' (default) or 'sd'.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure.

Return type:

Figure | None

ictonyx.plotting.plot_training_stability(stability_results=None, *, results=None, metric='loss', window_size=10, figsize=(12, 8), show=None)[source]

Plot training stability diagnostics from a stability analysis.

Creates a multi-panel figure showing convergence behavior, loss distributions, and coefficient of variation across runs.

Two ways to call this function:

  1. Direct: pass a VariabilityStudyResults via results= and the stability analysis will be computed for you:

    ix.plot_training_stability(results=my_results)

  2. Precomputed: pass a stability dict (useful when you want multiple views of the same analysis):

    stability = ix.assess_training_stability(
    [df['loss'] for df in my_results.all_runs_metrics]
)
ix.plot_training_stability(stability)
Parameters:

  • stability_results (Dict[str, Any] | None) – Dict produced by assess_training_stability(), containing keys such as 'n_runs', 'common_length', 'final_loss_mean', 'final_loss_std', 'final_loss_cv', 'stability_assessment', 'converged_runs', and 'final_losses_list'. Mutually exclusive with results.

  • results (VariabilityStudyResults | None) – A VariabilityStudyResults object. When provided, the stability analysis is computed internally using the specified metric column from each run’s history. Mutually exclusive with stability_results.

  • metric (str) – Name of the column in each run’s history DataFrame to analyze. Default "loss". Only used when results is provided.

  • window_size (int) – Number of recent epochs for convergence statistics. Default 10. Only used when results is provided.

  • figsize (Tuple[int, int]) – Figure dimensions in inches. Default (12, 8).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Raises:

  • ValueError – If both stability_results and results are provided, or if neither is.

  • KeyError – If metric is not found in a run’s history.

Return type:

Figure | None

ictonyx.plotting.plot_autocorr_vs_lag(data, max_lag=20, title='Autocorrelation of Loss', ax=None, show=None)[source]

Plot autocorrelation of a metric series as a function of lag.

Useful for diagnosing sequential dependence between runs — if autocorrelation is high at lag 1, consecutive runs may not be independent (e.g. due to incomplete GPU memory cleanup).

Parameters:
Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_averaged_autocorr(lags, mean_autocorr, std_autocorr, title='Averaged Autocorrelation of Loss', ax=None, show=None)[source]

Plot averaged autocorrelation with error bands across multiple studies.

Shows the mean autocorrelation at each lag with ±1 standard deviation shaded, summarizing sequential dependence across repeated studies.

Parameters:
Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_pacf_vs_lag(data, max_lag=20, title='Partial Autocorrelation of Loss', alpha=0.05, ax=None, show=None)[source]

Plot partial autocorrelation function (PACF) with confidence bands.

Requires statsmodels. PACF isolates the direct correlation at each lag, removing the influence of intermediate lags.

Parameters:

  • data (Series | List[float]) – A pd.Series or list of metric values ordered by run.

  • max_lag (int) – Maximum lag to compute. Default 20.

  • title (str) – Plot title. Default 'Partial Autocorrelation of Loss'.

  • alpha (float) – Confidence level for the bands. Default 0.05 (95% CI).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

  • ax (Axes | None)

Returns:

The matplotlib.figure.Figure, or None if statsmodels is not installed or the series is too short.

Return type:

Figure | None

ictonyx.plotting.plot_averaged_pacf(lags, mean_pacf, std_pacf, n_series=0, title='Averaged Partial Autocorrelation of Loss', conf_level=0.95, show=None)[source]

Plot averaged partial autocorrelation with error bands across runs.

Shows the mean PACF at each lag with ±1 standard deviation shaded.

Parameters:

  • lags (List[float]) – List of integer lag values.

  • mean_pacf (List[float]) – Mean PACF at each lag.

  • std_pacf (List[float]) – Standard deviation of PACF at each lag.

  • title (str) – Plot title. Default 'Averaged Partial Autocorrelation of Loss'.

  • conf_level (float) – Confidence level for reference lines. Default 0.95.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

  • n_series (int)

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

deprecated::

plot_averaged_pacf is deprecated and will be removed in v0.5.0. Use plot_run_independence_diagnostics() (v0.4.4) instead.

ictonyx.plotting.plot_paired_deltas(results_a, results_b, metric=None, *, name_a='Model A', name_b='Model B', show_ci=True, ci_confidence=0.95, ax=None, figsize=(8, 6), show=None)[source]

Per-run paired differences between two model studies.

Shows each run’s metric_a[i] - metric_b[i] as a point on a strip, with a horizontal zero line separating “Model A wins” (above) from “Model B wins” (below). When points appear on both sides of zero, the winner reverses depending on which seeds you picked — a phenomenon invisible from mean comparisons alone.

Requires equal run counts in both studies (paired comparison is undefined otherwise).

Parameters:

  • results_a (VariabilityStudyResults) – First model’s variability study.

  • results_b (VariabilityStudyResults) – Second model’s variability study.

  • metric (str | None) – Metric key to compare. If None, resolved from results_a.preferred_metric(context="scalar").

  • name_a (str) – Label for model A in the legend and axes. Default "Model A".

  • name_b (str) – Label for model B. Default "Model B".

  • show_ci (bool) – Whether to overlay a bootstrap CI band on the mean delta. Default True.

  • ci_confidence (float) – Confidence level for the mean-delta CI. Default 0.95.

  • ax (Axes | None) – Optional existing Axes. Creates a new figure if None.

  • figsize (Tuple[float, float]) – Figure dimensions when ax is None. Default (8, 6).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Raises:

ValueError – If the two studies have different run counts.

Return type:

Figure | None

ictonyx.plotting.plot_run_independence_diagnostics(results, metric=None, max_lag=5, *, alpha=0.05, ax=None, figsize=(8, 5), show=None)[source]

Autocorrelation diagnostic for run-level independence.

Runs from variability_study() should be IID by SeedSequence.spawn() construction. This plot helps detect violations — e.g. GPU state, wall-clock ordering, or caching leaking between runs — by showing autocorrelation at lags 1..max_lag of the run-indexed final-metric series.

Uses check_independence() internally, which applies a Bonferroni correction across the tested lags to control the familywise false-positive rate at alpha.

At small N the test has low power. With n=10 runs and max_lag=5, the plot bands are wide; absence of a flagged lag is not strong evidence of independence. Consider runs >= 20 before reading too much into a clean diagnostic.

Parameters:

  • results (VariabilityStudyResults) – A completed VariabilityStudyResults.

  • metric (str | None) – Metric key to diagnose. If None, resolved via results.preferred_metric(context="scalar").

  • max_lag (int) – Maximum lag to test. Default 5. Effective lag may be reduced if the run count is small.

  • alpha (float) – Familywise significance level. Default 0.05.

  • ax (Axes | None) – Optional existing Axes. Creates a new figure if None.

  • figsize (Tuple[float, float]) – Figure dimensions when ax is None. Default (8, 5).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_epoch_run_heatmap(results, metric=None, *, cmap='viridis', ax=None, figsize=None, show=None)[source]

Epoch × run heatmap of a per-epoch metric.

Rows are runs in run order (top = run 1). Columns are epochs. Cells are the metric value, colored by the supplied cmap. Reveals at a glance which runs train fast, which plateau early, which diverge, and how run-to-run consistency evolves over training.

Runs with fewer recorded epochs (early stopping, crashes, varied grid configs) have trailing NaN cells, which render as background. This is intentional — padding with zero or extrapolating would mislead.

Uses the epoch-context metric (preferred_metric() with context="epoch") because per-epoch plotters show training dynamics, and test-set metrics are single scalars not captured across epochs.

Parameters:

  • results (VariabilityStudyResults) – A completed VariabilityStudyResults.

  • metric (str | None) – Metric to plot. If None, resolved via results.preferred_metric(context="epoch").

  • cmap (str) – Matplotlib colormap name. Default "viridis".

  • ax (Axes | None) – Optional existing Axes. Creates a new figure if None.

  • figsize (Tuple[float, float] | None) – Figure dimensions. If None, auto-sized from the matrix shape.

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if the metric is not found in any run or display is enabled.

Return type:

Figure | None

ictonyx.plotting.plot_sequential_ci(results, metric=None, *, min_n=3, method='bca', confidence=0.95, ax=None, figsize=(8, 5), show=None)[source]

Bootstrap CI width as a function of number of runs.

For each n in [min_n, N], computes a bootstrap CI for the mean of the first n runs and plots the upper/lower bounds along with the running mean. The narrowing band illustrates how quickly additional runs reduce uncertainty and where returns diminish.

Useful for the “how many runs do I need?” methodology argument. Look for the point where the band stops meaningfully narrowing — additional runs past that point buy little precision.

Parameters:

  • results (VariabilityStudyResults) – A completed VariabilityStudyResults.

  • metric (str | None) – Metric key. If None, resolved via results.preferred_metric(context="scalar").

  • min_n (int) – Smallest N at which to compute a CI. Floored at 3 because bootstrap CIs are meaningless with fewer observations. Default 3.

  • method (str) – Bootstrap CI method. "bca" (default) or "percentile".

  • confidence (float) – Confidence level in (0, 1). Default 0.95.

  • ax (Axes | None) – Optional existing Axes. Creates a new figure if None.

  • figsize (Tuple[float, float]) – Figure dimensions when ax is None. Default (8, 5).

  • show (bool | None) – Display behavior. See plot_confusion_matrix().

Returns:

The matplotlib.figure.Figure, or None if display is enabled or fewer than min_n runs are available.

Raises:

ValueError – If min_n < 3 or if the study has fewer than min_n runs.

Return type:

Figure | None