ictonyx.analysis

Statistical tests, effect sizes, independence checks, and assumption validation.

class ictonyx.analysis.StatisticalTestResult(test_name, statistic, p_value, effect_size=None, effect_size_name=None, effect_size_interpretation=None, effect_size_secondary=None, effect_size_secondary_name=None, effect_size_secondary_interpretation=None, sample_sizes=None, assumptions_met=<factory>, assumption_details=<factory>, warnings=<factory>, recommendations=<factory>, corrected_p_value=None, correction_method=None, conclusion='', detailed_interpretation='', confidence_interval=None, ci_confidence_level=None, ci_method=None, ci_effect_size=None)[source]

Bases: object

Comprehensive result from a statistical comparison test.

Returned by all statistical test functions in this module. Contains the test statistic, p-value, effect size, assumption checks, and human-readable interpretation.

Parameters:

  • test_name (str)

  • statistic (float)

  • p_value (float)

  • effect_size (float | None)

  • effect_size_name (str | None)

  • effect_size_interpretation (str | None)

  • effect_size_secondary (float | None)

  • effect_size_secondary_name (str | None)

  • effect_size_secondary_interpretation (str | None)

  • sample_sizes (Dict[str, int] | None)

  • assumptions_met (Dict[str, bool | None])

  • assumption_details (Dict[str, Any])

  • warnings (List[str])

  • recommendations (List[str])

  • corrected_p_value (float | None)

  • correction_method (str | None)

  • conclusion (str)

  • detailed_interpretation (str)

  • confidence_interval (Tuple[float, float] | None)

  • ci_confidence_level (float | None)

  • ci_method (str | None)

  • ci_effect_size (Tuple[float, float] | None)

test_name

Name of the test performed (e.g. 'Mann-Whitney U').

Type:

str

statistic

The test statistic value.

Type:

float

p_value

The raw (uncorrected) p-value.

Type:

float

effect_size

Primary effect size estimate (Cohen’s d, rank-biserial, eta-squared-H, etc., depending on the test).

Type:

float | None

effect_size_name

Label for the primary effect size metric.

Type:

str | None

effect_size_interpretation

Qualitative label ('small', 'medium', 'large') per conventional thresholds.

Type:

str | None

effect_size_secondary

Optional secondary effect size for tests that report multiple conventional measures (e.g. Kruskal-Wallis reports both η²_H and ε²_R).

Type:

float | None

effect_size_secondary_name

Label for the secondary metric.

Type:

str | None

effect_size_secondary_interpretation

Qualitative label for the secondary effect size.

Type:

str | None

sample_sizes

Dict mapping group names to sample counts.

Type:

Dict[str, int] | None

assumptions_met

Dict mapping assumption names (e.g. 'normality_group1') to boolean pass/fail.

Type:

Dict[str, bool | None]

assumption_details

Dict with detailed assumption check outputs (e.g. Shapiro-Wilk statistics).

Type:

Dict[str, Any]

warnings

List of warning strings (e.g. small sample size caveats).

Type:

List[str]

recommendations

List of suggested next steps.

Type:

List[str]

corrected_p_value

P-value after multiple comparison correction, if applicable.

Type:

float | None

correction_method

Name of correction ('bonferroni', 'holm', 'benjamini-hochberg'), if applied.

Type:

str | None

conclusion

One-sentence summary of the result.

Type:

str

detailed_interpretation

Multi-sentence explanation suitable for a report.

Type:

str

confidence_interval

Bootstrap CI for the mean difference, as a (lower, upper) tuple.

Type:

Tuple[float, float] | None

ci_confidence_level

Confidence level (e.g. 0.95).

Type:

float | None

ci_method

CI construction method ('bca' or 'percentile').

Type:

str | None

ci_effect_size

Bootstrap CI for the effect size, as a (lower, upper) tuple.

Type:

Tuple[float, float] | None

test_name: str
statistic: float
p_value: float
effect_size: float | None = None
effect_size_name: str | None = None
effect_size_interpretation: str | None = None
effect_size_secondary: float | None = None
effect_size_secondary_name: str | None = None
effect_size_secondary_interpretation: str | None = None
sample_sizes: Dict[str, int] | None = None
assumptions_met: Dict[str, bool | None]
assumption_details: Dict[str, Any]
warnings: List[str]
recommendations: List[str]
corrected_p_value: float | None = None
correction_method: str | None = None
conclusion: str = ''
detailed_interpretation: str = ''
confidence_interval: Tuple[float, float] | None = None
ci_confidence_level: float | None = None
ci_method: str | None = None
ci_effect_size: Tuple[float, float] | None = None
is_significant(alpha=0.05)[source]

Check if result is statistically significant.

Parameters:

alpha (float)

Return type:

bool

get_summary()[source]

Get a concise summary of the test result.

Displays the corrected p-value (corrected_p_value) when a multiple-comparison correction has been applied, labelled p_corr. Displays the raw p-value otherwise, labelled p. The significance marker always derives from the same p-value shown, so the label and marker are never contradictory.

Return type:

str

ictonyx.analysis.validate_sample_sizes(data, min_size, test_name)[source]

Validate that sample sizes are adequate for a statistical test.

Checks each group against a minimum size threshold and generates warnings for small samples.

Parameters:

  • data (Series | List[Series]) – A single pd.Series or list of pd.Series to check.

  • min_size (int) – Minimum acceptable sample size per group.

  • test_name (str) – Name of the test, used in warning messages.

Returns:

Tuple of (adequate, warnings) where adequate is True if all groups meet the minimum size.

Return type:

Tuple[bool, List[str]]

ictonyx.analysis.check_normality(data, alpha=0.05, require_all_tests=True)[source]

Test for normality using Shapiro-Wilk (and D’Agostino-Pearson if n >= 20).

Used internally by assumption-checking logic in statistical tests.

Returns None when the sample is too small to test (n < 3), True when the sample does not reject normality, and False when at least one test rejects normality (or no tests could run). Callers must treat None as “untestable” and fall back to the non-parametric path — the same convention used by check_independence().

Parameters:

  • data (Series) – A pd.Series of metric values.

  • alpha (float) – Significance level. Default 0.05.

  • require_all_tests (bool) – If True (default), normality is declared only when all available tests pass — the conservative conjunction. If False, any single passing test declares normality (lenient disjunction). For n < 20, only Shapiro-Wilk runs, so the distinction has no effect below that threshold.

Returns:

Tuple of (is_normal, details_dict) where is_normal is True if normality is not rejected, False if it is rejected, or None if the sample is too small to test (n < 3). details_dict contains individual test statistics and p-values.

Return type:

Tuple[bool | None, Dict[str, Any]]

ictonyx.analysis.check_equal_variances(data1, data2, alpha=0.05)[source]

Test for equal variances between two groups using Levene’s test.

Used internally to choose between Student’s t-test (equal variance) and Welch’s t-test (unequal variance).

Parameters:

  • data1 (Series) – First group’s metric values.

  • data2 (Series) – Second group’s metric values.

  • alpha (float) – Significance level. Default 0.05.

Returns:

Tuple of (variances_equal, details_dict) where variances_equal is True if Levene’s test does not reject the null hypothesis of equal variances.

Return type:

Tuple[bool, Dict[str, Any]]

ictonyx.analysis.check_independence(data, max_lag=5, alpha=0.05)[source]

Check for autocorrelation that would violate the independence assumption.

Computes autocorrelation at lags 1 through effective_max_lag and flags the data as non-independent if any lag exceeds the significance threshold derived from alpha using a standard normal approximation.

The effective maximum lag is min(max_lag, n - 2) where n is the number of valid observations, ensuring the loop is meaningful for small samples. For a typical variability study with n=10 runs and max_lag=5, lags 1 through 5 are all checked.

Parameters:

  • data (Series) – A pd.Series of metric values ordered by run.

  • max_lag (int) – Maximum lag to check. Default 5. The actual number of lags checked may be fewer if n - 2 < max_lag.

  • alpha (float) – Significance level for autocorrelation detection. Default 0.05. A lag is flagged as significant if its autocorrelation exceeds norm.ppf(1 - alpha/2) / sqrt(n).

Return type:

Tuple[bool | None, Dict[str, Any]]

Note

At n ≤ 10 this test has very low power to detect autocorrelation. Absence of a warning is not a strong independence guarantee at typical variability study sizes (n = 20 runs).

A Bonferroni correction is applied across effective_max_lag lags, controlling the familywise false-positive rate at alpha. Without correction, testing 5 lags at α=0.05 would produce a 22.6% familywise rate for a spurious warning on independent data; the correction reduces this to the nominal α.

Returns:

  • is_independent is True if no lag exceeds the threshold, False if at least one lag does, or None if the series is too short to test (fewer than max_lag + 2 observations). Callers must handle None as “untestable”, not as “dependent”.

Return type:

Tuple of (is_independent, details_dict) where

Parameters:
ictonyx.analysis.cohens_d(group1, group2, pooled=True)[source]

Calculate Cohen’s d effect size between two groups.

Uses pooled standard deviation by default. Thresholds: small (0.2), medium (0.5), large (0.8).

Parameters:

  • group1 (Series) – First group’s metric values.

  • group2 (Series) – Second group’s metric values.

  • pooled (bool) – If True (default), use pooled standard deviation (Cohen’s d). If False, divide by group2.std() only — this is Glass’s delta, which uses the control group as the sole reference. Default True.

Returns:

Tuple of (d_value, interpretation) where interpretation is one of 'negligible', 'small', 'medium', 'large'.

Return type:

Tuple[float, str]

ictonyx.analysis.rank_biserial_correlation(group1, group2, U=None)[source]

Calculate rank-biserial correlation as the effect size for the Mann-Whitney U test.

Algebraically equivalent to Cliff’s delta (Cliff 1993): r = 2U / (n₁·n₂) − 1 = (concordant pairs − discordant pairs) / (n₁·n₂).

Ranges from -1 to 1, where 0 indicates no effect. Thresholds: small (0.1), medium (0.3), large (0.5).

Parameters:

  • group1 (Series) – First group’s metric values.

  • group2 (Series) – Second group’s metric values.

  • U (float | None)

Returns:

Tuple of (r_value, interpretation) where interpretation is 'negligible', 'small', 'medium', or 'large'.

Return type:

Tuple[float, str]

ictonyx.analysis.eta_squared(groups)[source]

Calculate eta-squared effect size for ANOVA.

For Kruskal-Wallis, use the epsilon-squared value embedded in the returned StatisticalTestResult — do not call this function on KW results. See also anova_test(), which now reports omega-squared (bias-corrected) rather than eta-squared.

Represents the proportion of variance in the dependent variable explained by group membership. Thresholds: small (0.01), medium (0.06), large (0.14).

Parameters:

groups (List[Series]) – List of pd.Series, one per group.

Returns:

Tuple of (eta_sq, interpretation) where interpretation is 'negligible', 'small', 'medium', or 'large'.

Return type:

Tuple[float, str]

ictonyx.analysis.apply_multiple_comparison_correction(p_values, method='holm')[source]

Apply multiple comparison correction to a list of p-values.

Adjusts p-values to control the family-wise error rate when performing many pairwise comparisons.

Parameters:

  • p_values (List[float]) – List of raw p-values from pairwise tests.

  • method (str) – Correction method — 'bonferroni', 'holm' (recommended), or 'fdr_bh' (Benjamini-Hochberg). Default 'holm'.

Returns:

List of corrected p-values in the same order as the input.

Return type:

Tuple[List[float], str]

ictonyx.analysis.mann_whitney_test(model1_metrics, model2_metrics, alternative='two-sided', alpha=0.05, random_state=None)[source]

Mann-Whitney U test for comparing two independent groups.

A non-parametric test that does not assume normal distributions. Includes sample-size validation, independence checks, rank-biserial effect size, and auto-generated interpretation text.

Parameters:

  • model1_metrics (Series) – Metric values for the first model.

  • model2_metrics (Series) – Metric values for the second model.

  • alternative (str) – 'two-sided', 'less', or 'greater'. Default 'two-sided'.

  • alpha (float) – Significance level for assumption checks. Default 0.05.

  • random_state (int | None)

Returns:

StatisticalTestResult with test statistic, p-value, rank-biserial correlation effect size, and interpretation.

Raises:

TypeError – If inputs are not pd.Series.

Return type:

StatisticalTestResult

ictonyx.analysis.wilcoxon_signed_rank_test(model_metrics, null_value=0.5, alternative='two-sided', alpha=0.05)[source]

Wilcoxon signed-rank test for a single sample against a null value.

Tests whether the median of the sample differs from null_value. Useful for testing whether a model’s accuracy is significantly different from chance.

Parameters:

  • model_metrics (Series) – Metric values for the model.

  • null_value (float) – Hypothesized median to test against. Default 0.5.

  • alternative (str) – 'two-sided', 'less', or 'greater'. Default 'two-sided'.

  • alpha (float) – Significance level. Default 0.05.

Returns:

StatisticalTestResult with test statistic, p-value, effect size (Wilcoxon r), and interpretation.

Return type:

StatisticalTestResult

ictonyx.analysis.anova_test(model_metrics, alpha=0.05)[source]

One-way ANOVA for comparing three or more independent groups.

A parametric test that assumes normality and equal variances. If assumptions are violated, consider kruskal_wallis_test() instead. Emits a UserWarning when any group has fewer than 30 samples, as normality cannot be reliably assumed at small n.

Parameters:

  • model_metrics (Dict[str, Series]) – Dict mapping model names to pd.Series of metric values.

  • alpha (float) – Significance level. Default 0.05.

Returns:

StatisticalTestResult with F-statistic, p-value, eta-squared effect size, and interpretation.

Return type:

StatisticalTestResult

ictonyx.analysis.kruskal_wallis_test(model_metrics, alpha=0.05)[source]

Kruskal-Wallis H-test for comparing three or more independent groups.

A non-parametric alternative to ANOVA that does not assume normality or equal variances. Recommended when sample sizes are small or distributions are skewed.

Parameters:

  • model_metrics (Dict[str, Series]) – Dict mapping model names to pd.Series of metric values.

  • alpha (float) – Significance level. Default 0.05.

Returns:

StatisticalTestResult with H-statistic, p-value, and two effect sizes: η²_H (primary, variance-explained analog) and ε²_R (secondary, rank-correlation analog). Pre-v0.4.7 the library computed η²_H but labeled it “epsilon-squared” — v0.4.7 corrects the labeling and adds the actual ε²_R value as the secondary effect size.

Return type:

StatisticalTestResult

ictonyx.analysis.friedman_test(model_metrics, alpha=0.05)[source]

Friedman chi-squared test for three or more related (paired) groups.

Non-parametric analog of repeated-measures ANOVA. Use when comparing 3+ models evaluated on the same runs (same seeds, same folds) — each “subject” is a run and each “condition” is a model.

The Friedman test ranks values within each subject (row), then tests whether the rank distributions across groups (columns) differ significantly. Paired counterpart to kruskal_wallis_test().

All groups must have the same number of observations, matched by index (implicit pairing). Mismatched lengths are a methodological error and raise ValueError.

Parameters:

  • model_metrics (Dict[str, Series]) – Dict mapping model names to pd.Series of metric values. All Series must have the same length (matched pairs).

  • alpha (float) – Significance level. Default 0.05.

Returns:

StatisticalTestResult with chi-squared statistic, p-value, Kendall’s W effect size, and interpretation.

Raises:

ValueError – If fewer than 3 groups are provided (use compare_two_models() for 2 groups), or if groups have mismatched lengths.

Return type:

StatisticalTestResult

Note

Recommended n (runs per model) >= 10 for reliable inference. At n < 6, exact Friedman null distributions from tables should be preferred over the chi-squared approximation; scipy’s implementation uses the chi-squared approximation throughout.

Example

>>> metrics = {
...     "BERT":      pd.Series([0.85, 0.86, 0.84, ...]),
...     "RoBERTa":   pd.Series([0.87, 0.88, 0.86, ...]),
...     "DeBERTa":   pd.Series([0.89, 0.90, 0.88, ...]),
... }
>>> result = friedman_test(metrics)
>>> if result.is_significant(alpha=0.05):
...     print(f"Models differ (W = {result.effect_size:.3f})")
ictonyx.analysis.shapiro_wilk_test(model_metrics, alpha=0.05)[source]

Shapiro-Wilk test for normality.

Tests whether a sample comes from a normally distributed population. Used internally by compare_two_models() to select between parametric and non-parametric tests.

Parameters:

  • model_metrics (Series) – Metric values to test.

  • alpha (float) – Significance level. Default 0.05. A significant result (p < alpha) indicates the data is not normally distributed.

Returns:

StatisticalTestResult with W-statistic, p-value, and interpretation.

Return type:

StatisticalTestResult

ictonyx.analysis.paired_wilcoxon_test(series_a, series_b, alpha=0.05, random_state=None, deterministic_tol=1e-10)[source]

Paired Wilcoxon signed-rank test for two matched samples.

Tests whether the median difference between series_a and series_b differs from zero. Appropriate when both models are evaluated on the same random seeds.

Parameters:

  • series_a (Series) – Per-run metric values for model A.

  • series_b (Series) – Per-run metric values for model B.

  • alpha (float) – Significance threshold. Default 0.05.

  • random_state (int | None)

  • deterministic_tol (float)

Returns:

StatisticalTestResult with paired-comparison conclusion text.

Return type:

StatisticalTestResult

ictonyx.analysis.compare_two_models(model1_results, model2_results, paired=False, alpha=0.05, random_state=None, ci_target='auto', test_method='auto')[source]

Compares two models using intelligent, assumption-driven test selection.

This function automatically selects the correct statistical test based on the data’s properties and whether the samples are paired.

  • If paired=True, performs a Wilcoxon signed-rank test on the differences.

  • If paired=False, the test is selected by test_method. The default is "auto" which applies data-driven selection (deprecated — see test_method parameter); v0.5.0 will default to "mann_whitney".

Parameters:

  • model1_results (pd.Series) – A Series of metric results for model 1.

  • model2_results (pd.Series) – A Series of metric results for model 2.

  • paired (bool, optional) – Whether the samples are paired (e.g., results from the same k-folds). Defaults to False.

  • alpha (float, optional) – Significance level used for assumption checks (normality, variance). Defaults to 0.05.

  • ci_target (str, optional) –

    Which quantity the bootstrap CI should be computed on. Relevant only for unpaired comparisons; paired comparisons always use paired-difference bootstrap.

    • "mean_difference" (legacy): bootstrap CI on the mean difference. Mismatched to the Mann-Whitney null hypothesis when MW is dispatched but preserved for backward compatibility.

    • "median_difference": bootstrap CI on the Hodges-Lehmann estimator (median of pairwise differences). Matches the Mann-Whitney null and is the methodologically correct CI when MW is dispatched. Also valid for parametric tests, where it is a robust alternative to the mean-difference CI.

    • "auto" (default in v0.4.7): emits a DeprecationWarning and uses "mean_difference" for backward compatibility. The default changes to "median_difference" in v0.5.0.

  • test_method (str, optional) –

    Which test to use for the unpaired comparison. Ignored when paired=True (paired comparisons always use paired Wilcoxon).

    • "mann_whitney": Mann-Whitney U test. The non-parametric default, methodologically appropriate for small-n variability studies where distributional assumptions are fragile.

    • "student_t": Student’s t-test (assumes normality + equal variances).

    • "welch_t": Welch’s t-test (assumes normality; does not assume equal variances).

    • "parametric": auto-choose between Student’s and Welch’s based on an equal-variance pre-test. Does not pre-test normality; caller is asserting normality holds.

    • "auto" (default in v0.4.7): emits a DeprecationWarning and applies data-driven test selection (pre-tests normality via Shapiro-Wilk, then equal variances via Levene, then dispatches to Student/Welch/Mann-Whitney accordingly). This pre-test-then-choose pattern inflates Type I error rates and is methodologically criticized. The default changes to "mann_whitney" in v0.5.0.

  • random_state (int | None)

Returns:

A rich object containing the test name, statistic, p-value, effect size, and details on which assumptions were met.

Return type:

StatisticalTestResult

Note

compare_models() — the primary public entry point — routes all comparisons through compare_multiple_models(), which applies Kruskal-Wallis + Mann-Whitney unconditionally regardless of data properties. The assumption-driven test selection in this function (normality → variance → Student/Welch/Mann-Whitney) is only exercised when calling it directly from ictonyx.analysis.

class ictonyx.analysis.ModelComparisonResults(overall_test, raw_data, pairwise_comparisons=<factory>, significant_comparisons=<factory>, correction_method='holm', n_models=0, metric=None)[source]

Bases: object

Results from a multi-model statistical comparison.

Returned by compare_models(). Contains the omnibus test result, pairwise comparisons, raw metric distributions, and convenience methods for summarization.

Parameters:
overall_test

StatisticalTestResult for the Kruskal-Wallis omnibus test across all models.

Type:

ictonyx.analysis.StatisticalTestResult

raw_data

Dict mapping model names to pd.Series of final metric values — one value per run. Use for custom analysis or plotting.

Type:

Dict[str, pandas.Series]

pairwise_comparisons

Dict mapping comparison names (e.g. 'ModelA_vs_ModelB') to StatisticalTestResult. Empty if the omnibus test was not significant.

Type:

Dict[str, ictonyx.analysis.StatisticalTestResult]

significant_comparisons

List of pairwise comparison names that survived multiple comparison correction.

Type:

List[str]

correction_method

The multiple comparison correction applied ('holm', 'bonferroni', or 'fdr_bh').

Type:

str

n_models

Number of models compared.

Type:

int

metric

The metric that was compared.

Type:

str | None

overall_test: StatisticalTestResult
raw_data: Dict[str, Series]
pairwise_comparisons: Dict[str, StatisticalTestResult]
significant_comparisons: List[str]
correction_method: str = 'holm'
n_models: int = 0
metric: str | None = None
is_significant(alpha=0.05)[source]

True if the omnibus test is significant at the given alpha.

Parameters:

alpha (float)

Return type:

bool

get_summary()[source]

Concise text summary of the comparison results.

Return type:

str

ictonyx.analysis.compare_multiple_models(model_results, alpha=0.05, correction_method='holm', metric=None, random_state=None)[source]

Compares three or more models using a robust, two-step procedure.

This function is the standard way to compare multiple groups in a statistically sound manner, protecting against inflating the error rate by running too many tests.

The procedure is: 1. Omnibus Test: First, it runs a single “overall” test

(Kruskal-Wallis) to see if any significant difference exists among all the model groups.

  1. Post-Hoc Tests: If, and only if, the omnibus test is significant, it then performs pairwise Mann-Whitney U tests for every possible combination of models to find out which specific pairs are different.

  2. Correction: The p-values from all post-hoc tests are automatically corrected using the specified correction_method (e.g., ‘holm’) to control the family-wise error rate.

Parameters:

  • model_results (Dict[str, pd.Series]) – A dictionary where keys are model names (str) and values are the metric results (pd.Series) for that model.

  • alpha (float, optional) – The significance level to use for the tests. Defaults to 0.05.

  • correction_method (str, optional) – The method to use for correcting p-values in the post-hoc pairwise comparisons. Options: ‘holm’, ‘bonferroni’, ‘fdr_bh’. Defaults to ‘holm’.

  • metric (str | None)

  • random_state (int | None)

Returns:

ModelComparisonResults containing the omnibus test, pairwise comparisons, and significant comparison names.

Raises:

ValueError – If fewer than two models are provided in model_results.

Return type:

ModelComparisonResults

ictonyx.analysis.calculate_autocorr(data, lag=1)[source]

Calculate autocorrelation of a metric series at a given lag.

Parameters:

  • data (Series | List[float]) – A pd.Series or list of metric values ordered by run.

  • lag (int) – The lag at which to compute autocorrelation. Default 1.

Returns:

The autocorrelation coefficient as a float, or None if the series is too short or the computation fails.

Return type:

float | None

ictonyx.analysis.calculate_averaged_autocorr(series_list, max_lag=20)[source]

Compute mean and std of autocorrelation across multiple run histories.

Parameters:

  • series_list (List[Series]) – List of pd.Series, each representing a training history (e.g. per-epoch loss values from one run).

  • max_lag (int) – Maximum lag to compute. Default 20.

Returns:

Tuple of (lags, mean_autocorr, std_autocorr) — three lists of equal length.

Raises:

ValueError – If series_list is empty or none are long enough for the requested max_lag.

Return type:

Tuple[List[int], List[float], List[float]]

ictonyx.analysis.check_convergence(data, window_size=10, slope_threshold=0.0001)[source]

Check whether a metric series has converged using a linear slope test.

Fits a linear regression to the most recent window_size values. Convergence is declared when the absolute slope is below slope_threshold (flat by magnitude) OR when the slope is not statistically significant at p > 0.10 (flat by evidence) — meaning the recent trend is indistinguishable from noise.

This replaces the previous OR-of-autocorrelation-and-variance-ratio heuristic, which fired on virtually every monotonically decreasing loss curve regardless of whether training had truly plateaued.

Parameters:

  • data (Series | List[float]) – A pd.Series or list of metric values ordered by epoch.

  • window_size (int) – Number of recent values to examine. Default 10.

  • slope_threshold (float) – Maximum absolute slope to declare convergence by the magnitude criterion. Default 1e-4.

Returns:

True if the series appears converged by either criterion. False if the series is too short (len(data) < window_size * 2) or if regression fails.

Return type:

bool

ictonyx.analysis.get_confusion_matrix_df(predictions, true_labels, class_names)[source]

Generate a confusion matrix as a labeled DataFrame.

Parameters:

  • predictions (ndarray) – Array of predicted class indices.

  • true_labels (ndarray) – Array of true class indices.

  • class_names (Dict[int, str]) – Dict mapping class indices to human-readable names, e.g. {0: 'cat', 1: 'dog'}.

Returns:

A square pd.DataFrame with class names as both row and column labels, suitable for plot_confusion_matrix().

Raises:

ValueError – If predictions and true_labels differ in length.

Return type:

DataFrame

ictonyx.analysis.generate_statistical_summary(results)[source]

Generates a human-readable, multi-line string summary of test results.

This function takes a list of StatisticalTestResult objects and formats them into a clean, printable report. The report includes: - An overall summary (total tests, number significant). - A one-line summary for each test (e.g., “Mann-Whitney U: … p=0.002 *”). - The effect size and interpretation for each test. - Any major warnings (e.g., assumption violations) for each test. - A de-duplicated list of key recommendations.

Parameters:

results (List[StatisticalTestResult]) – A list of one or more StatisticalTestResult objects to summarize.

Returns:

A formatted, multi-line string ready to be printed to the console or saved to a file.

Return type:

str

ictonyx.analysis.create_results_dataframe(results)[source]

Converts a list of statistical results into a structured pandas DataFrame.

This function is ideal for programmatic analysis or for exporting results to a file (e.g., a CSV). Each row in the returned DataFrame represents a single test result.

Parameters:

results (List[StatisticalTestResult]) – A list of one or more StatisticalTestResult objects.

Returns:

A DataFrame where each row is a test. Key columns include:

  • ’test_name’: The name of the test performed.

  • ’statistic’: The test statistic (e..g, U, H, or F-value).

  • ’p_value’: The original, uncorrected p-value.

  • ’significant’: Boolean (True/False) if the test is significant.

  • ’effect_size’: The calculated effect size.

  • ’effect_size_interpretation’: (e.g., “small”, “large”).

  • ’n_warnings’: Count of warnings (e.g., assumption violations).

  • ’assumptions_met’: Boolean (True/False) if all assumptions passed.

  • ’n_group1’, ‘n_group2’, etc.: Sample sizes for each group.

  • ’corrected_p_value’: The p-value after correction (if applied).

  • ’correction_method’: The correction method used (if applied).

Return type:

pd.DataFrame

ictonyx.analysis.assess_training_stability(loss_histories, window_size=10)[source]

Assesses training stability by analyzing loss histories from multiple runs.

This function calculates a suite of metrics to quantify the consistency and convergence of a model’s training process. It truncates all histories to the shortest common length for a fair comparison.

Parameters:

  • loss_histories (List[pd.Series]) – A list where each item is a pandas Series representing the loss history of a single training run.

  • window_size (int, optional) – The number of recent epochs to use for calculating convergence and final window statistics. Defaults to 10.

Returns:

A dictionary containing key stability metrics:

  • ’n_runs’: Number of training runs analyzed.

  • ’common_length’: The number of epochs used for analysis (based on the shortest run).

  • ’final_loss_mean’: The average loss value at the final epoch across all runs.

  • ’final_loss_std’: The standard deviation of the final epoch loss.

  • ’final_loss_cv’: Coefficient of Variation (Std / Mean) of the final loss. A key stability metric; lower is more stable.

  • ’final_losses_list’: The raw list of all final loss values, used for plotting the true distribution.

  • ’convergence_rate’: The percentage (0.0 to 1.0) of runs that were flagged as “converged” by check_convergence.

  • ’converged_runs’: The absolute number of converged runs.

  • ’stability_assessment’: A qualitative judgment (“high”, “moderate”, “low”) based on the final loss CV.

  • ’between_run_variance’: The variance between the average loss of each run’s final window. High values mean runs ended in different places.

  • ’within_run_variance_mean’: The average variance within the final window of each run. High values mean the runs were still oscillating at the end.

Return type:

Dict[str, Any]

ictonyx.analysis.required_runs(effect_size, alpha=0.05, power=0.8, n_sim=1000, alternative='two-sided', random_state=42)[source]

Minimum runs per model to detect a given effect size at stated power.

Uses simulation-based power calculation for the Mann-Whitney U test, which is the library’s default comparison method.

Parameters:

  • effect_size (float) – Expected rank-biserial correlation (0–1). Conventions: small=0.1, medium=0.3, large=0.5 (Cohen 1988).

  • alpha (float) – Type I error rate. Default 0.05.

  • power (float) – Desired statistical power (0–1). Default 0.80.

  • n_sim (int) – Number of simulations per candidate n. Default 1000.

  • alternative (str) – 'two-sided', 'less', or 'greater'. Default 'two-sided'.

  • random_state (int | None)

Returns:

Minimum number of runs per model.

Raises:

ValueError – If effect_size is not in (0, 1) or power is not in (0, 1).

Return type:

int

Note

The simulation draws from unbounded standard normal distributions. For metrics concentrated near 0 or 1 (e.g. accuracy on easy tasks), the normal approximation overestimates spread and required_runs() will over-estimate the number of runs needed. Results are most reliable for loss-scale metrics and accuracy values in the 0.4–0.8 range.

ictonyx.analysis.required_runs_paired(effect_size, alpha=0.05, power=0.8, n_sim=1000, alternative='two-sided', random_state=42)[source]

Minimum paired runs to detect a given effect size at stated power.

Paired-comparison counterpart to required_runs(). Uses simulation-based power calculation for the paired Wilcoxon signed-rank test, which is the library’s default paired comparison method.

Parameters:

  • effect_size (float) – Expected matched-pairs rank-biserial correlation (0–1). Conventions parallel independent-comparison r: small=0.1, medium=0.3, large=0.5 (Cohen 1988).

  • alpha (float) – Type I error rate. Default 0.05.

  • power (float) – Desired statistical power (0–1). Default 0.80.

  • n_sim (int) – Number of simulations per candidate n. Default 1000.

  • alternative (str) – 'two-sided', 'less', or 'greater'. Default 'two-sided'.

  • random_state (int | None) – Seed for reproducibility. Default 42.

Returns:

Minimum number of paired observations (e.g., runs per model when both models are run with matched seeds).

Raises:

ValueError – If effect_size is not in (0, 1) or power is not in (0, 1).

Return type:

int

Note

The simulation models paired differences directly as Normal(shift, 1.0) with no correlation term — equivalent to the independent-pair case (ρ=0). Users whose paired runs have substantial correlation (same data split, matched seeds, etc.) will achieve the target power with fewer runs than this function estimates. Modeling correlation is planned for a future release; for now, this returns a conservative upper bound under ρ=0.

The simulation draws from unbounded standard normal differences. For metrics concentrated near 0 or 1 the normal approximation overestimates spread and this function will over-estimate n. Results are most reliable for loss-scale metrics and accuracy values in the 0.4–0.8 range.

ictonyx.analysis.minimum_detectable_effect(n_runs, alpha=0.05, power=0.8, n_sim=1000, alternative='two-sided', random_state=42)[source]

Minimum rank-biserial correlation detectable at given n, alpha, and power.

Parameters:

  • n_runs (int) – Number of runs per model.

  • alpha (float) – Type I error rate. Default 0.05.

  • power (float) – Desired statistical power (0–1). Default 0.80.

  • n_sim (int) – Number of simulations per candidate effect size. Default 1000.

  • alternative (str) – 'two-sided', 'less', or 'greater'. Default 'two-sided'.

  • random_state (int | None)

Returns:

Minimum detectable rank-biserial correlation in (0, 1).

Raises:

ValueError – If n_runs < 2 or power is not in (0, 1).

Return type:

float

Note

The simulation draws from unbounded standard normal distributions. For metrics concentrated near 0 or 1 (e.g. accuracy on easy tasks), the normal approximation overestimates spread and the detectable effect will be under-estimated. Results are most reliable for loss-scale metrics and accuracy values in the 0.4–0.8 range.