ictonyx.bootstrap

Bootstrap confidence interval estimation.

Bootstrap confidence interval estimation for Ictonyx.

Provides non-parametric confidence intervals for effect sizes, mean differences, and arbitrary statistics. Designed for the small-sample regime (5-30 runs) typical of ML model comparison experiments.

Methods implemented:

• Percentile: Simple quantile-based CI. Adequate for symmetric distributions.

• BCa (Bias-Corrected and Accelerated): Gold-standard bootstrap CI. Corrects for both median bias and skewness in the bootstrap distribution. Recommended default for most use cases.

References

• Efron, B. (1987). “Better Bootstrap Confidence Intervals.” Journal of the American Statistical Association, 82(397), 171-185.

• Efron, B. & Tibshirani, R. (1993). “An Introduction to the Bootstrap.” Chapman & Hall.

• DiCiccio, T.J. & Efron, B. (1996). “Bootstrap Confidence Intervals.” Statistical Science, 11(3), 189-228.

class ictonyx.bootstrap.BootstrapCIResult(ci_lower, ci_upper, point_estimate, confidence_level, method, n_bootstrap, se_bootstrap, bootstrap_distribution=None)

Bases: object

Result of a bootstrap confidence interval estimation.

Parameters:

• ci_lower (float)

• ci_upper (float)

• point_estimate (float)

• confidence_level (float)

• method (str)

• n_bootstrap (int)

• se_bootstrap (float)

• bootstrap_distribution (ndarray | None)

ci_lower

Lower bound of the confidence interval.

Type:

float

ci_upper

Upper bound of the confidence interval.

Type:

float

point_estimate

The statistic computed on the original (non-resampled) data.

Type:

float

confidence_level

The confidence level (e.g. 0.95 for a 95% CI).

Type:

float

method

The CI construction method (‘percentile’ or ‘bca’).

Type:

str

n_bootstrap

Number of bootstrap resamples used.

Type:

int

se_bootstrap

Standard error of the bootstrap distribution.

Type:

float

bootstrap_distribution

The full array of bootstrap replications (available for plotting or diagnostics).

Type:

numpy.ndarray | None

ci_lower: float

ci_upper: float

point_estimate: float

confidence_level: float

method: str

n_bootstrap: int

se_bootstrap: float

bootstrap_distribution: ndarray | None = None

ictonyx.bootstrap.bootstrap_ci(data, statistic_fn, n_bootstrap=10000, confidence=0.95, method='bca', random_state=None, return_distribution=False)

Compute a bootstrap confidence interval for an arbitrary statistic.

This is the general-purpose engine. For common use cases, prefer the convenience functions bootstrap_mean_difference_ci and bootstrap_effect_size_ci, which handle two-sample data and call this function internally.

Parameters:

• data (ndarray | Series | List[float]) – 1-D array-like of observations.

• statistic_fn (Callable[[ndarray], float]) – A callable that takes a 1-D numpy array and returns a scalar float. This function is applied to each bootstrap resample.

• n_bootstrap (int) – Number of bootstrap resamples (default 10 000). Higher values give more precise CIs at the cost of compute time.

• confidence (float) – Confidence level in (0, 1). Default 0.95.

• method (str) – CI construction method. 'percentile' — simple quantile method. 'bca' — bias-corrected and accelerated (recommended).

• random_state (int | Generator | None) – Seed for reproducibility. Pass an integer for deterministic results.

• return_distribution (bool) – If True, the full bootstrap distribution is stored in the result (useful for plotting). Default False.

Returns:

BootstrapCIResult with CI bounds, point estimate, and diagnostics.

Raises:

ValueError – If data has fewer than 2 observations, confidence is outside (0, 1), n_bootstrap < 100, or method is unrecognised.

Return type:

BootstrapCIResult

ictonyx.bootstrap.bootstrap_mean_difference_ci(group1, group2, n_bootstrap=10000, confidence=0.95, method='bca', random_state=None, return_distribution=False)

Bootstrap CI for the difference in means (group1 - group2).

Resamples each group independently, computes the difference in means for each bootstrap replicate, and constructs a CI on that distribution.

This is the most directly actionable CI for model comparison: it tells you the plausible range of the true performance gap.

Parameters:

• group1 (ndarray | Series | List[float]) – Metric values for model 1 (e.g. accuracies from 10 runs).

• group2 (ndarray | Series | List[float]) – Metric values for model 2.

• n_bootstrap (int) – Number of bootstrap resamples (default 10 000).

• confidence (float) – Confidence level (default 0.95).

• method (str) – ‘percentile’ or ‘bca’ (default ‘bca’).

• random_state (int | None) – Seed for reproducibility.

• return_distribution (bool) – If True, store the full bootstrap distribution.

Returns:

BootstrapCIResult. The point_estimate is mean(group1) - mean(group2).

Return type:

BootstrapCIResult

ictonyx.bootstrap.bootstrap_effect_size_ci(group1, group2, n_bootstrap=10000, confidence=0.95, method='bca', pooled=True, random_state=None, return_distribution=False)

Bootstrap CI for Cohen’s d effect size.

Computes Cohen’s d on each bootstrap resample and constructs a CI. This pairs naturally with the effect sizes already reported by compare_two_models — it adds the uncertainty range around that point estimate.

Parameters:

• group1 (ndarray | Series | List[float]) – Metric values for model 1.

• group2 (ndarray | Series | List[float]) – Metric values for model 2.

• n_bootstrap (int) – Number of bootstrap resamples (default 10 000).

• confidence (float) – Confidence level (default 0.95).

• method (str) – ‘percentile’ or ‘bca’ (default ‘bca’).

• pooled (bool) – If True (default), use pooled standard deviation. If False, use group2’s std (Glass’s delta variant).

• random_state (int | None) – Seed for reproducibility.

• return_distribution (bool) – If True, store the full bootstrap distribution.

Returns:

BootstrapCIResult. The point_estimate is Cohen’s d on the original data.

Return type:

BootstrapCIResult

ictonyx.bootstrap.bootstrap_hedges_g_ci(group1, group2, n_bootstrap=10000, confidence=0.95, method='bca', random_state=None, return_distribution=False)

Bootstrap CI for Hedges’ g effect size.

Applies the Hedges’ J correction on every bootstrap resample so that the CI and point estimate are for the same estimator. Use this instead of bootstrap_effect_size_ci() when the point estimate is Hedges’ g (i.e., the Welch’s t-test path in compare_two_models()).

Parameters:

• group1 (ndarray | Series | List[float]) – Metric values for model 1.

• group2 (ndarray | Series | List[float]) – Metric values for model 2.

• n_bootstrap (int) – Number of bootstrap resamples. Default 10 000.

• confidence (float) – Confidence level. Default 0.95.

• method (str) – 'percentile' or 'bca'. Default 'bca'.

• random_state (int | None) – Seed for reproducibility.

• return_distribution (bool) – If True, store the full bootstrap distribution.

Returns:

BootstrapCIResult. point_estimate is Hedges’ g computed on the original (non-resampled) data.

Return type:

BootstrapCIResult

ictonyx.bootstrap.bootstrap_paired_difference_ci(group1, group2, n_bootstrap=10000, confidence=0.95, method='bca', random_state=None, return_distribution=False)

Bootstrap CI for paired mean differences.

For paired data (e.g. two models evaluated on the same folds), this computes the CI on the mean of the pairwise differences. This is more powerful than the independent two-sample version because it removes between-fold variance.

Parameters:

• group1 (ndarray | Series | List[float]) – Metric values for model 1.

• group2 (ndarray | Series | List[float]) – Metric values for model 2 (same length, paired with group1).

• n_bootstrap (int) – Number of bootstrap resamples (default 10 000).

• confidence (float) – Confidence level (default 0.95).

• method (str) – ‘percentile’ or ‘bca’ (default ‘bca’).

• random_state (int | None) – Seed for reproducibility.

• return_distribution (bool) – If True, store the full bootstrap distribution.

Returns:

BootstrapCIResult. The point_estimate is mean(group1 - group2).

Raises:

ValueError – If groups differ in length (pairing is impossible).

Return type:

BootstrapCIResult

ictonyx.bootstrap.bootstrap_hodges_lehmann_ci(group1, group2, n_bootstrap=10000, confidence=0.95, method='bca', random_state=None, return_distribution=False)

Bootstrap CI for the Hodges-Lehmann estimator of the location shift between two independent groups (group1 - group2).

The Hodges-Lehmann estimator is the median of all pairwise differences {x_i - y_j} across the two samples. It is the location-shift estimator that matches the null hypothesis of the Mann-Whitney U test (distributional equality / median difference under a shift model), unlike a bootstrap of the mean difference which is mismatched to MW’s inference target.

Parameters:

• group1 (ndarray | Series | List[float]) – Metric values for model 1 (e.g. accuracies from 10 runs).

• group2 (ndarray | Series | List[float]) – Metric values for model 2.

• n_bootstrap (int) – Number of bootstrap resamples (default 10 000).

• confidence (float) – Confidence level (default 0.95).

• method (str) – ‘percentile’ or ‘bca’ (default ‘bca’).

• random_state (int | None) – Seed for reproducibility.

• return_distribution (bool) – If True, store the full bootstrap distribution.

Returns:

BootstrapCIResult. The point_estimate is the Hodges-Lehmann estimator of the location shift: median of all pairwise (g1_i - g2_j).

Return type:

BootstrapCIResult

Notes

Pairwise difference computation is O(n1 * n2) per bootstrap sample. At n1 = n2 = 50 and n_bootstrap = 10000, this is ~25M floating-point operations — fast (well under 1 second). For very large samples (n > 500) consider reducing n_bootstrap or using the percentile method.