ictonyx.tuning

Hyperparameter tuning with variability-aware objectives.

class ictonyx.tuning.HyperparameterTuner(model_builder, data_handler, model_config, metric='val_loss', n_evals_per_trial=3, stability_weight=0.0)

Bases: object

Hyperparameter tuner using Optuna as the primary backend.

Trains each trial configuration n_evals_per_trial times and uses the mean metric as the objective, consistent with the library’s thesis that a single training run is not a reliable measurement.

Requires optuna: pip install ictonyx[tuning]

The legacy Hyperopt backend is still available but deprecated and will be removed in v0.5.0.

Parameters:

• model_builder (Callable[[ModelConfig], BaseModelWrapper])

• data_handler (DataHandler)

• model_config (ModelConfig)

• metric (str)

• n_evals_per_trial (int)

• stability_weight (float)

__init__(model_builder, data_handler, model_config, metric='val_loss', n_evals_per_trial=3, stability_weight=0.0)

Parameters:

• model_builder (Callable[[ModelConfig], BaseModelWrapper]) – Function returning BaseModelWrapper given ModelConfig.

• data_handler (DataHandler) – DataHandler for loading data. Data is loaded lazily at tune() time, not during construction.

• model_config (ModelConfig) – Base ModelConfig updated with trial parameters.

• metric (str) – Metric to optimize. Default ‘val_loss’.

• n_evals_per_trial (int) – Independent training runs per trial. The trial objective is the mean metric across these runs. Default 3. Set to 1 to reproduce single-run (old) behavior.

• stability_weight (float) – If > 0, penalizes run-to-run variance. Minimisation objectives (e.g. 'val_loss'): objective = mean + stability_weight * std. Maximisation objectives (e.g. 'val_accuracy'): objective = mean - stability_weight * std. In both cases, higher variance worsens the objective score. Default 0.0 (variance not penalized).

tune(param_space, max_evals=100, direction='auto', timeout=None, n_jobs=1)

Run hyperparameter optimisation using Optuna.

Parameters:

• param_space (Dict[str, Any]) –

  Dict mapping parameter names to Optuna distributions. Example:

  import optuna
  {
      "learning_rate": optuna.distributions.FloatDistribution(
          1e-4, 1e-1, log=True),
      "n_estimators": optuna.distributions.IntDistribution(50, 500),
  }
  

• max_evals (int) – Number of trials. Default 100.

• direction (str) – 'minimize', 'maximize', or 'auto'. When 'auto', infers from metric name: maximise for accuracy/f1/r2/auc; minimise for loss/mse/mae. Default 'auto'.

• timeout (float | None) – Optional wall-clock time limit in seconds.

• n_jobs (int) – Number of parallel Optuna workers. Default 1.

Returns:

Dict of best hyperparameters found.

Return type:

Dict[str, Any]

get_best_trial()

Get details about the best trial after optimisation.

Returns:

Dict with best trial information.

Raises:

RuntimeError – If tune() has not been called yet.

Return type:

Dict[str, Any]

get_trials_dataframe()

Get a DataFrame with all trial results.

Returns:

DataFrame with trial parameters and results.

Raises:

RuntimeError – If tune() has not been called yet.

Return type:

DataFrame

ictonyx.tuning.create_search_space()

Creates common hyperparameter search spaces for different model types.

Returns:

Dictionary of example search space definitions

Return type:

Dict[str, Any]