17 HPT: sklearn SVC VBDP Data

This document refers to the following software versions:

python: 3.10.10

pip list | grep  "spot[RiverPython]"

spotPython                 0.2.34

spotRiver                  0.0.93

Note: you may need to restart the kernel to use updated packages.

spotPython can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.

!pip install spotPython

Uncomment the following lines if you want to for (re-)installation the latest version of spotPython from gitHub.

# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

17.1 Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time and the initial design size.

MAX_TIME = 1
INIT_SIZE = 5
ORIGINAL = False

import os
import copy
import socket
from datetime import datetime
from dateutil.tz import tzlocal
start_time = datetime.now(tzlocal())
HOSTNAME = socket.gethostname().split(".")[0]
experiment_name = '18-svc-sklearn' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
experiment_name = experiment_name.replace(':', '-')
print(experiment_name)
if not os.path.exists('./figures'):
    os.makedirs('./figures')

18-svc-sklearn_p040025_1min_5init_2023-06-17_17-07-21

import warnings
warnings.filterwarnings("ignore")

17.2 Step 1: Initialization of the Empty `fun_control` Dictionary

from spotPython.utils.init import fun_control_init
fun_control = fun_control_init(task="classification",
    tensorboard_path="runs/16_spot_hpt_sklearn_classification")

17.3 1. Load Data: Classification

17.4 VBDP

import pandas as pd
if ORIGINAL == True:
    train_df = pd.read_csv('./data/VBDP/trainn.csv')
    test_df = pd.read_csv('./data/VBDP/testt.csv')
else:
    train_df = pd.read_csv('./data/VBDP/train.csv')
    # remove the id column
    train_df = train_df.drop(columns=['id'])

from sklearn.preprocessing import OrdinalEncoder
n_samples = train_df.shape[0]
n_features = train_df.shape[1] - 1
target_column = "prognosis"
# Encoder our prognosis labels as integers for easier decoding later
enc = OrdinalEncoder()
train_df[target_column] = enc.fit_transform(train_df[[target_column]])
train_df.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
print(train_df.shape)
train_df.head()

(707, 65)

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10	...	x56	x57	x58	x59	x60	x61	x62	x63	x64	prognosis
0	1.0	1.0	0.0	1.0	1.0	1.0	1.0	0.0	1.0	1.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	3.0
1	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	1.0	0.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	7.0
2	0.0	1.0	1.0	1.0	0.0	1.0	1.0	1.0	1.0	1.0	...	1.0	1.0	1.0	1.0	1.0	0.0	1.0	1.0	1.0	3.0
3	0.0	0.0	1.0	1.0	1.0	1.0	0.0	1.0	0.0	1.0	...	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	10.0
4	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	1.0	0.0	...	0.0	1.0	0.0	0.0	1.0	1.0	1.0	0.0	0.0	6.0

5 rows × 65 columns

The full data set train_df 64 features. The target column is labeled as prognosis.

17.5 Holdout Train and Test Data

We split out a hold-out test set (25% of the data) so we can calculate an example MAP@K

import numpy as np
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train_df.drop(target_column, axis=1), train_df[target_column],
                                                    random_state=42,
                                                    test_size=0.25,
                                                    stratify=train_df[target_column])
train = pd.DataFrame(np.hstack((X_train, np.array(y_train).reshape(-1, 1))))
test = pd.DataFrame(np.hstack((X_test, np.array(y_test).reshape(-1, 1))))
train.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
test.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
print(train.shape)
print(test.shape)
train.head()

(530, 65)
(177, 65)

	x1	x2	x3	x4	x5	x6	x7	x8	x9	...	x60	x61	x62	x64	prognosis
0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	2.0
1	0.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	...	0.0	0.0	0.0	0.0	4.0
2	0.0	0.0	0.0	1.0	1.0	1.0	0.0	0.0	0.0	...	1.0	1.0	1.0	1.0	1.0
3	1.0	1.0	0.0	1.0	1.0	1.0	0.0	0.0	0.0	...	0.0	0.0	0.0	0.0	6.0
4	0.0	0.0	0.0	1.0	0.0	0.0	1.0	1.0	0.0	...	0.0	0.0	0.0	0.0	5.0

5 rows × 65 columns

# add the dataset to the fun_control
fun_control.update({"data": train_df, # full dataset,
               "train": train,
               "test": test,
               "n_samples": n_samples,
               "target_column": target_column})

17.6 Step 3: Specification of the Preprocessing Model

Data preprocesssing can be very simple, e.g., you can ignore it. Then you would choose the prep_model “None”:

prep_model = None
fun_control.update({"prep_model": prep_model})

A default approach for numerical data is the StandardScaler (mean 0, variance 1). This can be selected as follows:

# prep_model = StandardScaler()
# fun_control.update({"prep_model": prep_model})

Even more complicated pre-processing steps are possible, e.g., the follwing pipeline:

# categorical_columns = []
# one_hot_encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
# prep_model = ColumnTransformer(
#         transformers=[
#             ("categorical", one_hot_encoder, categorical_columns),
#         ],
#         remainder=StandardScaler(),
#     )

17.7 Step 4: Select `algorithm` and `core_model_hyper_dict`

The selection of the algorithm (ML model) that should be tuned is done by specifying the its name from the sklearn implementation. For example, the SVC support vector machine classifier is selected as follows:

fun_control = add_core_model_to_fun_control(SVC, fun_control, SklearnHyperDict)

Other core_models are, e.g.,:

RidgeCV
GradientBoostingRegressor
ElasticNet
RandomForestClassifier
LogisticRegression
KNeighborsClassifier
RandomForestClassifier
GradientBoostingClassifier
HistGradientBoostingClassifier

We will use the RandomForestClassifier classifier in this example.

from sklearn.linear_model import RidgeCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import ElasticNet
from spotPython.hyperparameters.values import add_core_model_to_fun_control
from spotPython.data.sklearn_hyper_dict import SklearnHyperDict
from spotPython.fun.hypersklearn import HyperSklearn

# core_model  = RidgeCV
# core_model = GradientBoostingRegressor
# core_model = ElasticNet
# core_model = RandomForestClassifier
core_model = SVC
# core_model = LogisticRegression
# core_model = KNeighborsClassifier
# core_model = GradientBoostingClassifier
# core_model = HistGradientBoostingClassifier
fun_control = add_core_model_to_fun_control(core_model=core_model,
                              fun_control=fun_control,
                              hyper_dict=SklearnHyperDict,
                              filename=None)

Now fun_control has the information from the JSON file. The available hyperparameters are:

print(*fun_control["core_model_hyper_dict"].keys(), sep="\n")

C
kernel
degree
gamma
coef0
shrinking
probability
tol
cache_size
break_ties

17.8 Step 5: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

17.8.1 Modify hyperparameter of type numeric and integer (boolean)

Numeric and boolean values can be modified using the modify_hyper_parameter_bounds method. For example, to change the tol hyperparameter of the SVC model to the interval [1e-3, 1e-2], the following code can be used:

fun_control = modify_hyper_parameter_bounds(fun_control, "tol", bounds=[1e-3, 1e-2])

from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
fun_control = modify_hyper_parameter_bounds(fun_control, "probability", bounds=[1, 1])

17.8.2 Modify hyperparameter of type factor

spotPython provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 14.5.3.

Factors can be modified with the modify_hyper_parameter_levels function. For example, to exclude the sigmoid kernel from the tuning, the kernel hyperparameter of the SVC model can be modified as follows:

fun_control = modify_hyper_parameter_levels(fun_control, "kernel", ["linear", "rbf"])

The new setting can be controlled via:

fun_control["core_model_hyper_dict"]["kernel"]

from spotPython.hyperparameters.values import modify_hyper_parameter_levels
fun_control = modify_hyper_parameter_levels(fun_control, "kernel", ["rbf"])

17.8.3 Optimizers

Optimizers are described in Section 14.6.

17.9 5. Selection of the Objective: Metric and Loss Functions

Machine learning models are optimized with respect to a metric, for example, the accuracy function.
Deep learning, e.g., neural networks are optimized with respect to a loss function, for example, the cross_entropy function and evaluated with respect to a metric, for example, the accuracy function.

17.10 Step 6: Selection of the Objective (Loss) Function

The loss function, that is usually used in deep learning for optimizing the weights of the net, is stored in the fun_control dictionary as "loss_function".

17.10.1 Metric Function

There are two different types of metrics in spotPython:

"metric_river" is used for the river based evaluation via eval_oml_iter_progressive.
"metric_sklearn" is used for the sklearn based evaluation.

We will consider multi-class classification metrics, e.g., mapk_score and top_k_accuracy_score.

Predict Probabilities

In this multi-class classification example the machine learning algorithm should return the probabilities of the specific classes ("predict_proba") instead of the predicted values.

We set "predict_proba" to True in the fun_control dictionary.

17.10.1.1 The MAPK Metric

To select the MAPK metric, the following two entries can be added to the fun_control dictionary:

"metric_sklearn": mapk_score"

"metric_params": {"k": 3}.

17.10.1.2 Other Metrics

Alternatively, other metrics for multi-class classification can be used, e.g.,: * top_k_accuracy_score or * roc_auc_score

The metric roc_auc_score requires the parameter "multi_class", e.g.,

"multi_class": "ovr".

This is set in the fun_control dictionary.

Weights

spotPython performs a minimization, therefore, metrics that should be maximized have to be multiplied by -1. This is done by setting "weights" to -1.

The complete setup for the metric in our example is:

from spotPython.utils.metrics import mapk_score
fun_control.update({
               "weights": -1,
               "metric_sklearn": mapk_score,
               "predict_proba": True,
               "metric_params": {"k": 3},
               })

17.11 Evaluation on Hold-out Data

The default method for computing the performance is "eval_holdout".
Alternatively, cross-validation can be used for every machine learning model.
Specifically for RandomForests, the OOB-score can be used.

fun_control.update({
    "eval": "train_hold_out",
})

17.11.0.1 Cross Validation

Instead of using the OOB-score, the classical cross validation can be used. The number of folds is set by the key "k_folds". For example, to use 5-fold cross validation, the key "k_folds" is set to 5. Uncomment the following line to use cross validation:

# fun_control.update({
#      "eval": "train_cv",
#      "k_folds": 10,
# })

17.12 6. Calling the SPOT Function

17.13 Preparing the SPOT Call

Get types and variable names as well as lower and upper bounds for the hyperparameters.

# extract the variable types, names, and bounds
from spotPython.hyperparameters.values import (get_bound_values,
    get_var_name,
    get_var_type,)
var_type = get_var_type(fun_control)
var_name = get_var_name(fun_control)
fun_control.update({"var_type": var_type,
                    "var_name": var_name})
lower = get_bound_values(fun_control, "lower")
upper = get_bound_values(fun_control, "upper")

from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))

| name        | type   | default   |    lower |   upper | transform   |
|-------------|--------|-----------|----------|---------|-------------|
| C           | float  | 1.0       |   0.1    |   10    | None        |
| kernel      | factor | rbf       |   0      |    0    | None        |
| degree      | int    | 3         |   3      |    3    | None        |
| gamma       | factor | scale     |   0      |    1    | None        |
| coef0       | float  | 0.0       |   0      |    0    | None        |
| shrinking   | factor | 0         |   0      |    1    | None        |
| probability | factor | 0         |   1      |    1    | None        |
| tol         | float  | 0.001     |   0.0001 |    0.01 | None        |
| cache_size  | float  | 200.0     | 100      |  400    | None        |
| break_ties  | factor | 0         |   0      |    1    | None        |

17.14 The Objective Function

The objective function is selected next. It implements an interface from sklearn’s training, validation, and testing methods to spotPython.

from spotPython.fun.hypersklearn import HyperSklearn
fun = HyperSklearn().fun_sklearn

17.14.1 Run the `Spot` Optimizer

Run SPOT for approx. x mins (max_time).
Note: the run takes longer, because the evaluation time of initial design (here: initi_size, 20 points) is not considered.

from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
hyper_dict=SklearnHyperDict().load()
X_start = get_default_hyperparameters_as_array(fun_control, hyper_dict)
X_start

array([[1.e+00, 2.e+00, 3.e+00, 0.e+00, 0.e+00, 0.e+00, 0.e+00, 1.e-03,
        2.e+02, 0.e+00]])

import numpy as np
from spotPython.spot import spot
from math import inf
spot_tuner = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = inf,
                   fun_repeats = 1,
                   max_time = MAX_TIME,
                   noise = False,
                   tolerance_x = np.sqrt(np.spacing(1)),
                   var_type = var_type,
                   var_name = var_name,
                   infill_criterion = "y",
                   n_points = 1,
                   seed=123,
                   log_level = 50,
                   show_models= False,
                   show_progress= True,
                   fun_control = fun_control,
                   design_control={"init_size": INIT_SIZE,
                                   "repeats": 1},
                   surrogate_control={"noise": True,
                                      "cod_type": "norm",
                                      "min_theta": -4,
                                      "max_theta": 3,
                                      "n_theta": len(var_name),
                                      "model_fun_evals": 10_000,
                                      "log_level": 50
                                      })
spot_tuner.run(X_start=X_start)

spotPython tuning: -0.3671679197994987 [----------] 0.34%

spotPython tuning: -0.3671679197994987 [----------] 0.74%

spotPython tuning: -0.3671679197994987 [----------] 1.13%

spotPython tuning: -0.37092731829573933 [----------] 1.48%

spotPython tuning: -0.37092731829573933 [----------] 1.78%

spotPython tuning: -0.380952380952381 [----------] 2.07%

spotPython tuning: -0.380952380952381 [----------] 2.47%

spotPython tuning: -0.380952380952381 [----------] 2.86%

spotPython tuning: -0.38345864661654133 [----------] 3.29%

spotPython tuning: -0.38345864661654133 [----------] 3.76%

spotPython tuning: -0.38345864661654133 [----------] 4.35%

spotPython tuning: -0.38345864661654133 [----------] 4.98%

spotPython tuning: -0.38345864661654133 [#---------] 6.90%

spotPython tuning: -0.38345864661654133 [#---------] 9.06%

spotPython tuning: -0.38345864661654133 [#---------] 10.38%

spotPython tuning: -0.38345864661654133 [#---------] 11.77%

spotPython tuning: -0.38345864661654133 [#---------] 13.13%

spotPython tuning: -0.38345864661654133 [#---------] 14.38%

spotPython tuning: -0.38345864661654133 [##--------] 15.56%

spotPython tuning: -0.38345864661654133 [##--------] 17.10%

spotPython tuning: -0.38345864661654133 [##--------] 18.43%

spotPython tuning: -0.38345864661654133 [##--------] 19.60%

spotPython tuning: -0.39223057644110276 [##--------] 20.76%

spotPython tuning: -0.39223057644110276 [##--------] 21.92%

spotPython tuning: -0.39223057644110276 [##--------] 23.10%

spotPython tuning: -0.39223057644110276 [##--------] 24.19%

spotPython tuning: -0.39223057644110276 [###-------] 25.51%

spotPython tuning: -0.39223057644110276 [###-------] 26.95%

spotPython tuning: -0.39223057644110276 [###-------] 28.74%

spotPython tuning: -0.39223057644110276 [###-------] 30.87%

spotPython tuning: -0.39223057644110276 [###-------] 32.48%

spotPython tuning: -0.39223057644110276 [###-------] 34.26%

spotPython tuning: -0.39223057644110276 [####------] 36.49%

spotPython tuning: -0.39223057644110276 [####------] 38.92%

spotPython tuning: -0.39223057644110276 [####------] 41.18%

spotPython tuning: -0.39223057644110276 [####------] 43.98%

spotPython tuning: -0.39223057644110276 [#####-----] 46.71%

spotPython tuning: -0.39223057644110276 [#####-----] 49.21%

spotPython tuning: -0.39223057644110276 [#####-----] 51.54%

spotPython tuning: -0.39223057644110276 [#####-----] 53.95%

spotPython tuning: -0.39223057644110276 [######----] 56.20%

spotPython tuning: -0.39223057644110276 [######----] 58.51%

spotPython tuning: -0.39223057644110276 [######----] 60.84%

spotPython tuning: -0.39223057644110276 [######----] 63.50%

spotPython tuning: -0.39223057644110276 [#######---] 66.40%

spotPython tuning: -0.39223057644110276 [#######---] 69.45%

spotPython tuning: -0.39223057644110276 [#######---] 72.12%

spotPython tuning: -0.39223057644110276 [########--] 75.34%

spotPython tuning: -0.39223057644110276 [########--] 78.15%

spotPython tuning: -0.39223057644110276 [########--] 81.26%

spotPython tuning: -0.39223057644110276 [########--] 83.65%

spotPython tuning: -0.39223057644110276 [#########-] 86.47%

spotPython tuning: -0.39223057644110276 [#########-] 89.44%

spotPython tuning: -0.39223057644110276 [#########-] 91.82%

spotPython tuning: -0.39223057644110276 [#########-] 94.22%

spotPython tuning: -0.39223057644110276 [##########] 96.69%

spotPython tuning: -0.39223057644110276 [##########] 99.27%

spotPython tuning: -0.39223057644110276 [##########] 100.00% Done...

<spotPython.spot.spot.Spot at 0x13f643be0>

17.14.2 Results

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.

spot_tuner.plot_progress(log_y=False,
    filename="./figures/" + experiment_name+"_progress.png")

Progress plot. *Black* dots denote results from the initial design. *Red* dots illustrate the improvement found by the surrogate model based optimization.

Print the results

print(gen_design_table(fun_control=fun_control,
    spot=spot_tuner))

| name        | type   | default   |   lower |   upper |                tuned | transform   |   importance | stars   |
|-------------|--------|-----------|---------|---------|----------------------|-------------|--------------|---------|
| C           | float  | 1.0       |     0.1 |    10.0 |   2.6058909396076704 | None        |         5.05 | *       |
| kernel      | factor | rbf       |     0.0 |     0.0 |                  0.0 | None        |         0.00 |         |
| degree      | int    | 3         |     3.0 |     3.0 |                  3.0 | None        |         0.00 |         |
| gamma       | factor | scale     |     0.0 |     1.0 |                  1.0 | None        |       100.00 | ***     |
| coef0       | float  | 0.0       |     0.0 |     0.0 |                  0.0 | None        |         0.00 |         |
| shrinking   | factor | 0         |     0.0 |     1.0 |                  1.0 | None        |         1.80 | *       |
| probability | factor | 0         |     1.0 |     1.0 |                  1.0 | None        |         0.00 |         |
| tol         | float  | 0.001     |  0.0001 |    0.01 | 0.003046537270455511 | None        |         0.55 | .       |
| cache_size  | float  | 200.0     |   100.0 |   400.0 |   155.29201232147747 | None        |         0.00 |         |
| break_ties  | factor | 0         |     0.0 |     1.0 |                  1.0 | None        |         0.00 |         |

17.15 Show variable importance

spot_tuner.plot_importance(threshold=0.025, filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

17.16 Get Default Hyperparameters

from spotPython.hyperparameters.values import get_default_values, transform_hyper_parameter_values
values_default = get_default_values(fun_control)
values_default = transform_hyper_parameter_values(fun_control=fun_control, hyper_parameter_values=values_default)
values_default

{'C': 1.0,
 'kernel': 'rbf',
 'degree': 3,
 'gamma': 'scale',
 'coef0': 0.0,
 'shrinking': 0,
 'probability': 0,
 'tol': 0.001,
 'cache_size': 200.0,
 'break_ties': 0}

from sklearn.pipeline import make_pipeline
model_default = make_pipeline(fun_control["prep_model"], fun_control["core_model"](**values_default))
model_default

Pipeline(steps=[('nonetype', None),
                ('svc',
                 SVC(break_ties=0, cache_size=200.0, probability=0,
                     shrinking=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Note

Default value for “probability” is False, but we need it to be True for the metric “mapk_score”.

values_default.update({"probability": 1})

17.17 Get SPOT Results

X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
print(X)

[[2.60589094e+00 0.00000000e+00 3.00000000e+00 1.00000000e+00
  0.00000000e+00 1.00000000e+00 1.00000000e+00 3.04653727e-03
  1.55292012e+02 1.00000000e+00]]

from spotPython.hyperparameters.values import assign_values, return_conf_list_from_var_dict
v_dict = assign_values(X, fun_control["var_name"])
return_conf_list_from_var_dict(var_dict=v_dict, fun_control=fun_control)

[{'C': 2.6058909396076704,
  'kernel': 'rbf',
  'degree': 3,
  'gamma': 'auto',
  'coef0': 0.0,
  'shrinking': 1,
  'probability': 1,
  'tol': 0.003046537270455511,
  'cache_size': 155.29201232147747,
  'break_ties': 1}]

from spotPython.hyperparameters.values import get_one_sklearn_model_from_X
model_spot = get_one_sklearn_model_from_X(X, fun_control)
model_spot

SVC(C=2.6058909396076704, break_ties=1, cache_size=155.29201232147747,
    gamma='auto', probability=1, shrinking=1, tol=0.003046537270455511)

17.18 Evaluate SPOT Results

Fetch the data.

from spotPython.utils.convert import get_Xy_from_df
X_train, y_train = get_Xy_from_df(fun_control["train"], fun_control["target_column"])
X_test, y_test = get_Xy_from_df(fun_control["test"], fun_control["target_column"])
X_test.shape, y_test.shape

((177, 64), (177,))

Fit the model with the tuned hyperparameters. This gives one result:

model_spot.fit(X_train, y_train)
y_pred = model_spot.predict_proba(X_test)
res = mapk_score(y_true=y_test, y_pred=y_pred, k=3)
res

0.37382297551789073

def repeated_eval(n, model):
    res_values = []
    for i in range(n):
        model.fit(X_train, y_train)
        y_pred = model.predict_proba(X_test)
        res = mapk_score(y_true=y_test, y_pred=y_pred, k=3)
        res_values.append(res)
    mean_res = np.mean(res_values)
    print(f"mean_res: {mean_res}")
    std_res = np.std(res_values)
    print(f"std_res: {std_res}")
    min_res = np.min(res_values)
    print(f"min_res: {min_res}")
    max_res = np.max(res_values)
    print(f"max_res: {max_res}")
    median_res = np.median(res_values)
    print(f"median_res: {median_res}")
    return mean_res, std_res, min_res, max_res, median_res

17.18.1 Handling Non-deterministic Results

Because the model is non-determinstic, we perform \(n=30\) runs and calculate the mean and standard deviation of the performance metric.

_ = repeated_eval(30, model_spot)

mean_res: 0.37457627118644066
std_res: 0.004979022247232497
min_res: 0.36346516007532964
max_res: 0.3851224105461393
median_res: 0.37382297551789073

17.18.2 Evalution of the Default Hyperparameters

model_default["svc"].probability = True
model_default.fit(X_train, y_train)["svc"]

SVC(break_ties=0, cache_size=200.0, probability=True, shrinking=0)

One evaluation of the default hyperparameters is performed on the hold-out test set.

y_pred = model_default.predict_proba(X_test)
mapk_score(y_true=y_test, y_pred=y_pred, k=3)

0.3870056497175141

Since one single evaluation is not meaningful, we perform, similar to the evaluation of the SPOT results, \(n=30\) runs of the default setting and and calculate the mean and standard deviation of the performance metric.

_ = repeated_eval(30, model_default)

mean_res: 0.38559322033898297
std_res: 0.0059689571783880695
min_res: 0.3747645951035781
max_res: 0.39924670433145004
median_res: 0.3851224105461394

17.19 Plot: Compare Predictions

from spotPython.plot.validation import plot_confusion_matrix
plot_confusion_matrix(model_default, fun_control, title = "Default")

plot_confusion_matrix(model_spot, fun_control, title="SPOT")

min(spot_tuner.y), max(spot_tuner.y)

(-0.39223057644110276, -0.3333333333333333)

17.20 Cross-validated Evaluations

from spotPython.sklearn.traintest import evaluate_cv
fun_control.update({
     "eval": "train_cv",
     "k_folds": 10,
})
evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

(0.3584905660377358, None)

fun_control.update({
     "eval": "test_cv",
     "k_folds": 10,
})
evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

(0.3528322440087146, None)

This is the evaluation that will be used in the comparison:

fun_control.update({
     "eval": "data_cv",
     "k_folds": 10,
})
evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

(0.35963447350771294, None)

17.20.1 Detailed Hyperparameter Plots

filename = "./figures/" + experiment_name
spot_tuner.plot_important_hyperparameter_contour(filename=filename)

C:  5.052197104676483
gamma:  100.0
shrinking:  1.7978334858401686
tol:  0.5508569476698155

17.21 Parallel Coordinates Plot

spot_tuner.parallel_plot()

17.22 Plot all Combinations of Hyperparameters

Warning: this may take a while.

PLOT_ALL = False
if PLOT_ALL:
    n = spot_tuner.k
    for i in range(n-1):
        for j in range(i+1, n):
            spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)

17.1 Setup

17.2 Step 1: Initialization of the Empty fun_control Dictionary

17.3 1. Load Data: Classification

17.4 VBDP

17.5 Holdout Train and Test Data

17.6 Step 3: Specification of the Preprocessing Model

17.7 Step 4: Select algorithm and core_model_hyper_dict

17.8 Step 5: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model

17.8.1 Modify hyperparameter of type numeric and integer (boolean)

17.8.2 Modify hyperparameter of type factor

17.8.3 Optimizers

17.9 5. Selection of the Objective: Metric and Loss Functions

17.10 Step 6: Selection of the Objective (Loss) Function

17.10.1 Metric Function

17.10.1.1 The MAPK Metric

17.10.1.2 Other Metrics

17.11 Evaluation on Hold-out Data

17.11.0.1 Cross Validation

17.12 6. Calling the SPOT Function

17.13 Preparing the SPOT Call

17.14 The Objective Function

17.14.1 Run the Spot Optimizer

17.14.2 Results

17.15 Show variable importance

17.16 Get Default Hyperparameters

17.17 Get SPOT Results

17.18 Evaluate SPOT Results

17.18.1 Handling Non-deterministic Results

17.18.2 Evalution of the Default Hyperparameters

17.19 Plot: Compare Predictions

17.20 Cross-validated Evaluations

17.20.1 Detailed Hyperparameter Plots

17.21 Parallel Coordinates Plot

17.22 Plot all Combinations of Hyperparameters

17.2 Step 1: Initialization of the Empty `fun_control` Dictionary

17.7 Step 4: Select `algorithm` and `core_model_hyper_dict`

17.8 Step 5: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

17.14.1 Run the `Spot` Optimizer