pip list | grep "spot[RiverPython]"spotPython 0.2.46
spotRiver 0.0.93Note: you may need to restart the kernel to use updated packages.20 HPT: PyTorch With VBDP
In this tutorial, we will show how spotPython can be integrated into the PyTorch training workflow for a classifiaction task.
Caution: Data must be downloaded manually
- Ensure that the correspondiing data is available as
./data/VBDP/train.csv.
This document refers to the following software versions:
python: 3.10.10torch: 2.0.1torchvision: 0.15.0
spotPython can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.
!pip install spotPython- Uncomment the following lines if you want to for (re-)installation the latest version of
spotPythonfrom gitHub.
# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython20.1 Step 1: Setup
Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.
Caution: Run time and initial design size should be increased for real experiments
- MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
- INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.
Note: Device selection
- The device can be selected by setting the variable
DEVICE. - Since we are using a simple neural net, the setting
"cpu"is preferred (on Mac). - If you have a GPU, you can use
"cuda:0"instead. - If DEVICE is set to
None,spotPythonwill automatically select the device.- This might result in
"mps"on Macs, which is not the best choice for simple neural nets.
- This might result in
MAX_TIME = 1
INIT_SIZE = 5
DEVICE = None # "cpu" # "cuda:0"from spotPython.utils.device import getDevice
DEVICE = getDevice(DEVICE)
print(DEVICE)mpsimport os
import copy
import socket
from datetime import datetime
from dateutil.tz import tzlocal
start_time = datetime.now(tzlocal())
HOSTNAME = socket.gethostname().split(".")[0]
experiment_name = '25-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
experiment_name = experiment_name.replace(':', '-')
print(experiment_name)
if not os.path.exists('./figures'):
os.makedirs('./figures')25-torch_bartz09_1min_5init_2023-06-27_04-11-2420.2 Step 2: Initialization of the fun_control Dictionary
Caution: Tensorboard does not work under Windows
- Since tensorboard does not work under Windows, we recommend setting the parameter
tensorboard_pathtoNoneif you are working under Windows.
spotPython uses a Python dictionary for storing the information required for the hyperparameter tuning process, which was described in Section 14.2, see Initialization of the fun_control Dictionary in the documentation.
from spotPython.utils.init import fun_control_init
fun_control = fun_control_init(task="classification",
tensorboard_path="runs/25_spot_torch_vbdp",
device=DEVICE)20.3 Step 3: PyTorch Data Loading
20.3.1 1. Load VBDP Data
import pandas as pd
from sklearn.preprocessing import OrdinalEncoder
train_df = pd.read_csv('./data/VBDP/train.csv')
# remove the id column
train_df = train_df.drop(columns=['id'])
n_samples = train_df.shape[0]
n_features = train_df.shape[1] - 1
target_column = "prognosis"
# # Encoder our prognosis labels as integers for easier decoding later
enc = OrdinalEncoder()
train_df[target_column] = enc.fit_transform(train_df[[target_column]])
train_df.head()
# convert all entries to int for faster processing
train_df = train_df.astype(int)- Add logical combinations (AND, OR, XOR) of the features to the data set:
from spotPython.utils.convert import add_logical_columns
df_new = train_df.copy()
# save the target column using "target_column" as the column name
target = train_df[target_column]
# remove the target column
df_new = df_new.drop(columns=[target_column])
train_df = add_logical_columns(df_new)
# add the target column back
train_df[target_column] = target
train_df = train_df.astype(int)
train_df.head()| sudden_fever | headache | mouth_bleed | nose_bleed | muscle_pain | joint_pain | vomiting | rash | diarrhea | hypotension | ... | 6039 | 6040 | 6041 | 6042 | 6043 | 6044 | 6045 | 6046 | 6047 | prognosis | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 |
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 |
| 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | ... | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 3 |
| 3 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | ... | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 6 |
5 rows × 6113 columns
from sklearn.model_selection import train_test_split
import numpy as np
n_samples = train_df.shape[0]
n_features = train_df.shape[1] - 1
train_df.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
train_df.head()| x1 | x2 | x3 | x4 | x5 | x6 | x7 | x8 | x9 | x10 | ... | x6104 | x6105 | x6106 | x6107 | x6108 | x6109 | x6110 | x6111 | x6112 | prognosis | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 |
| 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 7 |
| 2 | 0 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | ... | 1 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 3 |
| 3 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | ... | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 10 |
| 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | ... | 0 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 6 |
5 rows × 6113 columns
20.3.2 Check content of the target column
train_df[target_column].head()0 3
1 7
2 3
3 10
4 6
Name: prognosis, dtype: int64X_train, X_test, y_train, y_test = train_test_split(train_df.drop(target_column, axis=1), train_df[target_column],
random_state=42,
test_size=0.25,
stratify=train_df[target_column])
trainset = pd.DataFrame(np.hstack((X_train, np.array(y_train).reshape(-1, 1))))
testset = pd.DataFrame(np.hstack((X_test, np.array(y_test).reshape(-1, 1))))
trainset.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
testset.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
print(train_df.shape)
print(trainset.shape)
print(testset.shape)(707, 6113)
(530, 6113)
(177, 6113)import torch
from sklearn.model_selection import train_test_split
from spotPython.torch.dataframedataset import DataFrameDataset
dtype_x = torch.float32
dtype_y = torch.long
train_df = DataFrameDataset(train_df, target_column=target_column, dtype_x=dtype_x, dtype_y=dtype_y)
train = DataFrameDataset(trainset, target_column=target_column, dtype_x=dtype_x, dtype_y=dtype_y)
test = DataFrameDataset(testset, target_column=target_column, dtype_x=dtype_x, dtype_y=dtype_y)
n_samples = len(train)# add the dataset to the fun_control
fun_control.update({"data": train_df, # full dataset,
"train": train,
"test": test,
"n_samples": n_samples,
"target_column": target_column})20.4 Step 4: Specification of the Preprocessing Model
After the training and test data are specified and added to the fun_control dictionary, spotPython allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables, see Section 14.4. This feature is not used here, so we do not change the default value (which is None).
20.5 Step 5: Select algorithm and core_model_hyper_dict
20.5.1 Implementing a Configurable Neural Network With spotPython
spotPython includes the Net_vbdp class which is implemented in the file netvbdp.py. The class is imported here.
This class inherits from the class Net_Core which is implemented in the file netcore.py, see Section 14.5.1.
20.5.2 Add the NN Model to the fun_control Dictionary
from spotPython.torch.netvbdp import Net_vbdp
from spotPython.data.torch_hyper_dict import TorchHyperDict
from spotPython.hyperparameters.values import add_core_model_to_fun_control
fun_control = add_core_model_to_fun_control(core_model=Net_vbdp,
fun_control=fun_control,
hyper_dict=TorchHyperDict)The corresponding entries for the core_model class are shown below.
fun_control['core_model_hyper_dict']{'_L0': {'type': 'int',
'default': 64,
'transform': 'None',
'lower': 64,
'upper': 64},
'l1': {'type': 'int',
'default': 8,
'transform': 'transform_power_2_int',
'lower': 8,
'upper': 16},
'dropout_prob': {'type': 'float',
'default': 0.01,
'transform': 'None',
'lower': 0.0,
'upper': 0.9},
'lr_mult': {'type': 'float',
'default': 1.0,
'transform': 'None',
'lower': 0.1,
'upper': 10.0},
'batch_size': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 1,
'upper': 4},
'epochs': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 4,
'upper': 9},
'k_folds': {'type': 'int',
'default': 1,
'transform': 'None',
'lower': 1,
'upper': 1},
'patience': {'type': 'int',
'default': 2,
'transform': 'transform_power_2_int',
'lower': 1,
'upper': 5},
'optimizer': {'levels': ['Adadelta',
'Adagrad',
'Adam',
'AdamW',
'SparseAdam',
'Adamax',
'ASGD',
'NAdam',
'RAdam',
'RMSprop',
'Rprop',
'SGD'],
'type': 'factor',
'default': 'SGD',
'transform': 'None',
'class_name': 'torch.optim',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 12},
'sgd_momentum': {'type': 'float',
'default': 0.0,
'transform': 'None',
'lower': 0.0,
'upper': 1.0}}20.6 Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model
spotPython provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 14.6.
Caution: Small number of epochs for demonstration purposes
epochsandpatienceare set to small values for demonstration purposes. These values are too small for a real application.- More resonable values are, e.g.:
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[7, 9])andfun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 7])
from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
fun_control = modify_hyper_parameter_bounds(fun_control, "_L0", bounds=[n_features, n_features])
fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[6, 13])
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[2, 3])
fun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 2])from spotPython.hyperparameters.values import modify_hyper_parameter_levels
fun_control = modify_hyper_parameter_levels(fun_control, "optimizer",["Adam", "AdamW", "Adamax", "NAdam"])
# fun_control = modify_hyper_parameter_levels(fun_control, "optimizer", ["Adam"])
# fun_control["core_model_hyper_dict"]20.6.1 Optimizers
Optimizers are described in Section 14.6.1.
fun_control = modify_hyper_parameter_bounds(fun_control,
"lr_mult", bounds=[1e-3, 1e-3])
fun_control = modify_hyper_parameter_bounds(fun_control,
"sgd_momentum", bounds=[0.9, 0.9])20.7 Step 7: Selection of the Objective (Loss) Function
20.7.1 Evaluation
The evaluation procedure requires the specification of two elements:
- the way how the data is split into a train and a test set (see Section 14.7.1)
- the loss function (and a metric).
20.7.2 Loss Functions and Metrics
The loss function is specified by the key "loss_function". We will use CrossEntropy loss for the multiclass-classification task.
from torch.nn import CrossEntropyLoss
loss_function = CrossEntropyLoss()
fun_control.update({"loss_function": loss_function})20.7.3 Metric
- We will use the MAP@k metric for the evaluation of the model. Here is an example how this metric is calculated.
from spotPython.torch.mapk import MAPK
import torch
mapk = MAPK(k=2)
target = torch.tensor([0, 1, 2, 2])
preds = torch.tensor(
[
[0.5, 0.2, 0.2], # 0 is in top 2
[0.3, 0.4, 0.2], # 1 is in top 2
[0.2, 0.4, 0.3], # 2 is in top 2
[0.7, 0.2, 0.1], # 2 isn't in top 2
]
)
mapk.update(preds, target)
print(mapk.compute()) # tensor(0.6250)tensor(0.6250)from spotPython.torch.mapk import MAPK
import torchmetrics
metric_torch = MAPK(k=3)
fun_control.update({"metric_torch": metric_torch})20.8 Step 8: Calling the SPOT Function
20.8.1 Preparing the SPOT Call
The following code passes the information about the parameter ranges and bounds to spot.
# extract the variable types, names, and bounds
from spotPython.hyperparameters.values import (get_bound_values,
get_var_name,
get_var_type,)
var_type = get_var_type(fun_control)
var_name = get_var_name(fun_control)
fun_control.update({"var_type": var_type,
"var_name": var_name})
lower = get_bound_values(fun_control, "lower")
upper = get_bound_values(fun_control, "upper")Now, the dictionary fun_control contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method gen_design_table generates a design table as follows:
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))| name | type | default | lower | upper | transform |
|--------------|--------|-----------|----------|----------|-----------------------|
| _L0 | int | 64 | 6112 | 6112 | None |
| l1 | int | 8 | 6 | 13 | transform_power_2_int |
| dropout_prob | float | 0.01 | 0 | 0.9 | None |
| lr_mult | float | 1.0 | 0.001 | 0.001 | None |
| batch_size | int | 4 | 1 | 4 | transform_power_2_int |
| epochs | int | 4 | 2 | 3 | transform_power_2_int |
| k_folds | int | 1 | 1 | 1 | None |
| patience | int | 2 | 2 | 2 | transform_power_2_int |
| optimizer | factor | SGD | 0 | 3 | None |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | None |This allows to check if all information is available and if the information is correct.
20.8.2 The Objective Function fun_torch
The objective function fun_torch is selected next. It implements an interface from PyTorch’s training, validation, and testing methods to spotPython.
from spotPython.fun.hypertorch import HyperTorch
fun = HyperTorch().fun_torchfrom spotPython.hyperparameters.values import get_default_hyperparameters_as_array
hyper_dict=TorchHyperDict().load()
X_start = get_default_hyperparameters_as_array(fun_control, hyper_dict)20.8.3 Starting the Hyperparameter Tuning
The spotPython hyperparameter tuning is started by calling the Spot function as described in Section 14.8.4.
import numpy as np
from spotPython.spot import spot
from math import inf
spot_tuner = spot.Spot(fun=fun,
lower = lower,
upper = upper,
fun_evals = inf,
fun_repeats = 1,
max_time = MAX_TIME,
noise = False,
tolerance_x = np.sqrt(np.spacing(1)),
var_type = var_type,
var_name = var_name,
infill_criterion = "y",
n_points = 1,
seed=123,
log_level = 50,
show_models= False,
show_progress= True,
fun_control = fun_control,
design_control={"init_size": INIT_SIZE,
"repeats": 1},
surrogate_control={"noise": True,
"cod_type": "norm",
"min_theta": -4,
"max_theta": 3,
"n_theta": len(var_name),
"model_fun_evals": 10_000,
"log_level": 50
})
spot_tuner.run(X_start=X_start)
config: {'_L0': 6112, 'l1': 2048, 'dropout_prob': 0.17031221661559992, 'lr_mult': 0.001, 'batch_size': 16, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1316964477300644 | Loss: 2.3979728562491283 | Acc: 0.0613207547169811.
Epoch: 2 | MAPK: 0.1555059701204300 | Loss: 2.3979720047542026 | Acc: 0.0613207547169811.
Epoch: 3 | MAPK: 0.1569940596818924 | Loss: 2.3978780848639354 | Acc: 0.0613207547169811.
Epoch: 4 | MAPK: 0.1517857164144516 | Loss: 2.3978367362703596 | Acc: 0.0613207547169811.
Epoch: 5 | MAPK: 0.1674107164144516 | Loss: 2.3977772338049754 | Acc: 0.0613207547169811.
Epoch: 6 | MAPK: 0.1562500000000000 | Loss: 2.3977687529155185 | Acc: 0.0613207547169811.
Epoch: 7 | MAPK: 0.1607142835855484 | Loss: 2.3977217163358415 | Acc: 0.0707547169811321.
Epoch: 8 | MAPK: 0.1636904776096344 | Loss: 2.3977202347346713 | Acc: 0.0613207547169811.
Returned to Spot: Validation loss: 2.3977202347346713
config: {'_L0': 6112, 'l1': 256, 'dropout_prob': 0.19379790035512987, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 4, 'k_folds': 1, 'patience': 4, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}Epoch: 1 | MAPK: 0.1358024775981903 | Loss: 2.3977888601797597 | Acc: 0.0660377358490566.
Epoch: 2 | MAPK: 0.1558642089366913 | Loss: 2.3976964862258345 | Acc: 0.0660377358490566.
Epoch: 3 | MAPK: 0.1435185223817825 | Loss: 2.3977048661973743 | Acc: 0.0660377358490566.
Epoch: 4 | MAPK: 0.1466049253940582 | Loss: 2.3977002744321472 | Acc: 0.0660377358490566.
Returned to Spot: Validation loss: 2.397700274432147
config: {'_L0': 6112, 'l1': 4096, 'dropout_prob': 0.6759063718076167, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1918238848447800 | Loss: 2.3977315853226862 | Acc: 0.1084905660377359.
Epoch: 2 | MAPK: 0.1902515292167664 | Loss: 2.3976550327157073 | Acc: 0.1132075471698113.
Epoch: 3 | MAPK: 0.1847484111785889 | Loss: 2.3975482909184582 | Acc: 0.1132075471698113.
Epoch: 4 | MAPK: 0.1823899298906326 | Loss: 2.3973221351515570 | Acc: 0.1132075471698113.
Epoch: 5 | MAPK: 0.1957546621561050 | Loss: 2.3971375384420717 | Acc: 0.1132075471698113.
Epoch: 6 | MAPK: 0.1839622706174850 | Loss: 2.3966186743862226 | Acc: 0.1132075471698113.
Epoch: 7 | MAPK: 0.1926099807024002 | Loss: 2.3964016257591969 | Acc: 0.1132075471698113.
Epoch: 8 | MAPK: 0.1996854692697525 | Loss: 2.3956185489330650 | Acc: 0.1132075471698113.
Returned to Spot: Validation loss: 2.395618548933065
config: {'_L0': 6112, 'l1': 128, 'dropout_prob': 0.37306669346546995, 'lr_mult': 0.001, 'batch_size': 4, 'epochs': 4, 'k_folds': 1, 'patience': 4, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1485849171876907 | Loss: 2.3982868779380366 | Acc: 0.0613207547169811.
Epoch: 2 | MAPK: 0.1485849171876907 | Loss: 2.3982661355216548 | Acc: 0.0613207547169811.
Epoch: 3 | MAPK: 0.1485849171876907 | Loss: 2.3982642371699496 | Acc: 0.0613207547169811.
Epoch: 4 | MAPK: 0.1485849171876907 | Loss: 2.3982994916304103 | Acc: 0.0613207547169811.
Returned to Spot: Validation loss: 2.3982994916304103
config: {'_L0': 6112, 'l1': 1024, 'dropout_prob': 0.870137281216666, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.2029320895671844 | Loss: 2.3974096156932689 | Acc: 0.1132075471698113.
Epoch: 2 | MAPK: 0.1828703880310059 | Loss: 2.3976021696020058 | Acc: 0.0896226415094340.
Epoch: 3 | MAPK: 0.1929012238979340 | Loss: 2.3976025757966219 | Acc: 0.1084905660377359.
Epoch: 4 | MAPK: 0.2206790149211884 | Loss: 2.3973903744309037 | Acc: 0.1320754716981132.
Epoch: 5 | MAPK: 0.1820987462997437 | Loss: 2.3972626880363181 | Acc: 0.0849056603773585.
Epoch: 6 | MAPK: 0.2075617015361786 | Loss: 2.3973924937071622 | Acc: 0.1367924528301887.
Epoch: 7 | MAPK: 0.2060185223817825 | Loss: 2.3974992522486933 | Acc: 0.1226415094339623.
Epoch: 8 | MAPK: 0.2183641791343689 | Loss: 2.3973672301680953 | Acc: 0.1320754716981132.
Returned to Spot: Validation loss: 2.3973672301680953
config: {'_L0': 6112, 'l1': 512, 'dropout_prob': 0.6757658889423604, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1929012387990952 | Loss: 2.3979867564307318 | Acc: 0.1367924528301887.
Epoch: 2 | MAPK: 0.1936728358268738 | Loss: 2.3980418752740928 | Acc: 0.1462264150943396.
Epoch: 3 | MAPK: 0.1898147910833359 | Loss: 2.3979952158751310 | Acc: 0.1462264150943396.
Epoch: 4 | MAPK: 0.1952160596847534 | Loss: 2.3978447560910827 | Acc: 0.1462264150943396.
Epoch: 5 | MAPK: 0.1882715970277786 | Loss: 2.3978271219465466 | Acc: 0.1462264150943396.
Epoch: 6 | MAPK: 0.1905864030122757 | Loss: 2.3979860058537237 | Acc: 0.1509433962264151.
Epoch: 7 | MAPK: 0.1921296119689941 | Loss: 2.3976853423648410 | Acc: 0.1509433962264151.
Epoch: 8 | MAPK: 0.1936728358268738 | Loss: 2.3979068243945085 | Acc: 0.1603773584905660.
Returned to Spot: Validation loss: 2.3979068243945085
spotPython tuning: 2.395618548933065 [#---------] 5.09%
config: {'_L0': 6112, 'l1': 4096, 'dropout_prob': 0.6451395692472426, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1643081903457642 | Loss: 2.3978354705954499 | Acc: 0.1084905660377359.
Epoch: 2 | MAPK: 0.2083333283662796 | Loss: 2.3975371527221969 | Acc: 0.1462264150943396.
Epoch: 3 | MAPK: 0.2256289124488831 | Loss: 2.3973583770248124 | Acc: 0.1509433962264151.
Epoch: 4 | MAPK: 0.2358490377664566 | Loss: 2.3971397112000665 | Acc: 0.1509433962264151.
Epoch: 5 | MAPK: 0.2272012084722519 | Loss: 2.3969265807349727 | Acc: 0.1273584905660377.
Epoch: 6 | MAPK: 0.2444967776536942 | Loss: 2.3964434727182926 | Acc: 0.1462264150943396.
Epoch: 7 | MAPK: 0.2429245263338089 | Loss: 2.3959800927144177 | Acc: 0.1462264150943396.
Epoch: 8 | MAPK: 0.2539307773113251 | Loss: 2.3953609129167952 | Acc: 0.1462264150943396.
Returned to Spot: Validation loss: 2.395360912916795
spotPython tuning: 2.395360912916795 [####------] 43.26%
config: {'_L0': 6112, 'l1': 1024, 'dropout_prob': 0.8297695457624555, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1581790298223495 | Loss: 2.3979962666829429 | Acc: 0.0990566037735849.
Epoch: 2 | MAPK: 0.1790123283863068 | Loss: 2.3980064215483488 | Acc: 0.0943396226415094.
Epoch: 3 | MAPK: 0.1442901343107224 | Loss: 2.3982671896616616 | Acc: 0.0896226415094340.
Epoch: 4 | MAPK: 0.1581790298223495 | Loss: 2.3978902675487377 | Acc: 0.0943396226415094.
Epoch: 5 | MAPK: 0.1628086417913437 | Loss: 2.3980069955190024 | Acc: 0.0896226415094340.
Epoch: 6 | MAPK: 0.1527777910232544 | Loss: 2.3981831426973694 | Acc: 0.0943396226415094.
Epoch: 7 | MAPK: 0.1666667014360428 | Loss: 2.3979461987813315 | Acc: 0.0990566037735849.
Epoch: 8 | MAPK: 0.1990740746259689 | Loss: 2.3977928426530628 | Acc: 0.1084905660377359.
Returned to Spot: Validation loss: 2.397792842653063
spotPython tuning: 2.395360912916795 [#####-----] 48.18%
config: {'_L0': 6112, 'l1': 4096, 'dropout_prob': 0.4132005099912892, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1548742502927780 | Loss: 2.3978520204436102 | Acc: 0.0660377358490566.
Epoch: 2 | MAPK: 0.1973270326852798 | Loss: 2.3974758229165709 | Acc: 0.0943396226415094.
Epoch: 3 | MAPK: 0.2177672535181046 | Loss: 2.3969496038724794 | Acc: 0.1320754716981132.
Epoch: 4 | MAPK: 0.1926100403070450 | Loss: 2.3964718895138435 | Acc: 0.0849056603773585.
Epoch: 5 | MAPK: 0.1894653737545013 | Loss: 2.3955456220878744 | Acc: 0.0660377358490566.
Epoch: 6 | MAPK: 0.1965408474206924 | Loss: 2.3942849186231507 | Acc: 0.0801886792452830.
Epoch: 7 | MAPK: 0.2012578547000885 | Loss: 2.3926740299980596 | Acc: 0.0801886792452830.
Epoch: 8 | MAPK: 0.2114779204130173 | Loss: 2.3907487392425537 | Acc: 0.0990566037735849.
Returned to Spot: Validation loss: 2.3907487392425537
spotPython tuning: 2.3907487392425537 [#########-] 86.28%
config: {'_L0': 6112, 'l1': 4096, 'dropout_prob': 0.39370640861812795, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 1, 'patience': 4, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1851851791143417 | Loss: 2.3977313571506076 | Acc: 0.0943396226415094.
Epoch: 2 | MAPK: 0.1751543134450912 | Loss: 2.3978461601116039 | Acc: 0.0896226415094340.
Epoch: 3 | MAPK: 0.1921296268701553 | Loss: 2.3977015283372669 | Acc: 0.0849056603773585.
Epoch: 4 | MAPK: 0.2083333134651184 | Loss: 2.3975946638319225 | Acc: 0.1179245283018868.
Epoch: 5 | MAPK: 0.2199074178934097 | Loss: 2.3976288106706409 | Acc: 0.1415094339622641.
Epoch: 6 | MAPK: 0.2091049551963806 | Loss: 2.3974509680712663 | Acc: 0.1132075471698113.
Epoch: 7 | MAPK: 0.2175926119089127 | Loss: 2.3973511678201183 | Acc: 0.1273584905660377.
Epoch: 8 | MAPK: 0.1944444626569748 | Loss: 2.3972847461700439 | Acc: 0.1084905660377359.
Returned to Spot: Validation loss: 2.397284746170044
spotPython tuning: 2.3907487392425537 [##########] 95.08%
config: {'_L0': 6112, 'l1': 256, 'dropout_prob': 0.3929325597144649, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 4, 'k_folds': 1, 'patience': 4, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
Epoch: 1 | MAPK: 0.1808176040649414 | Loss: 2.3983495617812536 | Acc: 0.1132075471698113.
Epoch: 2 | MAPK: 0.1894654184579849 | Loss: 2.3982177680393435 | Acc: 0.1226415094339623.
Epoch: 3 | MAPK: 0.2028301507234573 | Loss: 2.3982363466946586 | Acc: 0.1273584905660377.
Epoch: 4 | MAPK: 0.2044025212526321 | Loss: 2.3981041593371697 | Acc: 0.1273584905660377.
Returned to Spot: Validation loss: 2.3981041593371697
spotPython tuning: 2.3907487392425537 [##########] 100.00% Done...
<spotPython.spot.spot.Spot at 0x2c963b220>20.9 Step 9: Tensorboard
The textual output shown in the console (or code cell) can be visualized with Tensorboard as described in Section 14.9, see also the description in the documentation: Tensorboard.
20.10 Step 10: Results
After the hyperparameter tuning run is finished, the results can be analyzed as described in Section 14.10.
spot_tuner.plot_progress(log_y=False,
filename="./figures/" + experiment_name+"_progress.png")
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))| name | type | default | lower | upper | tuned | transform | importance | stars |
|--------------|--------|-----------|---------|---------|--------------------|-----------------------|--------------|---------|
| _L0 | int | 64 | 6112.0 | 6112.0 | 6112.0 | None | 0.00 | |
| l1 | int | 8 | 6.0 | 13.0 | 12.0 | transform_power_2_int | 0.00 | |
| dropout_prob | float | 0.01 | 0.0 | 0.9 | 0.4132005099912892 | None | 10.19 | * |
| lr_mult | float | 1.0 | 0.001 | 0.001 | 0.001 | None | 0.00 | |
| batch_size | int | 4 | 1.0 | 4.0 | 1.0 | transform_power_2_int | 5.01 | * |
| epochs | int | 4 | 2.0 | 3.0 | 3.0 | transform_power_2_int | 100.00 | *** |
| k_folds | int | 1 | 1.0 | 1.0 | 1.0 | None | 0.00 | |
| patience | int | 2 | 2.0 | 2.0 | 2.0 | transform_power_2_int | 0.00 | |
| optimizer | factor | SGD | 0.0 | 3.0 | 3.0 | None | 12.23 | * |
| sgd_momentum | float | 0.0 | 0.9 | 0.9 | 0.9 | None | 0.00 | |spot_tuner.plot_importance(threshold=0.025,
filename="./figures/" + experiment_name+"_importance.png")
20.10.1 Get the Tuned Architecture
from spotPython.hyperparameters.values import get_one_core_model_from_X
X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
model_spot = get_one_core_model_from_X(X, fun_control)
model_spotNet_vbdp(
(fc1): Linear(in_features=6112, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=2048, bias=True)
(fc3): Linear(in_features=2048, out_features=1024, bias=True)
(fc4): Linear(in_features=1024, out_features=512, bias=True)
(fc5): Linear(in_features=512, out_features=11, bias=True)
(relu): ReLU()
(softmax): Softmax(dim=1)
(dropout1): Dropout(p=0.4132005099912892, inplace=False)
(dropout2): Dropout(p=0.2066002549956446, inplace=False)
)20.10.2 Evaluation of the Tuned Architecture
from spotPython.torch.traintest import (
train_tuned,
test_tuned,
)
train_tuned(net=model_spot, train_dataset=train,
loss_function=fun_control["loss_function"],
metric=fun_control["metric_torch"],
shuffle=True,
device = fun_control["device"],
path=None,
task=fun_control["task"],)Epoch: 1 | MAPK: 0.1894654035568237 | Loss: 2.3973950687444434 | Acc: 0.1037735849056604.
Epoch: 2 | MAPK: 0.1965408623218536 | Loss: 2.3972069587347642 | Acc: 0.0943396226415094.
Epoch: 3 | MAPK: 0.2067609876394272 | Loss: 2.3968847630158909 | Acc: 0.1179245283018868.
Epoch: 4 | MAPK: 0.2201257497072220 | Loss: 2.3962177645485356 | Acc: 0.1273584905660377.
Epoch: 5 | MAPK: 0.2224842458963394 | Loss: 2.3954857655291288 | Acc: 0.1084905660377359.
Epoch: 6 | MAPK: 0.2311320453882217 | Loss: 2.3944394700932055 | Acc: 0.1132075471698113.
Epoch: 7 | MAPK: 0.2413521707057953 | Loss: 2.3928272521720744 | Acc: 0.1132075471698113.
Epoch: 8 | MAPK: 0.2452829629182816 | Loss: 2.3908711379429080 | Acc: 0.1179245283018868.
Returned to Spot: Validation loss: 2.390871137942908If path is set to a filename, e.g., path = "model_spot_trained.pt", the weights of the trained model will be loaded from this file.
test_tuned(net=model_spot, test_dataset=test,
shuffle=False,
loss_function=fun_control["loss_function"],
metric=fun_control["metric_torch"],
device = fun_control["device"],
task=fun_control["task"],)MAPK: 0.2668538987636566 | Loss: 2.3900089906842523 | Acc: 0.1242937853107345.
Final evaluation: Validation loss: 2.3900089906842523
Final evaluation: Validation metric: 0.2668538987636566
----------------------------------------------(2.3900089906842523, nan, tensor(0.2669))20.10.3 Cross-validated Evaluations
- This is the evaluation that will be used in the comparison.
Caution: Cross-validated Evaluations
- The number of folds is set to 1 by default.
- Here it was changed to 3 for demonstration purposes.
- Set the number of folds to a reasonable value, e.g., 10.
- This can be done by setting the
k_foldsattribute of the model as follows: setattr(model_spot, "k_folds", 10)
from spotPython.torch.traintest import evaluate_cv
# modify k-kolds:
setattr(model_spot, "k_folds", 3)
df_eval, df_preds, df_metrics = evaluate_cv(net=model_spot,
dataset=fun_control["data"],
loss_function=fun_control["loss_function"],
metric=fun_control["metric_torch"],
task=fun_control["task"],
writer=fun_control["writer"],
writerId="model_spot_cv",
device = fun_control["device"])Fold: 1
Epoch: 1 | MAPK: 0.1850282251834869 | Loss: 2.3975222999766723 | Acc: 0.0677966101694915.
Epoch: 2 | MAPK: 0.1793785244226456 | Loss: 2.3967978146116611 | Acc: 0.0677966101694915.
Epoch: 3 | MAPK: 0.1800847351551056 | Loss: 2.3957625566902809 | Acc: 0.0805084745762712.
Epoch: 4 | MAPK: 0.2040960192680359 | Loss: 2.3941442420927146 | Acc: 0.1186440677966102.
Epoch: 5 | MAPK: 0.2132768034934998 | Loss: 2.3915036492428539 | Acc: 0.1271186440677966.
Epoch: 6 | MAPK: 0.2372880876064301 | Loss: 2.3879980717675160 | Acc: 0.1398305084745763.
Epoch: 7 | MAPK: 0.2394067347049713 | Loss: 2.3837011587821832 | Acc: 0.1313559322033898.
Epoch: 8 | MAPK: 0.2535310387611389 | Loss: 2.3797799409446068 | Acc: 0.1355932203389831.
Fold: 2
Epoch: 1 | MAPK: 0.1426553726196289 | Loss: 2.3975857617491383 | Acc: 0.0677966101694915.
Epoch: 2 | MAPK: 0.2238699793815613 | Loss: 2.3969454644090038 | Acc: 0.0932203389830508.
Epoch: 3 | MAPK: 0.2281072586774826 | Loss: 2.3962483406066895 | Acc: 0.1398305084745763.
Epoch: 4 | MAPK: 0.2125706076622009 | Loss: 2.3951340610698120 | Acc: 0.1228813559322034.
Epoch: 5 | MAPK: 0.2217513769865036 | Loss: 2.3931047977027244 | Acc: 0.1228813559322034.
Epoch: 6 | MAPK: 0.2443502694368362 | Loss: 2.3899877354250116 | Acc: 0.1228813559322034.
Epoch: 7 | MAPK: 0.2627117931842804 | Loss: 2.3872380761776939 | Acc: 0.1610169491525424.
Epoch: 8 | MAPK: 0.2690677940845490 | Loss: 2.3835569216033159 | Acc: 0.1737288135593220.
Fold: 3
Epoch: 1 | MAPK: 0.2372881621122360 | Loss: 2.3973031831999956 | Acc: 0.1234042553191489.
Epoch: 2 | MAPK: 0.2471750527620316 | Loss: 2.3967881950281433 | Acc: 0.1148936170212766.
Epoch: 3 | MAPK: 0.2323445826768875 | Loss: 2.3959370588852189 | Acc: 0.1148936170212766.
Epoch: 4 | MAPK: 0.2224575579166412 | Loss: 2.3945886300782027 | Acc: 0.1148936170212766.
Epoch: 5 | MAPK: 0.2153954803943634 | Loss: 2.3923960176564880 | Acc: 0.1191489361702128.
Epoch: 6 | MAPK: 0.2097457349300385 | Loss: 2.3890991493807001 | Acc: 0.1234042553191489.
Epoch: 7 | MAPK: 0.2175140529870987 | Loss: 2.3851537078113880 | Acc: 0.1276595744680851.
Epoch: 8 | MAPK: 0.2387004941701889 | Loss: 2.3817195932743913 | Acc: 0.1361702127659574.metric_name = type(fun_control["metric_torch"]).__name__
print(f"loss: {df_eval}, Cross-validated {metric_name}: {df_metrics}")loss: 2.3816854852741045, Cross-validated MAPK: 0.253766447305679320.10.4 Detailed Hyperparameter Plots
filename = "./figures/" + experiment_name
spot_tuner.plot_important_hyperparameter_contour(filename=filename)dropout_prob: 10.186513486511242
batch_size: 5.011514729894648
epochs: 100.0
optimizer: 12.227028376797849
20.10.5 Parallel Coordinates Plot
spot_tuner.parallel_plot()Parallel coordinates plots
# close tensorbaoard writer
if fun_control["writer"] is not None:
fun_control["writer"].close()20.10.6 Plot all Combinations of Hyperparameters
- Warning: this may take a while.
PLOT_ALL = False
if PLOT_ALL:
n = spot_tuner.k
for i in range(n-1):
for j in range(i+1, n):
spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)