pip list | grep "spot[RiverPython]"spotPython 0.2.52
spotRiver 0.0.94Note: you may need to restart the kernel to use updated packages.21 HPT PyTorch Lightning: VBDP
In this tutorial, we will show how spotPython can be integrated into the PyTorch Lightning training workflow for a classifiaction task.
Caution: Data must be downloaded manually
- Ensure that the corresponding data is available as
./data/VBDP/train.csv.
This document refers to the following software versions:
python: 3.10.10torch: 2.0.1torchvision: 0.15.0
spotPython can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.
!pip install spotPython- Uncomment the following lines if you want to for (re-)installation the latest version of
spotPythonfrom GitHub.
# import sys
# !{sys.executable} -m pip install --upgrade build
# !{sys.executable} -m pip install --upgrade --force-reinstall spotPython21.1 Step 1: Setup
Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.
Caution: Run time and initial design size should be increased for real experiments
- MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
- INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.
- WORKERS is set to 0 for demonstration purposes. For real experiments, this should be increased. See the warnings that are printed when the number of workers is set to 0.
Note: Device selection
- The device can be selected by setting the variable
DEVICE. - Since we are using a simple neural net, the setting
"cpu"is preferred (on Mac). - If you have a GPU, you can use
"cuda:0"instead. - If DEVICE is set to
"auto"orNone,spotPythonwill automatically select the device.- This might result in
"mps"on Macs, which is not the best choice for simple neural nets.
- This might result in
Note: Prefix
- The prefix
PREFIXis used for the experiment name and the name of the log file.
MAX_TIME = 1
INIT_SIZE = 5
DEVICE = "cpu" #"cpu" # "cuda:0"
WORKERS = 0
PREFIX="31"from spotPython.utils.device import getDevice
DEVICE = getDevice(DEVICE)
print(DEVICE)cpuimport os
if not os.path.exists('./figures'):
os.makedirs('./figures')21.2 Step 2: Initialization of the fun_control Dictionary
Caution: Tensorboard does not work under Windows
- Since tensorboard does not work under Windows, we recommend setting the parameter
tensorboard_pathtoNoneif you are working under Windows.
spotPython uses a Python dictionary for storing the information required for the hyperparameter tuning process, which was described in Section 14.2, see Initialization of the fun_control Dictionary in the documentation.
from spotPython.utils.init import fun_control_init
from spotPython.utils.file import get_experiment_name
experiment_name = get_experiment_name(prefix=PREFIX)
fun_control = fun_control_init(
num_workers=WORKERS,
device=DEVICE,
_L_in=64,
_L_out=11)21.3 Step 3: PyTorch Data Loading
21.3.1 Lightning Dataset and DataModule
The data loading and preprocessing is handled by Lightning and PyTorch. It comprehends the following classes:
CSVDataset: A class that loads the data from a CSV file. [SOURCE]CSVDataModule: A class that prepares the data for training and testing. [SOURCE]
21.3.1.1 Taking a Look at the Data
import torch
from spotPython.light.csvdataset import CSVDataset
from torch.utils.data import DataLoader
from torchvision.transforms import ToTensor
# Create an instance of CSVDataset
dataset = CSVDataset(csv_file="./data/VBDP/train.csv", train=True)
# show the dimensions of the input data
print(dataset[0][0].shape)
# show the first element of the input data
print(dataset[0][0])
# show the size of the dataset
print(f"Dataset Size: {len(dataset)}")torch.Size([64])
tensor([1., 1., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 1., 1., 0., 0.,
1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0.,
1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 1., 0., 1.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
Dataset Size: 707# Set batch size for DataLoader
batch_size = 3
# Create DataLoader
dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
# Iterate over the data in the DataLoader
for batch in dataloader:
inputs, targets = batch
print(f"Batch Size: {inputs.size(0)}")
print("---------------")
print(f"Inputs: {inputs}")
print(f"Targets: {targets}")
breakBatch Size: 3
---------------
Inputs: tensor([[1., 1., 1., 1., 0., 1., 0., 1., 0., 1., 1., 0., 1., 0., 1., 0., 0., 0.,
0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1., 1., 1.,
1., 1., 0., 1., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 1., 0., 1., 0., 1., 1., 1., 1.,
1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[1., 0., 0., 1., 1., 0., 1., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 0.,
1., 0., 1., 1., 1., 0., 1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
Targets: tensor([1, 4, 1])
Caution: Data Loading in Lightning
- Data loading is handled independently from the
fun_controldictionary byLightningandPyTorch. - In contrast to
spotPythonwithtorch,riverandsklearn, the data sets are not added to thefun_controldictionary.
21.4 Step 4: Specification of the Preprocessing Model
The fun_control dictionary, the torch, sklearnand river versions of spotPython allow the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables, see Section 14.4. This feature is not used in the Lightning version.
Caution: Data preprocessing in Lightning
Lightning allows the data preprocessing to be specified in the LightningDataModule class. It is not considered here, because it should be computed at one location only.
21.5 Step 5: Select the NN Model (algorithm) and core_model_hyper_dict
21.5.1 Implementing a Configurable Neural Network With spotPython
spotPython includes the NetLightBase class [SOURCE] for configurable neural networks. The class is imported here. It inherits from the class Lightning.LightningModule, which is the base class for all models in Lightning. Lightning.LightningModule is a subclass of torch.nn.Module and provides additional functionality for the training and testing of neural networks. The class Lightning.LightningModule is described in the Lightning documentation.
21.5.2 Add the NN Model to the fun_control Dictionary
from spotPython.light.netlightbase import NetLightBase
from spotPython.data.light_hyper_dict import LightHyperDict
from spotPython.hyperparameters.values import add_core_model_to_fun_control
fun_control = add_core_model_to_fun_control(core_model=NetLightBase,
fun_control=fun_control,
hyper_dict= LightHyperDict)The default entries for the core_model class are shown below.
fun_control['core_model_hyper_dict']{'l1': {'type': 'int',
'default': 3,
'transform': 'transform_power_2_int',
'lower': 3,
'upper': 8},
'epochs': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 4,
'upper': 9},
'batch_size': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 1,
'upper': 4},
'act_fn': {'levels': ['Sigmoid', 'Tanh', 'ReLU', 'LeakyReLU', 'ELU', 'Swish'],
'type': 'factor',
'default': 'ReLU',
'transform': 'None',
'class_name': 'spotPython.torch.activation',
'core_model_parameter_type': 'instance()',
'lower': 0,
'upper': 5},
'optimizer': {'levels': ['Adadelta',
'Adagrad',
'Adam',
'AdamW',
'SparseAdam',
'Adamax',
'ASGD',
'NAdam',
'RAdam',
'RMSprop',
'Rprop',
'SGD'],
'type': 'factor',
'default': 'SGD',
'transform': 'None',
'class_name': 'torch.optim',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 11},
'dropout_prob': {'type': 'float',
'default': 0.01,
'transform': 'None',
'lower': 0.0,
'upper': 0.25},
'lr_mult': {'type': 'float',
'default': 1.0,
'transform': 'None',
'lower': 0.1,
'upper': 10.0},
'patience': {'type': 'int',
'default': 2,
'transform': 'transform_power_2_int',
'lower': 2,
'upper': 6},
'initialization': {'levels': ['Default', 'Kaiming', 'Xavier'],
'type': 'factor',
'default': 'Default',
'transform': 'None',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 2}}The NetLightBase is a configurable neural network. The hyperparameters of the model are specified in the core_model_hyper_dict dictionary [SOURCE].
21.6 Step 6: Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model
spotPython provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 14.6.
Caution: Small number of epochs for demonstration purposes
epochsandpatienceare set to small values for demonstration purposes. These values are too small for a real application.- More resonable values are, e.g.:
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[7, 9])andfun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 7])
from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[6,13])
fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[6,13])
fun_control = modify_hyper_parameter_bounds(fun_control, "batch_size", bounds=[2, 8])from spotPython.hyperparameters.values import modify_hyper_parameter_levels
fun_control = modify_hyper_parameter_levels(fun_control, "optimizer",["Adam", "AdamW", "Adamax", "NAdam"])
# fun_control = modify_hyper_parameter_levels(fun_control, "optimizer", ["Adam"])The updated fun_control dictionary is shown below.
fun_control["core_model_hyper_dict"]{'l1': {'type': 'int',
'default': 3,
'transform': 'transform_power_2_int',
'lower': 6,
'upper': 13},
'epochs': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 6,
'upper': 13},
'batch_size': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 2,
'upper': 8},
'act_fn': {'levels': ['Sigmoid', 'Tanh', 'ReLU', 'LeakyReLU', 'ELU', 'Swish'],
'type': 'factor',
'default': 'ReLU',
'transform': 'None',
'class_name': 'spotPython.torch.activation',
'core_model_parameter_type': 'instance()',
'lower': 0,
'upper': 5},
'optimizer': {'levels': ['Adam', 'AdamW', 'Adamax', 'NAdam'],
'type': 'factor',
'default': 'SGD',
'transform': 'None',
'class_name': 'torch.optim',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 3},
'dropout_prob': {'type': 'float',
'default': 0.01,
'transform': 'None',
'lower': 0.0,
'upper': 0.25},
'lr_mult': {'type': 'float',
'default': 1.0,
'transform': 'None',
'lower': 0.1,
'upper': 10.0},
'patience': {'type': 'int',
'default': 2,
'transform': 'transform_power_2_int',
'lower': 2,
'upper': 6},
'initialization': {'levels': ['Default', 'Kaiming', 'Xavier'],
'type': 'factor',
'default': 'Default',
'transform': 'None',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 2}}21.7 Step 7: Data Splitting, the Objective (Loss) Function and the Metric
21.7.1 Evaluation
The evaluation procedure requires the specification of two elements:
- the way how the data is split into a train and a test set (see Section 14.7.1)
- the loss function (and a metric).
Caution: Data Splitting in Lightning
- The data splitting is handled by
Lightning.
21.7.2 Loss Functions and Metrics
The loss function is specified in the configurable network class [SOURCE] We will use CrossEntropy loss for the multiclass-classification task.
21.7.3 Metric
- We will use the MAP@k metric [SOURCE] for the evaluation of the model. Here is an example how this metric is calculated.
from spotPython.torch.mapk import MAPK
import torch
mapk = MAPK(k=2)
target = torch.tensor([0, 1, 2, 2])
preds = torch.tensor(
[
[0.5, 0.2, 0.2], # 0 is in top 2
[0.3, 0.4, 0.2], # 1 is in top 2
[0.2, 0.4, 0.3], # 2 is in top 2
[0.7, 0.2, 0.1], # 2 isn't in top 2
]
)
mapk.update(preds, target)
print(mapk.compute()) # tensor(0.6250)tensor(0.6250)Similar to the loss function, the metric is specified in the configurable network class [SOURCE].
Caution: Loss Function and Metric in Lightning
- The loss function and the metric are not hyperparameters that can be tuned with
spotPython. - They are handled by
Lightning.
21.8 Step 8: Calling the SPOT Function
21.8.1 Preparing the SPOT Call
The following code passes the information about the parameter ranges and bounds to spot. It extracts the variable types, names, and bounds
from spotPython.hyperparameters.values import (get_bound_values,
get_var_name,
get_var_type,)
var_type = get_var_type(fun_control)
var_name = get_var_name(fun_control)
fun_control.update({"var_type": var_type,
"var_name": var_name})
lower = get_bound_values(fun_control, "lower")
upper = get_bound_values(fun_control, "upper")Now, the dictionary fun_control contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method gen_design_table [SOURCE] generates a design table as follows:
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))| name | type | default | lower | upper | transform |
|----------------|--------|-----------|---------|---------|-----------------------|
| l1 | int | 3 | 6 | 13 | transform_power_2_int |
| epochs | int | 4 | 6 | 13 | transform_power_2_int |
| batch_size | int | 4 | 2 | 8 | transform_power_2_int |
| act_fn | factor | ReLU | 0 | 5 | None |
| optimizer | factor | SGD | 0 | 3 | None |
| dropout_prob | float | 0.01 | 0 | 0.25 | None |
| lr_mult | float | 1.0 | 0.1 | 10 | None |
| patience | int | 2 | 2 | 6 | transform_power_2_int |
| initialization | factor | Default | 0 | 2 | None |This allows to check if all information is available and if the information is correct.
21.8.2 The Objective Function fun
The objective function fun from the class HyperLight [SOURCE] is selected next. It implements an interface from PyTorch’s training, validation, and testing methods to spotPython.
from spotPython.light.hyperlight import HyperLight
fun = HyperLight().funfun_control{'CHECKPOINT_PATH': 'saved_models/',
'DATASET_PATH': 'data/',
'RESULTS_PATH': 'results/',
'TENSORBOARD_PATH': 'runs/',
'_L_in': 64,
'_L_out': 11,
'data': None,
'data_dir': './data',
'device': 'cpu',
'enable_progress_bar': False,
'eval': None,
'k_folds': None,
'loss_function': None,
'metric_river': None,
'metric_sklearn': None,
'metric_torch': None,
'metric_params': {},
'model_dict': {},
'n_samples': None,
'num_workers': 0,
'optimizer': None,
'path': None,
'prep_model': None,
'save_model': False,
'show_batch_interval': 1000000,
'shuffle': None,
'target_column': None,
'train': None,
'test': None,
'task': 'classification',
'tensorboard_path': None,
'weights': 1.0,
'writer': None,
'core_model': spotPython.light.netlightbase.NetLightBase,
'core_model_hyper_dict': {'l1': {'type': 'int',
'default': 3,
'transform': 'transform_power_2_int',
'lower': 6,
'upper': 13},
'epochs': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 6,
'upper': 13},
'batch_size': {'type': 'int',
'default': 4,
'transform': 'transform_power_2_int',
'lower': 2,
'upper': 8},
'act_fn': {'levels': ['Sigmoid',
'Tanh',
'ReLU',
'LeakyReLU',
'ELU',
'Swish'],
'type': 'factor',
'default': 'ReLU',
'transform': 'None',
'class_name': 'spotPython.torch.activation',
'core_model_parameter_type': 'instance()',
'lower': 0,
'upper': 5},
'optimizer': {'levels': ['Adam', 'AdamW', 'Adamax', 'NAdam'],
'type': 'factor',
'default': 'SGD',
'transform': 'None',
'class_name': 'torch.optim',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 3},
'dropout_prob': {'type': 'float',
'default': 0.01,
'transform': 'None',
'lower': 0.0,
'upper': 0.25},
'lr_mult': {'type': 'float',
'default': 1.0,
'transform': 'None',
'lower': 0.1,
'upper': 10.0},
'patience': {'type': 'int',
'default': 2,
'transform': 'transform_power_2_int',
'lower': 2,
'upper': 6},
'initialization': {'levels': ['Default', 'Kaiming', 'Xavier'],
'type': 'factor',
'default': 'Default',
'transform': 'None',
'core_model_parameter_type': 'str',
'lower': 0,
'upper': 2}},
'var_type': ['int',
'int',
'int',
'factor',
'factor',
'float',
'float',
'int',
'factor'],
'var_name': ['l1',
'epochs',
'batch_size',
'act_fn',
'optimizer',
'dropout_prob',
'lr_mult',
'patience',
'initialization']}21.8.3 Starting the Hyperparameter Tuning
The spotPython hyperparameter tuning is started by calling the Spot function [SOURCE] as described in Section 14.8.4.
import numpy as np
from spotPython.spot import spot
from math import inf
spot_tuner = spot.Spot(fun=fun,
lower = lower,
upper = upper,
fun_evals = inf,
fun_repeats = 1,
max_time = MAX_TIME,
noise = False,
tolerance_x = np.sqrt(np.spacing(1)),
var_type = var_type,
var_name = var_name,
infill_criterion = "y",
n_points = 1,
seed=123,
log_level = 50,
show_models= False,
show_progress= True,
fun_control = fun_control,
design_control={"init_size": INIT_SIZE,
"repeats": 1},
surrogate_control={"noise": True,
"cod_type": "norm",
"min_theta": -4,
"max_theta": 3,
"n_theta": len(var_name),
"model_fun_evals": 10_000,
"log_level": 50
})
spot_tuner.run()
config: {'l1': 4096, 'epochs': 4096, 'batch_size': 32, 'act_fn': ReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.10939527466721133, 'lr_mult': 4.211776903906428, 'patience': 16, 'initialization': 'Default'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=4096, bias=True)
(1): ReLU()
(2): Dropout(p=0.10939527466721133, inplace=False)
(3): Linear(in_features=4096, out_features=2048, bias=True)
(4): ReLU()
(5): Dropout(p=0.10939527466721133, inplace=False)
(6): Linear(in_features=2048, out_features=2048, bias=True)
(7): ReLU()
(8): Dropout(p=0.10939527466721133, inplace=False)
(9): Linear(in_features=2048, out_features=1024, bias=True)
(10): ReLU()
(11): Dropout(p=0.10939527466721133, inplace=False)
(12): Linear(in_features=1024, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.433500051498413 │ │ val_acc │ 0.10954063385725021 │ │ val_loss │ 2.433500051498413 │ │ valid_mapk │ 0.1800411492586136 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.1800411492586136, 'val_loss': 2.433500051498413, 'val_acc': 0.10954063385725021, 'hp_metric': 2.433500051498413}
config: {'l1': 64, 'epochs': 128, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.012926647388264517, 'lr_mult': 0.832718394912432, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.238137722015381 │ │ val_acc │ 0.2862190902233124 │ │ val_loss │ 2.238137722015381 │ │ valid_mapk │ 0.45666956901550293 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.45666956901550293, 'val_loss': 2.238137722015381, 'val_acc': 0.2862190902233124, 'hp_metric': 2.238137722015381}
config: {'l1': 1024, 'epochs': 256, 'batch_size': 8, 'act_fn': Swish(), 'optimizer': 'NAdam', 'dropout_prob': 0.22086376796923401, 'lr_mult': 7.65501078489161, 'patience': 64, 'initialization': 'Xavier'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=1024, bias=True)
(1): Swish()
(2): Dropout(p=0.22086376796923401, inplace=False)
(3): Linear(in_features=1024, out_features=512, bias=True)
(4): Swish()
(5): Dropout(p=0.22086376796923401, inplace=False)
(6): Linear(in_features=512, out_features=512, bias=True)
(7): Swish()
(8): Dropout(p=0.22086376796923401, inplace=False)
(9): Linear(in_features=512, out_features=256, bias=True)
(10): Swish()
(11): Dropout(p=0.22086376796923401, inplace=False)
(12): Linear(in_features=256, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.4405665397644043 │ │ val_acc │ 0.10247349739074707 │ │ val_loss │ 2.4405665397644043 │ │ valid_mapk │ 0.18518517911434174 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.18518517911434174, 'val_loss': 2.4405665397644043, 'val_acc': 0.10247349739074707, 'hp_metric': 2.4405665397644043}
config: {'l1': 512, 'epochs': 512, 'batch_size': 16, 'act_fn': Sigmoid(), 'optimizer': 'Adam', 'dropout_prob': 0.1890928563375006, 'lr_mult': 2.3450676871382794, 'patience': 32, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=512, bias=True)
(1): Sigmoid()
(2): Dropout(p=0.1890928563375006, inplace=False)
(3): Linear(in_features=512, out_features=256, bias=True)
(4): Sigmoid()
(5): Dropout(p=0.1890928563375006, inplace=False)
(6): Linear(in_features=256, out_features=256, bias=True)
(7): Sigmoid()
(8): Dropout(p=0.1890928563375006, inplace=False)
(9): Linear(in_features=256, out_features=128, bias=True)
(10): Sigmoid()
(11): Dropout(p=0.1890928563375006, inplace=False)
(12): Linear(in_features=128, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.3434996604919434 │ │ val_acc │ 0.19434629380702972 │ │ val_loss │ 2.3434996604919434 │ │ valid_mapk │ 0.2698337733745575 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.2698337733745575, 'val_loss': 2.3434996604919434, 'val_acc': 0.19434629380702972, 'hp_metric': 2.3434996604919434}
config: {'l1': 256, 'epochs': 4096, 'batch_size': 64, 'act_fn': ReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.0708380794924471, 'lr_mult': 9.528945328733357, 'patience': 4, 'initialization': 'Xavier'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=256, bias=True)
(1): ReLU()
(2): Dropout(p=0.0708380794924471, inplace=False)
(3): Linear(in_features=256, out_features=128, bias=True)
(4): ReLU()
(5): Dropout(p=0.0708380794924471, inplace=False)
(6): Linear(in_features=128, out_features=128, bias=True)
(7): ReLU()
(8): Dropout(p=0.0708380794924471, inplace=False)
(9): Linear(in_features=128, out_features=64, bias=True)
(10): ReLU()
(11): Dropout(p=0.0708380794924471, inplace=False)
(12): Linear(in_features=64, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.292955160140991 │ │ val_acc │ 0.2473498284816742 │ │ val_loss │ 2.292955160140991 │ │ valid_mapk │ 0.3427276313304901 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3427276313304901, 'val_loss': 2.292955160140991, 'val_acc': 0.2473498284816742, 'hp_metric': 2.292955160140991}
config: {'l1': 64, 'epochs': 64, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.006064053615858084, 'lr_mult': 0.4763979523955304, 'patience': 8, 'initialization': 'Xavier'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.006064053615858084, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.006064053615858084, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.006064053615858084, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.006064053615858084, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.3466885089874268 │ │ val_acc │ 0.208480566740036 │ │ val_loss │ 2.3466885089874268 │ │ valid_mapk │ 0.3177203834056854 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3177203834056854, 'val_loss': 2.3466885089874268, 'val_acc': 0.208480566740036, 'hp_metric': 2.3466885089874268}spotPython tuning: 2.238137722015381 [----------] 3.16%
config: {'l1': 64, 'epochs': 512, 'batch_size': 256, 'act_fn': Sigmoid(), 'optimizer': 'Adam', 'dropout_prob': 0.030100713594192443, 'lr_mult': 3.4214626523101765, 'patience': 4, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): Sigmoid()
(2): Dropout(p=0.030100713594192443, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): Sigmoid()
(5): Dropout(p=0.030100713594192443, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): Sigmoid()
(8): Dropout(p=0.030100713594192443, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): Sigmoid()
(11): Dropout(p=0.030100713594192443, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.298098564147949 │ │ val_acc │ 0.22968198359012604 │ │ val_loss │ 2.298098564147949 │ │ valid_mapk │ 0.2535686790943146 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.2535686790943146, 'val_loss': 2.298098564147949, 'val_acc': 0.22968198359012604, 'hp_metric': 2.298098564147949}spotPython tuning: 2.238137722015381 [#---------] 7.64%
config: {'l1': 64, 'epochs': 128, 'batch_size': 256, 'act_fn': ReLU(), 'optimizer': 'NAdam', 'dropout_prob': 0.03832172101534319, 'lr_mult': 3.9929180674070883, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): ReLU()
(2): Dropout(p=0.03832172101534319, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): ReLU()
(5): Dropout(p=0.03832172101534319, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): ReLU()
(8): Dropout(p=0.03832172101534319, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): ReLU()
(11): Dropout(p=0.03832172101534319, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.271381378173828 │ │ val_acc │ 0.2614840865135193 │ │ val_loss │ 2.271381378173828 │ │ valid_mapk │ 0.402874231338501 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.402874231338501, 'val_loss': 2.271381378173828, 'val_acc': 0.2614840865135193, 'hp_metric': 2.271381378173828}spotPython tuning: 2.238137722015381 [#---------] 11.38%
config: {'l1': 64, 'epochs': 128, 'batch_size': 128, 'act_fn': Swish(), 'optimizer': 'AdamW', 'dropout_prob': 0.03644107482722437, 'lr_mult': 1.1031464719981001, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): Swish()
(2): Dropout(p=0.03644107482722437, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): Swish()
(5): Dropout(p=0.03644107482722437, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): Swish()
(8): Dropout(p=0.03644107482722437, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): Swish()
(11): Dropout(p=0.03644107482722437, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.278205156326294 │ │ val_acc │ 0.2473498284816742 │ │ val_loss │ 2.278205156326294 │ │ valid_mapk │ 0.3325938880443573 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3325938880443573, 'val_loss': 2.278205156326294, 'val_acc': 0.2473498284816742, 'hp_metric': 2.278205156326294}spotPython tuning: 2.238137722015381 [##--------] 15.93%
config: {'l1': 256, 'epochs': 256, 'batch_size': 8, 'act_fn': ReLU(), 'optimizer': 'Adam', 'dropout_prob': 0.07083807561978617, 'lr_mult': 7.5094994261143055, 'patience': 8, 'initialization': 'Xavier'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=256, bias=True)
(1): ReLU()
(2): Dropout(p=0.07083807561978617, inplace=False)
(3): Linear(in_features=256, out_features=128, bias=True)
(4): ReLU()
(5): Dropout(p=0.07083807561978617, inplace=False)
(6): Linear(in_features=128, out_features=128, bias=True)
(7): ReLU()
(8): Dropout(p=0.07083807561978617, inplace=False)
(9): Linear(in_features=128, out_features=64, bias=True)
(10): ReLU()
(11): Dropout(p=0.07083807561978617, inplace=False)
(12): Linear(in_features=64, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.394630193710327 │ │ val_acc │ 0.1484098881483078 │ │ val_loss │ 2.394630193710327 │ │ valid_mapk │ 0.22974535822868347 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.22974535822868347, 'val_loss': 2.394630193710327, 'val_acc': 0.1484098881483078, 'hp_metric': 2.394630193710327}spotPython tuning: 2.238137722015381 [##--------] 22.12%
config: {'l1': 64, 'epochs': 64, 'batch_size': 256, 'act_fn': Tanh(), 'optimizer': 'Adamax', 'dropout_prob': 0.0, 'lr_mult': 4.728881422091566, 'patience': 64, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): Tanh()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): Tanh()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): Tanh()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): Tanh()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.258221387863159 │ │ val_acc │ 0.2720848023891449 │ │ val_loss │ 2.258221387863159 │ │ valid_mapk │ 0.3440875709056854 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3440875709056854, 'val_loss': 2.258221387863159, 'val_acc': 0.2720848023891449, 'hp_metric': 2.258221387863159}spotPython tuning: 2.238137722015381 [###-------] 27.55%
config: {'l1': 64, 'epochs': 128, 'batch_size': 128, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.007097696158616737, 'lr_mult': 0.2005522500701167, 'patience': 16, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.007097696158616737, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.007097696158616737, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.007097696158616737, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.007097696158616737, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.3103928565979004 │ │ val_acc │ 0.22968198359012604 │ │ val_loss │ 2.3103928565979004 │ │ valid_mapk │ 0.2670235335826874 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.2670235335826874, 'val_loss': 2.3103928565979004, 'val_acc': 0.22968198359012604, 'hp_metric': 2.3103928565979004}spotPython tuning: 2.238137722015381 [####------] 35.06%
config: {'l1': 128, 'epochs': 128, 'batch_size': 256, 'act_fn': ReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.16965823251206952, 'lr_mult': 4.550881246576569, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=128, bias=True)
(1): ReLU()
(2): Dropout(p=0.16965823251206952, inplace=False)
(3): Linear(in_features=128, out_features=64, bias=True)
(4): ReLU()
(5): Dropout(p=0.16965823251206952, inplace=False)
(6): Linear(in_features=64, out_features=64, bias=True)
(7): ReLU()
(8): Dropout(p=0.16965823251206952, inplace=False)
(9): Linear(in_features=64, out_features=32, bias=True)
(10): ReLU()
(11): Dropout(p=0.16965823251206952, inplace=False)
(12): Linear(in_features=32, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.2878034114837646 │ │ val_acc │ 0.24381625652313232 │ │ val_loss │ 2.2878034114837646 │ │ valid_mapk │ 0.3672960102558136 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3672960102558136, 'val_loss': 2.2878034114837646, 'val_acc': 0.24381625652313232, 'hp_metric': 2.2878034114837646}spotPython tuning: 2.238137722015381 [####------] 37.93%
config: {'l1': 128, 'epochs': 64, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.13070801331028198, 'lr_mult': 5.664792875702936, 'patience': 64, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=128, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.13070801331028198, inplace=False)
(3): Linear(in_features=128, out_features=64, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.13070801331028198, inplace=False)
(6): Linear(in_features=64, out_features=64, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.13070801331028198, inplace=False)
(9): Linear(in_features=64, out_features=32, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.13070801331028198, inplace=False)
(12): Linear(in_features=32, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.259943962097168 │ │ val_acc │ 0.2826855182647705 │ │ val_loss │ 2.259943962097168 │ │ valid_mapk │ 0.39427807927131653 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.39427807927131653, 'val_loss': 2.259943962097168, 'val_acc': 0.2826855182647705, 'hp_metric': 2.259943962097168}spotPython tuning: 2.238137722015381 [####------] 42.77%
config: {'l1': 512, 'epochs': 64, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.0, 'lr_mult': 3.1952283197430966, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=512, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=512, out_features=256, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=256, out_features=256, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=256, out_features=128, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=128, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.246544599533081 │ │ val_acc │ 0.28975266218185425 │ │ val_loss │ 2.246544599533081 │ │ valid_mapk │ 0.3566141128540039 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3566141128540039, 'val_loss': 2.246544599533081, 'val_acc': 0.28975266218185425, 'hp_metric': 2.246544599533081}spotPython tuning: 2.238137722015381 [#####-----] 47.85%
config: {'l1': 1024, 'epochs': 128, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adam', 'dropout_prob': 0.0, 'lr_mult': 0.4729581593251257, 'patience': 4, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=1024, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=1024, out_features=512, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=512, out_features=512, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=512, out_features=256, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=256, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.267474889755249 │ │ val_acc │ 0.2614840865135193 │ │ val_loss │ 2.267474889755249 │ │ valid_mapk │ 0.3885995149612427 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3885995149612427, 'val_loss': 2.267474889755249, 'val_acc': 0.2614840865135193, 'hp_metric': 2.267474889755249}spotPython tuning: 2.238137722015381 [#####-----] 52.22%
config: {'l1': 64, 'epochs': 128, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.0, 'lr_mult': 4.783514932131345, 'patience': 64, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.2709927558898926 │ │ val_acc │ 0.2579505443572998 │ │ val_loss │ 2.2709927558898926 │ │ valid_mapk │ 0.2964892089366913 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.2964892089366913, 'val_loss': 2.2709927558898926, 'val_acc': 0.2579505443572998, 'hp_metric': 2.2709927558898926}spotPython tuning: 2.238137722015381 [######----] 58.65%
config: {'l1': 4096, 'epochs': 64, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.0, 'lr_mult': 1.8021236712633482, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=4096, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=4096, out_features=2048, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=2048, out_features=2048, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=2048, out_features=1024, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=1024, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.444101095199585 │ │ val_acc │ 0.0989399328827858 │ │ val_loss │ 2.444101095199585 │ │ valid_mapk │ 0.16326677799224854 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.16326677799224854, 'val_loss': 2.444101095199585, 'val_acc': 0.0989399328827858, 'hp_metric': 2.444101095199585}spotPython tuning: 2.238137722015381 [#######---] 68.52%
config: {'l1': 512, 'epochs': 64, 'batch_size': 256, 'act_fn': Swish(), 'optimizer': 'AdamW', 'dropout_prob': 0.044591380506204724, 'lr_mult': 2.359124011031594, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=512, bias=True)
(1): Swish()
(2): Dropout(p=0.044591380506204724, inplace=False)
(3): Linear(in_features=512, out_features=256, bias=True)
(4): Swish()
(5): Dropout(p=0.044591380506204724, inplace=False)
(6): Linear(in_features=256, out_features=256, bias=True)
(7): Swish()
(8): Dropout(p=0.044591380506204724, inplace=False)
(9): Linear(in_features=256, out_features=128, bias=True)
(10): Swish()
(11): Dropout(p=0.044591380506204724, inplace=False)
(12): Linear(in_features=128, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.2953412532806396 │ │ val_acc │ 0.2332155406475067 │ │ val_loss │ 2.2953412532806396 │ │ valid_mapk │ 0.3596402406692505 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3596402406692505, 'val_loss': 2.2953412532806396, 'val_acc': 0.2332155406475067, 'hp_metric': 2.2953412532806396}spotPython tuning: 2.238137722015381 [#######---] 72.69%
config: {'l1': 512, 'epochs': 128, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adam', 'dropout_prob': 0.004894161827505516, 'lr_mult': 2.1647113235226385, 'patience': 16, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=512, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.004894161827505516, inplace=False)
(3): Linear(in_features=512, out_features=256, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.004894161827505516, inplace=False)
(6): Linear(in_features=256, out_features=256, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.004894161827505516, inplace=False)
(9): Linear(in_features=256, out_features=128, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.004894161827505516, inplace=False)
(12): Linear(in_features=128, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.257730007171631 │ │ val_acc │ 0.2720848023891449 │ │ val_loss │ 2.257730007171631 │ │ valid_mapk │ 0.3320794701576233 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3320794701576233, 'val_loss': 2.257730007171631, 'val_acc': 0.2720848023891449, 'hp_metric': 2.257730007171631}spotPython tuning: 2.238137722015381 [########--] 78.25%
config: {'l1': 512, 'epochs': 1024, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.0, 'lr_mult': 9.138772126270819, 'patience': 16, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=512, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=512, out_features=256, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=256, out_features=256, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=256, out_features=128, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=128, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.3627161979675293 │ │ val_acc │ 0.18021202087402344 │ │ val_loss │ 2.3627161979675293 │ │ valid_mapk │ 0.2588855028152466 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.2588855028152466, 'val_loss': 2.3627161979675293, 'val_acc': 0.18021202087402344, 'hp_metric': 2.3627161979675293}spotPython tuning: 2.238137722015381 [########--] 83.23%
config: {'l1': 64, 'epochs': 64, 'batch_size': 128, 'act_fn': Swish(), 'optimizer': 'AdamW', 'dropout_prob': 0.09729428347287006, 'lr_mult': 0.6503127559977777, 'patience': 4, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): Swish()
(2): Dropout(p=0.09729428347287006, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): Swish()
(5): Dropout(p=0.09729428347287006, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): Swish()
(8): Dropout(p=0.09729428347287006, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): Swish()
(11): Dropout(p=0.09729428347287006, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.31158709526062 │ │ val_acc │ 0.23674911260604858 │ │ val_loss │ 2.31158709526062 │ │ valid_mapk │ 0.3402777910232544 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3402777910232544, 'val_loss': 2.31158709526062, 'val_acc': 0.23674911260604858, 'hp_metric': 2.31158709526062}spotPython tuning: 2.238137722015381 [#########-] 88.00%
config: {'l1': 128, 'epochs': 64, 'batch_size': 256, 'act_fn': Tanh(), 'optimizer': 'AdamW', 'dropout_prob': 0.0, 'lr_mult': 0.1, 'patience': 8, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=128, bias=True)
(1): Tanh()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=128, out_features=64, bias=True)
(4): Tanh()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=64, out_features=64, bias=True)
(7): Tanh()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=64, out_features=32, bias=True)
(10): Tanh()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=32, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.33255672454834 │ │ val_acc │ 0.2226148396730423 │ │ val_loss │ 2.33255672454834 │ │ valid_mapk │ 0.3295838236808777 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3295838236808777, 'val_loss': 2.33255672454834, 'val_acc': 0.2226148396730423, 'hp_metric': 2.33255672454834}spotPython tuning: 2.238137722015381 [#########-] 93.58%
config: {'l1': 1024, 'epochs': 256, 'batch_size': 256, 'act_fn': Swish(), 'optimizer': 'NAdam', 'dropout_prob': 0.0, 'lr_mult': 1.166260588985513, 'patience': 32, 'initialization': 'Kaiming'}
_L_in: 64
_L_out: 11
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=1024, bias=True)
(1): Swish()
(2): Dropout(p=0.0, inplace=False)
(3): Linear(in_features=1024, out_features=512, bias=True)
(4): Swish()
(5): Dropout(p=0.0, inplace=False)
(6): Linear(in_features=512, out_features=512, bias=True)
(7): Swish()
(8): Dropout(p=0.0, inplace=False)
(9): Linear(in_features=512, out_features=256, bias=True)
(10): Swish()
(11): Dropout(p=0.0, inplace=False)
(12): Linear(in_features=256, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.25215744972229 │ │ val_acc │ 0.2826855182647705 │ │ val_loss │ 2.25215744972229 │ │ valid_mapk │ 0.4092399775981903 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.4092399775981903, 'val_loss': 2.25215744972229, 'val_acc': 0.2826855182647705, 'hp_metric': 2.25215744972229}spotPython tuning: 2.238137722015381 [##########] 100.00% Done...
<spotPython.spot.spot.Spot at 0x18c45f310>21.9 Step 9: Tensorboard
The textual output shown in the console (or code cell) can be visualized with Tensorboard as described in Section 14.9, see also the description in the documentation: Tensorboard.
21.10 Step 10: Results
After the hyperparameter tuning run is finished, the results can be analyzed as described in Section 14.10.
spot_tuner.plot_progress(log_y=False,
filename="./figures/" + experiment_name+"_progress.png")
from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control=fun_control, spot=spot_tuner))| name | type | default | lower | upper | tuned | transform | importance | stars |
|----------------|--------|-----------|---------|---------|----------------------|-----------------------|--------------|---------|
| l1 | int | 3 | 6.0 | 13.0 | 6.0 | transform_power_2_int | 1.31 | * |
| epochs | int | 4 | 6.0 | 13.0 | 7.0 | transform_power_2_int | 0.14 | . |
| batch_size | int | 4 | 2.0 | 8.0 | 8.0 | transform_power_2_int | 0.59 | . |
| act_fn | factor | ReLU | 0.0 | 5.0 | 3.0 | None | 0.00 | |
| optimizer | factor | SGD | 0.0 | 3.0 | 2.0 | None | 0.00 | |
| dropout_prob | float | 0.01 | 0.0 | 0.25 | 0.012926647388264517 | None | 0.00 | |
| lr_mult | float | 1.0 | 0.1 | 10.0 | 0.832718394912432 | None | 0.22 | . |
| patience | int | 2 | 2.0 | 6.0 | 3.0 | transform_power_2_int | 0.00 | |
| initialization | factor | Default | 0.0 | 2.0 | 1.0 | None | 100.00 | *** |spot_tuner.plot_importance(threshold=0.025,
filename="./figures/" + experiment_name+"_importance.png")
21.10.1 Get the Tuned Architecture
from spotPython.light.utils import get_tuned_architecture
config = get_tuned_architecture(spot_tuner, fun_control)- Test on the full data set
from spotPython.light.traintest import test_model
test_model(config, fun_control)model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Test metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.1438722610473633 │ │ test_mapk_epoch │ 0.4611745774745941 │ │ val_acc │ 0.41442716121673584 │ │ val_loss │ 2.1438722610473633 │ └───────────────────────────┴───────────────────────────┘
test_model result: {'test_mapk_epoch': 0.4611745774745941, 'val_loss': 2.1438722610473633, 'val_acc': 0.41442716121673584, 'hp_metric': 2.1438722610473633}(2.1438722610473633, 0.41442716121673584)from spotPython.light.traintest import load_light_from_checkpoint
model_loaded = load_light_from_checkpoint(config, fun_control)Loading model from runs/lightning_logs/64_128_256_LeakyReLU()_Adamax_0.012926647388264517_0.832718394912432_8_Kaiming_TEST/checkpoints/last.ckpt21.10.2 Cross Validation With Lightning
- The
KFoldclass fromsklearn.model_selectionis used to generate the folds for cross-validation. - These mechanism is used to generate the folds for the final evaluation of the model.
- The
CrossValidationDataModuleclass [SOURCE] is used to generate the folds for the hyperparameter tuning process. - It is called from the
cv_modelfunction [SOURCE].
from spotPython.light.traintest import cv_model
cv_model(config, fun_control)model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)
k: 0
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.40067982673645 │ │ val_acc │ 0.1267605572938919 │ │ val_loss │ 2.40067982673645 │ │ valid_mapk │ 0.18309858441352844 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.18309858441352844, 'val_loss': 2.40067982673645, 'val_acc': 0.1267605572938919, 'hp_metric': 2.40067982673645}
k: 1
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.2687666416168213 │ │ val_acc │ 0.23943662643432617 │ │ val_loss │ 2.2687666416168213 │ │ valid_mapk │ 0.32863849401474 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.32863849401474, 'val_loss': 2.2687666416168213, 'val_acc': 0.23943662643432617, 'hp_metric': 2.2687666416168213}
k: 2
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.2070271968841553 │ │ val_acc │ 0.3380281627178192 │ │ val_loss │ 2.2070271968841553 │ │ valid_mapk │ 0.39906102418899536 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.39906102418899536, 'val_loss': 2.2070271968841553, 'val_acc': 0.3380281627178192, 'hp_metric': 2.2070271968841553}
k: 3
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.157003402709961 │ │ val_acc │ 0.39436620473861694 │ │ val_loss │ 2.157003402709961 │ │ valid_mapk │ 0.4600938856601715 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.4600938856601715, 'val_loss': 2.157003402709961, 'val_acc': 0.39436620473861694, 'hp_metric': 2.157003402709961}
k: 4
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.294954538345337 │ │ val_acc │ 0.23943662643432617 │ │ val_loss │ 2.294954538345337 │ │ valid_mapk │ 0.3004694879055023 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.3004694879055023, 'val_loss': 2.294954538345337, 'val_acc': 0.23943662643432617, 'hp_metric': 2.294954538345337}
k: 5
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.0768275260925293 │ │ val_acc │ 0.4647887349128723 │ │ val_loss │ 2.0768275260925293 │ │ valid_mapk │ 0.5446009635925293 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.5446009635925293, 'val_loss': 2.0768275260925293, 'val_acc': 0.4647887349128723, 'hp_metric': 2.0768275260925293}
k: 6
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.1443779468536377 │ │ val_acc │ 0.4225352108478546 │ │ val_loss │ 2.1443779468536377 │ │ valid_mapk │ 0.4530516564846039 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.4530516564846039, 'val_loss': 2.1443779468536377, 'val_acc': 0.4225352108478546, 'hp_metric': 2.1443779468536377}
k: 7
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.038654088973999 │ │ val_acc │ 0.5428571701049805 │ │ val_loss │ 2.038654088973999 │ │ valid_mapk │ 0.5761904716491699 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.5761904716491699, 'val_loss': 2.038654088973999, 'val_acc': 0.5428571701049805, 'hp_metric': 2.038654088973999}
k: 8
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.0311355590820312 │ │ val_acc │ 0.5428571701049805 │ │ val_loss │ 2.0311355590820312 │ │ valid_mapk │ 0.5642856955528259 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.5642856955528259, 'val_loss': 2.0311355590820312, 'val_acc': 0.5428571701049805, 'hp_metric': 2.0311355590820312}
k: 9
model: NetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Validate metric ┃ DataLoader 0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ hp_metric │ 2.099412441253662 │ │ val_acc │ 0.44285714626312256 │ │ val_loss │ 2.099412441253662 │ │ valid_mapk │ 0.4714285731315613 │ └───────────────────────────┴───────────────────────────┘
train_model result: {'valid_mapk': 0.4714285731315613, 'val_loss': 2.099412441253662, 'val_acc': 0.44285714626312256, 'hp_metric': 2.099412441253662}
cv_model mapk result: 0.42809188365936280.4280918836593628
Note: Evaluation for the Final Comaprison
- This is the evaluation that will be used in the comparison.
21.10.3 Detailed Hyperparameter Plots
filename = "./figures/" + experiment_name
spot_tuner.plot_important_hyperparameter_contour(filename=filename)l1: 1.3098014876342001
epochs: 0.140831886381637
batch_size: 0.5919967545081264
lr_mult: 0.22059848750984945
initialization: 100.0
21.10.4 Parallel Coordinates Plot
spot_tuner.parallel_plot()Parallel coordinates plots
21.10.5 Plot all Combinations of Hyperparameters
- Warning: this may take a while.
PLOT_ALL = False
if PLOT_ALL:
n = spot_tuner.k
for i in range(n-1):
for j in range(i+1, n):
spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)21.10.6 Visualizing the Activation Distribution
Reference:
- The following code is based on [PyTorch Lightning TUTORIAL 2: ACTIVATION FUNCTIONS], Author: Phillip Lippe, License: [CC BY-SA], Generated: 2023-03-15T09:52:39.179933.
After we have trained the models, we can look at the actual activation values that find inside the model. For instance, how many neurons are set to zero in ReLU? Where do we find most values in Tanh? To answer these questions, we can write a simple function which takes a trained model, applies it to a batch of images, and plots the histogram of the activations inside the network:
from spotPython.torch.activation import Sigmoid, Tanh, ReLU, LeakyReLU, ELU, Swish
act_fn_by_name = {"sigmoid": Sigmoid, "tanh": Tanh, "relu": ReLU, "leakyrelu": LeakyReLU, "elu": ELU, "swish": Swish}from spotPython.hyperparameters.values import get_one_config_from_X
X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
config = get_one_config_from_X(X, fun_control)
model = fun_control["core_model"](**config, _L_in=64, _L_out=11)
modelNetLightBase(
(train_mapk): MAPK()
(valid_mapk): MAPK()
(test_mapk): MAPK()
(layers): Sequential(
(0): Linear(in_features=64, out_features=64, bias=True)
(1): LeakyReLU()
(2): Dropout(p=0.012926647388264517, inplace=False)
(3): Linear(in_features=64, out_features=32, bias=True)
(4): LeakyReLU()
(5): Dropout(p=0.012926647388264517, inplace=False)
(6): Linear(in_features=32, out_features=32, bias=True)
(7): LeakyReLU()
(8): Dropout(p=0.012926647388264517, inplace=False)
(9): Linear(in_features=32, out_features=16, bias=True)
(10): LeakyReLU()
(11): Dropout(p=0.012926647388264517, inplace=False)
(12): Linear(in_features=16, out_features=11, bias=True)
)
)from spotPython.utils.eda import visualize_activations
visualize_activations(model, device="cpu", color=f"C{0}")
22 Submission
import numpy as np
import pandas as pd
from sklearn.preprocessing import OrdinalEncoderimport pandas as pd
from sklearn.preprocessing import OrdinalEncoder
train_df = pd.read_csv('./data/VBDP/train.csv', index_col=0)
# remove the id column
# train_df = train_df.drop(columns=['id'])
n_samples = train_df.shape[0]
n_features = train_df.shape[1] - 1
target_column = "prognosis"
# Encode our prognosis labels as integers for easier decoding later
enc = OrdinalEncoder()
y = enc.fit_transform(train_df[[target_column]])
test_df = pd.read_csv('./data/VBDP/test.csv', index_col=0)
test_df| sudden_fever | headache | mouth_bleed | nose_bleed | muscle_pain | joint_pain | vomiting | rash | diarrhea | hypotension | ... | lymph_swells | breathing_restriction | toe_inflammation | finger_inflammation | lips_irritation | itchiness | ulcers | toenail_loss | speech_problem | bullseye_rash | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||||||||||
| 707 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 708 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 709 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 710 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 711 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1005 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 1006 | 1.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 1007 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 1008 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| 1009 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
303 rows × 64 columns
# convert the test_df to a torch tensor
X_tensor = torch.tensor(test_df.values, dtype=torch.float32)
X_tensor.shapetorch.Size([303, 64])fun_control["device"]'cpu'y = model_loaded(X_tensor.to(fun_control["device"]))
y.shapetorch.Size([303, 11])# convert the predictions to a numpy array
y = y.cpu().detach().numpy()
yarray([[1.0339374e-03, 2.1091605e-02, 5.3262565e-02, ..., 4.0523899e-01,
2.5422257e-01, 5.0181095e-03],
[9.9639302e-01, 2.0018709e-03, 2.1086355e-06, ..., 3.3996358e-07,
3.5653440e-07, 3.2454190e-07],
[1.7835979e-05, 4.6545082e-07, 2.1162709e-04, ..., 7.9153629e-04,
1.5308069e-02, 2.5282228e-08],
...,
[7.2369519e-14, 2.3624997e-04, 6.5417282e-08, ..., 4.6419292e-03,
4.0949080e-06, 4.2314281e-09],
[7.1186376e-05, 1.7185325e-02, 1.7605823e-03, ..., 9.6896887e-02,
1.5296279e-01, 8.8034321e-05],
[4.2483574e-08, 6.4948481e-04, 3.2100389e-03, ..., 9.1423398e-01,
8.9441258e-03, 1.3803144e-05]], dtype=float32)test_sorted_prediction_ids = np.argsort(-y, axis=1)
test_top_3_prediction_ids = test_sorted_prediction_ids[:,:3]
original_shape = test_top_3_prediction_ids.shape
test_top_3_prediction = enc.inverse_transform(test_top_3_prediction_ids.reshape(-1, 1))
test_top_3_prediction = test_top_3_prediction.reshape(original_shape)
test_df['prognosis'] = np.apply_along_axis(lambda x: np.array(' '.join(x), dtype="object"), 1, test_top_3_prediction)
test_df['prognosis'].reset_index().to_csv('./data/VBDP/submission.csv', index=False)