11  River Hyperparameter Tuning with SPOT HATR

11.1 Step 1: Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.

Caution: Run time and initial design size should be increased for real experiments

• MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
• INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.
• K is set to 0.1 for demonstration purposes. For real experiments, this should be increased to at least 1.

MAX_TIME = 10
INIT_SIZE = 10
PREFIX="10-river"
K = .1

12 Chapter 10: Sequential Parameter Optimization

12.1 river Hyperparameter Tuning: HATR with Friedman Drift Data

• This notebook exemplifies hyperparameter tuning with SPOT (spotPython and spotRiver).
• The hyperparameter software SPOT was developed in R (statistical programming language), see Open Access book “Hyperparameter Tuning for Machine and Deep Learning with R - A Practical Guide”, available here: https://link.springer.com/book/10.1007/978-981-19-5170-1.
• This notebook demonstrates hyperparameter tuning for river. It is based on the notebook “Incremental decision trees in river: the Hoeffding Tree case”, see: https://riverml.xyz/0.15.0/recipes/on-hoeffding-trees/#42-regression-tree-splitters.
• Here we will use the river HTR and HATR functions as in “Incremental decision trees in river: the Hoeffding Tree case”, see: https://riverml.xyz/0.15.0/recipes/on-hoeffding-trees/#42-regression-tree-splitters.

12.2 Example 1: HATR Hyperparameter

12.3 0. Initialization of the Empty fun_control Dictionary

from spotPython.utils.init import fun_control_init
fun_control = fun_control_init(tensorboard_path=f"./runs/{experiment_name}/")

12.4 1. Load Data: The Friedman Drift Data

horizon = 7*24
k = K
n_total = int(k*100_000)
n_samples = n_total
p_1 = int(k*25_000)
p_2 = int(k*50_000)
position=(p_1, p_2)
n_train = 1_000
a = n_train + p_1 - 12
b = a + 12

• Since we also need a river version of the data below for plotting the model, the corresponding data set is generated here. Note: spotRiver uses the train and test data sets, while river uses the X and y data sets

from river.datasets import synth
import pandas as pd
dataset = synth.FriedmanDrift(
   drift_type='gra',
   position=position,
     seed=123
)

from spotRiver.utils.data_conversion import convert_to_df
target_column = "y"
df = convert_to_df(dataset, target_column=target_column, n_total=n_total)

# Add column names x1 until x10 to the first 10 columns of the dataframe and the column name y to the last column
df.columns = [f"x{i}" for i in range(1, 11)] + ["y"]

train = df[:n_train]
test = df[n_train:]
#
fun_control.update({"data": None, # dataset,
               "train": train,
               "test": test,
               "n_samples": n_samples,
               "target_column": target_column})

12.5 2. Specification of the Preprocessing Model

from river import preprocessing
prep_model = preprocessing.StandardScaler()
fun_control.update({"prep_model": prep_model})

12.6 3. Select algorithm and core_model_hyper_dict

• The river model (HATR) is selected.
• Furthermore, the corresponding hyperparameters, see: https://riverml.xyz/0.15.0/api/tree/HoeffdingTreeRegressor/ are selected (incl. type information, names, and bounds).
• The corresponding hyperparameter dictionary is added to the fun_control dictionary.
• Alternatively, you can load a local hyper_dict. Simply set river_hyper_dict.json as the filename. If filenameis set to None, the hyper_dict is loaded from the spotRiver package.

from river.tree import HoeffdingAdaptiveTreeRegressor
from spotRiver.data.river_hyper_dict import RiverHyperDict
from spotPython.hyperparameters.values import add_core_model_to_fun_control
core_model  = HoeffdingAdaptiveTreeRegressor
add_core_model_to_fun_control(core_model=core_model,
                              fun_control=fun_control,
                              hyper_dict=RiverHyperDict,
                              filename=None)

12.7 4. Modify hyper_dict Hyperparameters for the Selected Algorithm aka core_model

12.7.1 Modify hyperparameter of type factor

# modify_hyper_parameter_levels(fun_control, "leaf_model", ["LinearRegression"])
# fun_control["core_model_hyper_dict"]

12.7.2 Modify hyperparameter of type numeric and integer (boolean)

from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
modify_hyper_parameter_bounds(fun_control, "delta", bounds=[1e-10, 1e-6])
# modify_hyper_parameter_bounds(fun_control, "min_samples_split", bounds=[3, 20])
modify_hyper_parameter_bounds(fun_control, "merit_preprune", [0, 0])

12.8 5. Selection of the Objective (Loss) Function

There are two metrics:

1. `metric` is used for the river based evaluation via `eval_oml_iter_progressive`.
2. `metric_sklearn` is used for the sklearn based evaluation via `eval_oml_horizon`.

import numpy as np
from river import metrics
from sklearn.metrics import mean_absolute_error

weights = np.array([1, 1/1000, 1/1000])*10_000.0
horizon = 7*24
oml_grace_period = 2
step = 100
weight_coeff = 1.0

fun_control.update({
               "horizon": horizon,
               "oml_grace_period": oml_grace_period,
               "weights": weights,
               "step": step,
               "log_level": 50,
               "weight_coeff": weight_coeff,
               "metric": metrics.MAE(),
               "metric_sklearn": mean_absolute_error
               })

12.9 6. Calling the SPOT Function

12.9.1 Prepare the SPOT Parameters

• Get types and variable names as well as lower and upper bounds for the hyperparameters.

from spotPython.hyperparameters.values import (
    get_var_type,
    get_var_name,
    get_bound_values    
    )
var_type = get_var_type(fun_control)
var_name = get_var_name(fun_control)
lower = get_bound_values(fun_control, "lower")
upper = get_bound_values(fun_control, "upper")

from spotPython.utils.eda import gen_design_table
print(gen_design_table(fun_control))

12.9.2 Run the Spot Optimizer

from spotRiver.fun.hyperriver import HyperRiver
fun = HyperRiver().fun_oml_horizon

from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
X_start = get_default_hyperparameters_as_array(fun_control)

• Run SPOT for approx. x mins (max_time).
• Note: the run takes longer, because the evaluation time of initial design (here: init_size = INIT_SIZE as specified above) is not considered.

from spotPython.spot import spot
from math import inf
import numpy as np
spot_tuner = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = inf,
                   max_time = MAX_TIME,
                   tolerance_x = np.sqrt(np.spacing(1)),
                   var_type = var_type,
                   var_name = var_name,
                   show_progress= True,
                   fun_control = fun_control,
                   design_control={"init_size": INIT_SIZE},
                   surrogate_control={"noise": True,
                                      "cod_type": "norm",
                                      "min_theta": -4,
                                      "max_theta": 3,
                                      "n_theta": len(var_name),
                                      "model_fun_evals": 10_000})
spot_tuner.run(X_start=X_start)

12.9.3 4 Results

from spotPython.utils.file import save_pickle
save_pickle(spot_tuner, experiment_name)

from spotPython.utils.file import load_pickle
spot_tuner = load_pickle(experiment_name)

• Show the Progress of the hyperparameter tuning:

spot_tuner.plot_progress(log_y=True, filename="./figures/" + experiment_name+"_progress.pdf")

• Print the Results

print(gen_design_table(fun_control=fun_control, spot=spot_tuner))

12.10 Show variable importance

spot_tuner.plot_importance(threshold=0.0025, filename="./figures/" + experiment_name+"_importance.pdf")

12.11 Build and Evaluate HTR Model with Tuned Hyperparameters

m = test.shape[0]
a = int(m/2)-50
b = int(m/2)

13 Der große Datensatz

Caution: Increased Friedman-Drift Data Set

• The Friedman-Drift Data Set is increased by a factor of two to show the transferability of the hyperparameter tuning results.
• Larger values of k lead to a longer run time.

horizon = 7*24
k = 0.2
n_total = int(k*100_000)
n_samples = n_total
p_1 = int(k*25_000)
p_2 = int(k*50_000)
position=(p_1, p_2)
n_train = 1_000
a = n_train + p_1 - 12
b = a + 12

from river.datasets import synth
dataset = synth.FriedmanDrift(
   drift_type='gra',
   position=position,
     seed=123
)

from spotRiver.utils.data_conversion import convert_to_df
target_column = "y"
df = convert_to_df(dataset, target_column=target_column, n_total=n_total)
# Add column names x1 until x10 to the first 10 columns of the dataframe and the column name y to the last column
df.columns = [f"x{i}" for i in range(1, 11)] + ["y"]
train = df[:n_train]
test = df[n_train:]
target_column = "y"
#
fun_control.update({"data": None, # dataset,
               "train": train,
               "test": test,
               "n_samples": n_samples,
               "target_column": target_column})

13.1 Get Default Hyperparameters

# fun_control was modified, we generate a new one with the original 
# default hyperparameters
from spotPython.hyperparameters.values import get_one_core_model_from_X
from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
X_start = get_default_hyperparameters_as_array(fun_control)
model_default = get_one_core_model_from_X(X_start, fun_control)
model_default

from spotRiver.evaluation.eval_bml import eval_oml_horizon

df_eval_default, df_true_default = eval_oml_horizon(
                    model=model_default,
                    train=fun_control["train"],
                    test=fun_control["test"],
                    target_column=fun_control["target_column"],
                    horizon=fun_control["horizon"],
                    oml_grace_period=fun_control["oml_grace_period"],
                    metric=fun_control["metric_sklearn"],
                )

from spotRiver.evaluation.eval_bml import plot_bml_oml_horizon_metrics, plot_bml_oml_horizon_predictions
df_labels=["default"]
plot_bml_oml_horizon_metrics(df_eval = [df_eval_default], log_y=False, df_labels=df_labels, metric=fun_control["metric_sklearn"])
plot_bml_oml_horizon_predictions(df_true = [df_true_default[a:b]], target_column=target_column,  df_labels=df_labels)

13.2 Get SPOT Results

from spotPython.hyperparameters.values import get_one_core_model_from_X
X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
model_spot = get_one_core_model_from_X(X, fun_control)
model_spot

df_eval_spot, df_true_spot = eval_oml_horizon(
                    model=model_spot,
                    train=fun_control["train"],
                    test=fun_control["test"],
                    target_column=fun_control["target_column"],
                    horizon=fun_control["horizon"],
                    oml_grace_period=fun_control["oml_grace_period"],
                    metric=fun_control["metric_sklearn"],
                )

df_labels=["default", "spot"]
plot_bml_oml_horizon_metrics(df_eval = [df_eval_default, df_eval_spot], log_y=False, df_labels=df_labels, metric=fun_control["metric_sklearn"], filename="./figures/" + experiment_name+"_metrics.pdf")

a = int(m/2)+20
b = int(m/2)+50
plot_bml_oml_horizon_predictions(df_true = [df_true_default[a:b], df_true_spot[a:b]], target_column=target_column,  df_labels=df_labels, filename="./figures/" + experiment_name+"_predictions.pdf")

from spotPython.plot.validation import plot_actual_vs_predicted
plot_actual_vs_predicted(y_test=df_true_default["y"], y_pred=df_true_default["Prediction"], title="Default")
plot_actual_vs_predicted(y_test=df_true_spot["y"], y_pred=df_true_spot["Prediction"], title="SPOT")

13.3 Visualize Regression Trees

dataset_f = dataset.take(n_total)
for x, y in dataset_f:
    model_default.learn_one(x, y)

Caution: Large Trees

• Since the trees are large, the visualization is suppressed by default.
• To visualize the trees, uncomment the following line.

# model_default.draw()

model_default.summary

13.3.1 Spot Model

dataset_f = dataset.take(n_total)
for x, y in dataset_f:
    model_spot.learn_one(x, y)

Caution: Large Trees

• Since the trees are large, the visualization is suppressed by default.
• To visualize the trees, uncomment the following line.

# model_spot.draw()

model_spot.summary

from spotPython.utils.eda import compare_two_tree_models
print(compare_two_tree_models(model_default, model_spot))

13.4 Detailed Hyperparameter Plots

filename = "./figures/" + experiment_name
spot_tuner.plot_important_hyperparameter_contour(filename=filename)

13.5 Parallel Coordinates Plots

spot_tuner.parallel_plot()

13.6 Plot all Combinations of Hyperparameters

• Warning: this may take a while.

PLOT_ALL = False
if PLOT_ALL:
    n = spot_tuner.k
    for i in range(n-1):
        for j in range(i+1, n):
            spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)