4  Using sklearn Surrogates in spotPython

This notebook explains how different surrogate models from scikit-learn can be used as surrogates in spotPython optimization runs.

import numpy as np
from math import inf
from spotPython.fun.objectivefunctions import analytical
from spotPython.spot import spot
from scipy.optimize import shgo
from scipy.optimize import direct
from scipy.optimize import differential_evolution
import matplotlib.pyplot as plt

4.1 Example: Branin Function with spotPython’s Internal Kriging Surrogate

4.1.1 The Objective Function Branin

  • The spotPython package provides several classes of objective functions.

  • We will use an analytical objective function, i.e., a function that can be described by a (closed) formula.

  • Here we will use the Branin function:

      y = a * (x2 - b * x1**2 + c * x1 - r) ** 2 + s * (1 - t) * np.cos(x1) + s,
  where values of a, b, c, r, s and t are: a = 1, b = 5.1 / (4*pi**2),
  c = 5 / pi, r = 6, s = 10 and t = 1 / (8*pi).

  • It has three global minima:

      f(x) = 0.397887 at (-pi, 12.275), (pi, 2.275), and (9.42478, 2.475).
from spotPython.fun.objectivefunctions import analytical
lower = np.array([-5,-0])
upper = np.array([10,15])
fun = analytical().fun_branin

4.1.2 Running the surrogate model based optimizer Spot:

spot_2 = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = 20,
                   max_time = inf,
                   seed=123,
                   design_control={"init_size": 10})
spot_2.run()
<spotPython.spot.spot.Spot at 0x1033ae4a0>

4.1.4 Show the Progress and the Surrogate

spot_2.plot_progress(log_y=True)

spot_2.surrogate.plot()

4.2 Example: Using Surrogates From scikit-learn

  • Default is the spotPython (i.e., the internal) kriging surrogate.
  • It can be called explicitely and passed to Spot.
from spotPython.build.kriging import Kriging
S_0 = Kriging(name='kriging', seed=123)
  • Alternatively, models from scikit-learn can be selected, e.g., Gaussian Process, RBFs, Regression Trees, etc.
# Needed for the sklearn surrogates:
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn import linear_model
from sklearn import tree
import pandas as pd
  • Here are some additional models that might be useful later:
S_Tree = DecisionTreeRegressor(random_state=0)
S_LM = linear_model.LinearRegression()
S_Ridge = linear_model.Ridge()
S_RF = RandomForestRegressor(max_depth=2, random_state=0)

4.2.1 GaussianProcessRegressor as a Surrogate

  • To use a Gaussian Process model from sklearn, that is similar to spotPython’s Kriging, we can proceed as follows:
kernel = 1 * RBF(length_scale=1.0, length_scale_bounds=(1e-2, 1e2))
S_GP = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=9)

  • The scikit-learn GP model S_GP is selected for Spot as follows:

    surrogate = S_GP

  • We can check the kind of surogate model with the command isinstance:

isinstance(S_GP, GaussianProcessRegressor)
True
isinstance(S_0, Kriging)
True
  • Similar to the Spot run with the internal Kriging model, we can call the run with the scikit-learn surrogate:
fun = analytical(seed=123).fun_branin
spot_2_GP = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = 20,
                   seed=123,
                   design_control={"init_size": 10},
                   surrogate = S_GP)
spot_2_GP.run()
<spotPython.spot.spot.Spot at 0x16838d570>
spot_2_GP.plot_progress()

spot_2_GP.print_results()
min y: 0.39820887766041757
x0: 3.1496777572038104
x1: 2.2714748202662483
[['x0', 3.1496777572038104], ['x1', 2.2714748202662483]]

4.3 Example: One-dimensional Sphere Function With spotPython’s Kriging

  • In this example, we will use an one-dimensional function, which allows us to visualize the optimization process.
    • show_models= True is added to the argument list.
from spotPython.fun.objectivefunctions import analytical
lower = np.array([-1])
upper = np.array([1])
fun = analytical(seed=123).fun_sphere
spot_1 = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = 10,
                   max_time = inf,
                   seed=123,
                   show_models= True,
                   tolerance_x = np.sqrt(np.spacing(1)),
                   design_control={"init_size": 3},)
spot_1.run()

<spotPython.spot.spot.Spot at 0x16838f6d0>

4.3.1 Results

spot_1.print_results()
min y: 4.41925228274096e-08
x0: -0.00021022017702259125
[['x0', -0.00021022017702259125]]
spot_1.plot_progress(log_y=True)

  • The method plot_model plots the final surrogate:
spot_1.plot_model()

4.4 Example: Sklearn Model GaussianProcess

  • This example visualizes the search process on the GaussianProcessRegression surrogate from sklearn.
  • Therefore surrogate = S_GP is added to the argument list.
fun = analytical(seed=123).fun_sphere
spot_1_GP = spot.Spot(fun=fun,
                   lower = lower,
                   upper = upper,
                   fun_evals = 10,
                   max_time = inf,
                   seed=123,
                   show_models= True,
                   design_control={"init_size": 3},
                   surrogate = S_GP)
spot_1_GP.run()

<spotPython.spot.spot.Spot at 0x16802dba0>
spot_1_GP.print_results()
min y: 1.1056384820938813e-09
x0: 3.3251142568246905e-05
[['x0', 3.3251142568246905e-05]]
spot_1_GP.plot_progress(log_y=True)

spot_1_GP.plot_model()

4.5 Exercises

4.5.1 DecisionTreeRegressor

  • Describe the surrogate model.
  • Use the surrogate as the model for optimization.

4.5.2 RandomForestRegressor

  • Describe the surrogate model.
  • Use the surrogate as the model for optimization.

4.5.3 linear_model.LinearRegression

  • Describe the surrogate model.
  • Use the surrogate as the model for optimization.

4.5.4 linear_model.Ridge

  • Describe the surrogate model.
  • Use the surrogate as the model for optimization.

4.6 Exercise 2

  • Compare the performance of the five different surrogates on both objective functions:

    • spotPython’s internal Kriging
    • DecisionTreeRegressor
    • RandomForestRegressor
    • linear_model.LinearRegression
    • linear_model.Ridge