# 4 Using `sklearn` Surrogates in `spotPython`

This notebook explains how different surrogate models from `scikit-learn` can be used as surrogates in `spotPython` optimization runs.

    import numpy as np
    from math import inf
    from spotPython.fun.objectivefunctions import analytical
    from spotPython.spot import spot
    from scipy.optimize import shgo
    from scipy.optimize import direct
    from scipy.optimize import differential_evolution
    import matplotlib.pyplot as plt

## 4.1 Example: Branin Function with `spotPython`’s Internal Kriging Surrogate

### 4.1.1 The Objective Function Branin

- The `spotPython` package provides several classes of objective functions.

- We will use an analytical objective function, i.e., a function that can be described by a (closed) formula.

- Here we will use the Branin function:

        y = a * (x2 - b * x1**2 + c * x1 - r) ** 2 + s * (1 - t) * np.cos(x1) + s,
        where values of a, b, c, r, s and t are: a = 1, b = 5.1 / (4*pi**2),
        c = 5 / pi, r = 6, s = 10 and t = 1 / (8*pi).

- It has three global minima:

        f(x) = 0.397887 at (-pi, 12.275), (pi, 2.275), and (9.42478, 2.475).

    from spotPython.fun.objectivefunctions import analytical
    lower = np.array([-5,-0])
    upper = np.array([10,15])

    fun = analytical().fun_branin

### 4.1.2 Running the surrogate model based optimizer `Spot`:

    spot_2 = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = 20,
                       max_time = inf,
                       seed=123,
                       design_control={"init_size": 10})

    spot_2.run()

    <spotPython.spot.spot.Spot at 0x296198d60>

### 4.1.3 Print the Results

    spot_2.print_results()

    min y: 0.3982295132785083
    x0: 3.135528626303215
    x1: 2.2926027772585886

    [['x0', 3.135528626303215], ['x1', 2.2926027772585886]]

### 4.1.4 Show the Progress and the Surrogate

    spot_2.plot_progress(log_y=True)

    spot_2.surrogate.plot()

## 4.2 Example: Using Surrogates From scikit-learn

- Default is the `spotPython` (i.e., the internal) `kriging` surrogate.
- It can be called explicitely and passed to `Spot`.

    from spotPython.build.kriging import Kriging
    S_0 = Kriging(name='kriging', seed=123)

- Alternatively, models from `scikit-learn` can be selected, e.g., Gaussian Process, RBFs, Regression Trees, etc.

    # Needed for the sklearn surrogates:
    from sklearn.gaussian_process import GaussianProcessRegressor
    from sklearn.gaussian_process.kernels import RBF
    from sklearn.tree import DecisionTreeRegressor
    from sklearn.ensemble import RandomForestRegressor
    from sklearn import linear_model
    from sklearn import tree
    import pandas as pd

- Here are some additional models that might be useful later:

    S_Tree = DecisionTreeRegressor(random_state=0)
    S_LM = linear_model.LinearRegression()
    S_Ridge = linear_model.Ridge()
    S_RF = RandomForestRegressor(max_depth=2, random_state=0)

### 4.2.1 GaussianProcessRegressor as a Surrogate

- To use a Gaussian Process model from `sklearn`, that is similar to `spotPython`’s `Kriging`, we can proceed as follows:

    kernel = 1 * RBF(length_scale=1.0, length_scale_bounds=(1e-2, 1e2))
    S_GP = GaussianProcessRegressor(kernel=kernel, n_restarts_optimizer=9)

- The scikit-learn GP model `S_GP` is selected for `Spot` as follows:

  `surrogate = S_GP`

- We can check the kind of surogate model with the command `isinstance`:

    isinstance(S_GP, GaussianProcessRegressor)

    True

    isinstance(S_0, Kriging)

    True

- Similar to the `Spot` run with the internal `Kriging` model, we can call the run with the `scikit-learn` surrogate:

    fun = analytical(seed=123).fun_branin
    spot_2_GP = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = 20,
                       seed=123,
                       design_control={"init_size": 10},
                       surrogate = S_GP)
    spot_2_GP.run()

    <spotPython.spot.spot.Spot at 0x2b21a49d0>

    spot_2_GP.plot_progress()

    spot_2_GP.print_results()

    min y: 0.39795801268872033
    x0: 3.1437193954948177
    x1: 2.280337712260447

    [['x0', 3.1437193954948177], ['x1', 2.280337712260447]]

## 4.3 Example: One-dimensional Sphere Function With `spotPython`’s Kriging

- In this example, we will use an one-dimensional function, which allows us to visualize the optimization process.
  - `show_models= True` is added to the argument list.

    from spotPython.fun.objectivefunctions import analytical
    lower = np.array([-1])
    upper = np.array([1])
    fun = analytical(seed=123).fun_sphere

    spot_1 = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = 10,
                       max_time = inf,
                       seed=123,
                       show_models= True,
                       tolerance_x = np.sqrt(np.spacing(1)),
                       design_control={"init_size": 3},)
    spot_1.run()

    <spotPython.spot.spot.Spot at 0x2b21c4400>

### 4.3.1 Results

    spot_1.print_results()

    min y: 4.41925228274096e-08
    x0: -0.00021022017702259125

    [['x0', -0.00021022017702259125]]

    spot_1.plot_progress(log_y=True)

- The method `plot_model` plots the final surrogate:

    spot_1.plot_model()

## 4.4 Example: `Sklearn` Model GaussianProcess

- This example visualizes the search process on the `GaussianProcessRegression` surrogate from `sklearn`.
- Therefore `surrogate = S_GP` is added to the argument list.

    fun = analytical(seed=123).fun_sphere
    spot_1_GP = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = 10,
                       max_time = inf,
                       seed=123,
                       show_models= True,
                       design_control={"init_size": 3},
                       surrogate = S_GP)
    spot_1_GP.run()

    <spotPython.spot.spot.Spot at 0x297e98d90>

    spot_1_GP.print_results()

    min y: 1.4655862288352007e-09
    x0: -3.8282975705072886e-05

    [['x0', -3.8282975705072886e-05]]

    spot_1_GP.plot_progress(log_y=True)

    spot_1_GP.plot_model()

## 4.5 Exercises

### 4.5.1 `DecisionTreeRegressor`

- Describe the surrogate model.
- Use the surrogate as the model for optimization.

### 4.5.2 `RandomForestRegressor`

- Describe the surrogate model.
- Use the surrogate as the model for optimization.

### 4.5.3 `linear_model.LinearRegression`

- Describe the surrogate model.
- Use the surrogate as the model for optimization.

### 4.5.4 `linear_model.Ridge`

- Describe the surrogate model.
- Use the surrogate as the model for optimization.

## 4.6 Exercise 2

- Compare the performance of the five different surrogates on both objective functions:

  - `spotPython`’s internal Kriging
  - `DecisionTreeRegressor`
  - `RandomForestRegressor`
  - `linear_model.LinearRegression`
  - `linear_model.Ridge`