MixCOBRARegressor of gradientcobra v1.0.8 package

Author

Installing and importing packages

gradientcobra can be installed from pypi using pip:

pip install gradientcobra

Importing packages

# Metric of error
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_absolute_percentage_error

# Plotting figures
import matplotlib.pyplot as plt
from matplotlib import cm

# Import class MixCOBRA from the mixcobra module of gradientcobra library
from gradientcobra.mixcobra import MixCOBRARegressor

import seaborn as sns
sns.set()

Simulated data

We simulate a regression data with \(1000\) observations and \(10\) inputs variables.

# For simulating dataset
from sklearn.datasets import make_regression

X1, y1 = make_regression(n_samples=1000, n_features=10, noise=1)

Now, let’s randoly split the simulated data into \(80\%-20\%\) training-testing data.

from sklearn.model_selection import train_test_split

X_train1, X_test1, y_train1, y_test1 = train_test_split(X1, y1, test_size=0.2)
print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(
    X_train1.shape, 
    X_test1.shape, 
    y_train1.shape, 
    y_test1.shape))
shape: x_train = (800, 10) , x_train = (200, 10) , y_train = (800,) , y_test = (200,)

MixCOBRARegressor with default parameters

We create MixCOBRARegressor object called gc1, with the default parameters, then fit it to the training data.

Note that by default, we use grid search algorithm to estimate the optimal parameter of the method.

gc1 = MixCOBRARegressor()
gc1_fit = gc1.fit(X_train1, y_train1)

    * Grid search algorithm of two parameters with radial kernel is in progress...
        ~ Full process|--------------------------------------------------|100%
        ~   Processing|==================================================|100%

The estimated optimal bandwidth is given by gc1.optimization_outputs['opt_bandwidth'].

# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc1_fit.optimization_outputs['opt_alpha'], gc1_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (1e-05, 0.7575842424242424)

We can look at the learning curve (surface) of the algorithm using draw_learning_curve() method.

gc1_fit.draw_learning_curve()

We evaluate the performance of the method on the testing data using MSE and MAPE.

from sklearn.metrics import mean_absolute_percentage_error
y_pred1 = gc1_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred1))
print(mean_squared_error(y_test1, y_pred1))
0.16552630550806227
150.23447041060408

Let’s look at qq-plot of the predictions and the actual response values using draw_learning_curve() method.

gc1_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')

Use one bandwidth parameter instead of (\(\alpha, \beta\))

You can also try to implement MixCOBRARegressor using only one bandwidth parameter. To do so, just set one_parameter = True when fitting the method. We create another MixCOBRARegressor object with gradient descent optimization method, then fit it on the same data as in the previous case using only one bandwidth parameter as follow.

# MixCOBRA with one parameter
gc2 = MixCOBRARegressor(opt_method="grad")
gc2_fit = gc2.fit(X_train1, y_train1, one_parameter=True)

    * Gradient descent of one parameter with radial kernel is implemented...
        ~ Initial t = 0:        ~ bandwidth: 2.507  ~ gradient: 33.439   ~ threshold: 1e-05
        ~     Iteration: 1  ~ bandwidth: 2.498  ~ gradient: 33.439  ~ stopping criterion: 16.721        ~     Iteration: 2  ~ bandwidth: 2.498  ~ gradient: 33.439  ~ stopping criterion: 16.721        ~     Iteration: 3  ~ bandwidth: 2.498  ~ gradient: 33.439  ~ stopping criterion: 16.721        ~     Iteration: 4  ~ bandwidth: 2.498  ~ gradient: 33.439  ~ stopping criterion: 16.721        ~     Iteration: 5  ~ bandwidth: 2.498  ~ gradient: 32.570  ~ stopping criterion: 16.719        ~     Iteration: 6  ~ bandwidth: 2.488  ~ gradient: 31.760  ~ stopping criterion: 16.285        ~     Iteration: 7  ~ bandwidth: 2.478  ~ gradient: 33.054  ~ stopping criterion: 15.880        ~     Iteration: 8  ~ bandwidth: 2.468  ~ gradient: 32.551  ~ stopping criterion: 16.527        ~     Iteration: 9  ~ bandwidth: 2.459  ~ gradient: 32.933  ~ stopping criterion: 16.276        ~     Iteration: 10     ~ bandwidth: 2.449  ~ gradient: 31.902  ~ stopping criterion: 16.467        ~     Iteration: 11     ~ bandwidth: 2.439  ~ gradient: 32.765  ~ stopping criterion: 15.951        ~     Iteration: 12     ~ bandwidth: 2.429  ~ gradient: 30.846  ~ stopping criterion: 16.382        ~     Iteration: 13     ~ bandwidth: 2.420  ~ gradient: 33.097  ~ stopping criterion: 15.423        ~     Iteration: 14     ~ bandwidth: 2.410  ~ gradient: 32.646  ~ stopping criterion: 16.549        ~     Iteration: 15     ~ bandwidth: 2.401  ~ gradient: 32.505  ~ stopping criterion: 16.323        ~     Iteration: 16     ~ bandwidth: 2.391  ~ gradient: 33.117  ~ stopping criterion: 16.252        ~     Iteration: 17     ~ bandwidth: 2.381  ~ gradient: 34.084  ~ stopping criterion: 16.559        ~     Iteration: 18     ~ bandwidth: 2.371  ~ gradient: 34.117  ~ stopping criterion: 17.042        ~     Iteration: 19     ~ bandwidth: 2.361  ~ gradient: 34.410  ~ stopping criterion: 17.058        ~     Iteration: 20     ~ bandwidth: 2.350  ~ gradient: 33.783  ~ stopping criterion: 17.205        ~     Iteration: 21     ~ bandwidth: 2.340  ~ gradient: 33.098  ~ stopping criterion: 16.891        ~     Iteration: 22     ~ bandwidth: 2.330  ~ gradient: 33.975  ~ stopping criterion: 16.549        ~     Iteration: 23     ~ bandwidth: 2.320  ~ gradient: 34.338  ~ stopping criterion: 16.987        ~     Iteration: 24     ~ bandwidth: 2.310  ~ gradient: 35.558  ~ stopping criterion: 17.169        ~     Iteration: 25     ~ bandwidth: 2.299  ~ gradient: 34.418  ~ stopping criterion: 17.779        ~     Iteration: 26     ~ bandwidth: 2.289  ~ gradient: 33.465  ~ stopping criterion: 17.209        ~     Iteration: 27     ~ bandwidth: 2.279  ~ gradient: 32.339  ~ stopping criterion: 16.733        ~     Iteration: 28     ~ bandwidth: 2.269  ~ gradient: 33.435  ~ stopping criterion: 16.169        ~     Iteration: 29     ~ bandwidth: 2.259  ~ gradient: 33.759  ~ stopping criterion: 16.717        ~     Iteration: 30     ~ bandwidth: 2.249  ~ gradient: 34.556  ~ stopping criterion: 16.879        ~     Iteration: 31     ~ bandwidth: 2.239  ~ gradient: 34.685  ~ stopping criterion: 17.278        ~     Iteration: 32     ~ bandwidth: 2.228  ~ gradient: 35.063  ~ stopping criterion: 17.343        ~     Iteration: 33     ~ bandwidth: 2.218  ~ gradient: 36.170  ~ stopping criterion: 17.532        ~     Iteration: 34     ~ bandwidth: 2.207  ~ gradient: 33.235  ~ stopping criterion: 18.085        ~     Iteration: 35     ~ bandwidth: 2.197  ~ gradient: 35.208  ~ stopping criterion: 16.618        ~     Iteration: 36     ~ bandwidth: 2.187  ~ gradient: 37.075  ~ stopping criterion: 17.604        ~     Iteration: 37     ~ bandwidth: 2.176  ~ gradient: 35.903  ~ stopping criterion: 18.537        ~     Iteration: 38     ~ bandwidth: 2.165  ~ gradient: 34.320  ~ stopping criterion: 17.952        ~     Iteration: 39     ~ bandwidth: 2.155  ~ gradient: 35.160  ~ stopping criterion: 17.160        ~     Iteration: 40     ~ bandwidth: 2.144  ~ gradient: 35.873  ~ stopping criterion: 17.580        ~     Iteration: 41     ~ bandwidth: 2.133  ~ gradient: 35.583  ~ stopping criterion: 17.937        ~     Iteration: 42     ~ bandwidth: 2.123  ~ gradient: 35.432  ~ stopping criterion: 17.792        ~     Iteration: 43     ~ bandwidth: 2.112  ~ gradient: 36.362  ~ stopping criterion: 17.716        ~     Iteration: 44     ~ bandwidth: 2.101  ~ gradient: 36.246  ~ stopping criterion: 18.181        ~     Iteration: 45     ~ bandwidth: 2.090  ~ gradient: 36.555  ~ stopping criterion: 18.123        ~     Iteration: 46     ~ bandwidth: 2.079  ~ gradient: 36.160  ~ stopping criterion: 18.278        ~     Iteration: 47     ~ bandwidth: 2.069  ~ gradient: 36.117  ~ stopping criterion: 18.080        ~     Iteration: 48     ~ bandwidth: 2.058  ~ gradient: 36.913  ~ stopping criterion: 18.059        ~     Iteration: 49     ~ bandwidth: 2.047  ~ gradient: 37.435  ~ stopping criterion: 18.457        ~     Iteration: 50     ~ bandwidth: 2.036  ~ gradient: 36.893  ~ stopping criterion: 18.718        ~     Iteration: 51     ~ bandwidth: 2.025  ~ gradient: 34.964  ~ stopping criterion: 18.446        ~     Iteration: 52     ~ bandwidth: 2.014  ~ gradient: 35.160  ~ stopping criterion: 17.482        ~     Iteration: 53     ~ bandwidth: 2.004  ~ gradient: 37.250  ~ stopping criterion: 17.580        ~     Iteration: 54     ~ bandwidth: 1.992  ~ gradient: 37.434  ~ stopping criterion: 18.625        ~     Iteration: 55     ~ bandwidth: 1.981  ~ gradient: 36.231  ~ stopping criterion: 18.717        ~     Iteration: 56     ~ bandwidth: 1.970  ~ gradient: 36.648  ~ stopping criterion: 18.115        ~     Iteration: 57     ~ bandwidth: 1.959  ~ gradient: 37.566  ~ stopping criterion: 18.324        ~     Iteration: 58     ~ bandwidth: 1.948  ~ gradient: 37.458  ~ stopping criterion: 18.783        ~     Iteration: 59     ~ bandwidth: 1.937  ~ gradient: 37.723  ~ stopping criterion: 18.729        ~     Iteration: 60     ~ bandwidth: 1.926  ~ gradient: 37.482  ~ stopping criterion: 18.861        ~     Iteration: 61     ~ bandwidth: 1.915  ~ gradient: 37.073  ~ stopping criterion: 18.741        ~     Iteration: 62     ~ bandwidth: 1.903  ~ gradient: 37.777  ~ stopping criterion: 18.537        ~     Iteration: 63     ~ bandwidth: 1.892  ~ gradient: 37.617  ~ stopping criterion: 18.888        ~     Iteration: 64     ~ bandwidth: 1.881  ~ gradient: 36.882  ~ stopping criterion: 18.809        ~     Iteration: 65     ~ bandwidth: 1.870  ~ gradient: 38.111  ~ stopping criterion: 18.441        ~     Iteration: 66     ~ bandwidth: 1.858  ~ gradient: 36.905  ~ stopping criterion: 19.055        ~     Iteration: 67     ~ bandwidth: 1.847  ~ gradient: 38.639  ~ stopping criterion: 18.453        ~     Iteration: 68     ~ bandwidth: 1.836  ~ gradient: 37.235  ~ stopping criterion: 19.320        ~     Iteration: 69     ~ bandwidth: 1.825  ~ gradient: 37.896  ~ stopping criterion: 18.618        ~     Iteration: 70     ~ bandwidth: 1.813  ~ gradient: 38.226  ~ stopping criterion: 18.948        ~     Iteration: 71     ~ bandwidth: 1.802  ~ gradient: 39.661  ~ stopping criterion: 19.113        ~     Iteration: 72     ~ bandwidth: 1.790  ~ gradient: 38.067  ~ stopping criterion: 19.830        ~     Iteration: 73     ~ bandwidth: 1.779  ~ gradient: 38.549  ~ stopping criterion: 19.033        ~     Iteration: 74     ~ bandwidth: 1.767  ~ gradient: 38.525  ~ stopping criterion: 19.274        ~     Iteration: 75     ~ bandwidth: 1.756  ~ gradient: 37.115  ~ stopping criterion: 19.262        ~     Iteration: 76     ~ bandwidth: 1.745  ~ gradient: 38.392  ~ stopping criterion: 18.558        ~     Iteration: 77     ~ bandwidth: 1.733  ~ gradient: 38.877  ~ stopping criterion: 19.196        ~     Iteration: 78     ~ bandwidth: 1.721  ~ gradient: 39.536  ~ stopping criterion: 19.439        ~     Iteration: 79     ~ bandwidth: 1.710  ~ gradient: 38.347  ~ stopping criterion: 19.768        ~     Iteration: 80     ~ bandwidth: 1.698  ~ gradient: 37.458  ~ stopping criterion: 19.173        ~     Iteration: 81     ~ bandwidth: 1.687  ~ gradient: 38.791  ~ stopping criterion: 18.729        ~     Iteration: 82     ~ bandwidth: 1.675  ~ gradient: 38.535  ~ stopping criterion: 19.395        ~     Iteration: 83     ~ bandwidth: 1.664  ~ gradient: 38.680  ~ stopping criterion: 19.267        ~     Iteration: 84     ~ bandwidth: 1.652  ~ gradient: 38.911  ~ stopping criterion: 19.340        ~     Iteration: 85     ~ bandwidth: 1.641  ~ gradient: 39.169  ~ stopping criterion: 19.456        ~     Iteration: 86     ~ bandwidth: 1.629  ~ gradient: 39.254  ~ stopping criterion: 19.584        ~     Iteration: 87     ~ bandwidth: 1.617  ~ gradient: 38.863  ~ stopping criterion: 19.627        ~     Iteration: 88     ~ bandwidth: 1.606  ~ gradient: 38.706  ~ stopping criterion: 19.432        ~     Iteration: 89     ~ bandwidth: 1.594  ~ gradient: 38.867  ~ stopping criterion: 19.353        ~     Iteration: 90     ~ bandwidth: 1.582  ~ gradient: 38.863  ~ stopping criterion: 19.434        ~     Iteration: 91     ~ bandwidth: 1.571  ~ gradient: 40.212  ~ stopping criterion: 19.432        ~     Iteration: 92     ~ bandwidth: 1.559  ~ gradient: 40.882  ~ stopping criterion: 20.106        ~     Iteration: 93     ~ bandwidth: 1.547  ~ gradient: 39.742  ~ stopping criterion: 20.441        ~     Iteration: 94     ~ bandwidth: 1.535  ~ gradient: 40.603  ~ stopping criterion: 19.871        ~     Iteration: 95     ~ bandwidth: 1.522  ~ gradient: 38.723  ~ stopping criterion: 20.302        ~     Iteration: 96     ~ bandwidth: 1.511  ~ gradient: 39.437  ~ stopping criterion: 19.361        ~     Iteration: 97     ~ bandwidth: 1.499  ~ gradient: 39.963  ~ stopping criterion: 19.719        ~     Iteration: 98     ~ bandwidth: 1.487  ~ gradient: 40.406  ~ stopping criterion: 19.981        ~     Iteration: 99     ~ bandwidth: 1.475  ~ gradient: 40.569  ~ stopping criterion: 20.203        ~     Iteration: 100    ~ bandwidth: 1.463  ~ gradient: 41.067  ~ stopping criterion: 20.285        ~     Iteration: 101    ~ bandwidth: 1.451  ~ gradient: 40.289  ~ stopping criterion: 20.533        ~     Iteration: 102    ~ bandwidth: 1.439  ~ gradient: 39.731  ~ stopping criterion: 20.144        ~     Iteration: 103    ~ bandwidth: 1.427  ~ gradient: 41.624  ~ stopping criterion: 19.865        ~     Iteration: 104    ~ bandwidth: 1.414  ~ gradient: 39.657  ~ stopping criterion: 20.812        ~     Iteration: 105    ~ bandwidth: 1.402  ~ gradient: 40.531  ~ stopping criterion: 19.829        ~     Iteration: 106    ~ bandwidth: 1.390  ~ gradient: 41.455  ~ stopping criterion: 20.266        ~     Iteration: 107    ~ bandwidth: 1.378  ~ gradient: 40.318  ~ stopping criterion: 20.727        ~     Iteration: 108    ~ bandwidth: 1.366  ~ gradient: 39.433  ~ stopping criterion: 20.159        ~     Iteration: 109    ~ bandwidth: 1.354  ~ gradient: 41.338  ~ stopping criterion: 19.716        ~     Iteration: 110    ~ bandwidth: 1.342  ~ gradient: 41.455  ~ stopping criterion: 20.669        ~     Iteration: 111    ~ bandwidth: 1.329  ~ gradient: 40.924  ~ stopping criterion: 20.727        ~     Iteration: 112    ~ bandwidth: 1.317  ~ gradient: 40.149  ~ stopping criterion: 20.462        ~     Iteration: 113    ~ bandwidth: 1.305  ~ gradient: 40.132  ~ stopping criterion: 20.074        ~     Iteration: 114    ~ bandwidth: 1.293  ~ gradient: 39.529  ~ stopping criterion: 20.066        ~     Iteration: 115    ~ bandwidth: 1.281  ~ gradient: 39.958  ~ stopping criterion: 19.765        ~     Iteration: 116    ~ bandwidth: 1.269  ~ gradient: 40.801  ~ stopping criterion: 19.979        ~     Iteration: 117    ~ bandwidth: 1.257  ~ gradient: 39.030  ~ stopping criterion: 20.401        ~     Iteration: 118    ~ bandwidth: 1.245  ~ gradient: 39.225  ~ stopping criterion: 19.515        ~     Iteration: 119    ~ bandwidth: 1.234  ~ gradient: 40.178  ~ stopping criterion: 19.612        ~     Iteration: 120    ~ bandwidth: 1.222  ~ gradient: 40.319  ~ stopping criterion: 20.089        ~     Iteration: 121    ~ bandwidth: 1.210  ~ gradient: 40.496  ~ stopping criterion: 20.160        ~     Iteration: 122    ~ bandwidth: 1.197  ~ gradient: 39.627  ~ stopping criterion: 20.248        ~     Iteration: 123    ~ bandwidth: 1.186  ~ gradient: 39.677  ~ stopping criterion: 19.813        ~     Iteration: 124    ~ bandwidth: 1.174  ~ gradient: 39.699  ~ stopping criterion: 19.838        ~     Iteration: 125    ~ bandwidth: 1.162  ~ gradient: 38.777  ~ stopping criterion: 19.850        ~     Iteration: 126    ~ bandwidth: 1.150  ~ gradient: 39.688  ~ stopping criterion: 19.388        ~     Iteration: 127    ~ bandwidth: 1.138  ~ gradient: 38.588  ~ stopping criterion: 19.844        ~     Iteration: 128    ~ bandwidth: 1.127  ~ gradient: 39.553  ~ stopping criterion: 19.294        ~     Iteration: 129    ~ bandwidth: 1.115  ~ gradient: 36.344  ~ stopping criterion: 19.777        ~     Iteration: 130    ~ bandwidth: 1.104  ~ gradient: 38.735  ~ stopping criterion: 18.172        ~     Iteration: 131    ~ bandwidth: 1.093  ~ gradient: 36.876  ~ stopping criterion: 19.367        ~     Iteration: 132    ~ bandwidth: 1.082  ~ gradient: 38.788  ~ stopping criterion: 18.438        ~     Iteration: 133    ~ bandwidth: 1.070  ~ gradient: 38.341  ~ stopping criterion: 19.394        ~     Iteration: 134    ~ bandwidth: 1.059  ~ gradient: 37.313  ~ stopping criterion: 19.171        ~     Iteration: 135    ~ bandwidth: 1.047  ~ gradient: 37.391  ~ stopping criterion: 18.656        ~     Iteration: 136    ~ bandwidth: 1.036  ~ gradient: 36.883  ~ stopping criterion: 18.696        ~     Iteration: 137    ~ bandwidth: 1.025  ~ gradient: 35.974  ~ stopping criterion: 18.442        ~     Iteration: 138    ~ bandwidth: 1.014  ~ gradient: 35.982  ~ stopping criterion: 17.987        ~     Iteration: 139    ~ bandwidth: 1.004  ~ gradient: 37.327  ~ stopping criterion: 17.991        ~     Iteration: 140    ~ bandwidth: 0.992  ~ gradient: 34.022  ~ stopping criterion: 18.663        ~     Iteration: 141    ~ bandwidth: 0.982  ~ gradient: 35.532  ~ stopping criterion: 17.011        ~     Iteration: 142    ~ bandwidth: 0.972  ~ gradient: 35.211  ~ stopping criterion: 17.766        ~     Iteration: 143    ~ bandwidth: 0.961  ~ gradient: 32.485  ~ stopping criterion: 17.605        ~     Iteration: 144    ~ bandwidth: 0.951  ~ gradient: 32.277  ~ stopping criterion: 16.242        ~     Iteration: 145    ~ bandwidth: 0.942  ~ gradient: 33.378  ~ stopping criterion: 16.138        ~     Iteration: 146    ~ bandwidth: 0.932  ~ gradient: 32.958  ~ stopping criterion: 16.689        ~     Iteration: 147    ~ bandwidth: 0.922  ~ gradient: 32.462  ~ stopping criterion: 16.479        ~     Iteration: 148    ~ bandwidth: 0.912  ~ gradient: 30.227  ~ stopping criterion: 16.231        ~     Iteration: 149    ~ bandwidth: 0.903  ~ gradient: 30.399  ~ stopping criterion: 15.113        ~     Iteration: 150    ~ bandwidth: 0.894  ~ gradient: 28.374  ~ stopping criterion: 15.199        ~     Iteration: 151    ~ bandwidth: 0.886  ~ gradient: 28.302  ~ stopping criterion: 14.187        ~     Iteration: 152    ~ bandwidth: 0.877  ~ gradient: 27.961  ~ stopping criterion: 14.151        ~     Iteration: 153    ~ bandwidth: 0.869  ~ gradient: 26.522  ~ stopping criterion: 13.981        ~     Iteration: 154    ~ bandwidth: 0.861  ~ gradient: 26.798  ~ stopping criterion: 13.261        ~     Iteration: 155    ~ bandwidth: 0.853  ~ gradient: 25.203  ~ stopping criterion: 13.399        ~     Iteration: 156    ~ bandwidth: 0.845  ~ gradient: 24.713  ~ stopping criterion: 12.601        ~     Iteration: 157    ~ bandwidth: 0.838  ~ gradient: 23.031  ~ stopping criterion: 12.356        ~     Iteration: 158    ~ bandwidth: 0.831  ~ gradient: 22.019  ~ stopping criterion: 11.515        ~     Iteration: 159    ~ bandwidth: 0.824  ~ gradient: 21.954  ~ stopping criterion: 11.010        ~     Iteration: 160    ~ bandwidth: 0.818  ~ gradient: 21.373  ~ stopping criterion: 10.977        ~     Iteration: 161    ~ bandwidth: 0.811  ~ gradient: 22.419  ~ stopping criterion: 10.686        ~     Iteration: 162    ~ bandwidth: 0.805  ~ gradient: 18.759  ~ stopping criterion: 11.209        ~     Iteration: 163    ~ bandwidth: 0.799  ~ gradient: 19.891  ~ stopping criterion: 9.380     ~     Iteration: 164    ~ bandwidth: 0.793  ~ gradient: 18.285  ~ stopping criterion: 9.945     ~     Iteration: 165    ~ bandwidth: 0.788  ~ gradient: 17.670  ~ stopping criterion: 9.143     ~     Iteration: 166    ~ bandwidth: 0.782  ~ gradient: 15.641  ~ stopping criterion: 8.835     ~     Iteration: 167    ~ bandwidth: 0.778  ~ gradient: 15.377  ~ stopping criterion: 7.821     ~     Iteration: 168    ~ bandwidth: 0.773  ~ gradient: 13.831  ~ stopping criterion: 7.688     ~     Iteration: 169    ~ bandwidth: 0.769  ~ gradient: 14.703  ~ stopping criterion: 6.915     ~     Iteration: 170    ~ bandwidth: 0.765  ~ gradient: 15.614  ~ stopping criterion: 7.351     ~     Iteration: 171    ~ bandwidth: 0.760  ~ gradient: 12.007  ~ stopping criterion: 7.807     ~     Iteration: 172    ~ bandwidth: 0.756  ~ gradient: 13.628  ~ stopping criterion: 6.004     ~     Iteration: 173    ~ bandwidth: 0.752  ~ gradient: 11.557  ~ stopping criterion: 6.814     ~     Iteration: 174    ~ bandwidth: 0.749  ~ gradient: 9.434   ~ stopping criterion: 5.779     ~     Iteration: 175    ~ bandwidth: 0.746  ~ gradient: 8.647   ~ stopping criterion: 4.717     ~     Iteration: 176    ~ bandwidth: 0.743  ~ gradient: 8.527   ~ stopping criterion: 4.324     ~     Iteration: 177    ~ bandwidth: 0.741  ~ gradient: 8.860   ~ stopping criterion: 4.263     ~     Iteration: 178    ~ bandwidth: 0.738  ~ gradient: 8.644   ~ stopping criterion: 4.430     ~     Iteration: 179    ~ bandwidth: 0.736  ~ gradient: 8.655   ~ stopping criterion: 4.322     ~     Iteration: 180    ~ bandwidth: 0.733  ~ gradient: 6.855   ~ stopping criterion: 4.328     ~     Iteration: 181    ~ bandwidth: 0.731  ~ gradient: 7.078   ~ stopping criterion: 3.427     ~     Iteration: 182    ~ bandwidth: 0.729  ~ gradient: 8.211   ~ stopping criterion: 3.539     ~     Iteration: 183    ~ bandwidth: 0.726  ~ gradient: 7.145   ~ stopping criterion: 4.105     ~     Iteration: 184    ~ bandwidth: 0.724  ~ gradient: 5.995   ~ stopping criterion: 3.573     ~     Iteration: 185    ~ bandwidth: 0.723  ~ gradient: 5.043   ~ stopping criterion: 2.998     ~     Iteration: 186    ~ bandwidth: 0.721  ~ gradient: 4.909   ~ stopping criterion: 2.521     ~     Iteration: 187    ~ bandwidth: 0.720  ~ gradient: 5.327   ~ stopping criterion: 2.455     ~     Iteration: 188    ~ bandwidth: 0.718  ~ gradient: 2.895   ~ stopping criterion: 2.663     ~     Iteration: 189    ~ bandwidth: 0.717  ~ gradient: 3.976   ~ stopping criterion: 1.447     ~     Iteration: 190    ~ bandwidth: 0.716  ~ gradient: 3.597   ~ stopping criterion: 1.988     ~     Iteration: 191    ~ bandwidth: 0.715  ~ gradient: 3.651   ~ stopping criterion: 1.799     ~     Iteration: 192    ~ bandwidth: 0.714  ~ gradient: 4.651   ~ stopping criterion: 1.826     ~     Iteration: 193    ~ bandwidth: 0.712  ~ gradient: 4.288   ~ stopping criterion: 2.326     ~     Iteration: 194    ~ bandwidth: 0.711  ~ gradient: 3.022   ~ stopping criterion: 2.144     ~     Iteration: 195    ~ bandwidth: 0.710  ~ gradient: 3.597   ~ stopping criterion: 1.511     ~     Iteration: 196    ~ bandwidth: 0.709  ~ gradient: 2.005   ~ stopping criterion: 1.799     ~     Iteration: 197    ~ bandwidth: 0.708  ~ gradient: 1.798   ~ stopping criterion: 1.003     ~     Iteration: 198    ~ bandwidth: 0.708  ~ gradient: 2.078   ~ stopping criterion: 0.899     ~     Iteration: 199    ~ bandwidth: 0.707  ~ gradient: 2.216   ~ stopping criterion: 1.039     ~     Iteration: 200    ~ bandwidth: 0.707  ~ gradient: 3.766   ~ stopping criterion: 1.108     ~     Iteration: 201    ~ bandwidth: 0.706  ~ gradient: 1.895   ~ stopping criterion: 1.883     ~     Iteration: 202    ~ bandwidth: 0.705  ~ gradient: 1.989   ~ stopping criterion: 0.948     ~     Iteration: 203    ~ bandwidth: 0.704  ~ gradient: 0.884   ~ stopping criterion: 0.995     ~     Iteration: 204    ~ bandwidth: 0.704  ~ gradient: 0.034   ~ stopping criterion: 0.442     ~     Iteration: 205    ~ bandwidth: 0.704  ~ gradient: 1.075   ~ stopping criterion: 0.017     ~     Iteration: 206    ~ bandwidth: 0.704  ~ gradient: 0.809   ~ stopping criterion: 0.538     ~     Iteration: 207    ~ bandwidth: 0.704  ~ gradient: 0.974   ~ stopping criterion: 0.404     ~     Iteration: 208    ~ bandwidth: 0.703  ~ gradient: 1.272   ~ stopping criterion: 0.487     ~     Iteration: 209    ~ bandwidth: 0.703  ~ gradient: 1.031   ~ stopping criterion: 0.636     ~     Iteration: 210    ~ bandwidth: 0.703  ~ gradient: 0.824   ~ stopping criterion: 0.516     ~     Iteration: 211    ~ bandwidth: 0.702  ~ gradient: 0.574   ~ stopping criterion: 0.412     ~     Iteration: 212    ~ bandwidth: 0.702  ~ gradient: -0.045  ~ stopping criterion: 0.287     ~     Iteration: 213    ~ bandwidth: 0.702  ~ gradient: -1.370  ~ stopping criterion: 0.022     ~     Iteration: 214    ~ bandwidth: 0.702  ~ gradient: -0.081  ~ stopping criterion: 0.685     ~     Iteration: 215    ~ bandwidth: 0.703  ~ gradient: 1.688   ~ stopping criterion: 0.040     ~     Iteration: 216    ~ bandwidth: 0.702  ~ gradient: -0.780  ~ stopping criterion: 0.844     ~     Iteration: 217    ~ bandwidth: 0.702  ~ gradient: -0.101  ~ stopping criterion: 0.390     ~     Iteration: 218    ~ bandwidth: 0.702  ~ gradient: -0.119  ~ stopping criterion: 0.051     ~     Iteration: 219    ~ bandwidth: 0.702  ~ gradient: 0.531   ~ stopping criterion: 0.059     ~     Iteration: 220    ~ bandwidth: 0.702  ~ gradient: 0.786   ~ stopping criterion: 0.265     ~     Iteration: 221    ~ bandwidth: 0.702  ~ gradient: 0.487   ~ stopping criterion: 0.393     ~     Iteration: 222    ~ bandwidth: 0.702  ~ gradient: 1.407   ~ stopping criterion: 0.244     ~     Iteration: 223    ~ bandwidth: 0.702  ~ gradient: 0.562   ~ stopping criterion: 0.704     ~     Iteration: 224    ~ bandwidth: 0.702  ~ gradient: -0.860  ~ stopping criterion: 0.281     ~     Iteration: 225    ~ bandwidth: 0.702  ~ gradient: 1.547   ~ stopping criterion: 0.430     ~     Iteration: 226    ~ bandwidth: 0.702  ~ gradient: 0.296   ~ stopping criterion: 0.774     ~     Iteration: 227    ~ bandwidth: 0.702  ~ gradient: 0.446   ~ stopping criterion: 0.148     ~     Iteration: 228    ~ bandwidth: 0.702  ~ gradient: 1.167   ~ stopping criterion: 0.223     ~     Iteration: 229    ~ bandwidth: 0.702  ~ gradient: 1.101   ~ stopping criterion: 0.584     ~     Iteration: 230    ~ bandwidth: 0.702  ~ gradient: -0.906  ~ stopping criterion: 0.550     ~     Iteration: 231    ~ bandwidth: 0.702  ~ gradient: 0.829   ~ stopping criterion: 0.453     ~     Iteration: 232    ~ bandwidth: 0.702  ~ gradient: 0.730   ~ stopping criterion: 0.415     ~     Iteration: 233    ~ bandwidth: 0.702  ~ gradient: -0.245  ~ stopping criterion: 0.365     ~     Iteration: 234    ~ bandwidth: 0.702  ~ gradient: 1.577   ~ stopping criterion: 0.122     ~     Iteration: 235    ~ bandwidth: 0.701  ~ gradient: 0.024   ~ stopping criterion: 0.788     ~     Iteration: 236    ~ bandwidth: 0.701  ~ gradient: 1.736   ~ stopping criterion: 0.012     ~     Iteration: 237    ~ bandwidth: 0.701  ~ gradient: 0.633   ~ stopping criterion: 0.868     ~     Iteration: 238    ~ bandwidth: 0.701  ~ gradient: 1.599   ~ stopping criterion: 0.317     ~     Iteration: 239    ~ bandwidth: 0.701  ~ gradient: 1.140   ~ stopping criterion: 0.800     ~     Iteration: 240    ~ bandwidth: 0.701  ~ gradient: 1.434   ~ stopping criterion: 0.570     ~     Iteration: 241    ~ bandwidth: 0.701  ~ gradient: -0.243  ~ stopping criterion: 0.717     ~     Iteration: 242    ~ bandwidth: 0.701  ~ gradient: 1.831   ~ stopping criterion: 0.122     ~     Iteration: 243    ~ bandwidth: 0.701  ~ gradient: -0.497  ~ stopping criterion: 0.916     ~     Iteration: 244    ~ bandwidth: 0.701  ~ gradient: 1.226   ~ stopping criterion: 0.249     ~     Iteration: 245    ~ bandwidth: 0.701  ~ gradient: -0.292  ~ stopping criterion: 0.613     ~     Iteration: 246    ~ bandwidth: 0.701  ~ gradient: -0.067  ~ stopping criterion: 0.146     ~     Iteration: 247    ~ bandwidth: 0.701  ~ gradient: 0.449   ~ stopping criterion: 0.033     ~     Iteration: 248    ~ bandwidth: 0.701  ~ gradient: 0.290   ~ stopping criterion: 0.224     ~     Iteration: 249    ~ bandwidth: 0.701  ~ gradient: -0.736  ~ stopping criterion: 0.145     ~     Iteration: 250    ~ bandwidth: 0.701  ~ gradient: 0.039   ~ stopping criterion: 0.368     ~     Iteration: 251    ~ bandwidth: 0.701  ~ gradient: 1.392   ~ stopping criterion: 0.019     ~     Iteration: 252    ~ bandwidth: 0.701  ~ gradient: 0.187   ~ stopping criterion: 0.696     ~     Iteration: 253    ~ bandwidth: 0.701  ~ gradient: -0.352  ~ stopping criterion: 0.094     ~     Iteration: 254    ~ bandwidth: 0.701  ~ gradient: -0.693  ~ stopping criterion: 0.176     ~     Iteration: 255    ~ bandwidth: 0.701  ~ gradient: -0.410  ~ stopping criterion: 0.347     ~     Iteration: 256    ~ bandwidth: 0.701  ~ gradient: -0.858  ~ stopping criterion: 0.205     ~     Iteration: 257    ~ bandwidth: 0.701  ~ gradient: -0.014  ~ stopping criterion: 0.429     ~     Iteration: 258    ~ bandwidth: 0.701  ~ gradient: -0.138  ~ stopping criterion: 0.007     ~     Iteration: 259    ~ bandwidth: 0.701  ~ gradient: -0.207  ~ stopping criterion: 0.069     ~     Iteration: 260    ~ bandwidth: 0.701  ~ gradient: -1.444  ~ stopping criterion: 0.103     ~     Iteration: 261    ~ bandwidth: 0.701  ~ gradient: 0.497   ~ stopping criterion: 0.722     ~     Iteration: 262    ~ bandwidth: 0.701  ~ gradient: -0.539  ~ stopping criterion: 0.248     ~     Iteration: 263    ~ bandwidth: 0.701  ~ gradient: 1.030   ~ stopping criterion: 0.269     ~     Iteration: 264    ~ bandwidth: 0.701  ~ gradient: 0.294   ~ stopping criterion: 0.515     ~     Iteration: 265    ~ bandwidth: 0.701  ~ gradient: 0.303   ~ stopping criterion: 0.147     ~     Iteration: 266    ~ bandwidth: 0.701  ~ gradient: -0.277  ~ stopping criterion: 0.152     ~     Iteration: 267    ~ bandwidth: 0.701  ~ gradient: 0.243   ~ stopping criterion: 0.138     ~     Iteration: 268    ~ bandwidth: 0.701  ~ gradient: 0.500   ~ stopping criterion: 0.121     ~     Iteration: 269    ~ bandwidth: 0.701  ~ gradient: -0.835  ~ stopping criterion: 0.250     ~     Iteration: 270    ~ bandwidth: 0.701  ~ gradient: 0.442   ~ stopping criterion: 0.417     ~     Iteration: 271    ~ bandwidth: 0.701  ~ gradient: 1.868   ~ stopping criterion: 0.221     ~     Iteration: 272    ~ bandwidth: 0.701  ~ gradient: -0.799  ~ stopping criterion: 0.934     ~     Iteration: 273    ~ bandwidth: 0.701  ~ gradient: -1.488  ~ stopping criterion: 0.400     ~     Iteration: 274    ~ bandwidth: 0.701  ~ gradient: 0.798   ~ stopping criterion: 0.744     ~     Iteration: 275    ~ bandwidth: 0.701  ~ gradient: 0.647   ~ stopping criterion: 0.399     ~     Iteration: 276    ~ bandwidth: 0.701  ~ gradient: 0.525   ~ stopping criterion: 0.324     ~     Iteration: 277    ~ bandwidth: 0.701  ~ gradient: 0.455   ~ stopping criterion: 0.263     ~     Iteration: 278    ~ bandwidth: 0.701  ~ gradient: -0.345  ~ stopping criterion: 0.228     ~     Iteration: 279    ~ bandwidth: 0.701  ~ gradient: -0.541  ~ stopping criterion: 0.173     ~     Iteration: 280    ~ bandwidth: 0.701  ~ gradient: -1.220  ~ stopping criterion: 0.270     ~     Iteration: 281    ~ bandwidth: 0.701  ~ gradient: 0.927   ~ stopping criterion: 0.610     ~     Iteration: 282    ~ bandwidth: 0.701  ~ gradient: -0.659  ~ stopping criterion: 0.463     ~     Iteration: 283    ~ bandwidth: 0.701  ~ gradient: -0.248  ~ stopping criterion: 0.330     ~     Iteration: 284    ~ bandwidth: 0.701  ~ gradient: -0.298  ~ stopping criterion: 0.124     ~     Iteration: 285    ~ bandwidth: 0.701  ~ gradient: 0.263   ~ stopping criterion: 0.149     ~     Iteration: 286    ~ bandwidth: 0.701  ~ gradient: 0.964   ~ stopping criterion: 0.132     ~     Iteration: 287    ~ bandwidth: 0.701  ~ gradient: 0.492   ~ stopping criterion: 0.482     ~     Iteration: 288    ~ bandwidth: 0.701  ~ gradient: 0.355   ~ stopping criterion: 0.246     ~     Iteration: 289    ~ bandwidth: 0.701  ~ gradient: -0.169  ~ stopping criterion: 0.178     ~     Iteration: 290    ~ bandwidth: 0.701  ~ gradient: -1.281  ~ stopping criterion: 0.085     ~     Iteration: 291    ~ bandwidth: 0.701  ~ gradient: -0.711  ~ stopping criterion: 0.640     ~     Iteration: 292    ~ bandwidth: 0.701  ~ gradient: -0.046  ~ stopping criterion: 0.356     ~     Iteration: 293    ~ bandwidth: 0.701  ~ gradient: 0.493   ~ stopping criterion: 0.023     ~     Iteration: 294    ~ bandwidth: 0.701  ~ gradient: -0.283  ~ stopping criterion: 0.246     ~     Iteration: 295    ~ bandwidth: 0.701  ~ gradient: -1.623  ~ stopping criterion: 0.141     ~     Iteration: 296    ~ bandwidth: 0.701  ~ gradient: 1.777   ~ stopping criterion: 0.811     ~     Iteration: 297    ~ bandwidth: 0.701  ~ gradient: -0.091  ~ stopping criterion: 0.888     ~     Iteration: 298    ~ bandwidth: 0.701  ~ gradient: 0.148   ~ stopping criterion: 0.045     ~     Iteration: 299    ~ bandwidth: 0.701  ~ gradient: 0.214   ~ stopping criterion: 0.074     ~     Iteration: 300    ~ bandwidth: 0.701  ~ gradient: 0.181   ~ stopping criterion: 0.107                                                                                                                                                                                 ~    Stopped at: 300    ~ bandwidth: 0.701  ~ gradient: 0.181   ~ stopping criterion: 0.107

Let’s compare this to the previous case.

# MixCOBRA with default parameter
print("Estimated bandwidth = {}".format(gc2_fit.optimization_outputs['opt_bandwidth']))
Estimated bandwidth = 0.7012029287236681

The learning curve.

gc2_fit.draw_learning_curve()

Copare the MSE and MAPE.

from sklearn.metrics import mean_absolute_percentage_error
y_pred2 = gc2_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred2))
print(mean_squared_error(y_test1, y_pred2))
0.23524039384008133
300.2477389788587

Let’s look at the qq-plot.

gc2_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')

MixCOBRARegressor with non-default parameters

MixCOBRARegressor offers various options to enhance the performance of the method. By default, grid search algorithm is used to estimate the parameters \((\alpha, \beta)\), however, gradient descent algorithm can also be used to optimize the method by setting opt_method = "grad" as shown in case of one bandwidth parameter above. Moveover, you can control the hyperparameters of the basic estimators to enhance the aggregation performance just like in GradientCOBRA method. This can be done as follows:

  • learning_rate : control the learning rate of gradient descent in estimating the bandwidth parameter
  • speed : the speed of the learning rate.
  • kernel : the kernel function used for the aggregation
  • opt_method : the optimiztaion algorithm for estimating the bandwidth. It can be gradient descent (grad) or grid search (grid).
  • kernel : kernel function which is radial by default. It must be an element of ['exponential', 'gaussian', 'radial', 'cauchy', 'reverse_cosh', 'epanechnikov', 'biweight', 'triweight', 'triangular'].
  • alpha_list : list or array to search for \(\alpha\).
  • beta_list : list or array to search for \(\beta\).
  • bandwidth_list : : list or array to search for single bandwidth parameter \(h\) instead of \((\alpha, \beta)\). To do this, we have to set one_parameter = True when fitting the method.
  • max_iter : maximum iteration in gradient descent.
  • loss_function : control the type of loss function used for optimizing the bandwidth.
  • opt_params : control the optimization algorithm such as adjusting n_cv, start, n_tries
  • estiamtor_list : the list of basic estimators used for the aggregation.
  • estimator_params : controlling the hyperparameters of the basic estimators. It must be a dictionary with (key, dict) = (estimator, dict), i.e. the key must be the name of the basic estimator, and the value is a dictionary containing its hyperparamaters.

We create another object gc3 with non-default parameters, then fit it to the same training data as in the previous example.

gc3 = MixCOBRARegressor(learning_rate=0.1,
                    speed="linear",
                    kernel='radial',
                    opt_method='grad',
                    loss_function="weighted_mse",
                    max_iter=300,
                    estimator_list=['random_forest', 
                                    'adaboost', 
                                    'knn', 
                                    'svm', 
                                    'lasso', 
                                    'ridge'],
                    estimator_params={
                        'random_forest' : {
                            'n_estimators' : 300,
                            'min_samples_leaf' : 10},
                        'adaboost' : {
                            'n_estimators' : 300,
                            'max_depth' : 10},
                        'knn' : {
                            'n_neighbors' : 30},
                            'svm' : {'C' : 7}
                    })

gc3_fit = gc3.fit(X_train1, y_train1)

    * Gradient descent of two parameters with radial kernel is implemented...
        ~ Initial t = 0:        ~ (alpha, beta): (1.000, 1.325)     ~ |gradient|_1: 0.132    ~ threshold: 1e-05
        ~     Iteration: 1  ~ (alpha, beta): (1.000, 1.325)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.065924      ~     Iteration: 2  ~ (alpha, beta): (1.000, 1.325)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.087429      ~     Iteration: 3  ~ (alpha, beta): (1.000, 1.325)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.108935      ~     Iteration: 4  ~ (alpha, beta): (1.000, 1.325)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.130440      ~     Iteration: 5  ~ (alpha, beta): (1.025, 0.950)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.065924      ~     Iteration: 6  ~ (alpha, beta): (0.974, 1.530)     ~ |gradient|_1: -0.013  ~ stopping criterion: 0.083202      ~     Iteration: 7  ~ (alpha, beta): (1.020, 1.032)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.074755      ~     Iteration: 8  ~ (alpha, beta): (0.976, 1.178)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.028074      ~     Iteration: 9  ~ (alpha, beta): (0.971, 1.022)     ~ |gradient|_1: 0.015   ~ stopping criterion: 0.025987      ~     Iteration: 10     ~ (alpha, beta): (0.918, 1.209)     ~ |gradient|_1: -0.010  ~ stopping criterion: 0.040850      ~     Iteration: 11     ~ (alpha, beta): (0.959, 1.067)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.033706      ~     Iteration: 12     ~ (alpha, beta): (0.916, 1.111)     ~ |gradient|_1: -0.009  ~ stopping criterion: 0.016338      ~     Iteration: 13     ~ (alpha, beta): (0.943, 1.080)     ~ |gradient|_1: 0.017   ~ stopping criterion: 0.012792      ~     Iteration: 14     ~ (alpha, beta): (0.899, 1.078)     ~ |gradient|_1: 0.009   ~ stopping criterion: 0.008826      ~     Iteration: 15     ~ (alpha, beta): (0.879, 1.112)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.014056      ~     Iteration: 16     ~ (alpha, beta): (0.859, 1.100)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.008195      ~     Iteration: 17     ~ (alpha, beta): (0.827, 1.078)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.014637      ~     Iteration: 18     ~ (alpha, beta): (0.831, 1.110)     ~ |gradient|_1: -0.019  ~ stopping criterion: 0.012317      ~     Iteration: 19     ~ (alpha, beta): (0.874, 1.100)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.013762      ~     Iteration: 20     ~ (alpha, beta): (0.873, 1.099)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.000663      ~     Iteration: 21     ~ (alpha, beta): (0.871, 1.083)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.008359      ~     Iteration: 22     ~ (alpha, beta): (0.887, 1.088)     ~ |gradient|_1: -0.009  ~ stopping criterion: 0.006470      ~     Iteration: 23     ~ (alpha, beta): (0.903, 1.084)     ~ |gradient|_1: -0.017  ~ stopping criterion: 0.006947      ~     Iteration: 24     ~ (alpha, beta): (0.936, 1.095)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.015694      ~     Iteration: 25     ~ (alpha, beta): (0.924, 1.090)     ~ |gradient|_1: -0.012  ~ stopping criterion: 0.006975      ~     Iteration: 26     ~ (alpha, beta): (0.942, 1.092)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.007944      ~     Iteration: 27     ~ (alpha, beta): (0.939, 1.095)     ~ |gradient|_1: -0.017  ~ stopping criterion: 0.004901      ~     Iteration: 28     ~ (alpha, beta): (0.958, 1.078)     ~ |gradient|_1: -0.006  ~ stopping criterion: 0.027751      ~     Iteration: 29     ~ (alpha, beta): (0.963, 1.081)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.008136      ~     Iteration: 30     ~ (alpha, beta): (0.958, 1.085)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.007945      ~     Iteration: 31     ~ (alpha, beta): (0.947, 1.086)     ~ |gradient|_1: -0.010  ~ stopping criterion: 0.009480      ~     Iteration: 32     ~ (alpha, beta): (0.956, 1.087)     ~ |gradient|_1: 0.024   ~ stopping criterion: 0.005746      ~     Iteration: 33     ~ (alpha, beta): (0.939, 1.084)     ~ |gradient|_1: 0.022   ~ stopping criterion: 0.015416      ~     Iteration: 34     ~ (alpha, beta): (0.927, 1.087)     ~ |gradient|_1: 0.021   ~ stopping criterion: 0.016166      ~     Iteration: 35     ~ (alpha, beta): (0.915, 1.090)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.016617      ~     Iteration: 36     ~ (alpha, beta): (0.914, 1.089)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.003197      ~     Iteration: 37     ~ (alpha, beta): (0.910, 1.087)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.009037      ~     Iteration: 38     ~ (alpha, beta): (0.902, 1.081)     ~ |gradient|_1: 0.012   ~ stopping criterion: 0.019952      ~     Iteration: 39     ~ (alpha, beta): (0.894, 1.085)     ~ |gradient|_1: -0.011  ~ stopping criterion: 0.013848      ~     Iteration: 40     ~ (alpha, beta): (0.901, 1.088)     ~ |gradient|_1: 0.020   ~ stopping criterion: 0.016229      ~     Iteration: 41     ~ (alpha, beta): (0.890, 1.088)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.010511      ~     Iteration: 42     ~ (alpha, beta): (0.884, 1.086)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.014590      ~     Iteration: 43     ~ (alpha, beta): (0.882, 1.090)     ~ |gradient|_1: -0.013  ~ stopping criterion: 0.014406      ~     Iteration: 44     ~ (alpha, beta): (0.888, 1.091)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.010878      ~     Iteration: 45     ~ (alpha, beta): (0.887, 1.091)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001943      ~     Iteration: 46     ~ (alpha, beta): (0.887, 1.092)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.003673      ~     Iteration: 47     ~ (alpha, beta): (0.888, 1.093)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.005612      ~     Iteration: 48     ~ (alpha, beta): (0.887, 1.092)     ~ |gradient|_1: 0.016   ~ stopping criterion: 0.004393      ~     Iteration: 49     ~ (alpha, beta): (0.884, 1.091)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.011986      ~     Iteration: 50     ~ (alpha, beta): (0.882, 1.090)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.007660      ~     Iteration: 51     ~ (alpha, beta): (0.882, 1.089)     ~ |gradient|_1: -0.012  ~ stopping criterion: 0.004611      ~     Iteration: 52     ~ (alpha, beta): (0.884, 1.088)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.011304      ~     Iteration: 53     ~ (alpha, beta): (0.883, 1.089)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.006466      ~     Iteration: 54     ~ (alpha, beta): (0.883, 1.089)     ~ |gradient|_1: 0.016   ~ stopping criterion: 0.000902      ~     Iteration: 55     ~ (alpha, beta): (0.881, 1.089)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.008878      ~     Iteration: 56     ~ (alpha, beta): (0.878, 1.089)     ~ |gradient|_1: 0.009   ~ stopping criterion: 0.009695      ~     Iteration: 57     ~ (alpha, beta): (0.877, 1.088)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.009556      ~     Iteration: 58     ~ (alpha, beta): (0.877, 1.089)     ~ |gradient|_1: -0.005  ~ stopping criterion: 0.002561      ~     Iteration: 59     ~ (alpha, beta): (0.877, 1.089)     ~ |gradient|_1: -0.007  ~ stopping criterion: 0.003356      ~     Iteration: 60     ~ (alpha, beta): (0.878, 1.088)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.006347      ~     Iteration: 61     ~ (alpha, beta): (0.879, 1.088)     ~ |gradient|_1: 0.017   ~ stopping criterion: 0.001077      ~     Iteration: 62     ~ (alpha, beta): (0.876, 1.089)     ~ |gradient|_1: -0.006  ~ stopping criterion: 0.015848      ~     Iteration: 63     ~ (alpha, beta): (0.877, 1.089)     ~ |gradient|_1: 0.016   ~ stopping criterion: 0.004051      ~     Iteration: 64     ~ (alpha, beta): (0.875, 1.088)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.013425      ~     Iteration: 65     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.003705      ~     Iteration: 66     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.003234      ~     Iteration: 67     ~ (alpha, beta): (0.873, 1.089)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.007744      ~     Iteration: 68     ~ (alpha, beta): (0.873, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.004452      ~     Iteration: 69     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.002422      ~     Iteration: 70     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.007313      ~     Iteration: 71     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.003825      ~     Iteration: 72     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.012  ~ stopping criterion: 0.006748      ~     Iteration: 73     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.009112      ~     Iteration: 74     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.005  ~ stopping criterion: 0.010699      ~     Iteration: 75     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.012   ~ stopping criterion: 0.006512      ~     Iteration: 76     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.011142      ~     Iteration: 77     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.007410      ~     Iteration: 78     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.006  ~ stopping criterion: 0.013663      ~     Iteration: 79     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.008604      ~     Iteration: 80     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.005902      ~     Iteration: 81     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.003601      ~     Iteration: 82     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: -0.005  ~ stopping criterion: 0.011737      ~     Iteration: 83     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.018   ~ stopping criterion: 0.004572      ~     Iteration: 84     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.004  ~ stopping criterion: 0.014986      ~     Iteration: 85     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.028  ~ stopping criterion: 0.005204      ~     Iteration: 86     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.019777      ~     Iteration: 87     ~ (alpha, beta): (0.875, 1.089)     ~ |gradient|_1: 0.018   ~ stopping criterion: 0.003831      ~     Iteration: 88     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.009195      ~     Iteration: 89     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.010378      ~     Iteration: 90     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.011  ~ stopping criterion: 0.004439      ~     Iteration: 91     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.010082      ~     Iteration: 92     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.021   ~ stopping criterion: 0.003847      ~     Iteration: 93     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.011   ~ stopping criterion: 0.014638      ~     Iteration: 94     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.018533      ~     Iteration: 95     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.011874      ~     Iteration: 96     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.007  ~ stopping criterion: 0.001645      ~     Iteration: 97     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.005722      ~     Iteration: 98     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.005   ~ stopping criterion: 0.017228      ~     Iteration: 99     ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.025   ~ stopping criterion: 0.002783      ~     Iteration: 100    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.017   ~ stopping criterion: 0.017635      ~     Iteration: 101    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.014150      ~     Iteration: 102    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.012   ~ stopping criterion: 0.005757      ~     Iteration: 103    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.007102      ~     Iteration: 104    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.006087      ~     Iteration: 105    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.007154      ~     Iteration: 106    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.021   ~ stopping criterion: 0.011526      ~     Iteration: 107    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.015174      ~     Iteration: 108    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.005  ~ stopping criterion: 0.005473      ~     Iteration: 109    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.005   ~ stopping criterion: 0.004929      ~     Iteration: 110    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.006  ~ stopping criterion: 0.005358      ~     Iteration: 111    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.014516      ~     Iteration: 112    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.006655      ~     Iteration: 113    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.021  ~ stopping criterion: 0.011645      ~     Iteration: 114    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.011813      ~     Iteration: 115    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.007  ~ stopping criterion: 0.004324      ~     Iteration: 116    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.018   ~ stopping criterion: 0.012155      ~     Iteration: 117    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.012  ~ stopping criterion: 0.013049      ~     Iteration: 118    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.017   ~ stopping criterion: 0.008994      ~     Iteration: 119    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.017232      ~     Iteration: 120    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.010607      ~     Iteration: 121    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.018  ~ stopping criterion: 0.013039      ~     Iteration: 122    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.014739      ~     Iteration: 123    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001723      ~     Iteration: 124    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.002370      ~     Iteration: 125    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.005   ~ stopping criterion: 0.003934      ~     Iteration: 126    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.006   ~ stopping criterion: 0.002732      ~     Iteration: 127    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002904      ~     Iteration: 128    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.005  ~ stopping criterion: 0.010421      ~     Iteration: 129    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.007  ~ stopping criterion: 0.003750      ~     Iteration: 130    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.003929      ~     Iteration: 131    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.004891      ~     Iteration: 132    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.008  ~ stopping criterion: 0.007942      ~     Iteration: 133    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.008102      ~     Iteration: 134    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.012   ~ stopping criterion: 0.001128      ~     Iteration: 135    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.007495      ~     Iteration: 136    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.003812      ~     Iteration: 137    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.014   ~ stopping criterion: 0.005350      ~     Iteration: 138    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.006  ~ stopping criterion: 0.007249      ~     Iteration: 139    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.005   ~ stopping criterion: 0.005889      ~     Iteration: 140    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.010  ~ stopping criterion: 0.014089      ~     Iteration: 141    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.011  ~ stopping criterion: 0.005270      ~     Iteration: 142    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.006283      ~     Iteration: 143    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.001944      ~     Iteration: 144    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.002   ~ stopping criterion: 0.004854      ~     Iteration: 145    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.019   ~ stopping criterion: 0.005236      ~     Iteration: 146    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.008   ~ stopping criterion: 0.009927      ~     Iteration: 147    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.004823      ~     Iteration: 148    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.018   ~ stopping criterion: 0.005225      ~     Iteration: 149    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.018  ~ stopping criterion: 0.014587      ~     Iteration: 150    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.011   ~ stopping criterion: 0.010834      ~     Iteration: 151    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.016  ~ stopping criterion: 0.007541      ~     Iteration: 152    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.009899      ~     Iteration: 153    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.011   ~ stopping criterion: 0.009305      ~     Iteration: 154    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.004   ~ stopping criterion: 0.007607      ~     Iteration: 155    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.002029      ~     Iteration: 156    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.015   ~ stopping criterion: 0.000442      ~     Iteration: 157    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.001   ~ stopping criterion: 0.014383      ~     Iteration: 158    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.004  ~ stopping criterion: 0.010393      ~     Iteration: 159    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.008472      ~     Iteration: 160    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.004447      ~     Iteration: 161    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.001473      ~     Iteration: 162    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.004  ~ stopping criterion: 0.004006      ~     Iteration: 163    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.003   ~ stopping criterion: 0.003272      ~     Iteration: 164    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.003315      ~     Iteration: 165    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.004796      ~     Iteration: 166    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.007   ~ stopping criterion: 0.002243      ~     Iteration: 167    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.005328      ~     Iteration: 168    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: 0.013   ~ stopping criterion: 0.002214      ~     Iteration: 169    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.007727      ~     Iteration: 170    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.000812      ~     Iteration: 171    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.000804      ~     Iteration: 172    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001110      ~     Iteration: 173    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001133      ~     Iteration: 174    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.002281      ~     Iteration: 175    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002250      ~     Iteration: 176    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.002285      ~     Iteration: 177    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.000720      ~     Iteration: 178    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.000696      ~     Iteration: 179    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.001  ~ stopping criterion: 0.000863      ~     Iteration: 180    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.000817      ~     Iteration: 181    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001208      ~     Iteration: 182    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001103      ~     Iteration: 183    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001104      ~     Iteration: 184    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001107      ~     Iteration: 185    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001112      ~     Iteration: 186    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001148      ~     Iteration: 187    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001103      ~     Iteration: 188    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002278      ~     Iteration: 189    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001225      ~     Iteration: 190    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001226      ~     Iteration: 191    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001112      ~     Iteration: 192    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001113      ~     Iteration: 193    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001129      ~     Iteration: 194    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001114      ~     Iteration: 195    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001112      ~     Iteration: 196    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001112      ~     Iteration: 197    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001156      ~     Iteration: 198    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001217      ~     Iteration: 199    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001387      ~     Iteration: 200    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002079      ~     Iteration: 201    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001350      ~     Iteration: 202    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001355      ~     Iteration: 203    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001387      ~     Iteration: 204    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002079      ~     Iteration: 205    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001355      ~     Iteration: 206    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.001355      ~     Iteration: 207    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001387      ~     Iteration: 208    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002539      ~     Iteration: 209    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001816      ~     Iteration: 210    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001815      ~     Iteration: 211    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002539      ~     Iteration: 212    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001815      ~     Iteration: 213    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002429      ~     Iteration: 214    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001815      ~     Iteration: 215    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002429      ~     Iteration: 216    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002430      ~     Iteration: 217    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.001789      ~     Iteration: 218    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002410      ~     Iteration: 219    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002410      ~     Iteration: 220    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002404      ~     Iteration: 221    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002404      ~     Iteration: 222    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002404      ~     Iteration: 223    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002402      ~     Iteration: 224    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002402      ~     Iteration: 225    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002421      ~     Iteration: 226    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002421      ~     Iteration: 227    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002388      ~     Iteration: 228    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002388      ~     Iteration: 229    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002388      ~     Iteration: 230    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002389      ~     Iteration: 231    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002389      ~     Iteration: 232    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002352      ~     Iteration: 233    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002352      ~     Iteration: 234    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002351      ~     Iteration: 235    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002245      ~     Iteration: 236    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002245      ~     Iteration: 237    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002255      ~     Iteration: 238    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002255      ~     Iteration: 239    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002256      ~     Iteration: 240    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002266      ~     Iteration: 241    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002266      ~     Iteration: 242    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002266      ~     Iteration: 243    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002262      ~     Iteration: 244    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002262      ~     Iteration: 245    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002262      ~     Iteration: 246    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002262      ~     Iteration: 247    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002262      ~     Iteration: 248    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002259      ~     Iteration: 249    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002259      ~     Iteration: 250    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002259      ~     Iteration: 251    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002259      ~     Iteration: 252    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002259      ~     Iteration: 253    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002261      ~     Iteration: 254    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002261      ~     Iteration: 255    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002261      ~     Iteration: 256    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002261      ~     Iteration: 257    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002361      ~     Iteration: 258    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002418      ~     Iteration: 259    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002418      ~     Iteration: 260    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 261    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 262    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 263    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 264    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 265    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 266    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002415      ~     Iteration: 267    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002485      ~     Iteration: 268    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002580      ~     Iteration: 269    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002580      ~     Iteration: 270    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002580      ~     Iteration: 271    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002583      ~     Iteration: 272    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002583      ~     Iteration: 273    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002589      ~     Iteration: 274    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 275    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 276    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 277    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 278    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 279    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002561      ~     Iteration: 280    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002563      ~     Iteration: 281    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002563      ~     Iteration: 282    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002563      ~     Iteration: 283    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002563      ~     Iteration: 284    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002537      ~     Iteration: 285    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002540      ~     Iteration: 286    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002540      ~     Iteration: 287    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002541      ~     Iteration: 288    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002439      ~     Iteration: 289    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002272      ~     Iteration: 290    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002272      ~     Iteration: 291    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002260      ~     Iteration: 292    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.003  ~ stopping criterion: 0.002261      ~     Iteration: 293    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002289      ~     Iteration: 294    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002207      ~     Iteration: 295    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002186      ~     Iteration: 296    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002186      ~     Iteration: 297    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002184      ~     Iteration: 298    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002184      ~     Iteration: 299    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002184      ~     Iteration: 300    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002184                                                                                                                                                                                  ~    Stopped at: 300    ~ (alpha, beta): (0.874, 1.089)     ~ |gradient|_1: -0.002  ~ stopping criterion: 0.002184

Now, let’s compare it to the previous example.

# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc3_fit.optimization_outputs['opt_alpha'], gc3_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (0.8741706474293448, 1.0888201148722707)

Look at the learning surface using draw_learning_curve() method.

gc3_fit.draw_learning_curve()

We evaluate the performance of the method on the testing data using MSE and MAPE.

from sklearn.metrics import mean_absolute_percentage_error
y_pred3 = gc3_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred3))
print(mean_squared_error(y_test1, y_pred3))
0.1308662858088806
126.52951363322245

Let’s look at qq-plot using draw_learning_curve() method.

gc3_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')

Real dataset

We look at the California housing dataset from sklearn.datasets module. To illustrate the idea, we only work with the first \(1000\) observations.

from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
X_real, y_real = data['data'], data['target']

X_train_real, X_test_real, y_train_real, y_test_real = train_test_split(X_real[:1000,:], y_real[:1000], test_size=0.2)
print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(X_train_real.shape, X_test_real.shape, y_train_real.shape, y_test_real.shape))
shape: x_train = (800, 8) , x_train = (200, 8) , y_train = (800,) , y_test = (200,)

We gave some random parameters to the method as follows.

gc_real = MixCOBRARegressor(opt_method="grad",
                            estimator_list=['random_forest', 
                                            'knn', 
                                            'ridge', 
                                            'lasso'],
                            estimator_params={
                                'random_forest' : {'n_estimators': 300},
                                'knn' : {'n_neighbors' : 30}
                        })
gc_real_fit = gc_real.fit(X_train_real, y_train_real)

    * Gradient descent of two parameters with radial kernel is implemented...
        ~ Initial t = 0:        ~ (alpha, beta): (5.000, 2.550)     ~ |gradient|_1: 0.001    ~ threshold: 1e-05
        ~     Iteration: 1  ~ (alpha, beta): (5.000, 2.550)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.001329      ~     Iteration: 2  ~ (alpha, beta): (5.000, 2.550)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.001329      ~     Iteration: 3  ~ (alpha, beta): (5.000, 2.550)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.001329      ~     Iteration: 4  ~ (alpha, beta): (5.000, 2.550)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.001329      ~     Iteration: 5  ~ (alpha, beta): (5.003, 2.557)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000666      ~     Iteration: 6  ~ (alpha, beta): (5.003, 2.567)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000701      ~     Iteration: 7  ~ (alpha, beta): (5.004, 2.576)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000595      ~     Iteration: 8  ~ (alpha, beta): (5.003, 2.584)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000592      ~     Iteration: 9  ~ (alpha, beta): (5.003, 2.593)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000574      ~     Iteration: 10     ~ (alpha, beta): (5.005, 2.601)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000675      ~     Iteration: 11     ~ (alpha, beta): (5.005, 2.607)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000396      ~     Iteration: 12     ~ (alpha, beta): (5.005, 2.613)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000448      ~     Iteration: 13     ~ (alpha, beta): (5.004, 2.619)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000603      ~     Iteration: 14     ~ (alpha, beta): (5.004, 2.627)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000555      ~     Iteration: 15     ~ (alpha, beta): (5.003, 2.635)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000563      ~     Iteration: 16     ~ (alpha, beta): (5.004, 2.642)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000624      ~     Iteration: 17     ~ (alpha, beta): (5.004, 2.650)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000515      ~     Iteration: 18     ~ (alpha, beta): (5.004, 2.657)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000507      ~     Iteration: 19     ~ (alpha, beta): (5.004, 2.661)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000305      ~     Iteration: 20     ~ (alpha, beta): (5.003, 2.669)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000667      ~     Iteration: 21     ~ (alpha, beta): (5.003, 2.673)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000338      ~     Iteration: 22     ~ (alpha, beta): (5.003, 2.681)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000601      ~     Iteration: 23     ~ (alpha, beta): (5.003, 2.688)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000531      ~     Iteration: 24     ~ (alpha, beta): (5.003, 2.693)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000363      ~     Iteration: 25     ~ (alpha, beta): (5.003, 2.698)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000394      ~     Iteration: 26     ~ (alpha, beta): (5.003, 2.705)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000504      ~     Iteration: 27     ~ (alpha, beta): (5.003, 2.711)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000428      ~     Iteration: 28     ~ (alpha, beta): (5.003, 2.718)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000557      ~     Iteration: 29     ~ (alpha, beta): (5.003, 2.723)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000341      ~     Iteration: 30     ~ (alpha, beta): (5.004, 2.728)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000414      ~     Iteration: 31     ~ (alpha, beta): (5.004, 2.735)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000540      ~     Iteration: 32     ~ (alpha, beta): (5.004, 2.741)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000407      ~     Iteration: 33     ~ (alpha, beta): (5.004, 2.748)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000548      ~     Iteration: 34     ~ (alpha, beta): (5.004, 2.750)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000329      ~     Iteration: 35     ~ (alpha, beta): (5.004, 2.755)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000379      ~     Iteration: 36     ~ (alpha, beta): (5.004, 2.760)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000378      ~     Iteration: 37     ~ (alpha, beta): (5.004, 2.765)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000387      ~     Iteration: 38     ~ (alpha, beta): (5.004, 2.768)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000232      ~     Iteration: 39     ~ (alpha, beta): (5.004, 2.772)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000311      ~     Iteration: 40     ~ (alpha, beta): (5.004, 2.777)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000292      ~     Iteration: 41     ~ (alpha, beta): (5.004, 2.782)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000395      ~     Iteration: 42     ~ (alpha, beta): (5.004, 2.785)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000284      ~     Iteration: 43     ~ (alpha, beta): (5.004, 2.790)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000371      ~     Iteration: 44     ~ (alpha, beta): (5.004, 2.797)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000534      ~     Iteration: 45     ~ (alpha, beta): (5.004, 2.800)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000221      ~     Iteration: 46     ~ (alpha, beta): (5.004, 2.805)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000340      ~     Iteration: 47     ~ (alpha, beta): (5.004, 2.809)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000397      ~     Iteration: 48     ~ (alpha, beta): (5.004, 2.813)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000344      ~     Iteration: 49     ~ (alpha, beta): (5.004, 2.817)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000309      ~     Iteration: 50     ~ (alpha, beta): (5.004, 2.820)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000372      ~     Iteration: 51     ~ (alpha, beta): (5.004, 2.823)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000295      ~     Iteration: 52     ~ (alpha, beta): (5.004, 2.825)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000205      ~     Iteration: 53     ~ (alpha, beta): (5.004, 2.829)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000259      ~     Iteration: 54     ~ (alpha, beta): (5.004, 2.831)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000153      ~     Iteration: 55     ~ (alpha, beta): (5.004, 2.835)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000326      ~     Iteration: 56     ~ (alpha, beta): (5.004, 2.839)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000346      ~     Iteration: 57     ~ (alpha, beta): (5.004, 2.844)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000314      ~     Iteration: 58     ~ (alpha, beta): (5.004, 2.847)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000305      ~     Iteration: 59     ~ (alpha, beta): (5.004, 2.851)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000309      ~     Iteration: 60     ~ (alpha, beta): (5.004, 2.852)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000193      ~     Iteration: 61     ~ (alpha, beta): (5.004, 2.855)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000156      ~     Iteration: 62     ~ (alpha, beta): (5.004, 2.859)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000340      ~     Iteration: 63     ~ (alpha, beta): (5.004, 2.861)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000178      ~     Iteration: 64     ~ (alpha, beta): (5.004, 2.865)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000320      ~     Iteration: 65     ~ (alpha, beta): (5.004, 2.867)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000187      ~     Iteration: 66     ~ (alpha, beta): (5.004, 2.869)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000232      ~     Iteration: 67     ~ (alpha, beta): (5.004, 2.870)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000261      ~     Iteration: 68     ~ (alpha, beta): (5.004, 2.873)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000236      ~     Iteration: 69     ~ (alpha, beta): (5.004, 2.873)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000185      ~     Iteration: 70     ~ (alpha, beta): (5.004, 2.876)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000216      ~     Iteration: 71     ~ (alpha, beta): (5.004, 2.878)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000154      ~     Iteration: 72     ~ (alpha, beta): (5.004, 2.880)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000249      ~     Iteration: 73     ~ (alpha, beta): (5.004, 2.881)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000083      ~     Iteration: 74     ~ (alpha, beta): (5.004, 2.884)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000279      ~     Iteration: 75     ~ (alpha, beta): (5.004, 2.887)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000237      ~     Iteration: 76     ~ (alpha, beta): (5.004, 2.888)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000110      ~     Iteration: 77     ~ (alpha, beta): (5.004, 2.889)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000160      ~     Iteration: 78     ~ (alpha, beta): (5.004, 2.891)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000270      ~     Iteration: 79     ~ (alpha, beta): (5.004, 2.894)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000287      ~     Iteration: 80     ~ (alpha, beta): (5.004, 2.896)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000252      ~     Iteration: 81     ~ (alpha, beta): (5.004, 2.900)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000337      ~     Iteration: 82     ~ (alpha, beta): (5.004, 2.904)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000268      ~     Iteration: 83     ~ (alpha, beta): (5.004, 2.906)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000242      ~     Iteration: 84     ~ (alpha, beta): (5.004, 2.909)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000246      ~     Iteration: 85     ~ (alpha, beta): (5.004, 2.910)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000167      ~     Iteration: 86     ~ (alpha, beta): (5.004, 2.911)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000086      ~     Iteration: 87     ~ (alpha, beta): (5.004, 2.913)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000204      ~     Iteration: 88     ~ (alpha, beta): (5.004, 2.916)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000187      ~     Iteration: 89     ~ (alpha, beta): (5.004, 2.916)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000105      ~     Iteration: 90     ~ (alpha, beta): (5.004, 2.919)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000225      ~     Iteration: 91     ~ (alpha, beta): (5.004, 2.921)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000171      ~     Iteration: 92     ~ (alpha, beta): (5.004, 2.923)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000198      ~     Iteration: 93     ~ (alpha, beta): (5.004, 2.927)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000374      ~     Iteration: 94     ~ (alpha, beta): (5.004, 2.928)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000039      ~     Iteration: 95     ~ (alpha, beta): (5.004, 2.928)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000066      ~     Iteration: 96     ~ (alpha, beta): (5.004, 2.930)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000157      ~     Iteration: 97     ~ (alpha, beta): (5.004, 2.930)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000097      ~     Iteration: 98     ~ (alpha, beta): (5.004, 2.931)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000134      ~     Iteration: 99     ~ (alpha, beta): (5.004, 2.932)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000201      ~     Iteration: 100    ~ (alpha, beta): (5.004, 2.933)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000107      ~     Iteration: 101    ~ (alpha, beta): (5.004, 2.934)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000110      ~     Iteration: 102    ~ (alpha, beta): (5.004, 2.935)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000116      ~     Iteration: 103    ~ (alpha, beta): (5.004, 2.937)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000230      ~     Iteration: 104    ~ (alpha, beta): (5.004, 2.937)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000082      ~     Iteration: 105    ~ (alpha, beta): (5.004, 2.938)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000050      ~     Iteration: 106    ~ (alpha, beta): (5.004, 2.940)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000256      ~     Iteration: 107    ~ (alpha, beta): (5.004, 2.940)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000113      ~     Iteration: 108    ~ (alpha, beta): (5.004, 2.940)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000153      ~     Iteration: 109    ~ (alpha, beta): (5.004, 2.941)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000301      ~     Iteration: 110    ~ (alpha, beta): (5.004, 2.941)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000088      ~     Iteration: 111    ~ (alpha, beta): (5.004, 2.942)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000119      ~     Iteration: 112    ~ (alpha, beta): (5.004, 2.943)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000280      ~     Iteration: 113    ~ (alpha, beta): (5.004, 2.944)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000136      ~     Iteration: 114    ~ (alpha, beta): (5.004, 2.945)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000185      ~     Iteration: 115    ~ (alpha, beta): (5.004, 2.945)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000156      ~     Iteration: 116    ~ (alpha, beta): (5.004, 2.946)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000236      ~     Iteration: 117    ~ (alpha, beta): (5.004, 2.947)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000194      ~     Iteration: 118    ~ (alpha, beta): (5.004, 2.948)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000129      ~     Iteration: 119    ~ (alpha, beta): (5.004, 2.948)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000157      ~     Iteration: 120    ~ (alpha, beta): (5.004, 2.949)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000149      ~     Iteration: 121    ~ (alpha, beta): (5.004, 2.949)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000047      ~     Iteration: 122    ~ (alpha, beta): (5.004, 2.949)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000157      ~     Iteration: 123    ~ (alpha, beta): (5.004, 2.950)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000144      ~     Iteration: 124    ~ (alpha, beta): (5.004, 2.952)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000330      ~     Iteration: 125    ~ (alpha, beta): (5.004, 2.953)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000297      ~     Iteration: 126    ~ (alpha, beta): (5.004, 2.954)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000123      ~     Iteration: 127    ~ (alpha, beta): (5.004, 2.954)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000094      ~     Iteration: 128    ~ (alpha, beta): (5.004, 2.954)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000113      ~     Iteration: 129    ~ (alpha, beta): (5.004, 2.954)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000060      ~     Iteration: 130    ~ (alpha, beta): (5.004, 2.955)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000158      ~     Iteration: 131    ~ (alpha, beta): (5.004, 2.954)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000161      ~     Iteration: 132    ~ (alpha, beta): (5.004, 2.955)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000177      ~     Iteration: 133    ~ (alpha, beta): (5.004, 2.955)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000093      ~     Iteration: 134    ~ (alpha, beta): (5.004, 2.956)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000326      ~     Iteration: 135    ~ (alpha, beta): (5.004, 2.956)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000136      ~     Iteration: 136    ~ (alpha, beta): (5.004, 2.957)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000104      ~     Iteration: 137    ~ (alpha, beta): (5.004, 2.957)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000291      ~     Iteration: 138    ~ (alpha, beta): (5.004, 2.957)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000126      ~     Iteration: 139    ~ (alpha, beta): (5.004, 2.958)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000085      ~     Iteration: 140    ~ (alpha, beta): (5.004, 2.958)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000164      ~     Iteration: 141    ~ (alpha, beta): (5.004, 2.958)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000059      ~     Iteration: 142    ~ (alpha, beta): (5.004, 2.958)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000096      ~     Iteration: 143    ~ (alpha, beta): (5.004, 2.958)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000068      ~     Iteration: 144    ~ (alpha, beta): (5.004, 2.959)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000195      ~     Iteration: 145    ~ (alpha, beta): (5.004, 2.959)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000068      ~     Iteration: 146    ~ (alpha, beta): (5.004, 2.959)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000110      ~     Iteration: 147    ~ (alpha, beta): (5.004, 2.959)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000051      ~     Iteration: 148    ~ (alpha, beta): (5.004, 2.959)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000200      ~     Iteration: 149    ~ (alpha, beta): (5.004, 2.960)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000287      ~     Iteration: 150    ~ (alpha, beta): (5.004, 2.961)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000267      ~     Iteration: 151    ~ (alpha, beta): (5.004, 2.961)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000216      ~     Iteration: 152    ~ (alpha, beta): (5.004, 2.961)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000074      ~     Iteration: 153    ~ (alpha, beta): (5.004, 2.961)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000133      ~     Iteration: 154    ~ (alpha, beta): (5.004, 2.961)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000238      ~     Iteration: 155    ~ (alpha, beta): (5.004, 2.962)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000433      ~     Iteration: 156    ~ (alpha, beta): (5.004, 2.962)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000193      ~     Iteration: 157    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000185      ~     Iteration: 158    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000146      ~     Iteration: 159    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000049      ~     Iteration: 160    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000220      ~     Iteration: 161    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000091      ~     Iteration: 162    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000040      ~     Iteration: 163    ~ (alpha, beta): (5.004, 2.963)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000287      ~     Iteration: 164    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000152      ~     Iteration: 165    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000117      ~     Iteration: 166    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000032      ~     Iteration: 167    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000334      ~     Iteration: 168    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000156      ~     Iteration: 169    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000126      ~     Iteration: 170    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000138      ~     Iteration: 171    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000082      ~     Iteration: 172    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000258      ~     Iteration: 173    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000162      ~     Iteration: 174    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000231      ~     Iteration: 175    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000054      ~     Iteration: 176    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000066      ~     Iteration: 177    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000096      ~     Iteration: 178    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000075      ~     Iteration: 179    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000163      ~     Iteration: 180    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000087      ~     Iteration: 181    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000149      ~     Iteration: 182    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000047      ~     Iteration: 183    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000126      ~     Iteration: 184    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000220      ~     Iteration: 185    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000127      ~     Iteration: 186    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000236      ~     Iteration: 187    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000284      ~     Iteration: 188    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000276      ~     Iteration: 189    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000157      ~     Iteration: 190    ~ (alpha, beta): (5.004, 2.964)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000047      ~     Iteration: 191    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000114      ~     Iteration: 192    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000193      ~     Iteration: 193    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000068      ~     Iteration: 194    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000037      ~     Iteration: 195    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000314      ~     Iteration: 196    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000062      ~     Iteration: 197    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000053      ~     Iteration: 198    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000425      ~     Iteration: 199    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000262      ~     Iteration: 200    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000126      ~     Iteration: 201    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000094      ~     Iteration: 202    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000206      ~     Iteration: 203    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000130      ~     Iteration: 204    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000111      ~     Iteration: 205    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000333      ~     Iteration: 206    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000206      ~     Iteration: 207    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000251      ~     Iteration: 208    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000129      ~     Iteration: 209    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000078      ~     Iteration: 210    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000222      ~     Iteration: 211    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000122      ~     Iteration: 212    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000163      ~     Iteration: 213    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000040      ~     Iteration: 214    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000089      ~     Iteration: 215    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000076      ~     Iteration: 216    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000115      ~     Iteration: 217    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000183      ~     Iteration: 218    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000144      ~     Iteration: 219    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 220    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000078      ~     Iteration: 221    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000055      ~     Iteration: 222    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000119      ~     Iteration: 223    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000119      ~     Iteration: 224    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000036      ~     Iteration: 225    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000031      ~     Iteration: 226    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000164      ~     Iteration: 227    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000139      ~     Iteration: 228    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000160      ~     Iteration: 229    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000182      ~     Iteration: 230    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000093      ~     Iteration: 231    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000255      ~     Iteration: 232    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000254      ~     Iteration: 233    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000117      ~     Iteration: 234    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000076      ~     Iteration: 235    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000165      ~     Iteration: 236    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000130      ~     Iteration: 237    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000214      ~     Iteration: 238    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000032      ~     Iteration: 239    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000083      ~     Iteration: 240    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000056      ~     Iteration: 241    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000186      ~     Iteration: 242    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000022      ~     Iteration: 243    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000089      ~     Iteration: 244    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000093      ~     Iteration: 245    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000078      ~     Iteration: 246    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000036      ~     Iteration: 247    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000034      ~     Iteration: 248    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000026      ~     Iteration: 249    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000026      ~     Iteration: 250    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000031      ~     Iteration: 251    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000030      ~     Iteration: 252    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000026      ~     Iteration: 253    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 254    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000027      ~     Iteration: 255    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 256    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000030      ~     Iteration: 257    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000032      ~     Iteration: 258    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000016      ~     Iteration: 259    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000017      ~     Iteration: 260    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000016      ~     Iteration: 261    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 262    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000032      ~     Iteration: 263    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 264    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000032      ~     Iteration: 265    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 266    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000023      ~     Iteration: 267    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 268    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000017      ~     Iteration: 269    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 270    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000012      ~     Iteration: 271    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000010      ~     Iteration: 272    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000012      ~     Iteration: 273    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 274    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 275    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 276    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 277    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 278    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 279    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 280    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 281    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 282    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 283    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 284    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 285    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 286    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 287    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 288    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 289    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 290    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 291    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 292    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 293    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 294    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 295    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 296    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 297    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 298    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000011      ~     Iteration: 299    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010      ~     Iteration: 300    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010                                                                                                                                                                                  ~    Stopped at: 300    ~ (alpha, beta): (5.004, 2.965)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000010

Now, let’s look at the obtained bandwidth and the optimization result.

# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc_real_fit.optimization_outputs['opt_alpha'], gc_real_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (5.003947996785071, 2.9646284687052478)

We look at the numerical and graphical performance.

y_pred_real = gc_real_fit.predict(X_test_real)
print(mean_absolute_percentage_error(y_test_real, y_pred_real))
print(mean_squared_error(y_test_real, y_pred_real))
gc_real_fit.draw_learning_curve(y_test=y_test_real, fig_type='qq')
0.1533765484019444
0.16456235739517955

Compare to Adaboost

We campare the fitted method on California data with Adaboost method.

from sklearn.ensemble import AdaBoostRegressor

ada = AdaBoostRegressor(n_estimators=1000)
ada_fit = ada.fit(X_train_real, y_train_real)
ada_pred = ada_fit.predict(X_test_real)
print(mean_absolute_percentage_error(y_test_real, ada_pred))
print(mean_squared_error(y_test_real, ada_pred))
0.2723746701801255
0.25246896498101007

Pretrained basic estimators

You can also train other models then aggregate them using our method. Here, we build pretrained estimators including XGBoost then aggregate it to some sklearn basic estimators in MixCOBRARegressor.

We first split the training data into two parts: \(X_k\) for building basic estimators, and \(X_\ell\) for aggregation. We use the constructed estimators to predict the test data and both the input \(X_{\text{test}}\) and its predictions pred_feature_test are used in the final predictions.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor

# X_k and X_l split
id_k = np.random.permutation(range(len(y_train_real)))
k =int(.5 * len(y_train_real))
X_k, X_l, y_k, y_l = X_train_real[id_k[:k],:], X_train_real[id_k[k:],:], y_train_real[id_k[:k]], y_train_real[id_k[k:]]

# Building basic estiators
rf_real = RandomForestRegressor(n_estimators=300).fit(X_k, y_k)
lm_real = LinearRegression().fit(X_k, y_k)
knn_real = KNeighborsRegressor(n_neighbors=10).fit(X_k, y_k)
tr_real = DecisionTreeRegressor(min_samples_leaf=5).fit(X_k, y_k)

# External XGBoost estiator
import xgboost 
xgb = xgboost.XGBRegressor(n_estimators = 500)
xgb_real = xgb.fit(X_k, y_k)

# All pretrained estimators
basic_estimators = (rf_real, lm_real, knn_real, tr_real, xgb_real)

# Predicted features on X_l for aggregation
pred_feature_l = np.column_stack([est.predict(X_l) for est in basic_estimators])

# Predicted features on Testing data
pred_feature_test = np.column_stack([est.predict(X_test_real) for est in basic_estimators])

To fit the aggregation method on predicted features, we need to provide both input and predicted features to fit method.

gc4_fit = MixCOBRARegressor(opt_method='grad').fit(X=X_l,
                                                   Pred_features=pred_feature_l,
                                                   y=y_l)

    * Gradient descent of two parameters with radial kernel is implemented...
        ~ Initial t = 0:        ~ (alpha, beta): (5.000, 3.775)     ~ |gradient|_1: 0.000    ~ threshold: 1e-05
        ~     Iteration: 1  ~ (alpha, beta): (5.000, 3.775)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000598      ~     Iteration: 2  ~ (alpha, beta): (5.000, 3.775)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000598      ~     Iteration: 3  ~ (alpha, beta): (5.000, 3.775)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000598      ~     Iteration: 4  ~ (alpha, beta): (5.000, 3.775)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000598      ~     Iteration: 5  ~ (alpha, beta): (4.992, 3.777)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000029      ~     Iteration: 6  ~ (alpha, beta): (5.013, 3.820)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000185      ~     Iteration: 7  ~ (alpha, beta): (5.030, 3.840)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000118      ~     Iteration: 8  ~ (alpha, beta): (5.026, 3.868)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000092      ~     Iteration: 9  ~ (alpha, beta): (5.012, 3.894)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000140      ~     Iteration: 10     ~ (alpha, beta): (5.052, 3.936)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000297      ~     Iteration: 11     ~ (alpha, beta): (5.038, 3.999)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000260      ~     Iteration: 12     ~ (alpha, beta): (5.034, 4.032)     ~ |gradient|_1: 0.000   ~ stopping criterion: 0.000117      ~     Iteration: 13     ~ (alpha, beta): (5.034, 4.031)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000008                                                                                                                                                                                  ~    Stopped at: 13     ~ (alpha, beta): (5.034, 4.031)     ~ |gradient|_1: -0.000  ~ stopping criterion: 0.000008

We look at the optimization algorithm performance.

# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc4_fit.optimization_outputs['opt_alpha'], gc4_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (5.033650165115184, 4.030778809061925)

Now, let’s look at the result.

gc4_fit.draw_learning_curve()
pred_add = gc4_fit.predict(X = X_test_real, 
                           Pred_X = pred_feature_test)
print(mean_absolute_percentage_error(y_test_real, pred_add))
print(mean_squared_error(y_test_real, pred_add))
0.14222323551671262
0.15583368476512235