# Metric of error
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_absolute_percentage_error
# Plotting figures
import matplotlib.pyplot as plt
from matplotlib import cm
# Import class MixCOBRA from the mixcobra module of gradientcobra library
from gradientcobra.mixcobra import MixCOBRARegressor
import seaborn as sns
set() sns.
MixCOBRARegressor of gradientcobra v1.0.8
package
Installing and importing packages
gradientcobra can be installed from pypi using pip
:
pip install gradientcobra
Importing packages
Simulated data
We simulate a regression data with \(1000\) observations and \(10\) inputs variables.
# For simulating dataset
from sklearn.datasets import make_regression
= make_regression(n_samples=1000, n_features=10, noise=1) X1, y1
Now, let’s randoly split the simulated data into \(80\%-20\%\) training-testing data.
from sklearn.model_selection import train_test_split
= train_test_split(X1, y1, test_size=0.2)
X_train1, X_test1, y_train1, y_test1 print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(
X_train1.shape,
X_test1.shape,
y_train1.shape, y_test1.shape))
shape: x_train = (800, 10) , x_train = (200, 10) , y_train = (800,) , y_test = (200,)
MixCOBRARegressor
with default parameters
We create MixCOBRARegressor
object called gc1
, with the default parameters, then fit it to the training data.
Note that by default, we use grid search algorithm to estimate the optimal parameter of the method.
= MixCOBRARegressor()
gc1 = gc1.fit(X_train1, y_train1) gc1_fit
* Grid search algorithm of two parameters with radial kernel is in progress...
~ Full process|--------------------------------------------------|100%
~ Processing|==================================================|100%
The estimated optimal bandwidth is given by gc1.optimization_outputs['opt_bandwidth']
.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc1_fit.optimization_outputs['opt_alpha'], gc1_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (1e-05, 0.7575842424242424)
We can look at the learning curve (surface) of the algorithm using draw_learning_curve()
method.
gc1_fit.draw_learning_curve()
We evaluate the performance of the method on the testing data using MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
= gc1_fit.predict(X_test1)
y_pred1 print(mean_absolute_percentage_error(y_test1, y_pred1))
print(mean_squared_error(y_test1, y_pred1))
0.16552630550806227
150.23447041060408
Let’s look at qq-plot
of the predictions and the actual response values using draw_learning_curve()
method.
=y_test1, fig_type='qq') gc1_fit.draw_learning_curve(y_test
Use one bandwidth parameter instead of (\(\alpha, \beta\))
You can also try to implement MixCOBRARegressor
using only one bandwidth parameter. To do so, just set one_parameter = True
when fitting the method. We create another MixCOBRARegressor
object with gradient descent optimization method, then fit it on the same data as in the previous case using only one bandwidth parameter as follow.
# MixCOBRA with one parameter
= MixCOBRARegressor(opt_method="grad")
gc2 = gc2.fit(X_train1, y_train1, one_parameter=True) gc2_fit
* Gradient descent of one parameter with radial kernel is implemented...
~ Initial t = 0: ~ bandwidth: 2.507 ~ gradient: 33.439 ~ threshold: 1e-05
~ Iteration: 1 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 2 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 3 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 4 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 5 ~ bandwidth: 2.498 ~ gradient: 32.570 ~ stopping criterion: 16.719 ~ Iteration: 6 ~ bandwidth: 2.488 ~ gradient: 31.760 ~ stopping criterion: 16.285 ~ Iteration: 7 ~ bandwidth: 2.478 ~ gradient: 33.054 ~ stopping criterion: 15.880 ~ Iteration: 8 ~ bandwidth: 2.468 ~ gradient: 32.551 ~ stopping criterion: 16.527 ~ Iteration: 9 ~ bandwidth: 2.459 ~ gradient: 32.933 ~ stopping criterion: 16.276 ~ Iteration: 10 ~ bandwidth: 2.449 ~ gradient: 31.902 ~ stopping criterion: 16.467 ~ Iteration: 11 ~ bandwidth: 2.439 ~ gradient: 32.765 ~ stopping criterion: 15.951 ~ Iteration: 12 ~ bandwidth: 2.429 ~ gradient: 30.846 ~ stopping criterion: 16.382 ~ Iteration: 13 ~ bandwidth: 2.420 ~ gradient: 33.097 ~ stopping criterion: 15.423 ~ Iteration: 14 ~ bandwidth: 2.410 ~ gradient: 32.646 ~ stopping criterion: 16.549 ~ Iteration: 15 ~ bandwidth: 2.401 ~ gradient: 32.505 ~ stopping criterion: 16.323 ~ Iteration: 16 ~ bandwidth: 2.391 ~ gradient: 33.117 ~ stopping criterion: 16.252 ~ Iteration: 17 ~ bandwidth: 2.381 ~ gradient: 34.084 ~ stopping criterion: 16.559 ~ Iteration: 18 ~ bandwidth: 2.371 ~ gradient: 34.117 ~ stopping criterion: 17.042 ~ Iteration: 19 ~ bandwidth: 2.361 ~ gradient: 34.410 ~ stopping criterion: 17.058 ~ Iteration: 20 ~ bandwidth: 2.350 ~ gradient: 33.783 ~ stopping criterion: 17.205 ~ Iteration: 21 ~ bandwidth: 2.340 ~ gradient: 33.098 ~ stopping criterion: 16.891 ~ Iteration: 22 ~ bandwidth: 2.330 ~ gradient: 33.975 ~ stopping criterion: 16.549 ~ Iteration: 23 ~ bandwidth: 2.320 ~ gradient: 34.338 ~ stopping criterion: 16.987 ~ Iteration: 24 ~ bandwidth: 2.310 ~ gradient: 35.558 ~ stopping criterion: 17.169 ~ Iteration: 25 ~ bandwidth: 2.299 ~ gradient: 34.418 ~ stopping criterion: 17.779 ~ Iteration: 26 ~ bandwidth: 2.289 ~ gradient: 33.465 ~ stopping criterion: 17.209 ~ Iteration: 27 ~ bandwidth: 2.279 ~ gradient: 32.339 ~ stopping criterion: 16.733 ~ Iteration: 28 ~ bandwidth: 2.269 ~ gradient: 33.435 ~ stopping criterion: 16.169 ~ Iteration: 29 ~ bandwidth: 2.259 ~ gradient: 33.759 ~ stopping criterion: 16.717 ~ Iteration: 30 ~ bandwidth: 2.249 ~ gradient: 34.556 ~ stopping criterion: 16.879 ~ Iteration: 31 ~ bandwidth: 2.239 ~ gradient: 34.685 ~ stopping criterion: 17.278 ~ Iteration: 32 ~ bandwidth: 2.228 ~ gradient: 35.063 ~ stopping criterion: 17.343 ~ Iteration: 33 ~ bandwidth: 2.218 ~ gradient: 36.170 ~ stopping criterion: 17.532 ~ Iteration: 34 ~ bandwidth: 2.207 ~ gradient: 33.235 ~ stopping criterion: 18.085 ~ Iteration: 35 ~ bandwidth: 2.197 ~ gradient: 35.208 ~ stopping criterion: 16.618 ~ Iteration: 36 ~ bandwidth: 2.187 ~ gradient: 37.075 ~ stopping criterion: 17.604 ~ Iteration: 37 ~ bandwidth: 2.176 ~ gradient: 35.903 ~ stopping criterion: 18.537 ~ Iteration: 38 ~ bandwidth: 2.165 ~ gradient: 34.320 ~ stopping criterion: 17.952 ~ Iteration: 39 ~ bandwidth: 2.155 ~ gradient: 35.160 ~ stopping criterion: 17.160 ~ Iteration: 40 ~ bandwidth: 2.144 ~ gradient: 35.873 ~ stopping criterion: 17.580 ~ Iteration: 41 ~ bandwidth: 2.133 ~ gradient: 35.583 ~ stopping criterion: 17.937 ~ Iteration: 42 ~ bandwidth: 2.123 ~ gradient: 35.432 ~ stopping criterion: 17.792 ~ Iteration: 43 ~ bandwidth: 2.112 ~ gradient: 36.362 ~ stopping criterion: 17.716 ~ Iteration: 44 ~ bandwidth: 2.101 ~ gradient: 36.246 ~ stopping criterion: 18.181 ~ Iteration: 45 ~ bandwidth: 2.090 ~ gradient: 36.555 ~ stopping criterion: 18.123 ~ Iteration: 46 ~ bandwidth: 2.079 ~ gradient: 36.160 ~ stopping criterion: 18.278 ~ Iteration: 47 ~ bandwidth: 2.069 ~ gradient: 36.117 ~ stopping criterion: 18.080 ~ Iteration: 48 ~ bandwidth: 2.058 ~ gradient: 36.913 ~ stopping criterion: 18.059 ~ Iteration: 49 ~ bandwidth: 2.047 ~ gradient: 37.435 ~ stopping criterion: 18.457 ~ Iteration: 50 ~ bandwidth: 2.036 ~ gradient: 36.893 ~ stopping criterion: 18.718 ~ Iteration: 51 ~ bandwidth: 2.025 ~ gradient: 34.964 ~ stopping criterion: 18.446 ~ Iteration: 52 ~ bandwidth: 2.014 ~ gradient: 35.160 ~ stopping criterion: 17.482 ~ Iteration: 53 ~ bandwidth: 2.004 ~ gradient: 37.250 ~ stopping criterion: 17.580 ~ Iteration: 54 ~ bandwidth: 1.992 ~ gradient: 37.434 ~ stopping criterion: 18.625 ~ Iteration: 55 ~ bandwidth: 1.981 ~ gradient: 36.231 ~ stopping criterion: 18.717 ~ Iteration: 56 ~ bandwidth: 1.970 ~ gradient: 36.648 ~ stopping criterion: 18.115 ~ Iteration: 57 ~ bandwidth: 1.959 ~ gradient: 37.566 ~ stopping criterion: 18.324 ~ Iteration: 58 ~ bandwidth: 1.948 ~ gradient: 37.458 ~ stopping criterion: 18.783 ~ Iteration: 59 ~ bandwidth: 1.937 ~ gradient: 37.723 ~ stopping criterion: 18.729 ~ Iteration: 60 ~ bandwidth: 1.926 ~ gradient: 37.482 ~ stopping criterion: 18.861 ~ Iteration: 61 ~ bandwidth: 1.915 ~ gradient: 37.073 ~ stopping criterion: 18.741 ~ Iteration: 62 ~ bandwidth: 1.903 ~ gradient: 37.777 ~ stopping criterion: 18.537 ~ Iteration: 63 ~ bandwidth: 1.892 ~ gradient: 37.617 ~ stopping criterion: 18.888 ~ Iteration: 64 ~ bandwidth: 1.881 ~ gradient: 36.882 ~ stopping criterion: 18.809 ~ Iteration: 65 ~ bandwidth: 1.870 ~ gradient: 38.111 ~ stopping criterion: 18.441 ~ Iteration: 66 ~ bandwidth: 1.858 ~ gradient: 36.905 ~ stopping criterion: 19.055 ~ Iteration: 67 ~ bandwidth: 1.847 ~ gradient: 38.639 ~ stopping criterion: 18.453 ~ Iteration: 68 ~ bandwidth: 1.836 ~ gradient: 37.235 ~ stopping criterion: 19.320 ~ Iteration: 69 ~ bandwidth: 1.825 ~ gradient: 37.896 ~ stopping criterion: 18.618 ~ Iteration: 70 ~ bandwidth: 1.813 ~ gradient: 38.226 ~ stopping criterion: 18.948 ~ Iteration: 71 ~ bandwidth: 1.802 ~ gradient: 39.661 ~ stopping criterion: 19.113 ~ Iteration: 72 ~ bandwidth: 1.790 ~ gradient: 38.067 ~ stopping criterion: 19.830 ~ Iteration: 73 ~ bandwidth: 1.779 ~ gradient: 38.549 ~ stopping criterion: 19.033 ~ Iteration: 74 ~ bandwidth: 1.767 ~ gradient: 38.525 ~ stopping criterion: 19.274 ~ Iteration: 75 ~ bandwidth: 1.756 ~ gradient: 37.115 ~ stopping criterion: 19.262 ~ Iteration: 76 ~ bandwidth: 1.745 ~ gradient: 38.392 ~ stopping criterion: 18.558 ~ Iteration: 77 ~ bandwidth: 1.733 ~ gradient: 38.877 ~ stopping criterion: 19.196 ~ Iteration: 78 ~ bandwidth: 1.721 ~ gradient: 39.536 ~ stopping criterion: 19.439 ~ Iteration: 79 ~ bandwidth: 1.710 ~ gradient: 38.347 ~ stopping criterion: 19.768 ~ Iteration: 80 ~ bandwidth: 1.698 ~ gradient: 37.458 ~ stopping criterion: 19.173 ~ Iteration: 81 ~ bandwidth: 1.687 ~ gradient: 38.791 ~ stopping criterion: 18.729 ~ Iteration: 82 ~ bandwidth: 1.675 ~ gradient: 38.535 ~ stopping criterion: 19.395 ~ Iteration: 83 ~ bandwidth: 1.664 ~ gradient: 38.680 ~ stopping criterion: 19.267 ~ Iteration: 84 ~ bandwidth: 1.652 ~ gradient: 38.911 ~ stopping criterion: 19.340 ~ Iteration: 85 ~ bandwidth: 1.641 ~ gradient: 39.169 ~ stopping criterion: 19.456 ~ Iteration: 86 ~ bandwidth: 1.629 ~ gradient: 39.254 ~ stopping criterion: 19.584 ~ Iteration: 87 ~ bandwidth: 1.617 ~ gradient: 38.863 ~ stopping criterion: 19.627 ~ Iteration: 88 ~ bandwidth: 1.606 ~ gradient: 38.706 ~ stopping criterion: 19.432 ~ Iteration: 89 ~ bandwidth: 1.594 ~ gradient: 38.867 ~ stopping criterion: 19.353 ~ Iteration: 90 ~ bandwidth: 1.582 ~ gradient: 38.863 ~ stopping criterion: 19.434 ~ Iteration: 91 ~ bandwidth: 1.571 ~ gradient: 40.212 ~ stopping criterion: 19.432 ~ Iteration: 92 ~ bandwidth: 1.559 ~ gradient: 40.882 ~ stopping criterion: 20.106 ~ Iteration: 93 ~ bandwidth: 1.547 ~ gradient: 39.742 ~ stopping criterion: 20.441 ~ Iteration: 94 ~ bandwidth: 1.535 ~ gradient: 40.603 ~ stopping criterion: 19.871 ~ Iteration: 95 ~ bandwidth: 1.522 ~ gradient: 38.723 ~ stopping criterion: 20.302 ~ Iteration: 96 ~ bandwidth: 1.511 ~ gradient: 39.437 ~ stopping criterion: 19.361 ~ Iteration: 97 ~ bandwidth: 1.499 ~ gradient: 39.963 ~ stopping criterion: 19.719 ~ Iteration: 98 ~ bandwidth: 1.487 ~ gradient: 40.406 ~ stopping criterion: 19.981 ~ Iteration: 99 ~ bandwidth: 1.475 ~ gradient: 40.569 ~ stopping criterion: 20.203 ~ Iteration: 100 ~ bandwidth: 1.463 ~ gradient: 41.067 ~ stopping criterion: 20.285 ~ Iteration: 101 ~ bandwidth: 1.451 ~ gradient: 40.289 ~ stopping criterion: 20.533 ~ Iteration: 102 ~ bandwidth: 1.439 ~ gradient: 39.731 ~ stopping criterion: 20.144 ~ Iteration: 103 ~ bandwidth: 1.427 ~ gradient: 41.624 ~ stopping criterion: 19.865 ~ Iteration: 104 ~ bandwidth: 1.414 ~ gradient: 39.657 ~ stopping criterion: 20.812 ~ Iteration: 105 ~ bandwidth: 1.402 ~ gradient: 40.531 ~ stopping criterion: 19.829 ~ Iteration: 106 ~ bandwidth: 1.390 ~ gradient: 41.455 ~ stopping criterion: 20.266 ~ Iteration: 107 ~ bandwidth: 1.378 ~ gradient: 40.318 ~ stopping criterion: 20.727 ~ Iteration: 108 ~ bandwidth: 1.366 ~ gradient: 39.433 ~ stopping criterion: 20.159 ~ Iteration: 109 ~ bandwidth: 1.354 ~ gradient: 41.338 ~ stopping criterion: 19.716 ~ Iteration: 110 ~ bandwidth: 1.342 ~ gradient: 41.455 ~ stopping criterion: 20.669 ~ Iteration: 111 ~ bandwidth: 1.329 ~ gradient: 40.924 ~ stopping criterion: 20.727 ~ Iteration: 112 ~ bandwidth: 1.317 ~ gradient: 40.149 ~ stopping criterion: 20.462 ~ Iteration: 113 ~ bandwidth: 1.305 ~ gradient: 40.132 ~ stopping criterion: 20.074 ~ Iteration: 114 ~ bandwidth: 1.293 ~ gradient: 39.529 ~ stopping criterion: 20.066 ~ Iteration: 115 ~ bandwidth: 1.281 ~ gradient: 39.958 ~ stopping criterion: 19.765 ~ Iteration: 116 ~ bandwidth: 1.269 ~ gradient: 40.801 ~ stopping criterion: 19.979 ~ Iteration: 117 ~ bandwidth: 1.257 ~ gradient: 39.030 ~ stopping criterion: 20.401 ~ Iteration: 118 ~ bandwidth: 1.245 ~ gradient: 39.225 ~ stopping criterion: 19.515 ~ Iteration: 119 ~ bandwidth: 1.234 ~ gradient: 40.178 ~ stopping criterion: 19.612 ~ Iteration: 120 ~ bandwidth: 1.222 ~ gradient: 40.319 ~ stopping criterion: 20.089 ~ Iteration: 121 ~ bandwidth: 1.210 ~ gradient: 40.496 ~ stopping criterion: 20.160 ~ Iteration: 122 ~ bandwidth: 1.197 ~ gradient: 39.627 ~ stopping criterion: 20.248 ~ Iteration: 123 ~ bandwidth: 1.186 ~ gradient: 39.677 ~ stopping criterion: 19.813 ~ Iteration: 124 ~ bandwidth: 1.174 ~ gradient: 39.699 ~ stopping criterion: 19.838 ~ Iteration: 125 ~ bandwidth: 1.162 ~ gradient: 38.777 ~ stopping criterion: 19.850 ~ Iteration: 126 ~ bandwidth: 1.150 ~ gradient: 39.688 ~ stopping criterion: 19.388 ~ Iteration: 127 ~ bandwidth: 1.138 ~ gradient: 38.588 ~ stopping criterion: 19.844 ~ Iteration: 128 ~ bandwidth: 1.127 ~ gradient: 39.553 ~ stopping criterion: 19.294 ~ Iteration: 129 ~ bandwidth: 1.115 ~ gradient: 36.344 ~ stopping criterion: 19.777 ~ Iteration: 130 ~ bandwidth: 1.104 ~ gradient: 38.735 ~ stopping criterion: 18.172 ~ Iteration: 131 ~ bandwidth: 1.093 ~ gradient: 36.876 ~ stopping criterion: 19.367 ~ Iteration: 132 ~ bandwidth: 1.082 ~ gradient: 38.788 ~ stopping criterion: 18.438 ~ Iteration: 133 ~ bandwidth: 1.070 ~ gradient: 38.341 ~ stopping criterion: 19.394 ~ Iteration: 134 ~ bandwidth: 1.059 ~ gradient: 37.313 ~ stopping criterion: 19.171 ~ Iteration: 135 ~ bandwidth: 1.047 ~ gradient: 37.391 ~ stopping criterion: 18.656 ~ Iteration: 136 ~ bandwidth: 1.036 ~ gradient: 36.883 ~ stopping criterion: 18.696 ~ Iteration: 137 ~ bandwidth: 1.025 ~ gradient: 35.974 ~ stopping criterion: 18.442 ~ Iteration: 138 ~ bandwidth: 1.014 ~ gradient: 35.982 ~ stopping criterion: 17.987 ~ Iteration: 139 ~ bandwidth: 1.004 ~ gradient: 37.327 ~ stopping criterion: 17.991 ~ Iteration: 140 ~ bandwidth: 0.992 ~ gradient: 34.022 ~ stopping criterion: 18.663 ~ Iteration: 141 ~ bandwidth: 0.982 ~ gradient: 35.532 ~ stopping criterion: 17.011 ~ Iteration: 142 ~ bandwidth: 0.972 ~ gradient: 35.211 ~ stopping criterion: 17.766 ~ Iteration: 143 ~ bandwidth: 0.961 ~ gradient: 32.485 ~ stopping criterion: 17.605 ~ Iteration: 144 ~ bandwidth: 0.951 ~ gradient: 32.277 ~ stopping criterion: 16.242 ~ Iteration: 145 ~ bandwidth: 0.942 ~ gradient: 33.378 ~ stopping criterion: 16.138 ~ Iteration: 146 ~ bandwidth: 0.932 ~ gradient: 32.958 ~ stopping criterion: 16.689 ~ Iteration: 147 ~ bandwidth: 0.922 ~ gradient: 32.462 ~ stopping criterion: 16.479 ~ Iteration: 148 ~ bandwidth: 0.912 ~ gradient: 30.227 ~ stopping criterion: 16.231 ~ Iteration: 149 ~ bandwidth: 0.903 ~ gradient: 30.399 ~ stopping criterion: 15.113 ~ Iteration: 150 ~ bandwidth: 0.894 ~ gradient: 28.374 ~ stopping criterion: 15.199 ~ Iteration: 151 ~ bandwidth: 0.886 ~ gradient: 28.302 ~ stopping criterion: 14.187 ~ Iteration: 152 ~ bandwidth: 0.877 ~ gradient: 27.961 ~ stopping criterion: 14.151 ~ Iteration: 153 ~ bandwidth: 0.869 ~ gradient: 26.522 ~ stopping criterion: 13.981 ~ Iteration: 154 ~ bandwidth: 0.861 ~ gradient: 26.798 ~ stopping criterion: 13.261 ~ Iteration: 155 ~ bandwidth: 0.853 ~ gradient: 25.203 ~ stopping criterion: 13.399 ~ Iteration: 156 ~ bandwidth: 0.845 ~ gradient: 24.713 ~ stopping criterion: 12.601 ~ Iteration: 157 ~ bandwidth: 0.838 ~ gradient: 23.031 ~ stopping criterion: 12.356 ~ Iteration: 158 ~ bandwidth: 0.831 ~ gradient: 22.019 ~ stopping criterion: 11.515 ~ Iteration: 159 ~ bandwidth: 0.824 ~ gradient: 21.954 ~ stopping criterion: 11.010 ~ Iteration: 160 ~ bandwidth: 0.818 ~ gradient: 21.373 ~ stopping criterion: 10.977 ~ Iteration: 161 ~ bandwidth: 0.811 ~ gradient: 22.419 ~ stopping criterion: 10.686 ~ Iteration: 162 ~ bandwidth: 0.805 ~ gradient: 18.759 ~ stopping criterion: 11.209 ~ Iteration: 163 ~ bandwidth: 0.799 ~ gradient: 19.891 ~ stopping criterion: 9.380 ~ Iteration: 164 ~ bandwidth: 0.793 ~ gradient: 18.285 ~ stopping criterion: 9.945 ~ Iteration: 165 ~ bandwidth: 0.788 ~ gradient: 17.670 ~ stopping criterion: 9.143 ~ Iteration: 166 ~ bandwidth: 0.782 ~ gradient: 15.641 ~ stopping criterion: 8.835 ~ Iteration: 167 ~ bandwidth: 0.778 ~ gradient: 15.377 ~ stopping criterion: 7.821 ~ Iteration: 168 ~ bandwidth: 0.773 ~ gradient: 13.831 ~ stopping criterion: 7.688 ~ Iteration: 169 ~ bandwidth: 0.769 ~ gradient: 14.703 ~ stopping criterion: 6.915 ~ Iteration: 170 ~ bandwidth: 0.765 ~ gradient: 15.614 ~ stopping criterion: 7.351 ~ Iteration: 171 ~ bandwidth: 0.760 ~ gradient: 12.007 ~ stopping criterion: 7.807 ~ Iteration: 172 ~ bandwidth: 0.756 ~ gradient: 13.628 ~ stopping criterion: 6.004 ~ Iteration: 173 ~ bandwidth: 0.752 ~ gradient: 11.557 ~ stopping criterion: 6.814 ~ Iteration: 174 ~ bandwidth: 0.749 ~ gradient: 9.434 ~ stopping criterion: 5.779 ~ Iteration: 175 ~ bandwidth: 0.746 ~ gradient: 8.647 ~ stopping criterion: 4.717 ~ Iteration: 176 ~ bandwidth: 0.743 ~ gradient: 8.527 ~ stopping criterion: 4.324 ~ Iteration: 177 ~ bandwidth: 0.741 ~ gradient: 8.860 ~ stopping criterion: 4.263 ~ Iteration: 178 ~ bandwidth: 0.738 ~ gradient: 8.644 ~ stopping criterion: 4.430 ~ Iteration: 179 ~ bandwidth: 0.736 ~ gradient: 8.655 ~ stopping criterion: 4.322 ~ Iteration: 180 ~ bandwidth: 0.733 ~ gradient: 6.855 ~ stopping criterion: 4.328 ~ Iteration: 181 ~ bandwidth: 0.731 ~ gradient: 7.078 ~ stopping criterion: 3.427 ~ Iteration: 182 ~ bandwidth: 0.729 ~ gradient: 8.211 ~ stopping criterion: 3.539 ~ Iteration: 183 ~ bandwidth: 0.726 ~ gradient: 7.145 ~ stopping criterion: 4.105 ~ Iteration: 184 ~ bandwidth: 0.724 ~ gradient: 5.995 ~ stopping criterion: 3.573 ~ Iteration: 185 ~ bandwidth: 0.723 ~ gradient: 5.043 ~ stopping criterion: 2.998 ~ Iteration: 186 ~ bandwidth: 0.721 ~ gradient: 4.909 ~ stopping criterion: 2.521 ~ Iteration: 187 ~ bandwidth: 0.720 ~ gradient: 5.327 ~ stopping criterion: 2.455 ~ Iteration: 188 ~ bandwidth: 0.718 ~ gradient: 2.895 ~ stopping criterion: 2.663 ~ Iteration: 189 ~ bandwidth: 0.717 ~ gradient: 3.976 ~ stopping criterion: 1.447 ~ Iteration: 190 ~ bandwidth: 0.716 ~ gradient: 3.597 ~ stopping criterion: 1.988 ~ Iteration: 191 ~ bandwidth: 0.715 ~ gradient: 3.651 ~ stopping criterion: 1.799 ~ Iteration: 192 ~ bandwidth: 0.714 ~ gradient: 4.651 ~ stopping criterion: 1.826 ~ Iteration: 193 ~ bandwidth: 0.712 ~ gradient: 4.288 ~ stopping criterion: 2.326 ~ Iteration: 194 ~ bandwidth: 0.711 ~ gradient: 3.022 ~ stopping criterion: 2.144 ~ Iteration: 195 ~ bandwidth: 0.710 ~ gradient: 3.597 ~ stopping criterion: 1.511 ~ Iteration: 196 ~ bandwidth: 0.709 ~ gradient: 2.005 ~ stopping criterion: 1.799 ~ Iteration: 197 ~ bandwidth: 0.708 ~ gradient: 1.798 ~ stopping criterion: 1.003 ~ Iteration: 198 ~ bandwidth: 0.708 ~ gradient: 2.078 ~ stopping criterion: 0.899 ~ Iteration: 199 ~ bandwidth: 0.707 ~ gradient: 2.216 ~ stopping criterion: 1.039 ~ Iteration: 200 ~ bandwidth: 0.707 ~ gradient: 3.766 ~ stopping criterion: 1.108 ~ Iteration: 201 ~ bandwidth: 0.706 ~ gradient: 1.895 ~ stopping criterion: 1.883 ~ Iteration: 202 ~ bandwidth: 0.705 ~ gradient: 1.989 ~ stopping criterion: 0.948 ~ Iteration: 203 ~ bandwidth: 0.704 ~ gradient: 0.884 ~ stopping criterion: 0.995 ~ Iteration: 204 ~ bandwidth: 0.704 ~ gradient: 0.034 ~ stopping criterion: 0.442 ~ Iteration: 205 ~ bandwidth: 0.704 ~ gradient: 1.075 ~ stopping criterion: 0.017 ~ Iteration: 206 ~ bandwidth: 0.704 ~ gradient: 0.809 ~ stopping criterion: 0.538 ~ Iteration: 207 ~ bandwidth: 0.704 ~ gradient: 0.974 ~ stopping criterion: 0.404 ~ Iteration: 208 ~ bandwidth: 0.703 ~ gradient: 1.272 ~ stopping criterion: 0.487 ~ Iteration: 209 ~ bandwidth: 0.703 ~ gradient: 1.031 ~ stopping criterion: 0.636 ~ Iteration: 210 ~ bandwidth: 0.703 ~ gradient: 0.824 ~ stopping criterion: 0.516 ~ Iteration: 211 ~ bandwidth: 0.702 ~ gradient: 0.574 ~ stopping criterion: 0.412 ~ Iteration: 212 ~ bandwidth: 0.702 ~ gradient: -0.045 ~ stopping criterion: 0.287 ~ Iteration: 213 ~ bandwidth: 0.702 ~ gradient: -1.370 ~ stopping criterion: 0.022 ~ Iteration: 214 ~ bandwidth: 0.702 ~ gradient: -0.081 ~ stopping criterion: 0.685 ~ Iteration: 215 ~ bandwidth: 0.703 ~ gradient: 1.688 ~ stopping criterion: 0.040 ~ Iteration: 216 ~ bandwidth: 0.702 ~ gradient: -0.780 ~ stopping criterion: 0.844 ~ Iteration: 217 ~ bandwidth: 0.702 ~ gradient: -0.101 ~ stopping criterion: 0.390 ~ Iteration: 218 ~ bandwidth: 0.702 ~ gradient: -0.119 ~ stopping criterion: 0.051 ~ Iteration: 219 ~ bandwidth: 0.702 ~ gradient: 0.531 ~ stopping criterion: 0.059 ~ Iteration: 220 ~ bandwidth: 0.702 ~ gradient: 0.786 ~ stopping criterion: 0.265 ~ Iteration: 221 ~ bandwidth: 0.702 ~ gradient: 0.487 ~ stopping criterion: 0.393 ~ Iteration: 222 ~ bandwidth: 0.702 ~ gradient: 1.407 ~ stopping criterion: 0.244 ~ Iteration: 223 ~ bandwidth: 0.702 ~ gradient: 0.562 ~ stopping criterion: 0.704 ~ Iteration: 224 ~ bandwidth: 0.702 ~ gradient: -0.860 ~ stopping criterion: 0.281 ~ Iteration: 225 ~ bandwidth: 0.702 ~ gradient: 1.547 ~ stopping criterion: 0.430 ~ Iteration: 226 ~ bandwidth: 0.702 ~ gradient: 0.296 ~ stopping criterion: 0.774 ~ Iteration: 227 ~ bandwidth: 0.702 ~ gradient: 0.446 ~ stopping criterion: 0.148 ~ Iteration: 228 ~ bandwidth: 0.702 ~ gradient: 1.167 ~ stopping criterion: 0.223 ~ Iteration: 229 ~ bandwidth: 0.702 ~ gradient: 1.101 ~ stopping criterion: 0.584 ~ Iteration: 230 ~ bandwidth: 0.702 ~ gradient: -0.906 ~ stopping criterion: 0.550 ~ Iteration: 231 ~ bandwidth: 0.702 ~ gradient: 0.829 ~ stopping criterion: 0.453 ~ Iteration: 232 ~ bandwidth: 0.702 ~ gradient: 0.730 ~ stopping criterion: 0.415 ~ Iteration: 233 ~ bandwidth: 0.702 ~ gradient: -0.245 ~ stopping criterion: 0.365 ~ Iteration: 234 ~ bandwidth: 0.702 ~ gradient: 1.577 ~ stopping criterion: 0.122 ~ Iteration: 235 ~ bandwidth: 0.701 ~ gradient: 0.024 ~ stopping criterion: 0.788 ~ Iteration: 236 ~ bandwidth: 0.701 ~ gradient: 1.736 ~ stopping criterion: 0.012 ~ Iteration: 237 ~ bandwidth: 0.701 ~ gradient: 0.633 ~ stopping criterion: 0.868 ~ Iteration: 238 ~ bandwidth: 0.701 ~ gradient: 1.599 ~ stopping criterion: 0.317 ~ Iteration: 239 ~ bandwidth: 0.701 ~ gradient: 1.140 ~ stopping criterion: 0.800 ~ Iteration: 240 ~ bandwidth: 0.701 ~ gradient: 1.434 ~ stopping criterion: 0.570 ~ Iteration: 241 ~ bandwidth: 0.701 ~ gradient: -0.243 ~ stopping criterion: 0.717 ~ Iteration: 242 ~ bandwidth: 0.701 ~ gradient: 1.831 ~ stopping criterion: 0.122 ~ Iteration: 243 ~ bandwidth: 0.701 ~ gradient: -0.497 ~ stopping criterion: 0.916 ~ Iteration: 244 ~ bandwidth: 0.701 ~ gradient: 1.226 ~ stopping criterion: 0.249 ~ Iteration: 245 ~ bandwidth: 0.701 ~ gradient: -0.292 ~ stopping criterion: 0.613 ~ Iteration: 246 ~ bandwidth: 0.701 ~ gradient: -0.067 ~ stopping criterion: 0.146 ~ Iteration: 247 ~ bandwidth: 0.701 ~ gradient: 0.449 ~ stopping criterion: 0.033 ~ Iteration: 248 ~ bandwidth: 0.701 ~ gradient: 0.290 ~ stopping criterion: 0.224 ~ Iteration: 249 ~ bandwidth: 0.701 ~ gradient: -0.736 ~ stopping criterion: 0.145 ~ Iteration: 250 ~ bandwidth: 0.701 ~ gradient: 0.039 ~ stopping criterion: 0.368 ~ Iteration: 251 ~ bandwidth: 0.701 ~ gradient: 1.392 ~ stopping criterion: 0.019 ~ Iteration: 252 ~ bandwidth: 0.701 ~ gradient: 0.187 ~ stopping criterion: 0.696 ~ Iteration: 253 ~ bandwidth: 0.701 ~ gradient: -0.352 ~ stopping criterion: 0.094 ~ Iteration: 254 ~ bandwidth: 0.701 ~ gradient: -0.693 ~ stopping criterion: 0.176 ~ Iteration: 255 ~ bandwidth: 0.701 ~ gradient: -0.410 ~ stopping criterion: 0.347 ~ Iteration: 256 ~ bandwidth: 0.701 ~ gradient: -0.858 ~ stopping criterion: 0.205 ~ Iteration: 257 ~ bandwidth: 0.701 ~ gradient: -0.014 ~ stopping criterion: 0.429 ~ Iteration: 258 ~ bandwidth: 0.701 ~ gradient: -0.138 ~ stopping criterion: 0.007 ~ Iteration: 259 ~ bandwidth: 0.701 ~ gradient: -0.207 ~ stopping criterion: 0.069 ~ Iteration: 260 ~ bandwidth: 0.701 ~ gradient: -1.444 ~ stopping criterion: 0.103 ~ Iteration: 261 ~ bandwidth: 0.701 ~ gradient: 0.497 ~ stopping criterion: 0.722 ~ Iteration: 262 ~ bandwidth: 0.701 ~ gradient: -0.539 ~ stopping criterion: 0.248 ~ Iteration: 263 ~ bandwidth: 0.701 ~ gradient: 1.030 ~ stopping criterion: 0.269 ~ Iteration: 264 ~ bandwidth: 0.701 ~ gradient: 0.294 ~ stopping criterion: 0.515 ~ Iteration: 265 ~ bandwidth: 0.701 ~ gradient: 0.303 ~ stopping criterion: 0.147 ~ Iteration: 266 ~ bandwidth: 0.701 ~ gradient: -0.277 ~ stopping criterion: 0.152 ~ Iteration: 267 ~ bandwidth: 0.701 ~ gradient: 0.243 ~ stopping criterion: 0.138 ~ Iteration: 268 ~ bandwidth: 0.701 ~ gradient: 0.500 ~ stopping criterion: 0.121 ~ Iteration: 269 ~ bandwidth: 0.701 ~ gradient: -0.835 ~ stopping criterion: 0.250 ~ Iteration: 270 ~ bandwidth: 0.701 ~ gradient: 0.442 ~ stopping criterion: 0.417 ~ Iteration: 271 ~ bandwidth: 0.701 ~ gradient: 1.868 ~ stopping criterion: 0.221 ~ Iteration: 272 ~ bandwidth: 0.701 ~ gradient: -0.799 ~ stopping criterion: 0.934 ~ Iteration: 273 ~ bandwidth: 0.701 ~ gradient: -1.488 ~ stopping criterion: 0.400 ~ Iteration: 274 ~ bandwidth: 0.701 ~ gradient: 0.798 ~ stopping criterion: 0.744 ~ Iteration: 275 ~ bandwidth: 0.701 ~ gradient: 0.647 ~ stopping criterion: 0.399 ~ Iteration: 276 ~ bandwidth: 0.701 ~ gradient: 0.525 ~ stopping criterion: 0.324 ~ Iteration: 277 ~ bandwidth: 0.701 ~ gradient: 0.455 ~ stopping criterion: 0.263 ~ Iteration: 278 ~ bandwidth: 0.701 ~ gradient: -0.345 ~ stopping criterion: 0.228 ~ Iteration: 279 ~ bandwidth: 0.701 ~ gradient: -0.541 ~ stopping criterion: 0.173 ~ Iteration: 280 ~ bandwidth: 0.701 ~ gradient: -1.220 ~ stopping criterion: 0.270 ~ Iteration: 281 ~ bandwidth: 0.701 ~ gradient: 0.927 ~ stopping criterion: 0.610 ~ Iteration: 282 ~ bandwidth: 0.701 ~ gradient: -0.659 ~ stopping criterion: 0.463 ~ Iteration: 283 ~ bandwidth: 0.701 ~ gradient: -0.248 ~ stopping criterion: 0.330 ~ Iteration: 284 ~ bandwidth: 0.701 ~ gradient: -0.298 ~ stopping criterion: 0.124 ~ Iteration: 285 ~ bandwidth: 0.701 ~ gradient: 0.263 ~ stopping criterion: 0.149 ~ Iteration: 286 ~ bandwidth: 0.701 ~ gradient: 0.964 ~ stopping criterion: 0.132 ~ Iteration: 287 ~ bandwidth: 0.701 ~ gradient: 0.492 ~ stopping criterion: 0.482 ~ Iteration: 288 ~ bandwidth: 0.701 ~ gradient: 0.355 ~ stopping criterion: 0.246 ~ Iteration: 289 ~ bandwidth: 0.701 ~ gradient: -0.169 ~ stopping criterion: 0.178 ~ Iteration: 290 ~ bandwidth: 0.701 ~ gradient: -1.281 ~ stopping criterion: 0.085 ~ Iteration: 291 ~ bandwidth: 0.701 ~ gradient: -0.711 ~ stopping criterion: 0.640 ~ Iteration: 292 ~ bandwidth: 0.701 ~ gradient: -0.046 ~ stopping criterion: 0.356 ~ Iteration: 293 ~ bandwidth: 0.701 ~ gradient: 0.493 ~ stopping criterion: 0.023 ~ Iteration: 294 ~ bandwidth: 0.701 ~ gradient: -0.283 ~ stopping criterion: 0.246 ~ Iteration: 295 ~ bandwidth: 0.701 ~ gradient: -1.623 ~ stopping criterion: 0.141 ~ Iteration: 296 ~ bandwidth: 0.701 ~ gradient: 1.777 ~ stopping criterion: 0.811 ~ Iteration: 297 ~ bandwidth: 0.701 ~ gradient: -0.091 ~ stopping criterion: 0.888 ~ Iteration: 298 ~ bandwidth: 0.701 ~ gradient: 0.148 ~ stopping criterion: 0.045 ~ Iteration: 299 ~ bandwidth: 0.701 ~ gradient: 0.214 ~ stopping criterion: 0.074 ~ Iteration: 300 ~ bandwidth: 0.701 ~ gradient: 0.181 ~ stopping criterion: 0.107 ~ Stopped at: 300 ~ bandwidth: 0.701 ~ gradient: 0.181 ~ stopping criterion: 0.107
Let’s compare this to the previous case.
# MixCOBRA with default parameter
print("Estimated bandwidth = {}".format(gc2_fit.optimization_outputs['opt_bandwidth']))
Estimated bandwidth = 0.7012029287236681
The learning curve.
gc2_fit.draw_learning_curve()
Copare the MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
= gc2_fit.predict(X_test1)
y_pred2 print(mean_absolute_percentage_error(y_test1, y_pred2))
print(mean_squared_error(y_test1, y_pred2))
0.23524039384008133
300.2477389788587
Let’s look at the qq-plot
.
=y_test1, fig_type='qq') gc2_fit.draw_learning_curve(y_test
MixCOBRARegressor
with non-default parameters
MixCOBRARegressor
offers various options to enhance the performance of the method. By default, grid search algorithm is used to estimate the parameters \((\alpha, \beta)\), however, gradient descent algorithm can also be used to optimize the method by setting opt_method = "grad"
as shown in case of one bandwidth parameter above. Moveover, you can control the hyperparameters of the basic estimators to enhance the aggregation performance just like in GradientCOBRA
method. This can be done as follows:
- learning_rate : control the learning rate of gradient descent in estimating the bandwidth parameter
- speed : the speed of the learning rate.
- kernel : the kernel function used for the aggregation
- opt_method : the optimiztaion algorithm for estimating the bandwidth. It can be gradient descent (
grad
) or grid search (grid
). - kernel : kernel function which is
radial
by default. It must be an element of['exponential', 'gaussian', 'radial', 'cauchy', 'reverse_cosh', 'epanechnikov'
,'biweight', 'triweight', 'triangular']
. - alpha_list : list or array to search for \(\alpha\).
- beta_list : list or array to search for \(\beta\).
- bandwidth_list : : list or array to search for single bandwidth parameter \(h\) instead of \((\alpha, \beta)\). To do this, we have to set
one_parameter = True
when fitting the method. - max_iter : maximum iteration in gradient descent.
- loss_function : control the type of loss function used for optimizing the bandwidth.
- opt_params : control the optimization algorithm such as adjusting
n_cv
,start
,n_tries
… - estiamtor_list : the list of basic estimators used for the aggregation.
- estimator_params : controlling the hyperparameters of the basic estimators. It must be a dictionary with
(key, dict) = (estimator, dict)
, i.e. the key must be the name of the basic estimator, and the value is a dictionary containing its hyperparamaters.
We create another object gc3
with non-default parameters, then fit it to the same training data as in the previous example.
= MixCOBRARegressor(learning_rate=0.1,
gc3 ="linear",
speed='radial',
kernel='grad',
opt_method="weighted_mse",
loss_function=300,
max_iter=['random_forest',
estimator_list'adaboost',
'knn',
'svm',
'lasso',
'ridge'],
={
estimator_params'random_forest' : {
'n_estimators' : 300,
'min_samples_leaf' : 10},
'adaboost' : {
'n_estimators' : 300,
'max_depth' : 10},
'knn' : {
'n_neighbors' : 30},
'svm' : {'C' : 7}
})
= gc3.fit(X_train1, y_train1) gc3_fit
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: 0.132 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.065924 ~ Iteration: 2 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.087429 ~ Iteration: 3 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.108935 ~ Iteration: 4 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.130440 ~ Iteration: 5 ~ (alpha, beta): (1.025, 0.950) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.065924 ~ Iteration: 6 ~ (alpha, beta): (0.974, 1.530) ~ |gradient|_1: -0.013 ~ stopping criterion: 0.083202 ~ Iteration: 7 ~ (alpha, beta): (1.020, 1.032) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.074755 ~ Iteration: 8 ~ (alpha, beta): (0.976, 1.178) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.028074 ~ Iteration: 9 ~ (alpha, beta): (0.971, 1.022) ~ |gradient|_1: 0.015 ~ stopping criterion: 0.025987 ~ Iteration: 10 ~ (alpha, beta): (0.918, 1.209) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.040850 ~ Iteration: 11 ~ (alpha, beta): (0.959, 1.067) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.033706 ~ Iteration: 12 ~ (alpha, beta): (0.916, 1.111) ~ |gradient|_1: -0.009 ~ stopping criterion: 0.016338 ~ Iteration: 13 ~ (alpha, beta): (0.943, 1.080) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.012792 ~ Iteration: 14 ~ (alpha, beta): (0.899, 1.078) ~ |gradient|_1: 0.009 ~ stopping criterion: 0.008826 ~ Iteration: 15 ~ (alpha, beta): (0.879, 1.112) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.014056 ~ Iteration: 16 ~ (alpha, beta): (0.859, 1.100) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.008195 ~ Iteration: 17 ~ (alpha, beta): (0.827, 1.078) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.014637 ~ Iteration: 18 ~ (alpha, beta): (0.831, 1.110) ~ |gradient|_1: -0.019 ~ stopping criterion: 0.012317 ~ Iteration: 19 ~ (alpha, beta): (0.874, 1.100) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.013762 ~ Iteration: 20 ~ (alpha, beta): (0.873, 1.099) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.000663 ~ Iteration: 21 ~ (alpha, beta): (0.871, 1.083) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.008359 ~ Iteration: 22 ~ (alpha, beta): (0.887, 1.088) ~ |gradient|_1: -0.009 ~ stopping criterion: 0.006470 ~ Iteration: 23 ~ (alpha, beta): (0.903, 1.084) ~ |gradient|_1: -0.017 ~ stopping criterion: 0.006947 ~ Iteration: 24 ~ (alpha, beta): (0.936, 1.095) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.015694 ~ Iteration: 25 ~ (alpha, beta): (0.924, 1.090) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.006975 ~ Iteration: 26 ~ (alpha, beta): (0.942, 1.092) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.007944 ~ Iteration: 27 ~ (alpha, beta): (0.939, 1.095) ~ |gradient|_1: -0.017 ~ stopping criterion: 0.004901 ~ Iteration: 28 ~ (alpha, beta): (0.958, 1.078) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.027751 ~ Iteration: 29 ~ (alpha, beta): (0.963, 1.081) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.008136 ~ Iteration: 30 ~ (alpha, beta): (0.958, 1.085) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.007945 ~ Iteration: 31 ~ (alpha, beta): (0.947, 1.086) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.009480 ~ Iteration: 32 ~ (alpha, beta): (0.956, 1.087) ~ |gradient|_1: 0.024 ~ stopping criterion: 0.005746 ~ Iteration: 33 ~ (alpha, beta): (0.939, 1.084) ~ |gradient|_1: 0.022 ~ stopping criterion: 0.015416 ~ Iteration: 34 ~ (alpha, beta): (0.927, 1.087) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.016166 ~ Iteration: 35 ~ (alpha, beta): (0.915, 1.090) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.016617 ~ Iteration: 36 ~ (alpha, beta): (0.914, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.003197 ~ Iteration: 37 ~ (alpha, beta): (0.910, 1.087) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.009037 ~ Iteration: 38 ~ (alpha, beta): (0.902, 1.081) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.019952 ~ Iteration: 39 ~ (alpha, beta): (0.894, 1.085) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.013848 ~ Iteration: 40 ~ (alpha, beta): (0.901, 1.088) ~ |gradient|_1: 0.020 ~ stopping criterion: 0.016229 ~ Iteration: 41 ~ (alpha, beta): (0.890, 1.088) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.010511 ~ Iteration: 42 ~ (alpha, beta): (0.884, 1.086) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.014590 ~ Iteration: 43 ~ (alpha, beta): (0.882, 1.090) ~ |gradient|_1: -0.013 ~ stopping criterion: 0.014406 ~ Iteration: 44 ~ (alpha, beta): (0.888, 1.091) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.010878 ~ Iteration: 45 ~ (alpha, beta): (0.887, 1.091) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001943 ~ Iteration: 46 ~ (alpha, beta): (0.887, 1.092) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.003673 ~ Iteration: 47 ~ (alpha, beta): (0.888, 1.093) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.005612 ~ Iteration: 48 ~ (alpha, beta): (0.887, 1.092) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.004393 ~ Iteration: 49 ~ (alpha, beta): (0.884, 1.091) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.011986 ~ Iteration: 50 ~ (alpha, beta): (0.882, 1.090) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.007660 ~ Iteration: 51 ~ (alpha, beta): (0.882, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.004611 ~ Iteration: 52 ~ (alpha, beta): (0.884, 1.088) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.011304 ~ Iteration: 53 ~ (alpha, beta): (0.883, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.006466 ~ Iteration: 54 ~ (alpha, beta): (0.883, 1.089) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.000902 ~ Iteration: 55 ~ (alpha, beta): (0.881, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.008878 ~ Iteration: 56 ~ (alpha, beta): (0.878, 1.089) ~ |gradient|_1: 0.009 ~ stopping criterion: 0.009695 ~ Iteration: 57 ~ (alpha, beta): (0.877, 1.088) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.009556 ~ Iteration: 58 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.002561 ~ Iteration: 59 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.003356 ~ Iteration: 60 ~ (alpha, beta): (0.878, 1.088) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.006347 ~ Iteration: 61 ~ (alpha, beta): (0.879, 1.088) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.001077 ~ Iteration: 62 ~ (alpha, beta): (0.876, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.015848 ~ Iteration: 63 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.004051 ~ Iteration: 64 ~ (alpha, beta): (0.875, 1.088) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.013425 ~ Iteration: 65 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.003705 ~ Iteration: 66 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.003234 ~ Iteration: 67 ~ (alpha, beta): (0.873, 1.089) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.007744 ~ Iteration: 68 ~ (alpha, beta): (0.873, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.004452 ~ Iteration: 69 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.002422 ~ Iteration: 70 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.007313 ~ Iteration: 71 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.003825 ~ Iteration: 72 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.006748 ~ Iteration: 73 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.009112 ~ Iteration: 74 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.010699 ~ Iteration: 75 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.006512 ~ Iteration: 76 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.011142 ~ Iteration: 77 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007410 ~ Iteration: 78 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.013663 ~ Iteration: 79 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.008604 ~ Iteration: 80 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.005902 ~ Iteration: 81 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.003601 ~ Iteration: 82 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.011737 ~ Iteration: 83 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.004572 ~ Iteration: 84 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.014986 ~ Iteration: 85 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.028 ~ stopping criterion: 0.005204 ~ Iteration: 86 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.019777 ~ Iteration: 87 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.003831 ~ Iteration: 88 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.009195 ~ Iteration: 89 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.010378 ~ Iteration: 90 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.004439 ~ Iteration: 91 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.010082 ~ Iteration: 92 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.003847 ~ Iteration: 93 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.014638 ~ Iteration: 94 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.018533 ~ Iteration: 95 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.011874 ~ Iteration: 96 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.001645 ~ Iteration: 97 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.005722 ~ Iteration: 98 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.017228 ~ Iteration: 99 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.025 ~ stopping criterion: 0.002783 ~ Iteration: 100 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.017635 ~ Iteration: 101 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.014150 ~ Iteration: 102 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.005757 ~ Iteration: 103 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007102 ~ Iteration: 104 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.006087 ~ Iteration: 105 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.007154 ~ Iteration: 106 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.011526 ~ Iteration: 107 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.015174 ~ Iteration: 108 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.005473 ~ Iteration: 109 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.004929 ~ Iteration: 110 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.005358 ~ Iteration: 111 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.014516 ~ Iteration: 112 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.006655 ~ Iteration: 113 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.021 ~ stopping criterion: 0.011645 ~ Iteration: 114 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.011813 ~ Iteration: 115 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.004324 ~ Iteration: 116 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.012155 ~ Iteration: 117 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.013049 ~ Iteration: 118 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.008994 ~ Iteration: 119 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.017232 ~ Iteration: 120 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.010607 ~ Iteration: 121 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.018 ~ stopping criterion: 0.013039 ~ Iteration: 122 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.014739 ~ Iteration: 123 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001723 ~ Iteration: 124 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.002370 ~ Iteration: 125 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.003934 ~ Iteration: 126 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.002732 ~ Iteration: 127 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002904 ~ Iteration: 128 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.010421 ~ Iteration: 129 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.003750 ~ Iteration: 130 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.003929 ~ Iteration: 131 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.004891 ~ Iteration: 132 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007942 ~ Iteration: 133 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.008102 ~ Iteration: 134 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.001128 ~ Iteration: 135 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.007495 ~ Iteration: 136 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.003812 ~ Iteration: 137 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.005350 ~ Iteration: 138 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.007249 ~ Iteration: 139 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.005889 ~ Iteration: 140 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.014089 ~ Iteration: 141 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.005270 ~ Iteration: 142 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.006283 ~ Iteration: 143 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.001944 ~ Iteration: 144 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.004854 ~ Iteration: 145 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.019 ~ stopping criterion: 0.005236 ~ Iteration: 146 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.009927 ~ Iteration: 147 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.004823 ~ Iteration: 148 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.005225 ~ Iteration: 149 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.018 ~ stopping criterion: 0.014587 ~ Iteration: 150 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.010834 ~ Iteration: 151 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.016 ~ stopping criterion: 0.007541 ~ Iteration: 152 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.009899 ~ Iteration: 153 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.009305 ~ Iteration: 154 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.007607 ~ Iteration: 155 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.002029 ~ Iteration: 156 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.015 ~ stopping criterion: 0.000442 ~ Iteration: 157 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.014383 ~ Iteration: 158 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.010393 ~ Iteration: 159 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.008472 ~ Iteration: 160 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.004447 ~ Iteration: 161 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.001473 ~ Iteration: 162 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.004006 ~ Iteration: 163 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.003272 ~ Iteration: 164 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.003315 ~ Iteration: 165 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.004796 ~ Iteration: 166 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.002243 ~ Iteration: 167 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.005328 ~ Iteration: 168 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.002214 ~ Iteration: 169 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.007727 ~ Iteration: 170 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000812 ~ Iteration: 171 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.000804 ~ Iteration: 172 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001110 ~ Iteration: 173 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001133 ~ Iteration: 174 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.002281 ~ Iteration: 175 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002250 ~ Iteration: 176 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.002285 ~ Iteration: 177 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000720 ~ Iteration: 178 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000696 ~ Iteration: 179 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000863 ~ Iteration: 180 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.000817 ~ Iteration: 181 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001208 ~ Iteration: 182 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001103 ~ Iteration: 183 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001104 ~ Iteration: 184 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001107 ~ Iteration: 185 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 186 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001148 ~ Iteration: 187 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001103 ~ Iteration: 188 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002278 ~ Iteration: 189 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001225 ~ Iteration: 190 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001226 ~ Iteration: 191 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 192 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001113 ~ Iteration: 193 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001129 ~ Iteration: 194 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001114 ~ Iteration: 195 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 196 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 197 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001156 ~ Iteration: 198 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001217 ~ Iteration: 199 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001387 ~ Iteration: 200 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002079 ~ Iteration: 201 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001350 ~ Iteration: 202 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 203 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001387 ~ Iteration: 204 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002079 ~ Iteration: 205 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 206 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 207 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001387 ~ Iteration: 208 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002539 ~ Iteration: 209 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001816 ~ Iteration: 210 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 211 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002539 ~ Iteration: 212 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 213 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002429 ~ Iteration: 214 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 215 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002429 ~ Iteration: 216 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002430 ~ Iteration: 217 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001789 ~ Iteration: 218 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002410 ~ Iteration: 219 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002410 ~ Iteration: 220 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 221 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 222 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 223 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002402 ~ Iteration: 224 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002402 ~ Iteration: 225 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002421 ~ Iteration: 226 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002421 ~ Iteration: 227 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 228 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 229 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 230 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002389 ~ Iteration: 231 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002389 ~ Iteration: 232 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002352 ~ Iteration: 233 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002352 ~ Iteration: 234 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002351 ~ Iteration: 235 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002245 ~ Iteration: 236 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002245 ~ Iteration: 237 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002255 ~ Iteration: 238 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002255 ~ Iteration: 239 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002256 ~ Iteration: 240 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 241 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 242 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 243 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 244 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 245 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 246 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 247 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 248 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 249 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 250 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 251 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 252 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 253 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 254 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 255 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 256 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 257 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002361 ~ Iteration: 258 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002418 ~ Iteration: 259 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002418 ~ Iteration: 260 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 261 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 262 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 263 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 264 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 265 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 266 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 267 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002485 ~ Iteration: 268 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 269 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 270 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 271 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002583 ~ Iteration: 272 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002583 ~ Iteration: 273 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002589 ~ Iteration: 274 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 275 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 276 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 277 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 278 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 279 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 280 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 281 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 282 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 283 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 284 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002537 ~ Iteration: 285 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002540 ~ Iteration: 286 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002540 ~ Iteration: 287 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002541 ~ Iteration: 288 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002439 ~ Iteration: 289 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002272 ~ Iteration: 290 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002272 ~ Iteration: 291 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002260 ~ Iteration: 292 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 293 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002289 ~ Iteration: 294 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002207 ~ Iteration: 295 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002186 ~ Iteration: 296 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002186 ~ Iteration: 297 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 298 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 299 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 300 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Stopped at: 300 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184
Now, let’s compare it to the previous example.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc3_fit.optimization_outputs['opt_alpha'], gc3_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (0.8741706474293448, 1.0888201148722707)
Look at the learning surface using draw_learning_curve()
method.
gc3_fit.draw_learning_curve()
We evaluate the performance of the method on the testing data using MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
= gc3_fit.predict(X_test1)
y_pred3 print(mean_absolute_percentage_error(y_test1, y_pred3))
print(mean_squared_error(y_test1, y_pred3))
0.1308662858088806
126.52951363322245
Let’s look at qq-plot
using draw_learning_curve()
method.
=y_test1, fig_type='qq') gc3_fit.draw_learning_curve(y_test
Real dataset
We look at the California housing dataset from sklearn.datasets
module. To illustrate the idea, we only work with the first \(1000\) observations.
from sklearn.datasets import fetch_california_housing
= fetch_california_housing()
data = data['data'], data['target']
X_real, y_real
= train_test_split(X_real[:1000,:], y_real[:1000], test_size=0.2)
X_train_real, X_test_real, y_train_real, y_test_real print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(X_train_real.shape, X_test_real.shape, y_train_real.shape, y_test_real.shape))
shape: x_train = (800, 8) , x_train = (200, 8) , y_train = (800,) , y_test = (200,)
We gave some random parameters to the method as follows.
= MixCOBRARegressor(opt_method="grad",
gc_real =['random_forest',
estimator_list'knn',
'ridge',
'lasso'],
={
estimator_params'random_forest' : {'n_estimators': 300},
'knn' : {'n_neighbors' : 30}
})= gc_real.fit(X_train_real, y_train_real) gc_real_fit
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: 0.001 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 2 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 3 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 4 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 5 ~ (alpha, beta): (5.003, 2.557) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000666 ~ Iteration: 6 ~ (alpha, beta): (5.003, 2.567) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000701 ~ Iteration: 7 ~ (alpha, beta): (5.004, 2.576) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000595 ~ Iteration: 8 ~ (alpha, beta): (5.003, 2.584) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000592 ~ Iteration: 9 ~ (alpha, beta): (5.003, 2.593) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000574 ~ Iteration: 10 ~ (alpha, beta): (5.005, 2.601) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000675 ~ Iteration: 11 ~ (alpha, beta): (5.005, 2.607) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000396 ~ Iteration: 12 ~ (alpha, beta): (5.005, 2.613) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000448 ~ Iteration: 13 ~ (alpha, beta): (5.004, 2.619) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000603 ~ Iteration: 14 ~ (alpha, beta): (5.004, 2.627) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000555 ~ Iteration: 15 ~ (alpha, beta): (5.003, 2.635) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000563 ~ Iteration: 16 ~ (alpha, beta): (5.004, 2.642) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000624 ~ Iteration: 17 ~ (alpha, beta): (5.004, 2.650) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000515 ~ Iteration: 18 ~ (alpha, beta): (5.004, 2.657) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000507 ~ Iteration: 19 ~ (alpha, beta): (5.004, 2.661) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000305 ~ Iteration: 20 ~ (alpha, beta): (5.003, 2.669) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000667 ~ Iteration: 21 ~ (alpha, beta): (5.003, 2.673) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000338 ~ Iteration: 22 ~ (alpha, beta): (5.003, 2.681) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000601 ~ Iteration: 23 ~ (alpha, beta): (5.003, 2.688) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000531 ~ Iteration: 24 ~ (alpha, beta): (5.003, 2.693) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000363 ~ Iteration: 25 ~ (alpha, beta): (5.003, 2.698) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000394 ~ Iteration: 26 ~ (alpha, beta): (5.003, 2.705) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000504 ~ Iteration: 27 ~ (alpha, beta): (5.003, 2.711) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000428 ~ Iteration: 28 ~ (alpha, beta): (5.003, 2.718) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000557 ~ Iteration: 29 ~ (alpha, beta): (5.003, 2.723) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000341 ~ Iteration: 30 ~ (alpha, beta): (5.004, 2.728) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000414 ~ Iteration: 31 ~ (alpha, beta): (5.004, 2.735) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000540 ~ Iteration: 32 ~ (alpha, beta): (5.004, 2.741) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000407 ~ Iteration: 33 ~ (alpha, beta): (5.004, 2.748) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000548 ~ Iteration: 34 ~ (alpha, beta): (5.004, 2.750) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000329 ~ Iteration: 35 ~ (alpha, beta): (5.004, 2.755) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000379 ~ Iteration: 36 ~ (alpha, beta): (5.004, 2.760) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000378 ~ Iteration: 37 ~ (alpha, beta): (5.004, 2.765) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000387 ~ Iteration: 38 ~ (alpha, beta): (5.004, 2.768) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000232 ~ Iteration: 39 ~ (alpha, beta): (5.004, 2.772) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000311 ~ Iteration: 40 ~ (alpha, beta): (5.004, 2.777) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000292 ~ Iteration: 41 ~ (alpha, beta): (5.004, 2.782) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000395 ~ Iteration: 42 ~ (alpha, beta): (5.004, 2.785) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000284 ~ Iteration: 43 ~ (alpha, beta): (5.004, 2.790) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000371 ~ Iteration: 44 ~ (alpha, beta): (5.004, 2.797) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000534 ~ Iteration: 45 ~ (alpha, beta): (5.004, 2.800) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000221 ~ Iteration: 46 ~ (alpha, beta): (5.004, 2.805) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000340 ~ Iteration: 47 ~ (alpha, beta): (5.004, 2.809) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000397 ~ Iteration: 48 ~ (alpha, beta): (5.004, 2.813) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000344 ~ Iteration: 49 ~ (alpha, beta): (5.004, 2.817) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000309 ~ Iteration: 50 ~ (alpha, beta): (5.004, 2.820) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000372 ~ Iteration: 51 ~ (alpha, beta): (5.004, 2.823) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000295 ~ Iteration: 52 ~ (alpha, beta): (5.004, 2.825) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000205 ~ Iteration: 53 ~ (alpha, beta): (5.004, 2.829) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000259 ~ Iteration: 54 ~ (alpha, beta): (5.004, 2.831) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000153 ~ Iteration: 55 ~ (alpha, beta): (5.004, 2.835) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000326 ~ Iteration: 56 ~ (alpha, beta): (5.004, 2.839) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000346 ~ Iteration: 57 ~ (alpha, beta): (5.004, 2.844) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000314 ~ Iteration: 58 ~ (alpha, beta): (5.004, 2.847) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000305 ~ Iteration: 59 ~ (alpha, beta): (5.004, 2.851) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000309 ~ Iteration: 60 ~ (alpha, beta): (5.004, 2.852) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000193 ~ Iteration: 61 ~ (alpha, beta): (5.004, 2.855) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 62 ~ (alpha, beta): (5.004, 2.859) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000340 ~ Iteration: 63 ~ (alpha, beta): (5.004, 2.861) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000178 ~ Iteration: 64 ~ (alpha, beta): (5.004, 2.865) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000320 ~ Iteration: 65 ~ (alpha, beta): (5.004, 2.867) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000187 ~ Iteration: 66 ~ (alpha, beta): (5.004, 2.869) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000232 ~ Iteration: 67 ~ (alpha, beta): (5.004, 2.870) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000261 ~ Iteration: 68 ~ (alpha, beta): (5.004, 2.873) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 69 ~ (alpha, beta): (5.004, 2.873) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000185 ~ Iteration: 70 ~ (alpha, beta): (5.004, 2.876) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000216 ~ Iteration: 71 ~ (alpha, beta): (5.004, 2.878) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000154 ~ Iteration: 72 ~ (alpha, beta): (5.004, 2.880) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000249 ~ Iteration: 73 ~ (alpha, beta): (5.004, 2.881) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000083 ~ Iteration: 74 ~ (alpha, beta): (5.004, 2.884) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000279 ~ Iteration: 75 ~ (alpha, beta): (5.004, 2.887) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000237 ~ Iteration: 76 ~ (alpha, beta): (5.004, 2.888) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000110 ~ Iteration: 77 ~ (alpha, beta): (5.004, 2.889) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000160 ~ Iteration: 78 ~ (alpha, beta): (5.004, 2.891) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000270 ~ Iteration: 79 ~ (alpha, beta): (5.004, 2.894) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 80 ~ (alpha, beta): (5.004, 2.896) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000252 ~ Iteration: 81 ~ (alpha, beta): (5.004, 2.900) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000337 ~ Iteration: 82 ~ (alpha, beta): (5.004, 2.904) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000268 ~ Iteration: 83 ~ (alpha, beta): (5.004, 2.906) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000242 ~ Iteration: 84 ~ (alpha, beta): (5.004, 2.909) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000246 ~ Iteration: 85 ~ (alpha, beta): (5.004, 2.910) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000167 ~ Iteration: 86 ~ (alpha, beta): (5.004, 2.911) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000086 ~ Iteration: 87 ~ (alpha, beta): (5.004, 2.913) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000204 ~ Iteration: 88 ~ (alpha, beta): (5.004, 2.916) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000187 ~ Iteration: 89 ~ (alpha, beta): (5.004, 2.916) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000105 ~ Iteration: 90 ~ (alpha, beta): (5.004, 2.919) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000225 ~ Iteration: 91 ~ (alpha, beta): (5.004, 2.921) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000171 ~ Iteration: 92 ~ (alpha, beta): (5.004, 2.923) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000198 ~ Iteration: 93 ~ (alpha, beta): (5.004, 2.927) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000374 ~ Iteration: 94 ~ (alpha, beta): (5.004, 2.928) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000039 ~ Iteration: 95 ~ (alpha, beta): (5.004, 2.928) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000066 ~ Iteration: 96 ~ (alpha, beta): (5.004, 2.930) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 97 ~ (alpha, beta): (5.004, 2.930) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000097 ~ Iteration: 98 ~ (alpha, beta): (5.004, 2.931) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000134 ~ Iteration: 99 ~ (alpha, beta): (5.004, 2.932) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000201 ~ Iteration: 100 ~ (alpha, beta): (5.004, 2.933) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000107 ~ Iteration: 101 ~ (alpha, beta): (5.004, 2.934) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000110 ~ Iteration: 102 ~ (alpha, beta): (5.004, 2.935) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000116 ~ Iteration: 103 ~ (alpha, beta): (5.004, 2.937) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000230 ~ Iteration: 104 ~ (alpha, beta): (5.004, 2.937) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000082 ~ Iteration: 105 ~ (alpha, beta): (5.004, 2.938) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000050 ~ Iteration: 106 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000256 ~ Iteration: 107 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000113 ~ Iteration: 108 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000153 ~ Iteration: 109 ~ (alpha, beta): (5.004, 2.941) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000301 ~ Iteration: 110 ~ (alpha, beta): (5.004, 2.941) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000088 ~ Iteration: 111 ~ (alpha, beta): (5.004, 2.942) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000119 ~ Iteration: 112 ~ (alpha, beta): (5.004, 2.943) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000280 ~ Iteration: 113 ~ (alpha, beta): (5.004, 2.944) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000136 ~ Iteration: 114 ~ (alpha, beta): (5.004, 2.945) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000185 ~ Iteration: 115 ~ (alpha, beta): (5.004, 2.945) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 116 ~ (alpha, beta): (5.004, 2.946) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 117 ~ (alpha, beta): (5.004, 2.947) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000194 ~ Iteration: 118 ~ (alpha, beta): (5.004, 2.948) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000129 ~ Iteration: 119 ~ (alpha, beta): (5.004, 2.948) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 120 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000149 ~ Iteration: 121 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000047 ~ Iteration: 122 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 123 ~ (alpha, beta): (5.004, 2.950) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000144 ~ Iteration: 124 ~ (alpha, beta): (5.004, 2.952) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000330 ~ Iteration: 125 ~ (alpha, beta): (5.004, 2.953) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000297 ~ Iteration: 126 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000123 ~ Iteration: 127 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000094 ~ Iteration: 128 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000113 ~ Iteration: 129 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000060 ~ Iteration: 130 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000158 ~ Iteration: 131 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000161 ~ Iteration: 132 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000177 ~ Iteration: 133 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000093 ~ Iteration: 134 ~ (alpha, beta): (5.004, 2.956) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000326 ~ Iteration: 135 ~ (alpha, beta): (5.004, 2.956) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000136 ~ Iteration: 136 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000104 ~ Iteration: 137 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000291 ~ Iteration: 138 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000126 ~ Iteration: 139 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000085 ~ Iteration: 140 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000164 ~ Iteration: 141 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000059 ~ Iteration: 142 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000096 ~ Iteration: 143 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000068 ~ Iteration: 144 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000195 ~ Iteration: 145 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000068 ~ Iteration: 146 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000110 ~ Iteration: 147 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000051 ~ Iteration: 148 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000200 ~ Iteration: 149 ~ (alpha, beta): (5.004, 2.960) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 150 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000267 ~ Iteration: 151 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000216 ~ Iteration: 152 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000074 ~ Iteration: 153 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000133 ~ Iteration: 154 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000238 ~ Iteration: 155 ~ (alpha, beta): (5.004, 2.962) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000433 ~ Iteration: 156 ~ (alpha, beta): (5.004, 2.962) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000193 ~ Iteration: 157 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000185 ~ Iteration: 158 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000146 ~ Iteration: 159 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000049 ~ Iteration: 160 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000220 ~ Iteration: 161 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000091 ~ Iteration: 162 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000040 ~ Iteration: 163 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 164 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000152 ~ Iteration: 165 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000117 ~ Iteration: 166 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 167 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000334 ~ Iteration: 168 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 169 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000126 ~ Iteration: 170 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000138 ~ Iteration: 171 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000082 ~ Iteration: 172 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000258 ~ Iteration: 173 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000162 ~ Iteration: 174 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000231 ~ Iteration: 175 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000054 ~ Iteration: 176 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000066 ~ Iteration: 177 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000096 ~ Iteration: 178 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000075 ~ Iteration: 179 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000163 ~ Iteration: 180 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000087 ~ Iteration: 181 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000149 ~ Iteration: 182 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000047 ~ Iteration: 183 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000126 ~ Iteration: 184 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000220 ~ Iteration: 185 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000127 ~ Iteration: 186 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 187 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000284 ~ Iteration: 188 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000276 ~ Iteration: 189 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 190 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000047 ~ Iteration: 191 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000114 ~ Iteration: 192 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000193 ~ Iteration: 193 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000068 ~ Iteration: 194 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000037 ~ Iteration: 195 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000314 ~ Iteration: 196 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000062 ~ Iteration: 197 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000053 ~ Iteration: 198 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000425 ~ Iteration: 199 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000262 ~ Iteration: 200 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000126 ~ Iteration: 201 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000094 ~ Iteration: 202 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000206 ~ Iteration: 203 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000130 ~ Iteration: 204 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000111 ~ Iteration: 205 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000333 ~ Iteration: 206 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000206 ~ Iteration: 207 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000251 ~ Iteration: 208 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000129 ~ Iteration: 209 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 210 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000222 ~ Iteration: 211 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000122 ~ Iteration: 212 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000163 ~ Iteration: 213 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000040 ~ Iteration: 214 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000089 ~ Iteration: 215 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000076 ~ Iteration: 216 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000115 ~ Iteration: 217 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000183 ~ Iteration: 218 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000144 ~ Iteration: 219 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 220 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 221 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000055 ~ Iteration: 222 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000119 ~ Iteration: 223 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000119 ~ Iteration: 224 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000036 ~ Iteration: 225 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000031 ~ Iteration: 226 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000164 ~ Iteration: 227 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000139 ~ Iteration: 228 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000160 ~ Iteration: 229 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000182 ~ Iteration: 230 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000093 ~ Iteration: 231 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000255 ~ Iteration: 232 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000254 ~ Iteration: 233 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000117 ~ Iteration: 234 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000076 ~ Iteration: 235 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000165 ~ Iteration: 236 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000130 ~ Iteration: 237 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000214 ~ Iteration: 238 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000032 ~ Iteration: 239 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000083 ~ Iteration: 240 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000056 ~ Iteration: 241 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000186 ~ Iteration: 242 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000022 ~ Iteration: 243 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000089 ~ Iteration: 244 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000093 ~ Iteration: 245 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 246 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000036 ~ Iteration: 247 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000034 ~ Iteration: 248 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000026 ~ Iteration: 249 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000026 ~ Iteration: 250 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000031 ~ Iteration: 251 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000030 ~ Iteration: 252 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000026 ~ Iteration: 253 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 254 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000027 ~ Iteration: 255 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 256 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000030 ~ Iteration: 257 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 258 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000016 ~ Iteration: 259 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000017 ~ Iteration: 260 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000016 ~ Iteration: 261 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 262 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 263 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 264 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 265 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 266 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000023 ~ Iteration: 267 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 268 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000017 ~ Iteration: 269 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 270 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000012 ~ Iteration: 271 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 272 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000012 ~ Iteration: 273 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 274 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 275 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 276 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 277 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 278 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 279 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 280 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 281 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 282 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 283 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 284 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 285 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 286 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 287 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 288 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 289 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 290 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 291 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 292 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 293 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 294 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 295 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 296 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 297 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 298 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 299 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 300 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Stopped at: 300 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010
Now, let’s look at the obtained bandwidth and the optimization result.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc_real_fit.optimization_outputs['opt_alpha'], gc_real_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (5.003947996785071, 2.9646284687052478)
We look at the numerical and graphical performance.
= gc_real_fit.predict(X_test_real)
y_pred_real print(mean_absolute_percentage_error(y_test_real, y_pred_real))
print(mean_squared_error(y_test_real, y_pred_real))
=y_test_real, fig_type='qq') gc_real_fit.draw_learning_curve(y_test
0.1533765484019444
0.16456235739517955
Compare to Adaboost
We campare the fitted method on California
data with Adaboost method.
from sklearn.ensemble import AdaBoostRegressor
= AdaBoostRegressor(n_estimators=1000)
ada = ada.fit(X_train_real, y_train_real)
ada_fit = ada_fit.predict(X_test_real)
ada_pred print(mean_absolute_percentage_error(y_test_real, ada_pred))
print(mean_squared_error(y_test_real, ada_pred))
0.2723746701801255
0.25246896498101007
Pretrained basic estimators
You can also train other models then aggregate them using our method. Here, we build pretrained estimators including XGBoost
then aggregate it to some sklearn
basic estimators in MixCOBRARegressor
.
We first split the training data into two parts: \(X_k\) for building basic estimators, and \(X_\ell\) for aggregation. We use the constructed estimators to predict the test data and both the input \(X_{\text{test}}\) and its predictions pred_feature_test are used in the final predictions.
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
# X_k and X_l split
= np.random.permutation(range(len(y_train_real)))
id_k =int(.5 * len(y_train_real))
k = X_train_real[id_k[:k],:], X_train_real[id_k[k:],:], y_train_real[id_k[:k]], y_train_real[id_k[k:]]
X_k, X_l, y_k, y_l
# Building basic estiators
= RandomForestRegressor(n_estimators=300).fit(X_k, y_k)
rf_real = LinearRegression().fit(X_k, y_k)
lm_real = KNeighborsRegressor(n_neighbors=10).fit(X_k, y_k)
knn_real = DecisionTreeRegressor(min_samples_leaf=5).fit(X_k, y_k)
tr_real
# External XGBoost estiator
import xgboost
= xgboost.XGBRegressor(n_estimators = 500)
xgb = xgb.fit(X_k, y_k)
xgb_real
# All pretrained estimators
= (rf_real, lm_real, knn_real, tr_real, xgb_real)
basic_estimators
# Predicted features on X_l for aggregation
= np.column_stack([est.predict(X_l) for est in basic_estimators])
pred_feature_l
# Predicted features on Testing data
= np.column_stack([est.predict(X_test_real) for est in basic_estimators]) pred_feature_test
To fit the aggregation method on predicted features, we need to provide both input and predicted features to fit
method.
= MixCOBRARegressor(opt_method='grad').fit(X=X_l,
gc4_fit =pred_feature_l,
Pred_features=y_l) y
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 2 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 3 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 4 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 5 ~ (alpha, beta): (4.992, 3.777) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000029 ~ Iteration: 6 ~ (alpha, beta): (5.013, 3.820) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000185 ~ Iteration: 7 ~ (alpha, beta): (5.030, 3.840) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000118 ~ Iteration: 8 ~ (alpha, beta): (5.026, 3.868) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000092 ~ Iteration: 9 ~ (alpha, beta): (5.012, 3.894) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000140 ~ Iteration: 10 ~ (alpha, beta): (5.052, 3.936) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000297 ~ Iteration: 11 ~ (alpha, beta): (5.038, 3.999) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000260 ~ Iteration: 12 ~ (alpha, beta): (5.034, 4.032) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000117 ~ Iteration: 13 ~ (alpha, beta): (5.034, 4.031) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000008 ~ Stopped at: 13 ~ (alpha, beta): (5.034, 4.031) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000008
We look at the optimization algorithm performance.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc4_fit.optimization_outputs['opt_alpha'], gc4_fit.optimization_outputs['opt_beta']))
Estimated (alpha, beta) = (5.033650165115184, 4.030778809061925)
Now, let’s look at the result.
gc4_fit.draw_learning_curve()= gc4_fit.predict(X = X_test_real,
pred_add = pred_feature_test)
Pred_X print(mean_absolute_percentage_error(y_test_real, pred_add))
print(mean_squared_error(y_test_real, pred_add))
0.14222323551671262
0.15583368476512235