# Metric of error
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_absolute_percentage_error
# Plotting figures
import matplotlib.pyplot as plt
from matplotlib import cm
# Import class MixCOBRA from the mixcobra module of gradientcobra library
from gradientcobra.mixcobra import MixCOBRARegressor
import seaborn as sns
sns.set()MixCOBRARegressor of gradientcobra v1.0.8 package
Installing and importing packages
gradientcobra can be installed from pypi using pip:
pip install gradientcobra
Importing packages
Simulated data
We simulate a regression data with \(1000\) observations and \(10\) inputs variables.
# For simulating dataset
from sklearn.datasets import make_regression
X1, y1 = make_regression(n_samples=1000, n_features=10, noise=1)Now, let’s randoly split the simulated data into \(80\%-20\%\) training-testing data.
from sklearn.model_selection import train_test_split
X_train1, X_test1, y_train1, y_test1 = train_test_split(X1, y1, test_size=0.2)
print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(
X_train1.shape,
X_test1.shape,
y_train1.shape,
y_test1.shape))shape: x_train = (800, 10) , x_train = (200, 10) , y_train = (800,) , y_test = (200,)MixCOBRARegressor with default parameters
We create MixCOBRARegressor object called gc1, with the default parameters, then fit it to the training data.
Note that by default, we use grid search algorithm to estimate the optimal parameter of the method.
gc1 = MixCOBRARegressor()
gc1_fit = gc1.fit(X_train1, y_train1)
* Grid search algorithm of two parameters with radial kernel is in progress...
~ Full process|--------------------------------------------------|100%
~ Processing|==================================================|100%The estimated optimal bandwidth is given by gc1.optimization_outputs['opt_bandwidth'].
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc1_fit.optimization_outputs['opt_alpha'], gc1_fit.optimization_outputs['opt_beta']))Estimated (alpha, beta) = (1e-05, 0.7575842424242424)We can look at the learning curve (surface) of the algorithm using draw_learning_curve() method.
gc1_fit.draw_learning_curve()We evaluate the performance of the method on the testing data using MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
y_pred1 = gc1_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred1))
print(mean_squared_error(y_test1, y_pred1))0.16552630550806227
150.23447041060408Let’s look at qq-plot of the predictions and the actual response values using draw_learning_curve() method.
gc1_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')Use one bandwidth parameter instead of (\(\alpha, \beta\))
You can also try to implement MixCOBRARegressor using only one bandwidth parameter. To do so, just set one_parameter = True when fitting the method. We create another MixCOBRARegressor object with gradient descent optimization method, then fit it on the same data as in the previous case using only one bandwidth parameter as follow.
# MixCOBRA with one parameter
gc2 = MixCOBRARegressor(opt_method="grad")
gc2_fit = gc2.fit(X_train1, y_train1, one_parameter=True)
* Gradient descent of one parameter with radial kernel is implemented...
~ Initial t = 0: ~ bandwidth: 2.507 ~ gradient: 33.439 ~ threshold: 1e-05
~ Iteration: 1 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 2 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 3 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 4 ~ bandwidth: 2.498 ~ gradient: 33.439 ~ stopping criterion: 16.721 ~ Iteration: 5 ~ bandwidth: 2.498 ~ gradient: 32.570 ~ stopping criterion: 16.719 ~ Iteration: 6 ~ bandwidth: 2.488 ~ gradient: 31.760 ~ stopping criterion: 16.285 ~ Iteration: 7 ~ bandwidth: 2.478 ~ gradient: 33.054 ~ stopping criterion: 15.880 ~ Iteration: 8 ~ bandwidth: 2.468 ~ gradient: 32.551 ~ stopping criterion: 16.527 ~ Iteration: 9 ~ bandwidth: 2.459 ~ gradient: 32.933 ~ stopping criterion: 16.276 ~ Iteration: 10 ~ bandwidth: 2.449 ~ gradient: 31.902 ~ stopping criterion: 16.467 ~ Iteration: 11 ~ bandwidth: 2.439 ~ gradient: 32.765 ~ stopping criterion: 15.951 ~ Iteration: 12 ~ bandwidth: 2.429 ~ gradient: 30.846 ~ stopping criterion: 16.382 ~ Iteration: 13 ~ bandwidth: 2.420 ~ gradient: 33.097 ~ stopping criterion: 15.423 ~ Iteration: 14 ~ bandwidth: 2.410 ~ gradient: 32.646 ~ stopping criterion: 16.549 ~ Iteration: 15 ~ bandwidth: 2.401 ~ gradient: 32.505 ~ stopping criterion: 16.323 ~ Iteration: 16 ~ bandwidth: 2.391 ~ gradient: 33.117 ~ stopping criterion: 16.252 ~ Iteration: 17 ~ bandwidth: 2.381 ~ gradient: 34.084 ~ stopping criterion: 16.559 ~ Iteration: 18 ~ bandwidth: 2.371 ~ gradient: 34.117 ~ stopping criterion: 17.042 ~ Iteration: 19 ~ bandwidth: 2.361 ~ gradient: 34.410 ~ stopping criterion: 17.058 ~ Iteration: 20 ~ bandwidth: 2.350 ~ gradient: 33.783 ~ stopping criterion: 17.205 ~ Iteration: 21 ~ bandwidth: 2.340 ~ gradient: 33.098 ~ stopping criterion: 16.891 ~ Iteration: 22 ~ bandwidth: 2.330 ~ gradient: 33.975 ~ stopping criterion: 16.549 ~ Iteration: 23 ~ bandwidth: 2.320 ~ gradient: 34.338 ~ stopping criterion: 16.987 ~ Iteration: 24 ~ bandwidth: 2.310 ~ gradient: 35.558 ~ stopping criterion: 17.169 ~ Iteration: 25 ~ bandwidth: 2.299 ~ gradient: 34.418 ~ stopping criterion: 17.779 ~ Iteration: 26 ~ bandwidth: 2.289 ~ gradient: 33.465 ~ stopping criterion: 17.209 ~ Iteration: 27 ~ bandwidth: 2.279 ~ gradient: 32.339 ~ stopping criterion: 16.733 ~ Iteration: 28 ~ bandwidth: 2.269 ~ gradient: 33.435 ~ stopping criterion: 16.169 ~ Iteration: 29 ~ bandwidth: 2.259 ~ gradient: 33.759 ~ stopping criterion: 16.717 ~ Iteration: 30 ~ bandwidth: 2.249 ~ gradient: 34.556 ~ stopping criterion: 16.879 ~ Iteration: 31 ~ bandwidth: 2.239 ~ gradient: 34.685 ~ stopping criterion: 17.278 ~ Iteration: 32 ~ bandwidth: 2.228 ~ gradient: 35.063 ~ stopping criterion: 17.343 ~ Iteration: 33 ~ bandwidth: 2.218 ~ gradient: 36.170 ~ stopping criterion: 17.532 ~ Iteration: 34 ~ bandwidth: 2.207 ~ gradient: 33.235 ~ stopping criterion: 18.085 ~ Iteration: 35 ~ bandwidth: 2.197 ~ gradient: 35.208 ~ stopping criterion: 16.618 ~ Iteration: 36 ~ bandwidth: 2.187 ~ gradient: 37.075 ~ stopping criterion: 17.604 ~ Iteration: 37 ~ bandwidth: 2.176 ~ gradient: 35.903 ~ stopping criterion: 18.537 ~ Iteration: 38 ~ bandwidth: 2.165 ~ gradient: 34.320 ~ stopping criterion: 17.952 ~ Iteration: 39 ~ bandwidth: 2.155 ~ gradient: 35.160 ~ stopping criterion: 17.160 ~ Iteration: 40 ~ bandwidth: 2.144 ~ gradient: 35.873 ~ stopping criterion: 17.580 ~ Iteration: 41 ~ bandwidth: 2.133 ~ gradient: 35.583 ~ stopping criterion: 17.937 ~ Iteration: 42 ~ bandwidth: 2.123 ~ gradient: 35.432 ~ stopping criterion: 17.792 ~ Iteration: 43 ~ bandwidth: 2.112 ~ gradient: 36.362 ~ stopping criterion: 17.716 ~ Iteration: 44 ~ bandwidth: 2.101 ~ gradient: 36.246 ~ stopping criterion: 18.181 ~ Iteration: 45 ~ bandwidth: 2.090 ~ gradient: 36.555 ~ stopping criterion: 18.123 ~ Iteration: 46 ~ bandwidth: 2.079 ~ gradient: 36.160 ~ stopping criterion: 18.278 ~ Iteration: 47 ~ bandwidth: 2.069 ~ gradient: 36.117 ~ stopping criterion: 18.080 ~ Iteration: 48 ~ bandwidth: 2.058 ~ gradient: 36.913 ~ stopping criterion: 18.059 ~ Iteration: 49 ~ bandwidth: 2.047 ~ gradient: 37.435 ~ stopping criterion: 18.457 ~ Iteration: 50 ~ bandwidth: 2.036 ~ gradient: 36.893 ~ stopping criterion: 18.718 ~ Iteration: 51 ~ bandwidth: 2.025 ~ gradient: 34.964 ~ stopping criterion: 18.446 ~ Iteration: 52 ~ bandwidth: 2.014 ~ gradient: 35.160 ~ stopping criterion: 17.482 ~ Iteration: 53 ~ bandwidth: 2.004 ~ gradient: 37.250 ~ stopping criterion: 17.580 ~ Iteration: 54 ~ bandwidth: 1.992 ~ gradient: 37.434 ~ stopping criterion: 18.625 ~ Iteration: 55 ~ bandwidth: 1.981 ~ gradient: 36.231 ~ stopping criterion: 18.717 ~ Iteration: 56 ~ bandwidth: 1.970 ~ gradient: 36.648 ~ stopping criterion: 18.115 ~ Iteration: 57 ~ bandwidth: 1.959 ~ gradient: 37.566 ~ stopping criterion: 18.324 ~ Iteration: 58 ~ bandwidth: 1.948 ~ gradient: 37.458 ~ stopping criterion: 18.783 ~ Iteration: 59 ~ bandwidth: 1.937 ~ gradient: 37.723 ~ stopping criterion: 18.729 ~ Iteration: 60 ~ bandwidth: 1.926 ~ gradient: 37.482 ~ stopping criterion: 18.861 ~ Iteration: 61 ~ bandwidth: 1.915 ~ gradient: 37.073 ~ stopping criterion: 18.741 ~ Iteration: 62 ~ bandwidth: 1.903 ~ gradient: 37.777 ~ stopping criterion: 18.537 ~ Iteration: 63 ~ bandwidth: 1.892 ~ gradient: 37.617 ~ stopping criterion: 18.888 ~ Iteration: 64 ~ bandwidth: 1.881 ~ gradient: 36.882 ~ stopping criterion: 18.809 ~ Iteration: 65 ~ bandwidth: 1.870 ~ gradient: 38.111 ~ stopping criterion: 18.441 ~ Iteration: 66 ~ bandwidth: 1.858 ~ gradient: 36.905 ~ stopping criterion: 19.055 ~ Iteration: 67 ~ bandwidth: 1.847 ~ gradient: 38.639 ~ stopping criterion: 18.453 ~ Iteration: 68 ~ bandwidth: 1.836 ~ gradient: 37.235 ~ stopping criterion: 19.320 ~ Iteration: 69 ~ bandwidth: 1.825 ~ gradient: 37.896 ~ stopping criterion: 18.618 ~ Iteration: 70 ~ bandwidth: 1.813 ~ gradient: 38.226 ~ stopping criterion: 18.948 ~ Iteration: 71 ~ bandwidth: 1.802 ~ gradient: 39.661 ~ stopping criterion: 19.113 ~ Iteration: 72 ~ bandwidth: 1.790 ~ gradient: 38.067 ~ stopping criterion: 19.830 ~ Iteration: 73 ~ bandwidth: 1.779 ~ gradient: 38.549 ~ stopping criterion: 19.033 ~ Iteration: 74 ~ bandwidth: 1.767 ~ gradient: 38.525 ~ stopping criterion: 19.274 ~ Iteration: 75 ~ bandwidth: 1.756 ~ gradient: 37.115 ~ stopping criterion: 19.262 ~ Iteration: 76 ~ bandwidth: 1.745 ~ gradient: 38.392 ~ stopping criterion: 18.558 ~ Iteration: 77 ~ bandwidth: 1.733 ~ gradient: 38.877 ~ stopping criterion: 19.196 ~ Iteration: 78 ~ bandwidth: 1.721 ~ gradient: 39.536 ~ stopping criterion: 19.439 ~ Iteration: 79 ~ bandwidth: 1.710 ~ gradient: 38.347 ~ stopping criterion: 19.768 ~ Iteration: 80 ~ bandwidth: 1.698 ~ gradient: 37.458 ~ stopping criterion: 19.173 ~ Iteration: 81 ~ bandwidth: 1.687 ~ gradient: 38.791 ~ stopping criterion: 18.729 ~ Iteration: 82 ~ bandwidth: 1.675 ~ gradient: 38.535 ~ stopping criterion: 19.395 ~ Iteration: 83 ~ bandwidth: 1.664 ~ gradient: 38.680 ~ stopping criterion: 19.267 ~ Iteration: 84 ~ bandwidth: 1.652 ~ gradient: 38.911 ~ stopping criterion: 19.340 ~ Iteration: 85 ~ bandwidth: 1.641 ~ gradient: 39.169 ~ stopping criterion: 19.456 ~ Iteration: 86 ~ bandwidth: 1.629 ~ gradient: 39.254 ~ stopping criterion: 19.584 ~ Iteration: 87 ~ bandwidth: 1.617 ~ gradient: 38.863 ~ stopping criterion: 19.627 ~ Iteration: 88 ~ bandwidth: 1.606 ~ gradient: 38.706 ~ stopping criterion: 19.432 ~ Iteration: 89 ~ bandwidth: 1.594 ~ gradient: 38.867 ~ stopping criterion: 19.353 ~ Iteration: 90 ~ bandwidth: 1.582 ~ gradient: 38.863 ~ stopping criterion: 19.434 ~ Iteration: 91 ~ bandwidth: 1.571 ~ gradient: 40.212 ~ stopping criterion: 19.432 ~ Iteration: 92 ~ bandwidth: 1.559 ~ gradient: 40.882 ~ stopping criterion: 20.106 ~ Iteration: 93 ~ bandwidth: 1.547 ~ gradient: 39.742 ~ stopping criterion: 20.441 ~ Iteration: 94 ~ bandwidth: 1.535 ~ gradient: 40.603 ~ stopping criterion: 19.871 ~ Iteration: 95 ~ bandwidth: 1.522 ~ gradient: 38.723 ~ stopping criterion: 20.302 ~ Iteration: 96 ~ bandwidth: 1.511 ~ gradient: 39.437 ~ stopping criterion: 19.361 ~ Iteration: 97 ~ bandwidth: 1.499 ~ gradient: 39.963 ~ stopping criterion: 19.719 ~ Iteration: 98 ~ bandwidth: 1.487 ~ gradient: 40.406 ~ stopping criterion: 19.981 ~ Iteration: 99 ~ bandwidth: 1.475 ~ gradient: 40.569 ~ stopping criterion: 20.203 ~ Iteration: 100 ~ bandwidth: 1.463 ~ gradient: 41.067 ~ stopping criterion: 20.285 ~ Iteration: 101 ~ bandwidth: 1.451 ~ gradient: 40.289 ~ stopping criterion: 20.533 ~ Iteration: 102 ~ bandwidth: 1.439 ~ gradient: 39.731 ~ stopping criterion: 20.144 ~ Iteration: 103 ~ bandwidth: 1.427 ~ gradient: 41.624 ~ stopping criterion: 19.865 ~ Iteration: 104 ~ bandwidth: 1.414 ~ gradient: 39.657 ~ stopping criterion: 20.812 ~ Iteration: 105 ~ bandwidth: 1.402 ~ gradient: 40.531 ~ stopping criterion: 19.829 ~ Iteration: 106 ~ bandwidth: 1.390 ~ gradient: 41.455 ~ stopping criterion: 20.266 ~ Iteration: 107 ~ bandwidth: 1.378 ~ gradient: 40.318 ~ stopping criterion: 20.727 ~ Iteration: 108 ~ bandwidth: 1.366 ~ gradient: 39.433 ~ stopping criterion: 20.159 ~ Iteration: 109 ~ bandwidth: 1.354 ~ gradient: 41.338 ~ stopping criterion: 19.716 ~ Iteration: 110 ~ bandwidth: 1.342 ~ gradient: 41.455 ~ stopping criterion: 20.669 ~ Iteration: 111 ~ bandwidth: 1.329 ~ gradient: 40.924 ~ stopping criterion: 20.727 ~ Iteration: 112 ~ bandwidth: 1.317 ~ gradient: 40.149 ~ stopping criterion: 20.462 ~ Iteration: 113 ~ bandwidth: 1.305 ~ gradient: 40.132 ~ stopping criterion: 20.074 ~ Iteration: 114 ~ bandwidth: 1.293 ~ gradient: 39.529 ~ stopping criterion: 20.066 ~ Iteration: 115 ~ bandwidth: 1.281 ~ gradient: 39.958 ~ stopping criterion: 19.765 ~ Iteration: 116 ~ bandwidth: 1.269 ~ gradient: 40.801 ~ stopping criterion: 19.979 ~ Iteration: 117 ~ bandwidth: 1.257 ~ gradient: 39.030 ~ stopping criterion: 20.401 ~ Iteration: 118 ~ bandwidth: 1.245 ~ gradient: 39.225 ~ stopping criterion: 19.515 ~ Iteration: 119 ~ bandwidth: 1.234 ~ gradient: 40.178 ~ stopping criterion: 19.612 ~ Iteration: 120 ~ bandwidth: 1.222 ~ gradient: 40.319 ~ stopping criterion: 20.089 ~ Iteration: 121 ~ bandwidth: 1.210 ~ gradient: 40.496 ~ stopping criterion: 20.160 ~ Iteration: 122 ~ bandwidth: 1.197 ~ gradient: 39.627 ~ stopping criterion: 20.248 ~ Iteration: 123 ~ bandwidth: 1.186 ~ gradient: 39.677 ~ stopping criterion: 19.813 ~ Iteration: 124 ~ bandwidth: 1.174 ~ gradient: 39.699 ~ stopping criterion: 19.838 ~ Iteration: 125 ~ bandwidth: 1.162 ~ gradient: 38.777 ~ stopping criterion: 19.850 ~ Iteration: 126 ~ bandwidth: 1.150 ~ gradient: 39.688 ~ stopping criterion: 19.388 ~ Iteration: 127 ~ bandwidth: 1.138 ~ gradient: 38.588 ~ stopping criterion: 19.844 ~ Iteration: 128 ~ bandwidth: 1.127 ~ gradient: 39.553 ~ stopping criterion: 19.294 ~ Iteration: 129 ~ bandwidth: 1.115 ~ gradient: 36.344 ~ stopping criterion: 19.777 ~ Iteration: 130 ~ bandwidth: 1.104 ~ gradient: 38.735 ~ stopping criterion: 18.172 ~ Iteration: 131 ~ bandwidth: 1.093 ~ gradient: 36.876 ~ stopping criterion: 19.367 ~ Iteration: 132 ~ bandwidth: 1.082 ~ gradient: 38.788 ~ stopping criterion: 18.438 ~ Iteration: 133 ~ bandwidth: 1.070 ~ gradient: 38.341 ~ stopping criterion: 19.394 ~ Iteration: 134 ~ bandwidth: 1.059 ~ gradient: 37.313 ~ stopping criterion: 19.171 ~ Iteration: 135 ~ bandwidth: 1.047 ~ gradient: 37.391 ~ stopping criterion: 18.656 ~ Iteration: 136 ~ bandwidth: 1.036 ~ gradient: 36.883 ~ stopping criterion: 18.696 ~ Iteration: 137 ~ bandwidth: 1.025 ~ gradient: 35.974 ~ stopping criterion: 18.442 ~ Iteration: 138 ~ bandwidth: 1.014 ~ gradient: 35.982 ~ stopping criterion: 17.987 ~ Iteration: 139 ~ bandwidth: 1.004 ~ gradient: 37.327 ~ stopping criterion: 17.991 ~ Iteration: 140 ~ bandwidth: 0.992 ~ gradient: 34.022 ~ stopping criterion: 18.663 ~ Iteration: 141 ~ bandwidth: 0.982 ~ gradient: 35.532 ~ stopping criterion: 17.011 ~ Iteration: 142 ~ bandwidth: 0.972 ~ gradient: 35.211 ~ stopping criterion: 17.766 ~ Iteration: 143 ~ bandwidth: 0.961 ~ gradient: 32.485 ~ stopping criterion: 17.605 ~ Iteration: 144 ~ bandwidth: 0.951 ~ gradient: 32.277 ~ stopping criterion: 16.242 ~ Iteration: 145 ~ bandwidth: 0.942 ~ gradient: 33.378 ~ stopping criterion: 16.138 ~ Iteration: 146 ~ bandwidth: 0.932 ~ gradient: 32.958 ~ stopping criterion: 16.689 ~ Iteration: 147 ~ bandwidth: 0.922 ~ gradient: 32.462 ~ stopping criterion: 16.479 ~ Iteration: 148 ~ bandwidth: 0.912 ~ gradient: 30.227 ~ stopping criterion: 16.231 ~ Iteration: 149 ~ bandwidth: 0.903 ~ gradient: 30.399 ~ stopping criterion: 15.113 ~ Iteration: 150 ~ bandwidth: 0.894 ~ gradient: 28.374 ~ stopping criterion: 15.199 ~ Iteration: 151 ~ bandwidth: 0.886 ~ gradient: 28.302 ~ stopping criterion: 14.187 ~ Iteration: 152 ~ bandwidth: 0.877 ~ gradient: 27.961 ~ stopping criterion: 14.151 ~ Iteration: 153 ~ bandwidth: 0.869 ~ gradient: 26.522 ~ stopping criterion: 13.981 ~ Iteration: 154 ~ bandwidth: 0.861 ~ gradient: 26.798 ~ stopping criterion: 13.261 ~ Iteration: 155 ~ bandwidth: 0.853 ~ gradient: 25.203 ~ stopping criterion: 13.399 ~ Iteration: 156 ~ bandwidth: 0.845 ~ gradient: 24.713 ~ stopping criterion: 12.601 ~ Iteration: 157 ~ bandwidth: 0.838 ~ gradient: 23.031 ~ stopping criterion: 12.356 ~ Iteration: 158 ~ bandwidth: 0.831 ~ gradient: 22.019 ~ stopping criterion: 11.515 ~ Iteration: 159 ~ bandwidth: 0.824 ~ gradient: 21.954 ~ stopping criterion: 11.010 ~ Iteration: 160 ~ bandwidth: 0.818 ~ gradient: 21.373 ~ stopping criterion: 10.977 ~ Iteration: 161 ~ bandwidth: 0.811 ~ gradient: 22.419 ~ stopping criterion: 10.686 ~ Iteration: 162 ~ bandwidth: 0.805 ~ gradient: 18.759 ~ stopping criterion: 11.209 ~ Iteration: 163 ~ bandwidth: 0.799 ~ gradient: 19.891 ~ stopping criterion: 9.380 ~ Iteration: 164 ~ bandwidth: 0.793 ~ gradient: 18.285 ~ stopping criterion: 9.945 ~ Iteration: 165 ~ bandwidth: 0.788 ~ gradient: 17.670 ~ stopping criterion: 9.143 ~ Iteration: 166 ~ bandwidth: 0.782 ~ gradient: 15.641 ~ stopping criterion: 8.835 ~ Iteration: 167 ~ bandwidth: 0.778 ~ gradient: 15.377 ~ stopping criterion: 7.821 ~ Iteration: 168 ~ bandwidth: 0.773 ~ gradient: 13.831 ~ stopping criterion: 7.688 ~ Iteration: 169 ~ bandwidth: 0.769 ~ gradient: 14.703 ~ stopping criterion: 6.915 ~ Iteration: 170 ~ bandwidth: 0.765 ~ gradient: 15.614 ~ stopping criterion: 7.351 ~ Iteration: 171 ~ bandwidth: 0.760 ~ gradient: 12.007 ~ stopping criterion: 7.807 ~ Iteration: 172 ~ bandwidth: 0.756 ~ gradient: 13.628 ~ stopping criterion: 6.004 ~ Iteration: 173 ~ bandwidth: 0.752 ~ gradient: 11.557 ~ stopping criterion: 6.814 ~ Iteration: 174 ~ bandwidth: 0.749 ~ gradient: 9.434 ~ stopping criterion: 5.779 ~ Iteration: 175 ~ bandwidth: 0.746 ~ gradient: 8.647 ~ stopping criterion: 4.717 ~ Iteration: 176 ~ bandwidth: 0.743 ~ gradient: 8.527 ~ stopping criterion: 4.324 ~ Iteration: 177 ~ bandwidth: 0.741 ~ gradient: 8.860 ~ stopping criterion: 4.263 ~ Iteration: 178 ~ bandwidth: 0.738 ~ gradient: 8.644 ~ stopping criterion: 4.430 ~ Iteration: 179 ~ bandwidth: 0.736 ~ gradient: 8.655 ~ stopping criterion: 4.322 ~ Iteration: 180 ~ bandwidth: 0.733 ~ gradient: 6.855 ~ stopping criterion: 4.328 ~ Iteration: 181 ~ bandwidth: 0.731 ~ gradient: 7.078 ~ stopping criterion: 3.427 ~ Iteration: 182 ~ bandwidth: 0.729 ~ gradient: 8.211 ~ stopping criterion: 3.539 ~ Iteration: 183 ~ bandwidth: 0.726 ~ gradient: 7.145 ~ stopping criterion: 4.105 ~ Iteration: 184 ~ bandwidth: 0.724 ~ gradient: 5.995 ~ stopping criterion: 3.573 ~ Iteration: 185 ~ bandwidth: 0.723 ~ gradient: 5.043 ~ stopping criterion: 2.998 ~ Iteration: 186 ~ bandwidth: 0.721 ~ gradient: 4.909 ~ stopping criterion: 2.521 ~ Iteration: 187 ~ bandwidth: 0.720 ~ gradient: 5.327 ~ stopping criterion: 2.455 ~ Iteration: 188 ~ bandwidth: 0.718 ~ gradient: 2.895 ~ stopping criterion: 2.663 ~ Iteration: 189 ~ bandwidth: 0.717 ~ gradient: 3.976 ~ stopping criterion: 1.447 ~ Iteration: 190 ~ bandwidth: 0.716 ~ gradient: 3.597 ~ stopping criterion: 1.988 ~ Iteration: 191 ~ bandwidth: 0.715 ~ gradient: 3.651 ~ stopping criterion: 1.799 ~ Iteration: 192 ~ bandwidth: 0.714 ~ gradient: 4.651 ~ stopping criterion: 1.826 ~ Iteration: 193 ~ bandwidth: 0.712 ~ gradient: 4.288 ~ stopping criterion: 2.326 ~ Iteration: 194 ~ bandwidth: 0.711 ~ gradient: 3.022 ~ stopping criterion: 2.144 ~ Iteration: 195 ~ bandwidth: 0.710 ~ gradient: 3.597 ~ stopping criterion: 1.511 ~ Iteration: 196 ~ bandwidth: 0.709 ~ gradient: 2.005 ~ stopping criterion: 1.799 ~ Iteration: 197 ~ bandwidth: 0.708 ~ gradient: 1.798 ~ stopping criterion: 1.003 ~ Iteration: 198 ~ bandwidth: 0.708 ~ gradient: 2.078 ~ stopping criterion: 0.899 ~ Iteration: 199 ~ bandwidth: 0.707 ~ gradient: 2.216 ~ stopping criterion: 1.039 ~ Iteration: 200 ~ bandwidth: 0.707 ~ gradient: 3.766 ~ stopping criterion: 1.108 ~ Iteration: 201 ~ bandwidth: 0.706 ~ gradient: 1.895 ~ stopping criterion: 1.883 ~ Iteration: 202 ~ bandwidth: 0.705 ~ gradient: 1.989 ~ stopping criterion: 0.948 ~ Iteration: 203 ~ bandwidth: 0.704 ~ gradient: 0.884 ~ stopping criterion: 0.995 ~ Iteration: 204 ~ bandwidth: 0.704 ~ gradient: 0.034 ~ stopping criterion: 0.442 ~ Iteration: 205 ~ bandwidth: 0.704 ~ gradient: 1.075 ~ stopping criterion: 0.017 ~ Iteration: 206 ~ bandwidth: 0.704 ~ gradient: 0.809 ~ stopping criterion: 0.538 ~ Iteration: 207 ~ bandwidth: 0.704 ~ gradient: 0.974 ~ stopping criterion: 0.404 ~ Iteration: 208 ~ bandwidth: 0.703 ~ gradient: 1.272 ~ stopping criterion: 0.487 ~ Iteration: 209 ~ bandwidth: 0.703 ~ gradient: 1.031 ~ stopping criterion: 0.636 ~ Iteration: 210 ~ bandwidth: 0.703 ~ gradient: 0.824 ~ stopping criterion: 0.516 ~ Iteration: 211 ~ bandwidth: 0.702 ~ gradient: 0.574 ~ stopping criterion: 0.412 ~ Iteration: 212 ~ bandwidth: 0.702 ~ gradient: -0.045 ~ stopping criterion: 0.287 ~ Iteration: 213 ~ bandwidth: 0.702 ~ gradient: -1.370 ~ stopping criterion: 0.022 ~ Iteration: 214 ~ bandwidth: 0.702 ~ gradient: -0.081 ~ stopping criterion: 0.685 ~ Iteration: 215 ~ bandwidth: 0.703 ~ gradient: 1.688 ~ stopping criterion: 0.040 ~ Iteration: 216 ~ bandwidth: 0.702 ~ gradient: -0.780 ~ stopping criterion: 0.844 ~ Iteration: 217 ~ bandwidth: 0.702 ~ gradient: -0.101 ~ stopping criterion: 0.390 ~ Iteration: 218 ~ bandwidth: 0.702 ~ gradient: -0.119 ~ stopping criterion: 0.051 ~ Iteration: 219 ~ bandwidth: 0.702 ~ gradient: 0.531 ~ stopping criterion: 0.059 ~ Iteration: 220 ~ bandwidth: 0.702 ~ gradient: 0.786 ~ stopping criterion: 0.265 ~ Iteration: 221 ~ bandwidth: 0.702 ~ gradient: 0.487 ~ stopping criterion: 0.393 ~ Iteration: 222 ~ bandwidth: 0.702 ~ gradient: 1.407 ~ stopping criterion: 0.244 ~ Iteration: 223 ~ bandwidth: 0.702 ~ gradient: 0.562 ~ stopping criterion: 0.704 ~ Iteration: 224 ~ bandwidth: 0.702 ~ gradient: -0.860 ~ stopping criterion: 0.281 ~ Iteration: 225 ~ bandwidth: 0.702 ~ gradient: 1.547 ~ stopping criterion: 0.430 ~ Iteration: 226 ~ bandwidth: 0.702 ~ gradient: 0.296 ~ stopping criterion: 0.774 ~ Iteration: 227 ~ bandwidth: 0.702 ~ gradient: 0.446 ~ stopping criterion: 0.148 ~ Iteration: 228 ~ bandwidth: 0.702 ~ gradient: 1.167 ~ stopping criterion: 0.223 ~ Iteration: 229 ~ bandwidth: 0.702 ~ gradient: 1.101 ~ stopping criterion: 0.584 ~ Iteration: 230 ~ bandwidth: 0.702 ~ gradient: -0.906 ~ stopping criterion: 0.550 ~ Iteration: 231 ~ bandwidth: 0.702 ~ gradient: 0.829 ~ stopping criterion: 0.453 ~ Iteration: 232 ~ bandwidth: 0.702 ~ gradient: 0.730 ~ stopping criterion: 0.415 ~ Iteration: 233 ~ bandwidth: 0.702 ~ gradient: -0.245 ~ stopping criterion: 0.365 ~ Iteration: 234 ~ bandwidth: 0.702 ~ gradient: 1.577 ~ stopping criterion: 0.122 ~ Iteration: 235 ~ bandwidth: 0.701 ~ gradient: 0.024 ~ stopping criterion: 0.788 ~ Iteration: 236 ~ bandwidth: 0.701 ~ gradient: 1.736 ~ stopping criterion: 0.012 ~ Iteration: 237 ~ bandwidth: 0.701 ~ gradient: 0.633 ~ stopping criterion: 0.868 ~ Iteration: 238 ~ bandwidth: 0.701 ~ gradient: 1.599 ~ stopping criterion: 0.317 ~ Iteration: 239 ~ bandwidth: 0.701 ~ gradient: 1.140 ~ stopping criterion: 0.800 ~ Iteration: 240 ~ bandwidth: 0.701 ~ gradient: 1.434 ~ stopping criterion: 0.570 ~ Iteration: 241 ~ bandwidth: 0.701 ~ gradient: -0.243 ~ stopping criterion: 0.717 ~ Iteration: 242 ~ bandwidth: 0.701 ~ gradient: 1.831 ~ stopping criterion: 0.122 ~ Iteration: 243 ~ bandwidth: 0.701 ~ gradient: -0.497 ~ stopping criterion: 0.916 ~ Iteration: 244 ~ bandwidth: 0.701 ~ gradient: 1.226 ~ stopping criterion: 0.249 ~ Iteration: 245 ~ bandwidth: 0.701 ~ gradient: -0.292 ~ stopping criterion: 0.613 ~ Iteration: 246 ~ bandwidth: 0.701 ~ gradient: -0.067 ~ stopping criterion: 0.146 ~ Iteration: 247 ~ bandwidth: 0.701 ~ gradient: 0.449 ~ stopping criterion: 0.033 ~ Iteration: 248 ~ bandwidth: 0.701 ~ gradient: 0.290 ~ stopping criterion: 0.224 ~ Iteration: 249 ~ bandwidth: 0.701 ~ gradient: -0.736 ~ stopping criterion: 0.145 ~ Iteration: 250 ~ bandwidth: 0.701 ~ gradient: 0.039 ~ stopping criterion: 0.368 ~ Iteration: 251 ~ bandwidth: 0.701 ~ gradient: 1.392 ~ stopping criterion: 0.019 ~ Iteration: 252 ~ bandwidth: 0.701 ~ gradient: 0.187 ~ stopping criterion: 0.696 ~ Iteration: 253 ~ bandwidth: 0.701 ~ gradient: -0.352 ~ stopping criterion: 0.094 ~ Iteration: 254 ~ bandwidth: 0.701 ~ gradient: -0.693 ~ stopping criterion: 0.176 ~ Iteration: 255 ~ bandwidth: 0.701 ~ gradient: -0.410 ~ stopping criterion: 0.347 ~ Iteration: 256 ~ bandwidth: 0.701 ~ gradient: -0.858 ~ stopping criterion: 0.205 ~ Iteration: 257 ~ bandwidth: 0.701 ~ gradient: -0.014 ~ stopping criterion: 0.429 ~ Iteration: 258 ~ bandwidth: 0.701 ~ gradient: -0.138 ~ stopping criterion: 0.007 ~ Iteration: 259 ~ bandwidth: 0.701 ~ gradient: -0.207 ~ stopping criterion: 0.069 ~ Iteration: 260 ~ bandwidth: 0.701 ~ gradient: -1.444 ~ stopping criterion: 0.103 ~ Iteration: 261 ~ bandwidth: 0.701 ~ gradient: 0.497 ~ stopping criterion: 0.722 ~ Iteration: 262 ~ bandwidth: 0.701 ~ gradient: -0.539 ~ stopping criterion: 0.248 ~ Iteration: 263 ~ bandwidth: 0.701 ~ gradient: 1.030 ~ stopping criterion: 0.269 ~ Iteration: 264 ~ bandwidth: 0.701 ~ gradient: 0.294 ~ stopping criterion: 0.515 ~ Iteration: 265 ~ bandwidth: 0.701 ~ gradient: 0.303 ~ stopping criterion: 0.147 ~ Iteration: 266 ~ bandwidth: 0.701 ~ gradient: -0.277 ~ stopping criterion: 0.152 ~ Iteration: 267 ~ bandwidth: 0.701 ~ gradient: 0.243 ~ stopping criterion: 0.138 ~ Iteration: 268 ~ bandwidth: 0.701 ~ gradient: 0.500 ~ stopping criterion: 0.121 ~ Iteration: 269 ~ bandwidth: 0.701 ~ gradient: -0.835 ~ stopping criterion: 0.250 ~ Iteration: 270 ~ bandwidth: 0.701 ~ gradient: 0.442 ~ stopping criterion: 0.417 ~ Iteration: 271 ~ bandwidth: 0.701 ~ gradient: 1.868 ~ stopping criterion: 0.221 ~ Iteration: 272 ~ bandwidth: 0.701 ~ gradient: -0.799 ~ stopping criterion: 0.934 ~ Iteration: 273 ~ bandwidth: 0.701 ~ gradient: -1.488 ~ stopping criterion: 0.400 ~ Iteration: 274 ~ bandwidth: 0.701 ~ gradient: 0.798 ~ stopping criterion: 0.744 ~ Iteration: 275 ~ bandwidth: 0.701 ~ gradient: 0.647 ~ stopping criterion: 0.399 ~ Iteration: 276 ~ bandwidth: 0.701 ~ gradient: 0.525 ~ stopping criterion: 0.324 ~ Iteration: 277 ~ bandwidth: 0.701 ~ gradient: 0.455 ~ stopping criterion: 0.263 ~ Iteration: 278 ~ bandwidth: 0.701 ~ gradient: -0.345 ~ stopping criterion: 0.228 ~ Iteration: 279 ~ bandwidth: 0.701 ~ gradient: -0.541 ~ stopping criterion: 0.173 ~ Iteration: 280 ~ bandwidth: 0.701 ~ gradient: -1.220 ~ stopping criterion: 0.270 ~ Iteration: 281 ~ bandwidth: 0.701 ~ gradient: 0.927 ~ stopping criterion: 0.610 ~ Iteration: 282 ~ bandwidth: 0.701 ~ gradient: -0.659 ~ stopping criterion: 0.463 ~ Iteration: 283 ~ bandwidth: 0.701 ~ gradient: -0.248 ~ stopping criterion: 0.330 ~ Iteration: 284 ~ bandwidth: 0.701 ~ gradient: -0.298 ~ stopping criterion: 0.124 ~ Iteration: 285 ~ bandwidth: 0.701 ~ gradient: 0.263 ~ stopping criterion: 0.149 ~ Iteration: 286 ~ bandwidth: 0.701 ~ gradient: 0.964 ~ stopping criterion: 0.132 ~ Iteration: 287 ~ bandwidth: 0.701 ~ gradient: 0.492 ~ stopping criterion: 0.482 ~ Iteration: 288 ~ bandwidth: 0.701 ~ gradient: 0.355 ~ stopping criterion: 0.246 ~ Iteration: 289 ~ bandwidth: 0.701 ~ gradient: -0.169 ~ stopping criterion: 0.178 ~ Iteration: 290 ~ bandwidth: 0.701 ~ gradient: -1.281 ~ stopping criterion: 0.085 ~ Iteration: 291 ~ bandwidth: 0.701 ~ gradient: -0.711 ~ stopping criterion: 0.640 ~ Iteration: 292 ~ bandwidth: 0.701 ~ gradient: -0.046 ~ stopping criterion: 0.356 ~ Iteration: 293 ~ bandwidth: 0.701 ~ gradient: 0.493 ~ stopping criterion: 0.023 ~ Iteration: 294 ~ bandwidth: 0.701 ~ gradient: -0.283 ~ stopping criterion: 0.246 ~ Iteration: 295 ~ bandwidth: 0.701 ~ gradient: -1.623 ~ stopping criterion: 0.141 ~ Iteration: 296 ~ bandwidth: 0.701 ~ gradient: 1.777 ~ stopping criterion: 0.811 ~ Iteration: 297 ~ bandwidth: 0.701 ~ gradient: -0.091 ~ stopping criterion: 0.888 ~ Iteration: 298 ~ bandwidth: 0.701 ~ gradient: 0.148 ~ stopping criterion: 0.045 ~ Iteration: 299 ~ bandwidth: 0.701 ~ gradient: 0.214 ~ stopping criterion: 0.074 ~ Iteration: 300 ~ bandwidth: 0.701 ~ gradient: 0.181 ~ stopping criterion: 0.107 ~ Stopped at: 300 ~ bandwidth: 0.701 ~ gradient: 0.181 ~ stopping criterion: 0.107Let’s compare this to the previous case.
# MixCOBRA with default parameter
print("Estimated bandwidth = {}".format(gc2_fit.optimization_outputs['opt_bandwidth']))Estimated bandwidth = 0.7012029287236681The learning curve.
gc2_fit.draw_learning_curve()Copare the MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
y_pred2 = gc2_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred2))
print(mean_squared_error(y_test1, y_pred2))0.23524039384008133
300.2477389788587Let’s look at the qq-plot.
gc2_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')MixCOBRARegressor with non-default parameters
MixCOBRARegressor offers various options to enhance the performance of the method. By default, grid search algorithm is used to estimate the parameters \((\alpha, \beta)\), however, gradient descent algorithm can also be used to optimize the method by setting opt_method = "grad" as shown in case of one bandwidth parameter above. Moveover, you can control the hyperparameters of the basic estimators to enhance the aggregation performance just like in GradientCOBRA method. This can be done as follows:
- learning_rate : control the learning rate of gradient descent in estimating the bandwidth parameter
- speed : the speed of the learning rate.
- kernel : the kernel function used for the aggregation
- opt_method : the optimiztaion algorithm for estimating the bandwidth. It can be gradient descent (
grad) or grid search (grid). - kernel : kernel function which is
radialby default. It must be an element of['exponential', 'gaussian', 'radial', 'cauchy', 'reverse_cosh', 'epanechnikov','biweight', 'triweight', 'triangular']. - alpha_list : list or array to search for \(\alpha\).
- beta_list : list or array to search for \(\beta\).
- bandwidth_list : : list or array to search for single bandwidth parameter \(h\) instead of \((\alpha, \beta)\). To do this, we have to set
one_parameter = Truewhen fitting the method. - max_iter : maximum iteration in gradient descent.
- loss_function : control the type of loss function used for optimizing the bandwidth.
- opt_params : control the optimization algorithm such as adjusting
n_cv,start,n_tries… - estiamtor_list : the list of basic estimators used for the aggregation.
- estimator_params : controlling the hyperparameters of the basic estimators. It must be a dictionary with
(key, dict) = (estimator, dict), i.e. the key must be the name of the basic estimator, and the value is a dictionary containing its hyperparamaters.
We create another object gc3 with non-default parameters, then fit it to the same training data as in the previous example.
gc3 = MixCOBRARegressor(learning_rate=0.1,
speed="linear",
kernel='radial',
opt_method='grad',
loss_function="weighted_mse",
max_iter=300,
estimator_list=['random_forest',
'adaboost',
'knn',
'svm',
'lasso',
'ridge'],
estimator_params={
'random_forest' : {
'n_estimators' : 300,
'min_samples_leaf' : 10},
'adaboost' : {
'n_estimators' : 300,
'max_depth' : 10},
'knn' : {
'n_neighbors' : 30},
'svm' : {'C' : 7}
})
gc3_fit = gc3.fit(X_train1, y_train1)
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: 0.132 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.065924 ~ Iteration: 2 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.087429 ~ Iteration: 3 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.108935 ~ Iteration: 4 ~ (alpha, beta): (1.000, 1.325) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.130440 ~ Iteration: 5 ~ (alpha, beta): (1.025, 0.950) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.065924 ~ Iteration: 6 ~ (alpha, beta): (0.974, 1.530) ~ |gradient|_1: -0.013 ~ stopping criterion: 0.083202 ~ Iteration: 7 ~ (alpha, beta): (1.020, 1.032) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.074755 ~ Iteration: 8 ~ (alpha, beta): (0.976, 1.178) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.028074 ~ Iteration: 9 ~ (alpha, beta): (0.971, 1.022) ~ |gradient|_1: 0.015 ~ stopping criterion: 0.025987 ~ Iteration: 10 ~ (alpha, beta): (0.918, 1.209) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.040850 ~ Iteration: 11 ~ (alpha, beta): (0.959, 1.067) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.033706 ~ Iteration: 12 ~ (alpha, beta): (0.916, 1.111) ~ |gradient|_1: -0.009 ~ stopping criterion: 0.016338 ~ Iteration: 13 ~ (alpha, beta): (0.943, 1.080) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.012792 ~ Iteration: 14 ~ (alpha, beta): (0.899, 1.078) ~ |gradient|_1: 0.009 ~ stopping criterion: 0.008826 ~ Iteration: 15 ~ (alpha, beta): (0.879, 1.112) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.014056 ~ Iteration: 16 ~ (alpha, beta): (0.859, 1.100) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.008195 ~ Iteration: 17 ~ (alpha, beta): (0.827, 1.078) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.014637 ~ Iteration: 18 ~ (alpha, beta): (0.831, 1.110) ~ |gradient|_1: -0.019 ~ stopping criterion: 0.012317 ~ Iteration: 19 ~ (alpha, beta): (0.874, 1.100) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.013762 ~ Iteration: 20 ~ (alpha, beta): (0.873, 1.099) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.000663 ~ Iteration: 21 ~ (alpha, beta): (0.871, 1.083) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.008359 ~ Iteration: 22 ~ (alpha, beta): (0.887, 1.088) ~ |gradient|_1: -0.009 ~ stopping criterion: 0.006470 ~ Iteration: 23 ~ (alpha, beta): (0.903, 1.084) ~ |gradient|_1: -0.017 ~ stopping criterion: 0.006947 ~ Iteration: 24 ~ (alpha, beta): (0.936, 1.095) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.015694 ~ Iteration: 25 ~ (alpha, beta): (0.924, 1.090) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.006975 ~ Iteration: 26 ~ (alpha, beta): (0.942, 1.092) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.007944 ~ Iteration: 27 ~ (alpha, beta): (0.939, 1.095) ~ |gradient|_1: -0.017 ~ stopping criterion: 0.004901 ~ Iteration: 28 ~ (alpha, beta): (0.958, 1.078) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.027751 ~ Iteration: 29 ~ (alpha, beta): (0.963, 1.081) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.008136 ~ Iteration: 30 ~ (alpha, beta): (0.958, 1.085) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.007945 ~ Iteration: 31 ~ (alpha, beta): (0.947, 1.086) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.009480 ~ Iteration: 32 ~ (alpha, beta): (0.956, 1.087) ~ |gradient|_1: 0.024 ~ stopping criterion: 0.005746 ~ Iteration: 33 ~ (alpha, beta): (0.939, 1.084) ~ |gradient|_1: 0.022 ~ stopping criterion: 0.015416 ~ Iteration: 34 ~ (alpha, beta): (0.927, 1.087) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.016166 ~ Iteration: 35 ~ (alpha, beta): (0.915, 1.090) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.016617 ~ Iteration: 36 ~ (alpha, beta): (0.914, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.003197 ~ Iteration: 37 ~ (alpha, beta): (0.910, 1.087) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.009037 ~ Iteration: 38 ~ (alpha, beta): (0.902, 1.081) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.019952 ~ Iteration: 39 ~ (alpha, beta): (0.894, 1.085) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.013848 ~ Iteration: 40 ~ (alpha, beta): (0.901, 1.088) ~ |gradient|_1: 0.020 ~ stopping criterion: 0.016229 ~ Iteration: 41 ~ (alpha, beta): (0.890, 1.088) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.010511 ~ Iteration: 42 ~ (alpha, beta): (0.884, 1.086) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.014590 ~ Iteration: 43 ~ (alpha, beta): (0.882, 1.090) ~ |gradient|_1: -0.013 ~ stopping criterion: 0.014406 ~ Iteration: 44 ~ (alpha, beta): (0.888, 1.091) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.010878 ~ Iteration: 45 ~ (alpha, beta): (0.887, 1.091) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001943 ~ Iteration: 46 ~ (alpha, beta): (0.887, 1.092) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.003673 ~ Iteration: 47 ~ (alpha, beta): (0.888, 1.093) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.005612 ~ Iteration: 48 ~ (alpha, beta): (0.887, 1.092) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.004393 ~ Iteration: 49 ~ (alpha, beta): (0.884, 1.091) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.011986 ~ Iteration: 50 ~ (alpha, beta): (0.882, 1.090) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.007660 ~ Iteration: 51 ~ (alpha, beta): (0.882, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.004611 ~ Iteration: 52 ~ (alpha, beta): (0.884, 1.088) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.011304 ~ Iteration: 53 ~ (alpha, beta): (0.883, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.006466 ~ Iteration: 54 ~ (alpha, beta): (0.883, 1.089) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.000902 ~ Iteration: 55 ~ (alpha, beta): (0.881, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.008878 ~ Iteration: 56 ~ (alpha, beta): (0.878, 1.089) ~ |gradient|_1: 0.009 ~ stopping criterion: 0.009695 ~ Iteration: 57 ~ (alpha, beta): (0.877, 1.088) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.009556 ~ Iteration: 58 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.002561 ~ Iteration: 59 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.003356 ~ Iteration: 60 ~ (alpha, beta): (0.878, 1.088) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.006347 ~ Iteration: 61 ~ (alpha, beta): (0.879, 1.088) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.001077 ~ Iteration: 62 ~ (alpha, beta): (0.876, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.015848 ~ Iteration: 63 ~ (alpha, beta): (0.877, 1.089) ~ |gradient|_1: 0.016 ~ stopping criterion: 0.004051 ~ Iteration: 64 ~ (alpha, beta): (0.875, 1.088) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.013425 ~ Iteration: 65 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.003705 ~ Iteration: 66 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.003234 ~ Iteration: 67 ~ (alpha, beta): (0.873, 1.089) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.007744 ~ Iteration: 68 ~ (alpha, beta): (0.873, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.004452 ~ Iteration: 69 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.002422 ~ Iteration: 70 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.007313 ~ Iteration: 71 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.003825 ~ Iteration: 72 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.006748 ~ Iteration: 73 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.009112 ~ Iteration: 74 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.010699 ~ Iteration: 75 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.006512 ~ Iteration: 76 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.011142 ~ Iteration: 77 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007410 ~ Iteration: 78 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.013663 ~ Iteration: 79 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.008604 ~ Iteration: 80 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.005902 ~ Iteration: 81 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.003601 ~ Iteration: 82 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.011737 ~ Iteration: 83 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.004572 ~ Iteration: 84 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.014986 ~ Iteration: 85 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.028 ~ stopping criterion: 0.005204 ~ Iteration: 86 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.019777 ~ Iteration: 87 ~ (alpha, beta): (0.875, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.003831 ~ Iteration: 88 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.009195 ~ Iteration: 89 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.010378 ~ Iteration: 90 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.004439 ~ Iteration: 91 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.010082 ~ Iteration: 92 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.003847 ~ Iteration: 93 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.014638 ~ Iteration: 94 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.018533 ~ Iteration: 95 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.011874 ~ Iteration: 96 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.001645 ~ Iteration: 97 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.005722 ~ Iteration: 98 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.017228 ~ Iteration: 99 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.025 ~ stopping criterion: 0.002783 ~ Iteration: 100 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.017635 ~ Iteration: 101 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.014150 ~ Iteration: 102 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.005757 ~ Iteration: 103 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007102 ~ Iteration: 104 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.006087 ~ Iteration: 105 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.007154 ~ Iteration: 106 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.021 ~ stopping criterion: 0.011526 ~ Iteration: 107 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.015174 ~ Iteration: 108 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.005473 ~ Iteration: 109 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.004929 ~ Iteration: 110 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.005358 ~ Iteration: 111 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.014516 ~ Iteration: 112 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.006655 ~ Iteration: 113 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.021 ~ stopping criterion: 0.011645 ~ Iteration: 114 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.011813 ~ Iteration: 115 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.004324 ~ Iteration: 116 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.012155 ~ Iteration: 117 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.012 ~ stopping criterion: 0.013049 ~ Iteration: 118 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.017 ~ stopping criterion: 0.008994 ~ Iteration: 119 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.017232 ~ Iteration: 120 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.010607 ~ Iteration: 121 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.018 ~ stopping criterion: 0.013039 ~ Iteration: 122 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.014739 ~ Iteration: 123 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001723 ~ Iteration: 124 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.002370 ~ Iteration: 125 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.003934 ~ Iteration: 126 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.006 ~ stopping criterion: 0.002732 ~ Iteration: 127 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002904 ~ Iteration: 128 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.005 ~ stopping criterion: 0.010421 ~ Iteration: 129 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.007 ~ stopping criterion: 0.003750 ~ Iteration: 130 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.003929 ~ Iteration: 131 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.004891 ~ Iteration: 132 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.008 ~ stopping criterion: 0.007942 ~ Iteration: 133 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.008102 ~ Iteration: 134 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.012 ~ stopping criterion: 0.001128 ~ Iteration: 135 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.007495 ~ Iteration: 136 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.003812 ~ Iteration: 137 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.014 ~ stopping criterion: 0.005350 ~ Iteration: 138 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.006 ~ stopping criterion: 0.007249 ~ Iteration: 139 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.005 ~ stopping criterion: 0.005889 ~ Iteration: 140 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.010 ~ stopping criterion: 0.014089 ~ Iteration: 141 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.011 ~ stopping criterion: 0.005270 ~ Iteration: 142 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.006283 ~ Iteration: 143 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.001944 ~ Iteration: 144 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.002 ~ stopping criterion: 0.004854 ~ Iteration: 145 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.019 ~ stopping criterion: 0.005236 ~ Iteration: 146 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.008 ~ stopping criterion: 0.009927 ~ Iteration: 147 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.004823 ~ Iteration: 148 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.018 ~ stopping criterion: 0.005225 ~ Iteration: 149 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.018 ~ stopping criterion: 0.014587 ~ Iteration: 150 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.010834 ~ Iteration: 151 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.016 ~ stopping criterion: 0.007541 ~ Iteration: 152 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.009899 ~ Iteration: 153 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.011 ~ stopping criterion: 0.009305 ~ Iteration: 154 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.004 ~ stopping criterion: 0.007607 ~ Iteration: 155 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.002029 ~ Iteration: 156 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.015 ~ stopping criterion: 0.000442 ~ Iteration: 157 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.001 ~ stopping criterion: 0.014383 ~ Iteration: 158 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.010393 ~ Iteration: 159 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.008472 ~ Iteration: 160 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.004447 ~ Iteration: 161 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.001473 ~ Iteration: 162 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.004 ~ stopping criterion: 0.004006 ~ Iteration: 163 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.003 ~ stopping criterion: 0.003272 ~ Iteration: 164 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.003315 ~ Iteration: 165 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.004796 ~ Iteration: 166 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.007 ~ stopping criterion: 0.002243 ~ Iteration: 167 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.005328 ~ Iteration: 168 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: 0.013 ~ stopping criterion: 0.002214 ~ Iteration: 169 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.007727 ~ Iteration: 170 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000812 ~ Iteration: 171 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.000804 ~ Iteration: 172 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001110 ~ Iteration: 173 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001133 ~ Iteration: 174 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.002281 ~ Iteration: 175 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002250 ~ Iteration: 176 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.002285 ~ Iteration: 177 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000720 ~ Iteration: 178 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000696 ~ Iteration: 179 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.001 ~ stopping criterion: 0.000863 ~ Iteration: 180 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.000817 ~ Iteration: 181 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001208 ~ Iteration: 182 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001103 ~ Iteration: 183 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001104 ~ Iteration: 184 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001107 ~ Iteration: 185 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 186 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001148 ~ Iteration: 187 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001103 ~ Iteration: 188 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002278 ~ Iteration: 189 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001225 ~ Iteration: 190 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001226 ~ Iteration: 191 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 192 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001113 ~ Iteration: 193 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001129 ~ Iteration: 194 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001114 ~ Iteration: 195 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 196 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001112 ~ Iteration: 197 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001156 ~ Iteration: 198 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001217 ~ Iteration: 199 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001387 ~ Iteration: 200 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002079 ~ Iteration: 201 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001350 ~ Iteration: 202 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 203 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001387 ~ Iteration: 204 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002079 ~ Iteration: 205 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 206 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.001355 ~ Iteration: 207 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001387 ~ Iteration: 208 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002539 ~ Iteration: 209 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001816 ~ Iteration: 210 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 211 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002539 ~ Iteration: 212 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 213 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002429 ~ Iteration: 214 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001815 ~ Iteration: 215 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002429 ~ Iteration: 216 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002430 ~ Iteration: 217 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.001789 ~ Iteration: 218 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002410 ~ Iteration: 219 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002410 ~ Iteration: 220 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 221 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 222 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002404 ~ Iteration: 223 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002402 ~ Iteration: 224 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002402 ~ Iteration: 225 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002421 ~ Iteration: 226 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002421 ~ Iteration: 227 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 228 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 229 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002388 ~ Iteration: 230 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002389 ~ Iteration: 231 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002389 ~ Iteration: 232 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002352 ~ Iteration: 233 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002352 ~ Iteration: 234 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002351 ~ Iteration: 235 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002245 ~ Iteration: 236 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002245 ~ Iteration: 237 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002255 ~ Iteration: 238 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002255 ~ Iteration: 239 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002256 ~ Iteration: 240 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 241 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 242 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002266 ~ Iteration: 243 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 244 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 245 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 246 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 247 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002262 ~ Iteration: 248 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 249 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 250 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 251 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 252 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002259 ~ Iteration: 253 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 254 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 255 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 256 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 257 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002361 ~ Iteration: 258 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002418 ~ Iteration: 259 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002418 ~ Iteration: 260 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 261 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 262 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 263 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 264 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 265 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 266 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002415 ~ Iteration: 267 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002485 ~ Iteration: 268 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 269 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 270 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002580 ~ Iteration: 271 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002583 ~ Iteration: 272 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002583 ~ Iteration: 273 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002589 ~ Iteration: 274 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 275 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 276 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 277 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 278 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 279 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002561 ~ Iteration: 280 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 281 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 282 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 283 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002563 ~ Iteration: 284 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002537 ~ Iteration: 285 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002540 ~ Iteration: 286 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002540 ~ Iteration: 287 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002541 ~ Iteration: 288 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002439 ~ Iteration: 289 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002272 ~ Iteration: 290 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002272 ~ Iteration: 291 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002260 ~ Iteration: 292 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.003 ~ stopping criterion: 0.002261 ~ Iteration: 293 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002289 ~ Iteration: 294 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002207 ~ Iteration: 295 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002186 ~ Iteration: 296 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002186 ~ Iteration: 297 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 298 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 299 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Iteration: 300 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184 ~ Stopped at: 300 ~ (alpha, beta): (0.874, 1.089) ~ |gradient|_1: -0.002 ~ stopping criterion: 0.002184Now, let’s compare it to the previous example.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc3_fit.optimization_outputs['opt_alpha'], gc3_fit.optimization_outputs['opt_beta']))Estimated (alpha, beta) = (0.8741706474293448, 1.0888201148722707)Look at the learning surface using draw_learning_curve() method.
gc3_fit.draw_learning_curve()We evaluate the performance of the method on the testing data using MSE and MAPE.
from sklearn.metrics import mean_absolute_percentage_error
y_pred3 = gc3_fit.predict(X_test1)
print(mean_absolute_percentage_error(y_test1, y_pred3))
print(mean_squared_error(y_test1, y_pred3))0.1308662858088806
126.52951363322245Let’s look at qq-plot using draw_learning_curve() method.
gc3_fit.draw_learning_curve(y_test=y_test1, fig_type='qq')Real dataset
We look at the California housing dataset from sklearn.datasets module. To illustrate the idea, we only work with the first \(1000\) observations.
from sklearn.datasets import fetch_california_housing
data = fetch_california_housing()
X_real, y_real = data['data'], data['target']
X_train_real, X_test_real, y_train_real, y_test_real = train_test_split(X_real[:1000,:], y_real[:1000], test_size=0.2)
print('shape: x_train = {} , x_train = {} , y_train = {} , y_test = {}'.format(X_train_real.shape, X_test_real.shape, y_train_real.shape, y_test_real.shape))shape: x_train = (800, 8) , x_train = (200, 8) , y_train = (800,) , y_test = (200,)We gave some random parameters to the method as follows.
gc_real = MixCOBRARegressor(opt_method="grad",
estimator_list=['random_forest',
'knn',
'ridge',
'lasso'],
estimator_params={
'random_forest' : {'n_estimators': 300},
'knn' : {'n_neighbors' : 30}
})
gc_real_fit = gc_real.fit(X_train_real, y_train_real)
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: 0.001 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 2 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 3 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 4 ~ (alpha, beta): (5.000, 2.550) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.001329 ~ Iteration: 5 ~ (alpha, beta): (5.003, 2.557) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000666 ~ Iteration: 6 ~ (alpha, beta): (5.003, 2.567) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000701 ~ Iteration: 7 ~ (alpha, beta): (5.004, 2.576) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000595 ~ Iteration: 8 ~ (alpha, beta): (5.003, 2.584) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000592 ~ Iteration: 9 ~ (alpha, beta): (5.003, 2.593) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000574 ~ Iteration: 10 ~ (alpha, beta): (5.005, 2.601) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000675 ~ Iteration: 11 ~ (alpha, beta): (5.005, 2.607) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000396 ~ Iteration: 12 ~ (alpha, beta): (5.005, 2.613) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000448 ~ Iteration: 13 ~ (alpha, beta): (5.004, 2.619) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000603 ~ Iteration: 14 ~ (alpha, beta): (5.004, 2.627) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000555 ~ Iteration: 15 ~ (alpha, beta): (5.003, 2.635) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000563 ~ Iteration: 16 ~ (alpha, beta): (5.004, 2.642) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000624 ~ Iteration: 17 ~ (alpha, beta): (5.004, 2.650) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000515 ~ Iteration: 18 ~ (alpha, beta): (5.004, 2.657) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000507 ~ Iteration: 19 ~ (alpha, beta): (5.004, 2.661) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000305 ~ Iteration: 20 ~ (alpha, beta): (5.003, 2.669) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000667 ~ Iteration: 21 ~ (alpha, beta): (5.003, 2.673) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000338 ~ Iteration: 22 ~ (alpha, beta): (5.003, 2.681) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000601 ~ Iteration: 23 ~ (alpha, beta): (5.003, 2.688) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000531 ~ Iteration: 24 ~ (alpha, beta): (5.003, 2.693) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000363 ~ Iteration: 25 ~ (alpha, beta): (5.003, 2.698) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000394 ~ Iteration: 26 ~ (alpha, beta): (5.003, 2.705) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000504 ~ Iteration: 27 ~ (alpha, beta): (5.003, 2.711) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000428 ~ Iteration: 28 ~ (alpha, beta): (5.003, 2.718) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000557 ~ Iteration: 29 ~ (alpha, beta): (5.003, 2.723) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000341 ~ Iteration: 30 ~ (alpha, beta): (5.004, 2.728) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000414 ~ Iteration: 31 ~ (alpha, beta): (5.004, 2.735) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000540 ~ Iteration: 32 ~ (alpha, beta): (5.004, 2.741) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000407 ~ Iteration: 33 ~ (alpha, beta): (5.004, 2.748) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000548 ~ Iteration: 34 ~ (alpha, beta): (5.004, 2.750) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000329 ~ Iteration: 35 ~ (alpha, beta): (5.004, 2.755) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000379 ~ Iteration: 36 ~ (alpha, beta): (5.004, 2.760) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000378 ~ Iteration: 37 ~ (alpha, beta): (5.004, 2.765) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000387 ~ Iteration: 38 ~ (alpha, beta): (5.004, 2.768) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000232 ~ Iteration: 39 ~ (alpha, beta): (5.004, 2.772) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000311 ~ Iteration: 40 ~ (alpha, beta): (5.004, 2.777) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000292 ~ Iteration: 41 ~ (alpha, beta): (5.004, 2.782) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000395 ~ Iteration: 42 ~ (alpha, beta): (5.004, 2.785) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000284 ~ Iteration: 43 ~ (alpha, beta): (5.004, 2.790) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000371 ~ Iteration: 44 ~ (alpha, beta): (5.004, 2.797) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000534 ~ Iteration: 45 ~ (alpha, beta): (5.004, 2.800) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000221 ~ Iteration: 46 ~ (alpha, beta): (5.004, 2.805) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000340 ~ Iteration: 47 ~ (alpha, beta): (5.004, 2.809) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000397 ~ Iteration: 48 ~ (alpha, beta): (5.004, 2.813) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000344 ~ Iteration: 49 ~ (alpha, beta): (5.004, 2.817) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000309 ~ Iteration: 50 ~ (alpha, beta): (5.004, 2.820) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000372 ~ Iteration: 51 ~ (alpha, beta): (5.004, 2.823) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000295 ~ Iteration: 52 ~ (alpha, beta): (5.004, 2.825) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000205 ~ Iteration: 53 ~ (alpha, beta): (5.004, 2.829) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000259 ~ Iteration: 54 ~ (alpha, beta): (5.004, 2.831) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000153 ~ Iteration: 55 ~ (alpha, beta): (5.004, 2.835) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000326 ~ Iteration: 56 ~ (alpha, beta): (5.004, 2.839) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000346 ~ Iteration: 57 ~ (alpha, beta): (5.004, 2.844) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000314 ~ Iteration: 58 ~ (alpha, beta): (5.004, 2.847) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000305 ~ Iteration: 59 ~ (alpha, beta): (5.004, 2.851) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000309 ~ Iteration: 60 ~ (alpha, beta): (5.004, 2.852) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000193 ~ Iteration: 61 ~ (alpha, beta): (5.004, 2.855) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 62 ~ (alpha, beta): (5.004, 2.859) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000340 ~ Iteration: 63 ~ (alpha, beta): (5.004, 2.861) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000178 ~ Iteration: 64 ~ (alpha, beta): (5.004, 2.865) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000320 ~ Iteration: 65 ~ (alpha, beta): (5.004, 2.867) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000187 ~ Iteration: 66 ~ (alpha, beta): (5.004, 2.869) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000232 ~ Iteration: 67 ~ (alpha, beta): (5.004, 2.870) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000261 ~ Iteration: 68 ~ (alpha, beta): (5.004, 2.873) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 69 ~ (alpha, beta): (5.004, 2.873) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000185 ~ Iteration: 70 ~ (alpha, beta): (5.004, 2.876) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000216 ~ Iteration: 71 ~ (alpha, beta): (5.004, 2.878) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000154 ~ Iteration: 72 ~ (alpha, beta): (5.004, 2.880) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000249 ~ Iteration: 73 ~ (alpha, beta): (5.004, 2.881) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000083 ~ Iteration: 74 ~ (alpha, beta): (5.004, 2.884) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000279 ~ Iteration: 75 ~ (alpha, beta): (5.004, 2.887) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000237 ~ Iteration: 76 ~ (alpha, beta): (5.004, 2.888) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000110 ~ Iteration: 77 ~ (alpha, beta): (5.004, 2.889) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000160 ~ Iteration: 78 ~ (alpha, beta): (5.004, 2.891) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000270 ~ Iteration: 79 ~ (alpha, beta): (5.004, 2.894) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 80 ~ (alpha, beta): (5.004, 2.896) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000252 ~ Iteration: 81 ~ (alpha, beta): (5.004, 2.900) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000337 ~ Iteration: 82 ~ (alpha, beta): (5.004, 2.904) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000268 ~ Iteration: 83 ~ (alpha, beta): (5.004, 2.906) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000242 ~ Iteration: 84 ~ (alpha, beta): (5.004, 2.909) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000246 ~ Iteration: 85 ~ (alpha, beta): (5.004, 2.910) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000167 ~ Iteration: 86 ~ (alpha, beta): (5.004, 2.911) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000086 ~ Iteration: 87 ~ (alpha, beta): (5.004, 2.913) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000204 ~ Iteration: 88 ~ (alpha, beta): (5.004, 2.916) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000187 ~ Iteration: 89 ~ (alpha, beta): (5.004, 2.916) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000105 ~ Iteration: 90 ~ (alpha, beta): (5.004, 2.919) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000225 ~ Iteration: 91 ~ (alpha, beta): (5.004, 2.921) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000171 ~ Iteration: 92 ~ (alpha, beta): (5.004, 2.923) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000198 ~ Iteration: 93 ~ (alpha, beta): (5.004, 2.927) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000374 ~ Iteration: 94 ~ (alpha, beta): (5.004, 2.928) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000039 ~ Iteration: 95 ~ (alpha, beta): (5.004, 2.928) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000066 ~ Iteration: 96 ~ (alpha, beta): (5.004, 2.930) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 97 ~ (alpha, beta): (5.004, 2.930) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000097 ~ Iteration: 98 ~ (alpha, beta): (5.004, 2.931) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000134 ~ Iteration: 99 ~ (alpha, beta): (5.004, 2.932) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000201 ~ Iteration: 100 ~ (alpha, beta): (5.004, 2.933) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000107 ~ Iteration: 101 ~ (alpha, beta): (5.004, 2.934) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000110 ~ Iteration: 102 ~ (alpha, beta): (5.004, 2.935) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000116 ~ Iteration: 103 ~ (alpha, beta): (5.004, 2.937) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000230 ~ Iteration: 104 ~ (alpha, beta): (5.004, 2.937) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000082 ~ Iteration: 105 ~ (alpha, beta): (5.004, 2.938) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000050 ~ Iteration: 106 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000256 ~ Iteration: 107 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000113 ~ Iteration: 108 ~ (alpha, beta): (5.004, 2.940) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000153 ~ Iteration: 109 ~ (alpha, beta): (5.004, 2.941) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000301 ~ Iteration: 110 ~ (alpha, beta): (5.004, 2.941) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000088 ~ Iteration: 111 ~ (alpha, beta): (5.004, 2.942) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000119 ~ Iteration: 112 ~ (alpha, beta): (5.004, 2.943) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000280 ~ Iteration: 113 ~ (alpha, beta): (5.004, 2.944) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000136 ~ Iteration: 114 ~ (alpha, beta): (5.004, 2.945) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000185 ~ Iteration: 115 ~ (alpha, beta): (5.004, 2.945) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 116 ~ (alpha, beta): (5.004, 2.946) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 117 ~ (alpha, beta): (5.004, 2.947) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000194 ~ Iteration: 118 ~ (alpha, beta): (5.004, 2.948) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000129 ~ Iteration: 119 ~ (alpha, beta): (5.004, 2.948) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 120 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000149 ~ Iteration: 121 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000047 ~ Iteration: 122 ~ (alpha, beta): (5.004, 2.949) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 123 ~ (alpha, beta): (5.004, 2.950) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000144 ~ Iteration: 124 ~ (alpha, beta): (5.004, 2.952) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000330 ~ Iteration: 125 ~ (alpha, beta): (5.004, 2.953) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000297 ~ Iteration: 126 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000123 ~ Iteration: 127 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000094 ~ Iteration: 128 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000113 ~ Iteration: 129 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000060 ~ Iteration: 130 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000158 ~ Iteration: 131 ~ (alpha, beta): (5.004, 2.954) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000161 ~ Iteration: 132 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000177 ~ Iteration: 133 ~ (alpha, beta): (5.004, 2.955) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000093 ~ Iteration: 134 ~ (alpha, beta): (5.004, 2.956) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000326 ~ Iteration: 135 ~ (alpha, beta): (5.004, 2.956) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000136 ~ Iteration: 136 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000104 ~ Iteration: 137 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000291 ~ Iteration: 138 ~ (alpha, beta): (5.004, 2.957) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000126 ~ Iteration: 139 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000085 ~ Iteration: 140 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000164 ~ Iteration: 141 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000059 ~ Iteration: 142 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000096 ~ Iteration: 143 ~ (alpha, beta): (5.004, 2.958) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000068 ~ Iteration: 144 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000195 ~ Iteration: 145 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000068 ~ Iteration: 146 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000110 ~ Iteration: 147 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000051 ~ Iteration: 148 ~ (alpha, beta): (5.004, 2.959) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000200 ~ Iteration: 149 ~ (alpha, beta): (5.004, 2.960) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 150 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000267 ~ Iteration: 151 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000216 ~ Iteration: 152 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000074 ~ Iteration: 153 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000133 ~ Iteration: 154 ~ (alpha, beta): (5.004, 2.961) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000238 ~ Iteration: 155 ~ (alpha, beta): (5.004, 2.962) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000433 ~ Iteration: 156 ~ (alpha, beta): (5.004, 2.962) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000193 ~ Iteration: 157 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000185 ~ Iteration: 158 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000146 ~ Iteration: 159 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000049 ~ Iteration: 160 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000220 ~ Iteration: 161 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000091 ~ Iteration: 162 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000040 ~ Iteration: 163 ~ (alpha, beta): (5.004, 2.963) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000287 ~ Iteration: 164 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000152 ~ Iteration: 165 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000117 ~ Iteration: 166 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 167 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000334 ~ Iteration: 168 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000156 ~ Iteration: 169 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000126 ~ Iteration: 170 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000138 ~ Iteration: 171 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000082 ~ Iteration: 172 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000258 ~ Iteration: 173 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000162 ~ Iteration: 174 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000231 ~ Iteration: 175 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000054 ~ Iteration: 176 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000066 ~ Iteration: 177 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000096 ~ Iteration: 178 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000075 ~ Iteration: 179 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000163 ~ Iteration: 180 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000087 ~ Iteration: 181 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000149 ~ Iteration: 182 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000047 ~ Iteration: 183 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000126 ~ Iteration: 184 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000220 ~ Iteration: 185 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000127 ~ Iteration: 186 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000236 ~ Iteration: 187 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000284 ~ Iteration: 188 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000276 ~ Iteration: 189 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000157 ~ Iteration: 190 ~ (alpha, beta): (5.004, 2.964) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000047 ~ Iteration: 191 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000114 ~ Iteration: 192 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000193 ~ Iteration: 193 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000068 ~ Iteration: 194 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000037 ~ Iteration: 195 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000314 ~ Iteration: 196 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000062 ~ Iteration: 197 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000053 ~ Iteration: 198 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000425 ~ Iteration: 199 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000262 ~ Iteration: 200 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000126 ~ Iteration: 201 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000094 ~ Iteration: 202 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000206 ~ Iteration: 203 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000130 ~ Iteration: 204 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000111 ~ Iteration: 205 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000333 ~ Iteration: 206 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000206 ~ Iteration: 207 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000251 ~ Iteration: 208 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000129 ~ Iteration: 209 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 210 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000222 ~ Iteration: 211 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000122 ~ Iteration: 212 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000163 ~ Iteration: 213 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000040 ~ Iteration: 214 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000089 ~ Iteration: 215 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000076 ~ Iteration: 216 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000115 ~ Iteration: 217 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000183 ~ Iteration: 218 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000144 ~ Iteration: 219 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 220 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 221 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000055 ~ Iteration: 222 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000119 ~ Iteration: 223 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000119 ~ Iteration: 224 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000036 ~ Iteration: 225 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000031 ~ Iteration: 226 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000164 ~ Iteration: 227 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000139 ~ Iteration: 228 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000160 ~ Iteration: 229 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000182 ~ Iteration: 230 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000093 ~ Iteration: 231 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000255 ~ Iteration: 232 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000254 ~ Iteration: 233 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000117 ~ Iteration: 234 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000076 ~ Iteration: 235 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000165 ~ Iteration: 236 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000130 ~ Iteration: 237 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000214 ~ Iteration: 238 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000032 ~ Iteration: 239 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000083 ~ Iteration: 240 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000056 ~ Iteration: 241 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000186 ~ Iteration: 242 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000022 ~ Iteration: 243 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000089 ~ Iteration: 244 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000093 ~ Iteration: 245 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000078 ~ Iteration: 246 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000036 ~ Iteration: 247 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000034 ~ Iteration: 248 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000026 ~ Iteration: 249 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000026 ~ Iteration: 250 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000031 ~ Iteration: 251 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000030 ~ Iteration: 252 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000026 ~ Iteration: 253 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 254 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000027 ~ Iteration: 255 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 256 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000030 ~ Iteration: 257 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 258 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000016 ~ Iteration: 259 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000017 ~ Iteration: 260 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000016 ~ Iteration: 261 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 262 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 263 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 264 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000032 ~ Iteration: 265 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 266 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000023 ~ Iteration: 267 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 268 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000017 ~ Iteration: 269 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 270 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000012 ~ Iteration: 271 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000010 ~ Iteration: 272 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000012 ~ Iteration: 273 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 274 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 275 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 276 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 277 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 278 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 279 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 280 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 281 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 282 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 283 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 284 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 285 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 286 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 287 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 288 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 289 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 290 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 291 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 292 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 293 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 294 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 295 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 296 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 297 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 298 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000011 ~ Iteration: 299 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Iteration: 300 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010 ~ Stopped at: 300 ~ (alpha, beta): (5.004, 2.965) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000010Now, let’s look at the obtained bandwidth and the optimization result.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc_real_fit.optimization_outputs['opt_alpha'], gc_real_fit.optimization_outputs['opt_beta']))Estimated (alpha, beta) = (5.003947996785071, 2.9646284687052478)We look at the numerical and graphical performance.
y_pred_real = gc_real_fit.predict(X_test_real)
print(mean_absolute_percentage_error(y_test_real, y_pred_real))
print(mean_squared_error(y_test_real, y_pred_real))
gc_real_fit.draw_learning_curve(y_test=y_test_real, fig_type='qq')0.1533765484019444
0.16456235739517955Compare to Adaboost
We campare the fitted method on California data with Adaboost method.
from sklearn.ensemble import AdaBoostRegressor
ada = AdaBoostRegressor(n_estimators=1000)
ada_fit = ada.fit(X_train_real, y_train_real)
ada_pred = ada_fit.predict(X_test_real)
print(mean_absolute_percentage_error(y_test_real, ada_pred))
print(mean_squared_error(y_test_real, ada_pred))0.2723746701801255
0.25246896498101007Pretrained basic estimators
You can also train other models then aggregate them using our method. Here, we build pretrained estimators including XGBoost then aggregate it to some sklearn basic estimators in MixCOBRARegressor.
We first split the training data into two parts: \(X_k\) for building basic estimators, and \(X_\ell\) for aggregation. We use the constructed estimators to predict the test data and both the input \(X_{\text{test}}\) and its predictions pred_feature_test are used in the final predictions.
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor
from sklearn.tree import DecisionTreeRegressor
# X_k and X_l split
id_k = np.random.permutation(range(len(y_train_real)))
k =int(.5 * len(y_train_real))
X_k, X_l, y_k, y_l = X_train_real[id_k[:k],:], X_train_real[id_k[k:],:], y_train_real[id_k[:k]], y_train_real[id_k[k:]]
# Building basic estiators
rf_real = RandomForestRegressor(n_estimators=300).fit(X_k, y_k)
lm_real = LinearRegression().fit(X_k, y_k)
knn_real = KNeighborsRegressor(n_neighbors=10).fit(X_k, y_k)
tr_real = DecisionTreeRegressor(min_samples_leaf=5).fit(X_k, y_k)
# External XGBoost estiator
import xgboost
xgb = xgboost.XGBRegressor(n_estimators = 500)
xgb_real = xgb.fit(X_k, y_k)
# All pretrained estimators
basic_estimators = (rf_real, lm_real, knn_real, tr_real, xgb_real)
# Predicted features on X_l for aggregation
pred_feature_l = np.column_stack([est.predict(X_l) for est in basic_estimators])
# Predicted features on Testing data
pred_feature_test = np.column_stack([est.predict(X_test_real) for est in basic_estimators])To fit the aggregation method on predicted features, we need to provide both input and predicted features to fit method.
gc4_fit = MixCOBRARegressor(opt_method='grad').fit(X=X_l,
Pred_features=pred_feature_l,
y=y_l)
* Gradient descent of two parameters with radial kernel is implemented...
~ Initial t = 0: ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ threshold: 1e-05
~ Iteration: 1 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 2 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 3 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 4 ~ (alpha, beta): (5.000, 3.775) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000598 ~ Iteration: 5 ~ (alpha, beta): (4.992, 3.777) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000029 ~ Iteration: 6 ~ (alpha, beta): (5.013, 3.820) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000185 ~ Iteration: 7 ~ (alpha, beta): (5.030, 3.840) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000118 ~ Iteration: 8 ~ (alpha, beta): (5.026, 3.868) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000092 ~ Iteration: 9 ~ (alpha, beta): (5.012, 3.894) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000140 ~ Iteration: 10 ~ (alpha, beta): (5.052, 3.936) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000297 ~ Iteration: 11 ~ (alpha, beta): (5.038, 3.999) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000260 ~ Iteration: 12 ~ (alpha, beta): (5.034, 4.032) ~ |gradient|_1: 0.000 ~ stopping criterion: 0.000117 ~ Iteration: 13 ~ (alpha, beta): (5.034, 4.031) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000008 ~ Stopped at: 13 ~ (alpha, beta): (5.034, 4.031) ~ |gradient|_1: -0.000 ~ stopping criterion: 0.000008We look at the optimization algorithm performance.
# MixCOBRA with default parameter
print("Estimated (alpha, beta) = ({}, {})".format(gc4_fit.optimization_outputs['opt_alpha'], gc4_fit.optimization_outputs['opt_beta']))Estimated (alpha, beta) = (5.033650165115184, 4.030778809061925)Now, let’s look at the result.
gc4_fit.draw_learning_curve()
pred_add = gc4_fit.predict(X = X_test_real,
Pred_X = pred_feature_test)
print(mean_absolute_percentage_error(y_test_real, pred_add))
print(mean_squared_error(y_test_real, pred_add))0.14222323551671262
0.15583368476512235