🗺️ Content
Motivation & Tools
Bagging: Random Forest
Boosting: Adaboost, XGBoost…
Concensual aggregation: COBRA, KernelCOBRA, MixCOBRA
Stacking: SuperLearner
Advanced Machine Learning
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
- Train Models: Train a separate model (typically the same type) on each bootstrap sample.
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
- Train Models: Train a separate model (typically the same type) on each bootstrap sample.
- Aggregate Predictions: Combine the predictions of all models by averaging (for regression) or majority voting (for classification).
Bagging: Bootstrap Aggregating
Cool things in bagging/ random forest
Out-Of-Bag (OOB) Error
- For each bootstrap sample \(B_t\), \(\text{x}_O\notin B_t\) are called Out-Of-Bag samples (\(\approx 37\%\)).
- Tree’s OOB Error: The average error for a tree, calculated using its out-of-bag samples.
- OOB Sample Error: The average error for a given observation, computed from all trees whose bootstrap samples did not include that observation.
- Overall or Average OOB Error: The average of all OOB Sample Errors, providing an overall measure of the model’s performance.
- The Overall OOB Error can be used as an approximation of cross-validation error. However, it tends to overestimate the true error in classification according to Silke Janitza and Roman Hornung (2018).
Bagging: Bootstrap Aggregating
Extra-trees method
- Number of trees: \(T\)
for t = 1,2,...,T:- Build tree: \(f_t\) is built on the Full dataset but a bit differently. At each split,
- Random Features: A random subset \(S_t\) of the full input features is considered.
- Random Split Points: A random set of points \(\{a_1,\dots,a_p\}\) is considered.
- Build tree: \(f_t\) is built on the Full dataset but a bit differently. At each split,
Prediction: (same as before).
Boosting
Algorithm
- Combined model: \(\color{blue}{\hat{f}(x)=\sum_{t=1}^Tw_tf_t(x)}\).
Consensual Aggregation
COBRA by Beau et al. (2016)
Gradient COBRA by Has (2023)
- Given base regressors: \({\bf r}=(r_1,\dots,r_M)\), the combination takes the form: \[\hat{y}=\sum_{i=1}^nW_{n,i}(\text{x})y_i.\]
- Training such a model is equivalent to finding an optimal smoothing parameter \(h\) minimizing cross-validation error:
\[\varphi(h)=\frac{1}{K}\sum_{k=1}^K\sum_{(\text{x}_i,y_i)\in F_j}(\hat{y}_i-y_i)^2.\]
Consensual Aggregation
COBRA by Beau et al. (2016)
Gradient COBRA by Has (2023)
Consensual Aggregation
MixCOBRA by Fischer & Mougeot (2019)
- Given base regressors: \({\bf r}=(r_1,\dots,r_M)\), the combination takes the form: \[\hat{y}=\sum_{i=1}^nW_{n,i}(\text{x})y_i.\]
- In this case, we seek for parameter \((\alpha,\beta)\) minimizing cross-validation error:
\[\varphi(\alpha,\beta)=\frac{1}{K}\sum_{k=1}^K\sum_{(\text{x}_i,y_i)\in F_j}(\hat{y}_i-y_i)^2.\]
Consensual Aggregation
MixCOBRA by Fischer & Mougeot (2019)
Stacking
Super Learner by M. J. Van der Laan (2007)
- It can be used for both: classification and regression.
