🗺️ Content
Motivation & Introducion
Bootstrap Aggregating: Bagging
Random Forest
Boosting: Adaboost & XGBoost
Concensual aggregation: Gradient COBRA
🔗 Ensemble Learning Methods
ITM 390 004: Machine Learning
Motivation & Introducion
Motivation
Why do we trust a panel of judges more than a single judge in competitions?
In medicine, why do doctors often seek second or third opinions for complex cases?
Have you noticed that weather forecasts often give a probability of rain rather than a simple yes/no prediction?
Would the famous “Wisdom of Crowds” principle applies to machine learning?
The keys idea of ensemble learning is combining multiple base models/learners to create a better/stronger one.
Motivation & Tools
Main methods
- We’re going to explore \(3\) main EL methods:
Bagging: Combine nearly decorrelated high-varianced models.
Boosting: Sequentially combine weak learners to create a strong final model.
Stacking: Combination is based on the predicted features.
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
- Train Models: Train a separate model (typically the same type) on each bootstrap sample.
Bagging: Bootstrap Aggregating
Purpose
- Reduce variance of high-varianced base learners (trees) to produce a more stable and accurate final model.
Methodology
- Bootstrap Sampling: Generate multiple subsets of the training data by sampling with replacement.
- Train Models: Train a separate model (typically the same type) on each bootstrap sample.
- Aggregate Predictions: Combine the predictions of all models by averaging (for regression) or majority voting (for classification).
Bagging: Bootstrap Aggregating
Random Forest
- In practice, Bagging doesn’t work so well as the base learners (trees) are not as decorrelated as expected (due to correlated bootstrap samples).
- Random Forest: (What a cool name! 😎)
- Number of trees: \(T\)
for t = 1,2,...,T:- Bootstrap sampling: sample \(B_t\) with replacement from \(\cal D\).
- Random Features: select subset \(S_t\) from full \(d\) input features.
- Build tree \(f_t\) on \(B_t\) using only features from \(S_t\).
Prediction: (same as before).
- It works well as Random Features part introduces even more randomness on top of bootstrap sampling.
Bagging: Bootstrap Aggregating
Cool things in bagging/ random forest
Out-Of-Bag (OOB) Error
- For each bootstrap sample \(B_t\), \(\text{x}_O\notin B_t\) are called Out-Of-Bag samples (\(\approx 37\%\)).
- Tree’s OOB Error: The average error for a tree, calculated using its out-of-bag samples.
- OOB Sample Error: The average error for a given observation, computed from all trees whose bootstrap samples did not include that observation.
- Overall or Average OOB Error: The average of all OOB Sample Errors, providing an overall measure of the model’s performance.
- The Overall OOB Error can be used as an approximation of cross-validation error. However, it tends to overestimate the true error in classification according to Silke Janitza and Roman Hornung (2018).
Bagging: Bootstrap Aggregating
Cool things in bagging/ random forest
Feature Importances (FI)
Mean Decrease in Impurity (MDI)
- The Reduction of Impurity Measures when spliting at \(X_j\) is denoted by \[\color{blue}{\text{RIM}}_t(X_j)=\text{Imp}_{t-1}(X_j)-\text{Imp}_t(X_j).\]
- The importance of feature \(X_j\) within a tree: \(\sum_{t\in S_j}\color{blue}{\text{RIM}_t(X_j)},\) where \(S_j\) is the set of indices \(t\) when a split is performed at \(X_j\).
- Unnormaized MDI of variable \(X_j\) is the sum of FIs from all the built trees.
- ⚠️ Might be biased toward high cardinality features and might not be accurate with highly correlated features.
Bagging: Bootstrap Aggregating
Extremely Randomized Trees (Extra-trees)
- Number of trees: \(T\)
for t = 1,2,...,T:- Build tree: \(f_t\) is built on the Full dataset but a bit differently. At each split,
- Random Features: A random subset \(S_t\) of the full input features is considered.
- Random Split Points: Random points \(\{a_1,\dots,a_p\}\) is considered at each split.
- Build tree: \(f_t\) is built on the Full dataset but a bit differently. At each split,
- Prediction: (same as before).
Boosting
Algorithm
- Combined model: \(\color{blue}{\hat{f}(x)=\sum_{t=1}^Tw_tf_t(x)}\).
Stacking: Consensual Aggregation
General framework
- Stacking: Consensual Aggregation combines \(M\) base learners \(f_1,\dots,f_M\) based on how “close” the predictions given by these learners are.
- Roughly speaking, it’s the nonparametric methods on predicted features: \[\color{red}{\tilde{\text{x}}=(f_1(\text{x}),\dots,f_m(\text{x}))}.\]
- General form: \(\hat{y}=\sum_{i=1}^nW_{n,i}(\color{red}{\tilde{\text{x}}})y_i\)
Stacking: Consensual Aggregation
COBRA by Beau et al. (2016)
Gradient COBRA by Has (2023)
- Given base regressors: \({\bf r}=(r_1,\dots,r_M)\), the combination takes the form: \[\hat{y}=\sum_{i=1}^nW_{n,i}(\text{x})y_i.\]
- Training such a model is equivalent to finding an optimal smoothing parameter \(\color{blue}{h}\) minimizing cross-validation error:
\[\varphi(\color{blue}{h})=\frac{1}{K}\sum_{k=1}^K\sum_{(\text{x}_i,y_i)\in F_j}(\hat{y}_i-y_i)^2.\]
Stacking: Consensual Aggregation
COBRA by Beau et al. (2016)
Gradient COBRA by Has (2023)
- Python package available at
![]()
Gradient Cobra Library.
Gradient Cobra Library