Adaboost Using Caret Package in R

Last Updated : 23 Jul, 2025

Adaboost (Adaptive Boosting) is an ensemble learning technique that combines multiple weak classifiers to create a strong classifier. The caret package in R provides a convenient interface for training Adaboost models, along with numerous other machine-learning algorithms. This article will walk you through the theory behind Adaboost and demonstrate how to implement it using the caret package in the R Programming Language.

Overview of Adaboost

Adaboost works by sequentially training a series of weak classifiers, typically decision trees with a single split (also known as decision stumps). Each classifier is trained to correct the errors made by the previous classifiers. The main idea is to focus on the difficult-to-classify instances by adjusting their weights, thereby improving the overall performance of the model.

Steps of Adaboost Algorithm

  1. Initialize weights: Assign equal weights to all training instances.
  2. Train weak classifiers: Train a weak classifier on the training data.
  3. Evaluate and update weights: Increase the weights of misclassified instances so that the next classifier focuses more on them.
  4. Combine weak classifiers: Combine the weak classifiers into a single strong classifier, typically by taking a weighted vote.

The final model is a weighted sum of the weak classifiers, where the weights are based on the classifiers' accuracies.

Step 1: Install and Load Required Packages

First, ensure you have the caret and adabag packages installed. The adabag package is used by caret to implement Adaboost.

R 
install.packages("caret")
install.packages("adabag")
library(caret)
library(adabag)

Step 2: Prepare the Data

For this example, we'll use the well-known iris dataset, which is available in R by default.

R 
# Load the iris dataset
data(iris)

# Set a seed for reproducibility
set.seed(123)

Step 3: Define the Training Control

Define the training control using the trainControl function. Here, we'll use 10-fold cross-validation to evaluate the model's performance.

R 
# Define training control
train_control <- trainControl(method = "cv", number = 10)

Step 4: Train the Adaboost Model

Use the train function from the caret package to train the Adaboost model. The method parameter should be set to "AdaBoost.M1" to specify the Adaboost algorithm.

R 
# Train the Adaboost model
adaboost_model <- train(Species ~ ., data = iris, method = "AdaBoost.M1", 
                        trControl = train_control)

# Print the model
print(adaboost_model)

Output:

Boosted Classification Trees using AdaBoost.M1 

150 samples
4 predictor
3 classes: 'setosa', 'versicolor', 'virginica' 

No pre-processing
Resampling: Cross-Validated (10 fold) 
Summary of sample sizes: 135, 135, 135, 135, 135, 135, ... 
Resampling results across tuning parameters:
  
  Accuracy   Kappa
  0.96       0.94

Step 5: Evaluate the Model

After training, you can evaluate the model's performance by examining the results.

R 
# Predict on the training set
predictions <- predict(adaboost_model, newdata = iris)

# Create a confusion matrix
confusion_matrix <- confusionMatrix(predictions, iris$Species)
print(confusion_matrix)

Output:

Confusion Matrix and Statistics

            Reference
Prediction   setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         47         3
  virginica       0          3        47

Overall Statistics
                                          
               Accuracy : 0.9733          
                 95% CI : (0.9286, 0.9932)
    No Information Rate : 0.3333          
    P-Value [Acc > NIR] : < 2.2e-16       

Kappa : 0.96   

Mcnemar's Test P-Value : NA              

Statistics by Class:

                     Class: setosa Class: versicolor Class: virginica
Sensitivity                1.0000            0.9400           0.9400
Specificity                1.0000            0.9800           0.9800
Pos Pred Value             1.0000            0.9400           0.9400
Neg Pred Value             1.0000            0.9800           0.9800
Prevalence                 0.3333            0.3333           0.3333
Detection Rate             0.3333            0.3133           0.3133
Detection Prevalence       0.3333            0.3333           0.3333
Balanced Accuracy          1.0000            0.9600           0.9600

Conclusion

Adaboost is a powerful ensemble learning technique that can significantly improve the performance of weak classifiers. The caret package in R provides a user-friendly interface for implementing Adaboost and other machine learning algorithms. By following the steps outlined in this article, you can easily train and evaluate an Adaboost model on your dataset. Understanding the theory behind Adaboost and how to implement it using caret will help you leverage this technique effectively in your predictive modeling tasks.

Comment

Explore