Understanding the logic behind AdaBoost and implementing it using Python
In machine learning, boosting is a kind of ensemble learning method that combines several weak learners into a single strong learner. The idea is to train the weak learners sequentially, each trying to do its best not to repeat the mistakes its predecessor made, eventually building a strong ensemble.
In this article, we will learn about one of the popular boosting techniques known as AdaBoost and show how elegantly it allows each weak learner to pass on their mistakes to the next weak learner to improve the quality of predictions eventually.
We will cover the following topics in this article:
- Understand what is an ensemble and its different types. That’s where we will define boosting.
- Understand what is meant by a weak learner using an example since boosting involves fitting multiple weak learners.
- Understand why there was a need for boosting algorithms.
- Introduction to AdaBoost algorithm.
- Implement the AdaBoost algorithm for binary classification in Python and look at the individual weak learners as well as the final predictions with some interesting visualizations.
- Finally, we will wrap up.
In the opening sentence, I mentioned that boosting is a kind of ensemble learning that combines several weak learners into a single strong learner. Before digging in, we must be familiar with these terminologies first. So, let’s get straight into it!
An ensemble is a collection of multiple base models that can be aggregated together in such a way that the quality of the final predictions is much better than what an individual base model could have produced by itself.
In terms of the diversity of base models, ensembles can be of two types:
- Homogeneous ensemble, in which…
