Random forest is one of the widely used machine learning algorithms known for its performance and reliability on almost any kind of data. It is one of the most highly performing algorithms, which is a type of ensemble technique known as bagging in machine learning where the same algorithm is used multiple times, data samples are fed to them, and predictions are made.

Out of Bag score or OOB score is a type of method that helps us to identify if the model is making any mistake on the unknown data and help us refine the performance of the model. It can be used as a performance metric or a score, and based on that, the model’s reliability and flexibility can be determined.

In this article, we will discuss the out-of-bag score or OOB score in the random forest or bagging techniques, the core intuition behind it, how it works, its advantages, and the complete code example for the implementation of the same.

This article will help one to understand the significance of OOB score in bagging techniques, implement it very efficiently, and will help one to answer the interview questions related to the same very easily and efficiently.

Now before directly jumping into the OOB score, let us discuss a bit about the random forest algorithm and how it works.

How Does Random Forest Works?

Random forest is a type of machine learning algorithm that is a bagging ensemble model. In the case of bagging ensembles, multiple models of the same algorithms are taken as the base models, and then subparts or the samples of the data (bootstrapped data) are fed to them individually, now once they are trained; the predictions are taken from each base model, and the final prediction is decided.

Basically, in bagging algorithms, multiple models of the same algorithms are taken, and predictions are made based on their individuality, and to get the final result, the predictions from each model are appended with the help of averaging, weighted averaging, or the most frequent categories.

The random forest is a special case of bagging ensemble techniques, where all the base models are decision trees. So, in short, a bagging model with decision tree base models is known as a random forest. The training and prediction workflow in the random forest is the same as the bagging technique.

Now let us discuss the Out of Bag samples and their core intuition before discussing the OOB score.

The Out of Bag (OOB) Samples

As the name suggest, the Out of Bag (OOB) samples are the samples or the datapoints that are out of the bag or out o the training data. In short, they are the datapoints which are not included in the dataset which is used for the training of the base model. 

These samples are the data observations which were included in the training dataset, but while bootstraping the data, they were left out for training of the particular base model and were not used for the training of the same. 

As we can see in the above image, from the total five observations of the training data, three of them were used for the training of the base model as they were the bootstrapped data, but the two data points that were left and non-included in the bootstrapped data were considered as the Out of Bag samples. 

To clear the idea more clearly, let us take another example. 

Suppose we have a total of 300 data points, and we have three random forest base models to train for the final model. Now while bootstrapping, let us suppose that we are bootstrapping the data in such a way that three equal parts of the data (100 data points for each base model) are used for the training of each base model. 

Now when we go for the training of the first base model with the first bootstrapped data with 100 samples, the other 200 data samples will be considered as the OOb samples of the model. And hence, for each base model, the OOB samples are different and unseen. 

Now let us discuss the OOB Score and how it is calculated. 

The Out of Bag (OOB) Score 

Now as we have discussed in the above section, the OOB samples are the data observations or the data points that are not included in the training data of the models. These OOB samples are then used to evaluate each base model. Note that here for each base model, the OOB samples will be different and unseen. 

Now once we have trained each base model on the bootstrapped data, the OOB predictions are made in which the OOB samples are fed to the model, and the model makes the prediction for the same. These predictions are then compared with the actual value of those samples, and the final result (or OOB Score) is obtained by integrating the performance of each base model on the OOB samples. 

Note that here we can not evaluate each base model; the final OOB score is simply the representation of all of the base models on the OOB or the Unseen samples. 

Now let us discuss the difference between the OOB score and the validation score. 

OOB Score Vs. Validation Score

As we have discussed the OOB score in the above section, a gentle question may appear in your mind, how the OOB score is different from the validation score, and what makes them very different from each. 

However, the OOB score and validation score are both used for evaluation purposes of the model, and there are some major differences too between them. Let us discuss them one by one. 

1. Data Source

In the case of the OOB score, the OOB samples are part of the training data only, and the OOB score is calculated from the same, whereas in the case of validation sources, the completely separated data sources or the testing data sources are used to evaluate the model. 

2. Evaluation Process

In the case of the OOB score, the score is the result of the performance of the different base models on unseen training data, and hence the evaluation is somewhere related to the training process only, whereas in the case of the validation score, the complete separate data is used to evaluate the trained model. 

3. Availability

As the OOB score is obtained from the unseen training data samples, the score si easily available after the training of the model, and there is no need for any further process to obtain this score, whereas, in the case of validation score, we need to again make the prediction on the testing set with the model to get the validation score.   

Now let us discuss the advantages of the OOB score. 

Advantages of OOB Score

OOB score offers many advantages, which help while training and building a model. In most cases, the OOB score is used while training bagging or a Random Forest algorithm.

1. Efficiency

The OOB score is calculated without further separating the data, which reduces the time taken and computational power. Basically, efficient model training can be done with the help of the OOB score without any more separation of the dataset.

2. Performance Analysis

The OOB Score clearly gives us an idea about the performance of the model on unseen type of data, which help us determine the performance of the model on foreign data and help us enhance the performance of the model by tuning the parameters of the same.

3. Highly Replicable

The OOB score is completely replicable as the same model with the same dataset, and the same algorithms will produce the same OOB score every time. Basically, it will not change with time if the dataset and the algorithm used for the model are the same.

4. Robust to Overfitting

The OOB score is very robust to overfitting, and hence there is no effect of overfitting in the calculation of the OOB Score, making it more flexible and robust for the model.

5. Imbalanced Data

The OOB score can be used on imbalanced data as well, and it will have no effect on the imbalanced dataset. Here OOB score will be calculated the same as in the case of balanced data.

However, there are some disadvantages of the OOB Score, which limits the use of the same in some cases. Let us discuss those in the next section.

Disadvantages of OOB Score

There is no doubt in the efficiency and helpfulness of the OOB score, but still, there are some disadvantages to the OOB score, which sometimes limit its use of the same.

1. Overestimated Performace

Sometimes in the case of the OOB score, since we are cutting off some of the parts of the training data to get the OOB samples, we are basically cutting the information that is to be provided to the model. 

Hence the model may not capture the complete information of the training data as some parts of the data remain unseen to the model. In such cases, the model’s performance can be overestimated, or the model may not be reliable.

2. Limited Information

The OOB score for each complete model is single only, meaning that the OOB score is calculated for the whole model, not the base models, and hence we can not get the complete idea of where or in what base model the error is the maximum and hence the OOB score provided limited information of the performance of the model.

3. Baised OOB Score

In some cases, the OOB score can be biased as it is calculated by measuring the performance of the model on the unseen training data while the training process only. Hence the bias arises here as the OOB samples are not completely independent of the training process.

Now let us discuss the code example of the OOB score to implement the OOB score in the Random Forest Algorithm.

OOB Score: Code Example

OOB score can easily be calculated for the random forest algorithms with the help of the Sci-Kit Learn library. Here we will use the dummy dataset to get the OOB score for the model with the random forest algorithm.

Step 1: Import Required Libraries

Importing required libraries, which will be needed for getting the model trained and calculating the OOB score.

from sklearn.ensemble import RandomForestClassifier

from sklearn.datasets import make_classification

from sklearn.metrics import accuracy_score

Step 2: Generate the Dataset

Now let us generate the dataset with 1000 rows or samples and 10 columns or features with two classes, where the problem is now classification supervised machine learning.

element:not(.old)::after {
    content: "New!";
    color: deepPink;
}   

X, y = make_classification(n_samples=1000, n_features=10, n_classes=2)

Step 3: Training the Random Forest

Now training the random forest algorithm and fitting the training data with the same. Note that before training the model, the oob_score parameter is set to the True value. This parameter will help to calculate the OOB score for the model.

rf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
rf.fit(X, y)

Step 4: Getting the OOB Score

To get the OOB score of the model, we will need to use the oob_score_ function for the model.

oob_score = rf.oob_score_

Step 5: Printing the Accuracy

Getting the accuracy of the model on the training dataset and unseen dataset.

y_pred = rf.predict(X)

accuracy = accuracy_score(y, y_pred)

print("OOB score:", oob_score)

print("Accuracy on training data:", accuracy)

Output

OOB score: 0.931

Accuracy on training data: 1.0

Now let us discuss the advantages of the OOB score and how it helps in certain ways.

Key Takeaways

1. Bagging is an ensemble technique where multiple models of the same algorithm are taken, and the bootstrapped data is fed to each base model.

2. Random forest is a special type of bagging technique where all the base models are taken from decision trees.

3. It is very important to have an idea about the performance of the model on the unseen type of dataset.

4. OOB score is a type of score or a metric that signifies the performance of the model on unseen datasets with the help of the model’s prediction on the OOB samples that are taken from the bootstrapped data.

5. OOB score helps us to identify the performance of the model on the unseen dataset and gives an idea if any tuning for the model is needed.

Conclusion

In this article, we discussed the out-of-bag score or OOB score in bagging and random forest algorithms, it’s core intuition, how it is calculated, its advantages, and the complete code example for the implementation of the same.

To conclude, the OOB score, which is the performance metric of the model on the unseen dataset, can be used to identify the model’s actual performance on unseen or foreign datasets and can help optimize the performance of the model by giving the true scenario of model’s performance on the unseen training data.

This article will help one to understand the OOB score clearly, implement it whenever necessary, and will also help to answer interview questions related to the same.