Ensembling¶
What is ensembling?¶
Model ensembling is using a combination of models in order to increase performance beyond any of the individual models
An ensemble is an type of supervised algorithm where we give a weight to the output of each model under consideration. In practice, we tune the weights in order to maximize performance on a validation set. We also set a restriction on the weights making them sum to one.
Why use ensembling?¶
Different types of models have different strengths and weakness
Some models are better at seperating certain types of noise from signal
Some models are better at avoiding overfitting/underfitting to certain types of data
Having multiple models will likely reduce the variance in our predictions, reducing overfitting.
Example of ensembling two models with the Boston housing data set¶
- Ensemble of a support vector machine and a random forest model
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_boston
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
%matplotlib inline
Load data and split into training, validation, and test sets¶
In [2]:
np.random.seed(1)
X, y = load_boston(return_X_y=True)
# 50% train, 25% val, 25% test
num_train = int(round(len(X)*0.6, 0))
num_val = int(round(len(X)*0.2, 0))
random_indices = np.random.permutation(len(X))
train_indices = random_indices[:num_train]
val_indices = random_indices[num_train:(num_train+num_val)]
test_indices = random_indices[(num_train+num_val):]
X_train = X[train_indices]
y_train = y[train_indices]
X_val = X[val_indices]
y_val = y[val_indices]
X_test = X[test_indices]
y_test = y[test_indices]
print('X_train shape:', X_train.shape)
print('y_train shape:', y_train.shape)
print('X_val shape:', X_val.shape)
print('y_val shape:', y_val.shape)
print('X_test shape:', X_test.shape)
print('y_test shape:', y_test.shape)
In [3]:
pd.DataFrame(X_train).head()
Out[3]:
Scale inputs¶
- based on the mean and standard deviation of training data
In [4]:
X_train = (X_train - X_train.mean(axis=0))/X_train.std(axis=0)
X_val = (X_val - X_train.mean(axis=0))/X_train.std(axis=0)
X_test = (X_test - X_train.mean(axis=0))/X_train.std(axis=0)
In [5]:
pd.DataFrame(X_train).head()
Out[5]:
Define & train models¶
In [6]:
svm = SVR(kernel='linear', epsilon=8, C=0.0003)
rf = RandomForestRegressor(n_estimators=64, max_depth=2, max_features=3)
svm.fit(X_train, y_train)
rf.fit(X_train, y_train)
print('done training')
Evaluate performance¶
In [7]:
y_vpred_svm = svm.predict(X_val)
y_vpred_rf = rf.predict(X_val)
mse = []
for i in np.arange(0, 1.1, 0.1):
# start with 100% svm
ensemble_pred = y_vpred_svm*(1-i) + y_vpred_rf*i
mse.append(mean_squared_error(y_val, ensemble_pred))
plt.plot(np.arange(0, 1.1, 0.1), mse)
#plt.ylim(0)
plt.xlabel('Weight for random forest model')
plt.ylabel('MSE')
plt.show()
We can see from the plot above and the table below that the MSE is minimized on the validation set when we give the the random forest model's prediction a weight of 0.6 and the support vector machine's prediction a weight of 0.4¶
In [8]:
ensemble_scores = pd.DataFrame({
'RF Weight':np.arange(0, 1.1, 0.1),
'MSE':mse
})
ensemble_scores
Out[8]:
Evaluate on test set¶
In [9]:
rf_weight = ensemble_scores['RF Weight'].iloc[ensemble_scores['MSE'].idxmin()]
y_pred_svm = svm.predict(X_test)
y_pred_rf = rf.predict(X_test)
y_pred_ensemble = (1-rf_weight)*y_pred_svm + rf_weight*y_pred_rf
print('SVM MSE on test set:', mean_squared_error(y_test, y_pred_svm))
print('RF MSE on test set:', mean_squared_error(y_test, y_pred_rf))
print('Ensemble MSE on test set:', mean_squared_error(y_test, y_pred_ensemble))