How can we identify the weights of different models for ensemble?

One of the most common challenge with ensemble modeling is to find optimal weights to ensemble base models. In general, we assume equal weight for all models and takes the average of predictions. But, is this the best way to deal with this challenge?

There are various methods to find the optimal weight for combining all base learners. These methods provide a fair understanding about finding the right weight. I am listing some of the methods below:

• Find the collinearity between base learners and based on this table, then identify the base models to ensemble. After that look at the cross validation score (ratio of score) of identified base models to find the weight.
• Find the algorithm to return the optimal weight for base learners. You can refer article Finding Optimal Weights of Ensemble Learner using Neural Network to look at the method to find optimal weight.
• We can also solve the same problem using methods like:
  • Forward Selection of learners
  • Selection with Replacement
  • Bagging of ensemble methods

You can also look at the winning solution of Kaggle / data science competitions to understand other methods to deal with this challenge.