How do you merge Machine Learning models?

it's clear that many people already used model ensembling (combined models) : my contribution here is just to mention that during my work and when I used this technique, I had two ways to calculate the combine predictions : 1 - simple averaging like : final_pre = (pre1 + pre2) / 2 2 - Weighted average where I give more importance to the model that performs better : w_model1 = 0.6 w_model2 = 0.4 final_pred = ((w_model1*pre1)+(w_model2*pre2))/ 2

During my time as a data science student, I recall a project focused on predicting customer turnout for a telecom company. We tested diverse machine learning models like Random Forest and Gradient Boosting, each with its strengths and weaknesses. While they performed decently on their own, merging them using stacking, noticeably improved results. This showed me how blending model predictions enhanced accuracy by leveraging their unique insights, highlighting the performance of ensemble techniques.

For example, any recommendation engine will typically be optimizing various steps of the customer journey. Having a model for each of the steps and merging them for the overall outcome can be super helpful and reduce the class imbalance problem commonly faced in recommender systems for internet.

Another usecase is when there are large foundational models like LLMs that are too expensive to train. It is always better to merge the additional model layers alongside the frozen layers of the LLMs. This is successfully used for aligning LLMs or adapting them to related usecases.

I am kind of interested to understand how come LinkedIn came up with this topic. In my mind, "how to create ML model network" is a better topic. We can use some "simple" model to create a "bigger" model (network), for example, Stable Diffusion can be seen as a network with many models to fulfill each step functions. Similar, we can use existing models to create a (business) model network to fulfill some business cases, which is the topic of this blog, I think. The later model network can be run within same deployment, or multiple deployments.

In addition to standard decision tree techniques of bagging and boosting that the library will offer, one always has the option of taking the output of each of the models and using those as features for another model on top. This new model can be from a totally different technique. It can be a decision tree, a DNN or even a simple regression that most traditional systems would have used. Merging neural networks is even easier. You can feed the output of as many networks as you want to the next layer. Using any of the modern libraries like torch or tensorflow makes it simple to define your model architectures. In practice it is no different from treating the whole combined graph as one model with the input networks layers being frozen.

Merging the input features along with the output of simpler models is also seen to be successful. One could argue that a lot of the gains of the transformer models popular in generative AI come from merging the raw features via the residual connections with a set of outputs from the previous block and repeating the process.

Some other topic like "how to deploy merged models to gain best performance?" also need to considerate when deploying a merged network models.