Hybird Model = Distination + Fine-turning Model

Make mixed-boold children model with special professions

Recently Deepseek v3 have so great success about how to make a better and cheap model with combination of many methodologies, like DeepSeekMoE with Auxiliary-Loss-Free Load Balancing, DualPipe and Computation-Communication Overlap, Extremely Memory Saving with Minimal Overhead, FP8 traning...

However, I must say Model Distination is the key point to make great success about Deepseek v3. And I am happy to find it is a production function of Amazon Bedrock Model Distillation (please refer this demo just 4 mins).

So we got to an new age about create Hybird model with distination from supper model to make similar capability but better performance with lower cost. Well, with fine-turning (can also help from another strong model) as a widely accepted approach to enhance the distinated model.