Large number of ML models' parameter tuning

We have to do cross-validation for 1K models. each model needs to do parameter tuning over 100+ combinations of hyper-parameters. HOW?

So now we need thousands of models to train together, and each model has its own data set, but the parameters they need to tune are shared, so how to do it? Our specific approach is this: we will have an infinite loop, and each iteration in your loop will randomly select a model and a combination of parameters. If the combination of this model and this parameter has not been tested, then we will start the experiment and write the results of the experiment into the database. But the problem is that for different models, the number of experiments and resources they obtain are different, and the number of experiments required by each model is also different. Some models get very good results in the first experiment. Some models may be very difficult and require more than ten experiments. Our specific approach is to balance the performance of different models, as well as the number of experiments, that is, every time we randomly select a model. , we will see if the best result he has obtained now is better than 30% or 50%, or 70% of the other models in the database. If so, then he will pass it.