Transfer learning makes use of a pre-trained model to train another model in a similar domain Model pruning is the process of elimination of unnecessary parameters for desired task Hyperparameter tuning is the process of experimenting with different variables that influence model learning Neural architecture search (NAS) can facilitate faster development, improve network architecture compared to human-created designs AI quantization can lead to 4 × reduction in model size Deployment runtime modification : Open Neural Network Exchange Runtime (ONNX) simplifies, streamlines deployment Select libraries, frameworks that best fit your underlying hardware Regardless of the techniques, it is important to approach AI model optimization

Always begin with what you can comprehend and your machine can bear. - Start with defined performance thresholds - Begin with simpler algorithms, like KNN, Regressions etc. - Reduce dimensionality and isolate useful features with PCA, or others - Optimize smartly, use RandomGridSearch, Bayesian Optimization, or Early Stopping. - Approximate using quantization and pruning. - Reduce the model's complexity. Start with simpler architectures. - Subsample data! Stratified sampling or mini-batch training can do a great help. - Make smart use of hardware (GPUs), parallel processing, and distributed computing.

Balancing model accuracy and computational efficiency in ML projects is crucial. One effective strategy is small sample testing to evaluate speed, efficiency, and accuracy. By running experiments on smaller data subsets, you can quickly gauge a model's performance. For example, models like XGBoost and Random Forest often exhibit similar accuracy, but their computational times can vary significantly. XGBoost may run faster due to its optimized gradient boosting framework, making it more efficient on large datasets. Use small sample tests to identify such differences early, allowing informed decisions about which model best meets the project’s accuracy and efficiency needs while managing computational resources effectively.

The manner in which I have always tried to resolve the answer to this question is by asking myself the following question. What is the lift in performance that each added layer of complexity brings to my ML Model? For example, do I really need to build a neural network over my existing XG Boost model? What is the lift I am observing from doing so? Is this added complexity justified? What is the added benefit of having 40 features as compared to only 20? What is the benefit of using grid search for training vs bayes search cv for optimizing hyper parameters? At each step, it is important to measure the incremental impact and remove the layers that are not improving the performance drastically.