How do you evaluate the reliability of your metrics?

To assess metric reliability: 1. Understand theory and alignment with objectives. 2. Ensure data quality and address outliers. 3. Confirm consistency and reproducibility. 4. Validate with domain experts and benchmarks. 5. Analyze sensitivity and statistical significance. 6. Check for bias/fairness concerns. 7. Seek peer review for external validation. 8. Communicate limitations and relevance clearly.

In addition to this, always check for the imbalance in your dataset. Before choosing any metrics make sure that the result is not biased.

Also keep in mind that metrics and data selection are deeply connected. Sometimes you get better metrics if you do not use noisy data, but removing the noisy data can bias the distribution probability, and actually a worst metric when including the noisy data could mean a better model.

Skewed data can have a significant impact on metrics in production, especially when the data distribution is highly asymmetric. It can affect various metrics depending on the nature of the data and the specific metric being used. Dealing with skewed data in classification problems is crucial to ensure accurate and reliable modeling. Here are some commonly used techniques to address the issue of skewed data: - Resampling Techniques*, - Class Weighting, - Cost-Sensitive Learning, etc. *You have to be carefull with resampling, due to oversampling can lead to overfitting, while undersampling may result in loss of important information.

When working in a ensambling learning scenario. Is good practice to work blindling (not realising details of each individual model and data manipulation). That way confimation bias is avoided.

Comparing ML model errors against human performance (in some cases) allows us to evaluate the accuracy and effectiveness of the model. For instance: human performance serves as a benchmark for determining how well the model is performing in understanding and transcribing speech. It helps us assess the quality of the model's output and identify areas where improvements are needed.

As a data scientist, it is essential to communicate insights effectively. I strive to translate complex findings into compelling narratives that resonate with stakeholders. I make a conscious effort to relate my findings to business use cases and demonstrate how they impact the growth of the company. When delivering presentations, I structure my thoughts logically explain my approach, share findings, and discuss implications.

Carbon footprint is something you can measure when you run your code. It is good to think on this as a test of how efficient is your training code.

Explainable Artificial Intelligence (XAI) is a research area that aims to increase the transparency and interpretability of machine learning models. Its importance lies in its ability to provide insights into the decision-making process of models, and to enable users to evaluate their reliability, fairness, and potential biases. XAI techniques ensure that models are trustworthy, explainable, and suitable for their intended use cases. As an evolving field, ongoing research is necessary to improve the interpretability and evaluation of machine learning models.