What are some tips for visualizing neural network performance during training and testing?

If you have a long-running training, it makes sense to plot these curves live as the epochs progress. Depending on the framework used, this could be trivial. Most experiment trackers (like MLflow, neptune.ai, W&B) also track these live. So you can see how the training progresses in real-time and can abort training if a errors start increasing without waiting for the training to complete.

Reviewing learning curves during training is an essential way to diagnose problems with learning, such as an underfit or overfit model as well as whether the training and validation datasets are suitably representative of the problem domain. An underfit model can be identified from the learning curve of the training loss only. It may show a line or noisy values of relatively high loss, indicating that the model was unable to learn the training dataset at all. An overfit model can be identified by the plot of training loss decreasing with experience, whereas the plot of validation loss decreases to a point and then begins increasing again. The inflection point in validation loss may be the point at which training could be halted.

Showing how a model performs is tricky when you introduce variables like error rate, correlation, forecast, or bias. Since what can be considered positive for a model could be negative vs. similar iterations is not recommended to show in a good or bad scenario, try neutral colors. Data modeling is an iterative process by design. Added data noise can lead to bad interpretations so I recommend separating secondary metrics from the main chart, either in a table or in widgets near the data viz as complementary data.

For classification problems a confusion matrix is a visual summary of actual and predicted values that can be used as the basis for calculation of most accuracy metrics required by an analysis. It displays the actual number or ratio of True Positives, False Positives, True Negatives and False Negatives for classification problems. From these numbers one can calculate metrics such as Precision/ Recall, Sensitivity/ Specificity, F1 scores etc. depending on your use case. Confusion matrices can be used for binary or multi-class classification problems.

Embeddings are essentially ways to vectorise non-numeric variables for use in Machine Learning algorithms. The most basic embedding used in ML is probably One Hot Encoding used for encoding categorical variables or Bag of Words for encoding words in a corpus for Natural Language Processing. Both these suffer from several problems: you have to add a number for each category/ word, similar entities are not closer in the embedding space. To overcome these problems embeddings are typically learned from the problem space. For NLP problems Word2Vec has been used since 2013, but recent developments include the creation of contextualized word embeddings by using cutting-edge models like BERT and GPT2 to generate vectors representing words.

Imagine training a neural network as coaching a parrot to sing. Visualize the loss curve as the parrot's musical score, model output as its singing, and validation as the teacher's feedback. Adjust hyperparameters like changing the parrot's diet and watch for overfitting like the parrot singing differently at home vs. a talent show. Learning rate controls volume, batch size reflects the audience, and regularization is like voice lessons. Use monitoring tools to check the parrot's health.