How do you visualize your data with pandas and other libraries?

I agree! Loading and inspecting your data is absolutely the first step in data analysis. It’s very important to understand your data’s structure, characteristics, and quality to identify any potential issues. Pandas makes it super easy to load data from various sources to create a DataFrame, and begin your analysis. By thoroughly "investigating" your data, you’ll be equipped to make informed decisions and draw meaningful insights from it. Something that has been very helpful to me is not to refrain from drawing conclusions until theanalysis is complete. I want a 360-degree understanding of the data before I reach a conclusion and share my insights with others.

In Python, visualizing data with Pandas typically involves leveraging libraries like Matplotlib and Seaborn. Pandas itself can plot data directly from DataFrames and Series using its built-in methods that are essentially wrappers around Matplotlib. For more sophisticated visualizations, Seaborn provides a higher-level interface that is easier to use for creating attractive and informative statistical graphics. Additionally, libraries like Plotly offer interactive capabilities that allow users to explore data more deeply through dynamic visualizations.

Before we can visualize data, we first need to load it into Python using the pandas library. Pandas provides a number of functions for reading data from various sources, such as CSV and Excel files, databases, and web APIs. Once we have our data loaded, we may need to clean and transform it to make it suitable for visualization. This can involve tasks like removing missing values, converting data types, and aggregating data to a coarser level of granularity.

Let me talk about EDA on images. This is highly different from EDA on tabular data because you are dealing with pixels. Usually these kind of data are stored in datalakes such as GCS, S3, Azure blob store in case of well known cloud providers. You can use couple of tools such as labellerr, VisualLayer and CVAT or jupyter notebook to load the data from cloud. Once it is connected, visualize a few images randomly. If the data is raw and unlabeled, then EDA can be mainly around checking pixel distributions, clustering the images, brightness, contrasts, height, width distributions etc Similarily, other unstructured data such as videos, pdf, audio etc also has their own challenges and techniques.

Again i will limit my answer to image data type. Here transformations and cleaning could mean following 1. Color space transformations - Adjusting optimal brightness, contrast etc 2. Increasing the resolution in case of blurry images or 3. Doing some basic augmentations such as rotation, transformation 4. Generative new images with some domain inputs, thanks to generative AI models 5. Random erasing of the data to create new data points 6. Breaking down high resolution images into small tiles for better handling Such techniques help improve the performance of deep learning models, reduce overfitting, and increase the robustness of the models

Pandas provides a number of functions for basic data visualization, including line plots, scatter plots, and histograms. These functions are built on top of Matplotlib, a widely used plotting library in Python. We can customize the plot by passing additional arguments to the plot() method, such as title, xlabel, and ylabel.

While pandas provides basic data visualization capabilities, we may need to use Matplotlib directly to create more complex plots or customize existing ones. Matplotlib is a low-level plotting library that provides a high degree of control over the appearance of plots.