Exploratory Data Analysis Using D-Tale Library

D-Tale for interactive data exploration

D-Tale is python library allows us to visualize a Pandas DataFrame. D-Tale generates an interactive graphical interface.

D-Tale presents a variety of details about the data provided. It supports a wide range of file formats including CSV, TSV, XLS, XLSX. It is a Python library built with Flask backend and React as the frontend.

There are two ways in which we can start a D-Tale interface and load the data in Jupyter notebooks:

1. Either pass the dataframe object to the D-Tale function. This instantiates the GUI within the Jupyter cell only.

import dtale

dataset = pd.read_csv("eda_train.csv")

dtale.show(dataset) ?????        

2. Initialize the D-Tale interface without passing the dataframe. It will show up an interaction menu with the GUI to load the data and provide various other options too.

import dtale

dtale.show(open_browser=True)        

?As soon as we run this code, we will get this GUI menu:

First-time interface

Here we have the following options –

1. Loading data from a file
2. Loading data from websites. Here we need to pass the link of the website from where we can fetch files such as CSV, JSON, TSV, or Excel.
3. Loading sample datasets. These datasets may require some background downloading to fetch the datasets from the server.

As we load the dataset, a table will be displayed just like the pandas dataframe. All the cells of this table can be edited, and we can make direct changes to the values just like in excel.

Column Menu Functions

Whenever we click on the column header, we will get a list of options depending upon the type of data the column contains. The common thing in all three of them is sorting by Ascending or Descending order. Other than that, there would be different filter approaches for different types of data.

Also, in the string type column, there would be no heat map and Variance Report option but there will be a new option called Clean Columns which is not present in integer and datetime value column.

?1. Lock

?The lock option sticks the column to the left of the screen allowing we to freely scroll/navigate to other columns without the locked column being displaced. This can prove useful in cases when we want to have a look at columns that are placed apart.

?2. Hide and Delete

The hide option removes the column from the dataframe view. It is not deleted from the actual dataframe. We can simply unhide the column from the top right strip.

The delete option will remove the column from the dataframe permanently. It is similar to the pandas drop function. In the backend, it is iterating over the list of columns to select the column to delete from the dataframe.

?3. Replacements and Type conversion

A replacement option is used for replacing some values of the column with a constant or nan value. We can make this replacement in place or create a separate column. Replacement type can also be defined as replacing specific values, spaces, or specific string replacement.

?4. Describe

The describe function in pandas helps in providing a statistical summary of the column or the dataset. The describe option here works in the same way but it provides way more information than the normal pandas functions. As it is named column analysis, it provides a unique summary for each data type.

It also generates histogram and value_counts graph for the features.

For integer type columns, it provides measures of centrality and spread along with the frequency of most frequent value along with Kurtosis and Skewness. It also represents the data in the box plot, histogram, value_count plot, and Q-Q plot.

?Pic For string type columns, it provides the most frequent word and its frequency, detailed summary on characters present, word value count plot, and value counts plot.

??5. Filters

These are used to make a subset of the data. Filtering data in D-Tale is very easy and we just need to specify the type of filters we want.

?6. Variance Report

This option is not available for string-type values. Variance report shows whether the feature has low variance or not. It decides this based on two checks:

1. Count of unique values in a feature / sample size < 10%
2. Count of most common value / Count of second most common value > 20

?It displays the result with the calculations and a histogram to present the findings.

?7. Clean columns

This option is only available for string-type values. D-Tale provides all the possible text cleaning methods that can be applied to the text. We simply need to select the methods we want to apply to the text and the work will be done in the backend.

8. Formats

With the option Formats we can define how the numbers are displayed.

Main Menu Options

The main menu has almost all the same options as provided in the column menu but in the main menu, they are generalized, and we can do the operations in one place on multiple columns rather can manually picking them from the display. Here are some of the options which are exclusive to the Main menu and work differently.

1. Build Column

This option allows us to create new features/columns out of the already available columns. We can create these new features by performing arithmetic operations on columns or using two columns to perform operations. We can also provide the name of the new column to be made and its datatype.

2. Summarize Data

In pandas, we summarise the data via group-by or pivot tables. The same thing we can do with this package too. The pandas required us to write the code for every group by and pivot tables but with D-Tale, we can select the columns, the aggregation function, and the columns we want in the final dataset.

3. Missing Analysis

D-Tale uses missingno python package to visualize the missing values present in the dataset. It provides matrix, bar, heatmap, and dendrogram too.

4. Charts

D-Tale uses plotly to create interactive plots on the go. It offers Line, Bar, Scatter, Pie, word cloud, Heatmap, 3D scatter, Surface, Maps, Candlestick, Treemap, and funnel charts. Different types of data support different types of plots.

5. Highlighters

These are used to highlight some sections of the dataset. Like we use stylers in pandas to bring out the odd values, highlighters do the same job. We can highlight missing values, Data types, Outliers, and range.

6. Code Export and Data Export

All the operations we have done on our dataframe in D-Tale are automatically converted into their python/pandas/plotly equivalent code. They can be accessed by clicking on the export code option present at every operation and chart GUI.

The code export option in the main menu captures all the changes done on the dataframe. We can directly export the final dataset after changes to CSV or TSV using the export option.

You can check my GitHub profile for code.

?