How to incorporate automations in Data Virtualization Layer using Python !!

What is Data Virtualization?

Data virtualization is a logical data layer that

• unifies all enterprise data scattered across various systems
• controls the unified data for centralized security and governance
• and distributes it in real time to business users.

Note: We will mainly focus on Denodo (Data Virtualization tool) as the scope of this article.

How can we automate Data virtualization Layer?

If automation is your interest then Python can be one of the easiest way forward to incorporate in data virtualization.

Denodo specifically generates several VQL queries for each of the button clicks on the interface level of VDP admin tool it generates VQL in VDP Queries Log.

We can take those VQL statements and execute it on bulk for all the Denodo views as per our requirements using python.

Few Automation Ideas are:

1. Single Dimensional Row Level/Column Level Security implementation on all Denodo views.
2. Data count check/ Sanity Check if all Denodo views are executing fine or not.
3. Create Denodo views programmatically with details like source database name, schema name, and table names.
4. Finding Denodo Views and its source table mapping in Bulk.
5. Data ingestion to remote tables in applications like SAP-HANA in Bulk.

Frequently used Python libraries for these kind of Automations:

• jaydebeapi
• psycopg2
• numpy
• pandas

Frequently used stored procedures:

• GET_VIEWS()
• GET_ELEMENTS()
• GET_VIEW_COLUMNS()
• GET_SOURCE_TABLE()
• LIST_JDBC_DATASOURCE_TABLES()

How to Connect Denodo using Python?