Working with Dataverse data - the 'legacy' of the Common Data Service

In my last article on the Common Data Model, we learnt that out of the box, Power Apps provides you with two default unmanaged?solutions:

?? Common Data Service Default Solution?- this solution is identified as available for Makers to use by default for their customizations in an environment, and is noted as "useful when you want to evaluate or learn Power Apps".

and

???Default Solution?- this is a special solution that contains all components in the system, and is useful for discovering all the components and configurations in your system.

?

?If you've ever evaluated the available reporting and analytics options for Microsoft Power Platform you will, of course, have considered Power BI .?You might also have noticed (either then or now) the uncanny similarities in the way the two options for using Dataverse data in Power BI are laid out:

? ? Common Data Service (legacy) connector?- the earlier version of the connector that uses the OData connection to Dataverse. You should use this connector for large datasets that are greater than 80 MB. This version also supports paging of the query results and building reports that use the image data type. OData connections import data into a dataset and are refreshed on a regular interval.

?and

? Dataverse connector?- this option uses the Tabular Data Stream (TDS) protocol to connect to Dataverse.?This connector can either import the data into the dataset or use Direct Query, which permits real-time data.

?

This isn't a coincidence, and in this article, we'll explore:

• ?The breadcrumb trail of this alignment ???♀?
• ?How that trail enabled the multiple ways Microsoft Dataverse now offers to connect with business data from various sources ??

?and

• ?The deprecation of the Microsoft Dataverse (legacy) connector and what this could mean across the Power Platform ??

Abstraction

?As we learnt in my last Heuristics article, multiple systems and platforms currently implement a Common Data Model.?Internally to 微软 , this includes:

• Microsoft Azure data services
• Power BI dataflows
• Microsoft Dataverse
• and more…

?

When you provision a environment, the core Common Data Model schema is used to create the tables, columns, and relationships in the Dataverse database.

?Even though we traditionally talk about Dataverse being a database, it isn't actually a single data storage mechanism.?The data that's held in Dataverse is from the underlying storage facilities.

?At the physical level, the data could be stored in the feature of Azure SQL Database, Azure Storage, Azure Cosmos DB, or Azure Data Lake Storage.?Microsoft decides how to differentially store the data, depending on factors such as its type, e.g. files, images, text.

?

The doesn't have to concern themselves with this physical level.?Their work with the Common Data Model schema sits at the logical level.?They don't have to worry about the complex physical level structures that the implementation of these relatively simple structures are dependent on.

?

How come? ??

?

Because they can access the data's physical level through the APIs that are provided by Dataverse. ??

?APIs

?I love the definition of an Application Programming Interface () that Wikipedia, the Free Encyclopedia uses, as "a type of software interface offering a service to other pieces of software". ??

?The service the Dataverse API is providing is to enable Maker's App to reach through that abstraction layer to those low-level data structures. ??

? Microsoft Power Platform does this using a style of API known as - Representational State Transfer; a software architectural style that was created to define a set of constraints for how the architecture of an Internet-scale distributed hypermedia system, such as the Web, should behave.?

?All popular programming languages support a form of integration with REST-based APIs.?They are built on , with the Dataverse Web API implementing the (Open Data Protocol) version 4.0 - an standard for building and consuming APIs over rich data sources.

Dataverse takes an "API first" approach. This means that the service doesn't just provide a mechanism to query data, it also provides that rich semantic metadata from the Common Data Service on Business rules, constraints, and so on, that you can use to build intelligent, responsive apps and services.

Connectors

?From a Microsoft Power Platform Architecture perspective, Microsoft Dataverse is identified as being your data source of choice where you want to use either:

?? Dataverse capabilities, such as:

• Azure Active Directory () identity and access management mechanisms
• Custom from Business Rules, Classic Workflows and/or Power Automate
• Components metadata

?or

??? Features of Power Platform, such as:

• 微软 Power Apps portals Apps (aka. Power Pages )

??

Certainly, when you start building apps it's an easier app-building experience integrating data from more than one source into a common store, so that you have a single set of logic to maintain and operate over the data.

?There are many other use cases, however, where storing data in the Platform through integrating with Microsoft Dataverse isn't as optimal as accessing data from the Platform by using Connectors.?Particularly when you're needing to work with large datasets. ??????

?

This is where the architectural magic of the Microsoft Power Platform and its underlying Microsoft Azure platform come into play in clever ways. ??♀?

?What connectors do is create a or a around an API that allows the underlying App Service Environment to talk to Microsoft Power Automate, Power Apps, Azure Logic Apps and Power BI, enabling us to build cloud-connected and cross platform Business Apps.

?

Architecturally, this is achieved using the following components:

Credential and metadata store

• A service to store connector metadata (swagger, connection, ACLs, etc.), and credentials associated with a connection.

Connector

• Azure APIM (API Manager) to host all the swagger and policies. In addition to being the entry point for all calls that interact with the connector calls, Azure APIM verifies keys, tokens, certificates, and other credentials.
• ?App Service Environment to host connector webapps.

?

?Each connector offers a set of operations classified as??and?. Once you connect to the underlying service, these operations can be easily leveraged within your apps and workflows.

?

?Integrating and replicating large volumes of data into Microsoft Dataverse therefore isn't logical or cost-effective when you can make use of the functionality of the OData connection to Dataverse, which as we saw above, provides a range of important functionality, such as:

• Building reports that use the image data type
• Paging of query results
• Refreshing the data in the datasets on regular intervals

Deprecation

??So if the Common Data Service (Legacy) connector is so good, why is it being deprecated?

Well, this is related to the description we see for the new Dataverse connector that has and is taking over, which uses the Tabular Data Stream () protocol to connect to .?

?

In 1990, 微软 entered into a technology-sharing agreement with Sybase Inc, an enterprise software and services company, which resulted in Microsoft moving its own SQL Server over to Sybase code.

?During the early development of Sybase's own SQL Server, the developers at Sybase perceived the lack of a commonly accepted application level protocol to transfer data between a database server and its client.?In order to encourage the use of its products, Sybase promoted the use of a flexible pair of libraries, called netlib and db-lit, to implement standard SQL.?A further library was included in order to implement "Bulk Copy" called blk.

?Whilst netlib's job is to ferry data between two computers through the underlying network protocol, db-lib provides an API to the client program, and communicates with the server via netlib.?db-lib sends to the server a structured stream of bytes meant for tables of data, hence Tabular Data Stream.?blk provides, like db-lib, an API to the client programs and communicates with the server via netlib.

?

微软 kept the db-lib API and added Open Database Connectivity (), which is a standard API for accessing database management systems ().?It has a since added additional .?At about the same time, Sybase introduced a more power successor to db-lib, called ct-lib, and called the pair Open Client.

?I've included a link to the Wikipedia, the Free Encyclopedia article on in the comments below where you can read further details about its history.

?Fun fact ??: Sybase was acquired by SAP in 2010. ?? [SAP ceased using the Sybase name in 2014.]

I've also included a link to the Microsoft Learn Documentation article where you can read more about the timelines for the what you'll need to do if you're using one of the features not currently available in the Dataverse connector.

?

I hope you've found this follow-up deep dive into the Dataverse connectors of interest.?If there are any other topics or functionalities you'd be interested in seeing a deep dive on, do let me know. ??