Azure Confidential Ledger Deep Dive

This article is based on the transcript of my video on the same topic, https://youtu.be/Y5FnHs339o8. Was created using the OpenAI 4o model hosted in Azure AI Foundry with some human edit post generation.

For many industries and scenarios, there’s a critical need to validate various aspects related to data. Maybe it’s part of a supply chain, ensuring things were kept at certain temperatures, or verifying that certain documents or database records haven’t been tampered with. We need to ensure the integrity of our data. If someone changes it in any way, we want the ability to detect that tampering. This is where the concepts of tamper-evidence and encryption capabilities come into play. And we want all of this in a trusted, attested manner so we can guarantee these protections were and are in place.

Often, we see a need for end-to-end protection and attestation. This could involve relational data, blobs stored in a storage account, system records requiring identity protection, or confidential transaction data. Across all of these scenarios, we need an ironclad audit capability. We need a ledger—something immutable, meaning existing entries cannot be changed. New data can only be appended to the end of the ledger in a tamper-evident way.

Confidential Computing

When we talk about ledgers, particularly Azure Confidential Ledger, it’s worth noting that all cryptocurrencies are based on the idea of ledgers. Bitcoin, for example, uses a ledger behind the scenes. However, before diving into the specifics of Azure Confidential Ledger, I want to quickly refresh your understanding of confidential computing. I’ve covered this topic in a separate video, but let’s revisit a particular aspect of it.

Today, we often think about protecting our data in three states: at rest (e.g., stored on disk in a storage account), in transit (e.g., encrypted using TLS as it travels over wires or virtual wires), and, with confidential computing, in use. Confidential computing adds protection to data while it’s being processed, specifically in memory and CPU states.

Let’s consider a regular machine. It CPU, memory, and other components. Within this machine, we can create a secure enclave, also known as a trusted execution environment (TEE). For example, Intel SGX provides such an environment. This secure enclave is a component of the processor designed to ensure data security and integrity. It can even generate attestation reports from the hardware to confirm the use of the trusted execution environment.

What’s remarkable about these secure enclaves is that even if there’s a malicious process on the host, it cannot access or interfere with the data inside the trusted execution environment. This is important because, as you might have guessed, we’re going to leverage this capability when discussing the ledger.

Blockchains

The goal of this video is to explore Azure Confidential Ledger. This solution addresses the pain points we’ve discussed. It ensures the integrity of all transactions written to the ledger, making them tamper-evident and auditable. At any time, you can validate that you’re interacting with the ledger and confirm its use of trusted execution environments.

When using Azure Confidential Ledger, the data is immutable. Any data written to the ledger cannot be changed or removed. The only operation allowed is appending new transactions. For every transaction written to the ledger, you receive a receipt, which is crucial for auditing purposes.

A common question with cryptocurrencies and ledgers in general is: How do we ensure the integrity of the data? How can we detect tampering within the transactions? Azure Confidential Ledger leverages blockchain technology to address this. Specifically, it uses a tree of hash values for every block, forming a chain by pointing to the previous block. This structure is based on a Merkle tree, a concept that has been around for a long time and underpins much of modern cryptography.

Without delving too deeply into the technical details, let’s briefly discuss the Merkle tree to understand how it ensures data integrity. Imagine we have four data transactions: A, B, C, and D. (In reality, there would be far more transactions in a block, but let’s keep it simple.) For each transaction, we run a hash function, which generates a digest. These digests form the leaf nodes of our tree.

A hash function has several important properties:

1. Fixed Size: Regardless of the input size, the output digest is always a fixed size. This is useful for storage and processing.
2. Deterministic: The same input always produces the same output. For example, hashing "Hello, World!" with the same function will always yield the same digest.
3. One-Way: While the same input generates the same digest, knowing the digest doesn’t allow you to reverse-engineer the original input.
4. Collision Resistance: It’s extremely unlikely that two different inputs will produce the same digest.
5. Avalanche Effect: A small change in the input results in a vastly different digest.
6. Efficiency: Hash functions are computationally efficient.

These properties make digests ideal for detecting tampering. If we store the digest of some data and later recompute it, a match confirms the data hasn’t been altered.

In a Merkle tree, we hash each individual transaction (A, B, C, D) to create their respective digests. Then, we hash pairs of these digests (e.g., hash(A) and hash(B)) to create parent nodes. This process continues until we arrive at a single root hash, known as the Merkle root.

Here’s why this structure is powerful: If any individual transaction changes, its hash changes, which in turn invalidates all parent hashes up to the root. This makes tampering evident.

In a blockchain, each block contains two main components: the data (e.g., transactions) and a header. The header includes the Merkle root, a timestamp, a nonce (a "number used once"), its own hash, and the hash of the previous block’s header. By linking each block to the previous one, we create a chain of blocks—a blockchain.

This structure ensures that if any block is tampered with, it breaks the chain, making the tampering detectable. Azure Confidential Ledger leverages this blockchain technology, combined with trusted execution environments, to provide a secure, tamper-evident ledger for your data.

Let’s continue exploring how this works in practice. All of these elements together create a fantastic integrity-validating and proof mechanism—this becomes my blockchain. Because we are constantly hashing the headers, and the headers contain the Merkle root (which is based on the hashes that form the tree of the data), any tampering anywhere would invalidate the block. This would then indicate, "Hey, this isn’t matching the value I had in mind—this has been changed." As a result, I’ll always be able to verify if there’s been any change to anything.

The specific blockchain used in the Azure Confidential Ledger is CCF, the Confidential Consortium Framework, which is open-sourced and originates from Microsoft Research. As part of the CCF Ledger specification, the integrity of this system is further assured. For example, the head node in the cluster periodically signs the Merkle root of these various blocks.

Azure Confidential Ledger

I can store any type of data in the ledger. It’s a key-value store, but that value can be anything I want. Very commonly, we’ll want to store digests—essentially, the output from a hashing function based on input from databases. For instance, I could take n number of records, run them through a hash function, and get a digest. I would then store the digest as a transaction. Later, I could validate the records by running the hash function again on the same series of records. If the resulting digest matches the one I stored, I know those records have not been changed.

I could also take a blob, which can contain anything—scans of documents, other types of data, or even metadata from Internet of Things (IoT) sensors that I need for integrity protection. For example, I might need to prove that during the transport of some medicine, the temperature never dropped below a certain threshold. A blockchain would ensure that I couldn’t tamper with those temperature readings. In health and AI scenarios, I might want to store logs or other sensitive data.

However, just because I can store anything doesn’t mean I should store everything. Could I store the whole blob here? Yes, but there’s a much more efficient way to handle this. I would store the blob in a data lake in Azure Storage, create the digest, and then store only the digest in the ledger. Remember, there are other things I might want to do with those documents, and the ledger wouldn’t be particularly useful for regular, day-to-day operations. Generally, I’ll think of the ledger as a place to store either transaction data or immutable digests.

The Azure Confidential Ledger is also very developer friendly. There are software development kits (SDKs) for many languages that make it easy to interact with the ledger. Additionally, there’s a REST API and a portal where I can explore elements of the ledger.

Public vs Private

When I create a new ledger, one of the things I specify is whether it’s public or private. It’s important to note that this designation doesn’t determine who can access the data. Instead, it determines how the data is stored. If I specify the ledger as public, the transactions I write into it are stored as plain text. If I mark it as private, the data is encrypted. Of course, even with a public ledger, I could encrypt the data myself before writing it, as the ledger doesn’t care what’s in the value. These options simply give me flexibility.

Azure Confidential Ledger Instance

When I create an instance of the ledger, it’s tied to a specific region. What happens during this process is that a number of components are spun up. There’s a CCF application and a CCF blockchain. The application part of this confidential solution runs inside a secure enclave, also known as a trusted execution environment. The CCF includes both the app (which I interact with as a client) and the blockchain.

This setup isn’t a single instance. Instead, it’s a three-node cluster, all within the region I specified. Each of these nodes uses confidential compute, meaning each has its own trusted execution environment, CCF installation, app, and blockchain. The nodes replicate data among themselves. If the region supports availability zones, the nodes are distributed across those zones for added resilience. Additionally, the ledger data is persisted to Azure-backed storage for durability.

A recently announced feature allows for application-level logic. This means I can write my own JavaScript or TypeScript with business logic that triggers as part of a transactional operation. This logic runs inside the trusted execution environment, ensuring it remains secure. For example, I might write a function to increment a counter or perform some other operation on sensitive data. This capability is a powerful addition to the platform.

Example Scenario

Let’s consider a common interaction. Imagine I have an application and some data whose integrity I want to ensure. I don’t want to store all of the data in the ledger. Instead, I take the data (e.g., a blob or a set of database records) and run it through a hashing function. This produces a digest. I might also generate an identifier for the data, which, along with the digest, becomes the value of my transaction.

Step 1 - My application then establishes a TLS connection with the Azure Confidential Ledger. Importantly, this connection doesn’t terminate at the host—it passes through to the app inside the trusted execution environment. Currently, TLS 1.3 is used. This ensures that nothing in the host can compromise or observe the data.

Step 2 - Once the application sends the transaction to the app, the app commits it to a majority of the nodes (at least two out of three) before acknowledging it. At this point, the transaction is globally committed as it replicates to the remaining nodes.

Step 3 - The app then generates a receipt, which includes the transaction ID and proof of the transaction. This receipt is a signed token that serves as evidence the transaction was added to the ledger.

Step 4 - If the source data is a blob in an Azure Storage account, I could store the receipt as metadata for the blob. This makes it easier to look up and compare data in the future. Adding metadata doesn’t modify the digest value because the hash is based on the data content, not the metadata. However, storing the receipt as metadata is optional.

Disaster Recovery

Now, the next thing I want to talk about is: you have this critical ledger—what is the disaster recovery capability related to that?

There is a certain native set of backup and recovery. What the service provides is that it natively does a backup and then asynchronously replicates that to another region. If there were a regional disaster, it would then restore that within a couple of minutes. Now, it isn’t actually the Azure paired region—they have their own mapping to that.

As a customer, you could perform your own backup to your own storage and then restore that as another operation, which would create a new ledger in any region you want.

I have to pass two pieces of information to take a backup of a ledger instance:

• A shared access signature to an Azure file share. That is where it is going to go ahead and store the actual content. The backup process will create a new folder under the given share with a <ledger name>-<timestamp> format. The backup has its own serialization format that’s put into that Azure file share. I’m just going to see a whole set of files, and even those files would be tamper-evident based on the way they write it.
• The other thing I have to give is a region. The region is where I would want to be able to restore it to. I have to, in advance, specify the target region where I would do the restore. The file share doesn’t have to be in that region, but I have to tell it which region I would take this backup and restore it to using the restore operation. The reason for this is, as you could imagine, with something that’s all about integrity and potentially encryption, there’s key material involved. What’s also happening here is that within the region, there is an Azure Key Vault. That Azure Key Vault has keys that are associated with your Azure Confidential Ledger instance. Now, those keys may be used for encryption if it’s private, but they’re also used for various governance transactions. When I specify the target region, in the target region it’s going to take that key material and put it there as well. So, in the event that I had to restore from here to a new Azure Confidential Ledger, the key material is present so it can do its job.

The frequency with which you decide to do this backup operation is completely up to you. It really depends on the pattern of writes and what you may be willing to lose in a disaster. The only cost is the storage. What you’d probably want to do is write something to clean up the old folders because it’s very unlikely you’d ever want to restore to anything other than the most recent one. Remember, again, the keys rotate, so a really old backup would not be very useful anyway.

Where to use this

Now, you may already be seeing this and using this. Beyond your own scenarios—which are many—any time I want that integrity, that tamper evidence, this is fantastic. Any type of transactional data—this is great for. There are also some native Azure uses of the Azure Confidential Ledger. The integration is native, meaning you don’t need to write your own code—unlike some of the custom implementations I’ve worked on.

Blob

If you search for "Confidential Ledger," you’ll see an option for "Blob Storage Digest Backed by Confidential Ledger." Selecting this creates a managed app, which is free. This app, based on the storage account you provide, will automatically handle the process of generating digests after a certain number of blobs are added. It then adds those digests to the ledger. Essentially, it performs the same tasks I’ve described, but it does so automatically. Additionally, the managed app can assist with the audit process by adding results to an audit records container.

SQL Database

Azure SQL Database can natively integrate with Confidential Ledger to provide tamper-evidence for the records written to the database. For example, if you create a new SQL database, you’ll notice a "Ledger" section under the security settings.

Here, you have two options:

1. Enable the ledger for all tables, so that any changes or records added to the database are automatically added to the ledger.
2. Enable the ledger on a per-table basis.

The ledger must be in the same region as the database. Once enabled, the SQL database will use Azure Confidential Ledger to store digests. This ensures the highest level of integrity and tamper-evidence for your data. Blocks of records written to the database will have digests created and stored in the ledger, just as we saw earlier.

Pricing

From a pricing perspective, there are no SKU options for Confidential Ledger. You pay a per-hour, per-instance charge, along with a gigabyte-per-month fee. However, the base ledger price includes 100 GB of storage, so the additional storage cost only applies after the first 100 GB. In my experience, even ledgers that are several years old rarely exceed 100 GB, so most users will only need to consider the base ledger price. https://azure.microsoft.com/en-us/pricing/details/confidential-ledger/

Sample Applications

For those interested in building their own applications, there’s a GitHub repository with sample apps at https://github.com/microsoft/azureconfidentialledger-app-samples. The documentation is excellent, with detailed walkthroughs.

Summary

In summary, Azure Confidential Ledger is a powerful tool for ensuring the immutability and integrity of your data. Whether you’re working with Azure Blob Storage, SQL Database, or other applications, it provides a native and efficient way to store transactions and data, including digests, that you can guarantee have not been tampered with.

That’s it for this overview of Azure Confidential Ledger. I hope you found it useful. Until next time, take care! ??