TEEs: Intel SGX attestation

Disclamer:?the author made an attempt to present truthful unopinionated statements, all information in the post is publicly available, this is an introduction not an in-depth review/analysis.

---------------

Previous post introduced trusted execution environments and described Intel SGX enclaves. Quick recap of Intel SGX:

• An Intel SGX enclave is a region of memory protected by the CPU from any access from the outside of that memory regardless of whether it is kernel, driver or application.
• One can interact with an enclave using E/O calls, interface surface defined by the enclave's developer. The E/O calls and Intel's SDK is one programming model, it is not simple to use in applications with complex interactions or written in a language other than C.
• Gramine and other LibOS'es provide a simplified programming model via conversion of Docker container images to confidential container images that start enclaves.

An enclave's purpose is to process "confidential" data in the sense that a less secure environment would not be trusted to handle the data because of a possibility of a data leak or misuse. How can an enclave be entrusted with the data?

Let's consider how the data may get into an enclave. It is common to use blob storage for large volumes of data or a database for transactional/smaller data, the enclave would make a call to the respective REST/gRPC service: AWS S3, Azure Blob Storage, GCP Cloud Storage, GCP Spanner, etc. These services are not aware of the binary/process that makes the call, of course proper authentication/authorization is applied during the call, but it's not tied to the binary code making the call. Identity of the process making the call is not sufficient to guarantee the data will not be misused, identity can be assigned to a different deployment in Kubernetes after the parties reviewed it. How to assure the data owner that the data processor code is the one they agreed to release sensitive data to?

The solution is to return the data from the storage services in an encrypted form and only let the decryption key into the enclave if its binary measurement matches the previously approved policy. Once the enclave obtained the decryption key, the data is decrypted and processed. How the binary code's measurement is created and is passed to the attestors safely is the subject of the next few paragraphs.

Intel SGX platform provides a way to read attestation report from inside an enclave, the report is generated by the CPU. This is how the report looks like (link):

typedef struct _report_body_t
	{
	    sgx_cpu_svn_t           cpu_svn;        /* (  0) Security Version of the CPU */
	    sgx_misc_select_t       misc_select;    /* ( 16) Which fields defined in SSA.MISC */
	    uint8_t                 reserved1[SGX_REPORT_BODY_RESERVED1_BYTES];  /* ( 20) */
	    sgx_isvext_prod_id_t    isv_ext_prod_id;/* ( 32) ISV assigned Extended Product ID */
	    sgx_attributes_t        attributes;     /* ( 48) Any special Capabilities the Enclave possess */
	    sgx_measurement_t       mr_enclave;     /* ( 64) The value of the enclave's ENCLAVE measurement */
	    uint8_t                 reserved2[SGX_REPORT_BODY_RESERVED2_BYTES];  /* ( 96) */
	    sgx_measurement_t       mr_signer;      /* (128) The value of the enclave's SIGNER measurement */
	    uint8_t                 reserved3[SGX_REPORT_BODY_RESERVED3_BYTES];  /* (160) */
	    sgx_config_id_t         config_id;      /* (192) CONFIGID */
	    sgx_prod_id_t           isv_prod_id;    /* (256) Product ID of the Enclave */
	    sgx_isv_svn_t           isv_svn;        /* (258) Security Version of the Enclave */
	    sgx_config_svn_t        config_svn;     /* (260) CONFIGSVN */
	    uint8_t                 reserved4[SGX_REPORT_BODY_RESERVED4_BYTES];  /* (262) */
	    sgx_isvfamily_id_t      isv_family_id;  /* (304) ISV assigned Family ID */
	    sgx_report_data_t       report_data;    /* (320) Data provided by the user */
	} sgx_report_body_t;        

The two application-specific measurements in the report are:

• mr_enclave is SHA-256 of the application memory after the enclave is initialized (see previous post about enclave's life-cycle). From Intel's doc: "Enclave measurement represented as SHA256 digest (as defined in FIPS PUB 180-4).". The digest of the memory uniquely identifies the application binary. For LibOS'es mentioned in the previous article, the measurement includes the measurement of the manifest file that secures files the enclave is allowed to access.
• mr_signer - from Intel's doc: "SHA256 digest (as defined in FIPS PUB 180-4) of the big endian format modulus of the RSA public key of the enclave’s signing key pair". The digest of the signing RSA key modulus identifies the party who signed the enclave, the build process.

Other values include versioning and CPU platform information. The enclave can pass custom data as part of the report in the "report_data" field. Its size is 64 bytes, if the data exceeds the size, only the hash (e.g. SHA-256) is included in the report and the data is sent as an addition to the report. The receiver verifies validity of the data by comparing data hash to the one in the report.

An enclave can read this report and send it to an attestor for verification followed by release of confidential data or decryption keys to the enclave. There is one problem though. Per Intel's Developer Guide, the report can only be used for attestation locally on the machine with the same CPU that created the report. This is useful for enclave-to-enclave attestation on the same machine but not for much else. The reason for it is that the report is not signed by a key that can be verified outside of the machine. Other enclaves on the machine can verify the report as they have access to the same CPU. This ability to attest local enclaves is the basis for remote attestation.

Remote attestation

For remote attestation Intel provides "quote" data structure, it is signed by a key that can be verified remotely. The quote is obtained by the enclave in exchange for the report from an Intel's service hosted in another enclave running on the same machine. The application enclave reads the report, then passes it to the Intel service to obtain the quote. The service is called "quoting enclave". More on how the quoting enclave is started later. There are two types of quotes (and respectively two types of quoting enclaves) that differ by signing algorithms:

• EPID - privacy-preserving Intel's signing algorithm. Intel provides the quoting enclave. Only Intel Attestation Service can verify the signature. Privacy here is a reference to the privacy of the CPU, if the enclave runs on an individual's machine, the CPU's id can be correlated with user online behavior. The fact that only Intel Attestation Service can verify the quote may limit usability and resilience if every attestation call must go to Intel.
• ECDSA - standard elliptic-curve based signing algorithm. Intel provides the quoting enclave. The public certificates allow verification of the signature by anyone. The certificates don't have to be read from Intel on every call, this simplifies verification inside cloud data centers.

In addition to the quoting enclaves, there is the Provisioning Certification Enclave that attests the quoting enclaves. In summary, the "local" primitives (called data center attestation primitives or DCAP) include:

• Quoting enclaves to convert report to quote.
• Provisioning certification enclave to certify quoting enclaves.
• Other components for communication between enclaves. Intel's AESM (Architectural Enclave Service Manager ) is the component that hosts the enclaves.

These primitives enable remote attestation in Azure Kubernetes Service. The AKS plug-in for this is sgx-quote-helper. The QE and PCE enclaves are run inside a daemonset (AESMD) installed by the plug-in. Application connects to the daemonset via unix domain socket.

Intel's paper on this topic provides a chart for the signing chain between the enclaves:

Gramine

In the previous article, we discussed LibOS options for a simplified programming model with containers. The open-source Intel-backed option for LibOS is Gramine.

Gramine simplifies interactions with the other enclaves. From the user's point of view getting a quote is reduced to two steps:

1. Optionally, write user data (64 bytes) to include in the quote to /dev/attestation/user_report_data.
2. Read the quote to the end from /dev/attestation/quote.

This is much simpler, no need to know details about QE, PCE, AESMD. Enable the sgx-quote-helper in Azure Kubernetes and read the quote from the virtual device.

Gramine docs provide charts that help understand the quoting process:

After reading the above, one may ask: why the complexity? The author's limited knowledge allows for speculations only:

• CPU hardware is not updatable, keys need to rotate. That may be the reason for separating the local report from the quote.
• Need for different signing algorithms, desire to promote EPID vs standard ECDSA, could be another one.

Application in Azure

We discussed how to obtain the SGX quote suitable for remote attestation. What can do the attestation? It depends on system architecture, the party that holds the keys is the one that should verify the quote and provide the keys.

Azure provides Azure Attestation Service abbreviated as MAAS. The service understands the format of the SGX quote, can verify it against Intel's certificates and can convert the quote to a Json Web Token (JWT), which is easier to parse. The service is free and is easily enabled in any Azure subscription.

It would not be an attestation service if it only validated and converted the binary byte array of Intel's quote to JWT. Besides, performing the validation and conversion, the service checks the measurements (MRENCLAVE, MRSIGNER) against a configured attestation policy. The token is returned only if the measurements match the policy. The policy is a text document that looks like this:

version= 1.0
authorizationrules?
{
	[ type=="x-ms-sgx-is-debuggable", value==true ]&&
	[ type=="x-ms-sgx-mrsigner", value=="mrsigner1"] => permit();?
	[ type=="x-ms-sgx-is-debuggable", value==true ]&&?
	[ type=="x-ms-sgx-mrsigner", value=="mrsigner2"] => permit();?
};        

By convention, the values from the quote can be referenced with "x-ms-mrenclave", "x-ms-mrsigner" keys in the policy. It is important to emphasize that the measurements get converted to values in the JWT's payload. The user data passed to the attestation service is copied to "x-ms-runtime" node of the payload as-is. See example of the JWT in Azure documentation.

The API to the attestation service is straightforward, auth is done via managed identities.

user_data = """
{
   "custom_field": "any value"
}
"""
# Note that quote includes SHA256 of the user_data.
_, token = attestation_client.attest_sgx_enclave(quote, runtime_json=user_data)        

The obtained token, now, can be passed to other services. This assumes that both Intel and Microsoft can be trusted not to change evidence from the enclave.

Another service provided by Azure is called Managed HSM, it provides access to a hosted hardware security module to generate and manage encryption keys. The service can release the private key of an RSA key pair if it is allowed to be exported, subject to a release policy. The two services, managed HSM and the attestation service, are made to work with each other. The release policy can be configured to accept x-ms-mrenclave and x-ms-mrsigner measurements from the attestation JWT:

{
? "anyOf":
? [
? ? {
? ? ? "authority": "my.attestation.com",
? ? ? "allOf":
? ? ? [
? ? ? ? {?
? ? ? ? ? "claim": "mr-signer",?
? ? ? ? ? "equals": "0123456789"
? ? ? ? }
? ? ? ]
? ? }
? ]
}        

The policy is set up per key in the HSM, it is required for exportable keys. Important to note that authority is the endpoint that the HSM will call to verify the signature of the token. The endpoint is expected to expose OpenID metadata endpoint pointing to the certificate or certificate chain for the signing key.

The released key can then be used to decrypt the data.

HSM shares the API layer with Azure Key Vault. The API is straightforward to use, for example in Python:

# Attestation token is obtained from the attestation service.
encrypted_key = key_client.release_key("key-name", attestation_token).value        

It is somewhat tricky to figure out decryption of the released key. I am not going to go into those details here, only saying that:

• Once decrypted the key can be deserialized from DER format.
• The key is encrypted with an AES key.
• The AES key is encrypted with the user's public key that was passed as part of the x-ms-runtime claim.

Summary

The article discussed attestation of SGX enclaves and how to use attestation evidence to decrypt data inside enclaves securely. The process follows these steps:

• Read SGX enclave report.
• Exchange the SGX report for an SGX quote using Intel's DCAP infra hosted by Azure.
• Exchange the SGX quote for attestation JWT using Azure Attestation Service.
• Exchange the attestation token for a private key using Azure Managed HSM Service.
• Decrypt data using the private key.