Protecting Tezos Baking Secrets with IBM Confidential Computing

In a previous article, I detailed how to run Tezos Octez software on IBM Confidential Computing. In this new article, I explain how to leverage IBM Confidential Computing service to protect your Tezos keys and signing operations with FIPS 104-2 Level 4 certified hardware. For this purpose, I enabled a Tezos remote signer -provided by the Tezos Foundation, to IBM Confidential Computing services. This article provides the details of this integration and the configuration steps. IBM Crypto Express supports tz1, tz2 and tz3 addresses.

The source code of the programs used in this article can be retrieved at the following sites:

• Generating a Tezos address using IBM Hyper Protect grep11 service: https://github.com/jyg007/hpcstezos
• The remote signer with IBM Hyper Protect grep11 support: https://github.com/jyg007/remote-signer/tree/hpcsgrep11

If you want to test them, the IBM Hyper Protect software has to be installed and configured as a prerequisite. Online documentation is available at https://www.ibm.com/docs/en/hpvs/2.1.x. The service is also available in IBM Cloud.

What is Confidential Computing

Confidential Computing technology is used to protect applications dealing with sensitive data. It protects your IT infrastructure against attacks like admin privilege escalation or resulting of operating system/hypervisor vulnerabilities. Digital asset custodians use it to safeguard digital assets. IBM Confidential Computing is built on top of IBM Secure Execution?z/Architecture??hardware technology.

Confidential Computing can easily be compared with hardware security module (HSM) that you may already be familiar with:

• Hardware Security Modules?are hardware devices used to safeguard cryptographic keys and we will use one to protect our Tezos private key. The cryptographic signature operation will be technically impossible to process outside the HSM. Reading the clear text value of this key will be impossible. However, this does not protect the signing service itself.
• Confidential Computing?extends this capability to protect the?application?-including the signing service - and its data. The application runs in a secure enclave called?Trusted Execution Environment (TEE) that prevents tampering. Leveraging transparent hardware encryption, TEE hardware resources (like memory and processor state) are restricted to processes executed in the TEE and are made inaccessible to the operating system, firmware and any external application. The TEE is able to verify the application image signatures at run time in an auditable way. Opening a shell access into the TEE process is prohibited.

Both technologies will be jointly used to secure the Tezos remote signer. The TEE enclave will protect the signing program and the Tezos baker from malicious tampering.

IBM Confidential Computing allows access to an Hardware Security Module via a service providing the Enterprise PKCS #11 (EP11) API over gRPC (also called grep11 service). Mutual TLS secures the access to this service. The grep11 service connects an IBM Crypto Express Network API appliance that has the IBM Crypto Express HSM attached.

The next picture summarizes the overall architecture:

• The Tezos node, baker and remote signer run in a Trusted Execution Environment that protects the signing service from low-level unauthorized access. The TEE also stores the TLS certificates used to connect the grep11 service.
• The grep11 service runs in a second Trusted Execution Environment. The grep11 container image is built and signed by IBM. The TLS Certificates used to connect the crypto appliance using mTLS authentication are stored inside the TEE as well.
• The IBM Crypto Express Network API appliance runs in a third Trusted Execution Environment in a?Secure Service Container partition.

With the architecture:

• Tampering with these enclaves is impossible for system administrators or programs executed on the same server.
• The Tezos baker's key cryptographic signing can only be performed inside the HSM attached to the appliance via the remote signer.
• A custodian controls on used to baker signature processing as it controls the the HSM master key.
• The remote signer protects against funds draining and double signing. (https://news.tezoscommons.org/double-baking-and-remote-signers-639bf8ef7e65)

In the rest of this article, we assume that the connection between the grep11 server and the IBM Crypto Express Network API appliance and the grep11 services are properly configured and started.

We could also consider running the remote signer in a separate trusted execution environment on another server.

Creating your Tezos keys

We will create the Tezos key using the IBM Crypto Express HSM. For this purpose, I created the createzoskey tools available at the following link.

This program uses a grep11 service to perform cryptographic operations and to create the Tezos keys. You provide one parameter to specify the type of tezos key you need to generate. IBM Crypto Express cards support Prime256, Ed25519 and Secp256k1 curves so you can create tz1, tz2 or tz3 addresses. If you are familiar with PKCS11 API, you will easily understand how it works.

Having a look on the source code, you will notice that:

• It will create tz3 addresses if tz3 is provided as command parameter as the prime256 curve will be used to create the key pairs.
• The private key is made non-extractable which will make the HSM required for any signing operation. The private key is impossible to retrieve in clear text (technical assurance). Changing it to extractable would allow the key value to be extracted out of the HSM.
• CKM_EC_KEY_PAIR_GEN (elliptic curve key pair generation) specifies the operation to execute inside the hardware security module to create the Tezos key.

ecParameters, err := asn1.Marshal(asn1.ObjectIdentifier{1, 2, 840, 10045, 3, 1, 7})? //prime256

publicKeyTemplate := ep11.EP11Attributes{
   ep11.CKA_VERIFY:? ? ? ? ?true,
   ep11.CKA_EC_PARAMS: ecParameters, }

privateKeyTemplate := ep11.EP11Attributes{
   ep11.CKA_PRIVATE:? ? ?true,
   ep11.CKA_SENSITIVE:? ?true,
   ep11.CKA_SIGN:? ? ?true,
   ep11.CKA_EXTRACTABLE: false,}

generateKeypairRequest := &pb.GenerateKeyPairRequest{
	Mech:? &pb.Mechanism{Mechanism: ep11.CKM_EC_KEY_PAIR_GEN},
	PubKeyTemplate:? AttributeMap(publicKeyTemplate),
	PrivKeyTemplate: AttributeMap(privateKeyTemplate),
}        

Using the createtezoskey program:

Compiling the program will require you to install the golang software on your system. Issue the make command for compiling.

To configure it, you need to specify the grep11 service mTLS connection parameters in the credential.yaml file. It means:

• The grep11 url.
• The certificate authority root cert for the authority created for the grep11 service.
• The client public and private keys that you or your grep11 admin provided.

url: "mygrep11.ibm.com:9876"
cert_path: "/home/girardjy/mygrep11client.pem"
key_path: "/home/girardjy/mygrep11client-key.pem"
cacert_path: "/home/girardjy/mygrep11ca.pem"        

Running the program

In order to create a key pair, execute the program as in this example:

$ ./createtezoskey tz3
0000000......................................................................................................................................82324e449d24dbe2b7b9ed2e884250

p2pk66EnR4uFKpTmZj9z5Fg9J2J8KmLb7LCwZe5QPySs8JurcTP1PXM
tz3aiVJAbkWK5hCnB91wzkUMb9H9DmM9sSfs

$ ls tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd*
tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd.der
tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd.sk.hex        

In our example, you create the following files prefixed with the public Tezos key hash tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd:

• The public Tezos key p2pk66EnR4uFKpTmZj9z5Fg9J2J8KmLb7LCwZe5QPySs8JurcTP1PXM which is also stored in der format in the tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd.der file.
• The private key BLOB which is an AES-256 ciphered binary of the Tezos private key. The encryption key is kept securely inside the the IBM Crypto Express HSM device. The file tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd.sk.hex stores this blob in hexadecimal format.

Running your remote signer with grep11

Different implementations are available for remote signing. In this example, I enabled a signer written in the Python programming language and which provided by the Tezos Foundation. This signer is available at https://github.com/jyg007/remote-signer/tree/hpcsgrep11.

To configure this signer, you need to edit the keys.json file to specify:

• "grep11" for the signer attribute.
• The grep11 connection parameters: the "url" of the grep11 server, the certificate authority root certificate file ("cacert_path"), your client's public certificates and your grep11 client private key files ("cert_path" and "key_path").
• In the "keys" attribute, the Tezos public key hash along with the Tezos public key and the private key blob. The three values are retrieved from the previous createtezoskey command.

In our sample, the ratchet is mocked (mockery value for ratchet and chain_ratchet) but you may probably want to indicate a valid configuration for a production system. The repository includes a sample for SQLite chain ratchet backend.

{
???????"signer": "grep11",
???????"ratchet": "mockery",
???????"url": "grep11.metaco.com:9876",
???????"cert_path": "/data/tezos/remote-signer/grep11client.pem",
???????"key_path": "/data/tezos/remote-signer/grep11client-key.pem",
???????"cacert_path": "/data/tezos/remote-signer/grep11ca.pem" ,

???????"keys": {
???????????????"tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd": {
???????????????????????"public_key": "p2pk66EnR4uFKpTmZj9z5Fg9J2J8KmLb7LCwZe5QPySs8JurcTP1PXM",
???????????????????????"secret_key": "000000000000000000000000000000000000000000000000000000000000000058b25ed705767b2729e96514ee97a6b3000000000040012600000000000000011234fe9bd078342ef1f116ca2e86b7debc752ef181125d0cf3c803ac029e218fd2106be2882eccbc6b657b505d1d0c625c054da40399a35df06e190a511d2e0fd2a67d3411c6f0633cc75003e768e4125fae879c70ec436aa517c36e949b6d64930801c5310557edcb182c8ba1fd0a470fbf13b399b1322ab73565368501a2067e8a5d5c1be40caaa267c6271caee1cd76a1c14a3aa292f8670dc2413d8a90d2ecfdb4c20af8d0c518ca4b28ef4d4750fdcf1e99781de36432c14472fa9ed06bb06465438dfa3796a51048973294eef4bdb1bcacdc898a9708448f85d1626bb28d6177abe07030ad7a09d1438440fec5d62f7871f74a126eebd71b69a6d7b7caa1489c5fd945859b2d72a690eb67f66d39d75f3aafecaef1c25b9fb6503d66d74a302dcf6d6d8613a59ef89d2ba1c9e7b2a4bd441813361844622f0a23780d362d02a348c1517f79215d84152b8c80e3ebd6c1e4b1181e704e16d1e152eb5cdbb79385f9e3c9db4ca94dcb2a93beb627ac118674c3e4d2e5b880401ca1905c538840be3ba2e9755273d35075434ba8ecd9636707481786458928cdd4bd96758a9fb4e06fd0392d9e30a564b664dc650d55b4fd3e22d65d6a3a5e936b8115360c4c100c231ffcb5d541caa22eae0ffb3114431c13240bb71b58ad9100f0501930067d34728c3f8bb6ca655a1cb042082d4a825b0727444fc1a126e0c388e74c50"
???????????????}
???????},
???????"policy": {
???????????????"baking": 1,
???????????????"voting": ["aye", "pass"]
???????},
???????"chain_ratchet" : "mockery"
}        

You may notice I am not worried about sharing my encrypted private key as you have no access to the grep11 service that I use and AES-256 is considered resistant to quantum computer attack.

Start your service with the following command:

$ FLASK_APP=signer ?flask ?run --host=0.0.0.0

* Serving Flask app "signer
* Environment: production
??WARNING: This is a development server. Do not use it in a production   deployment.
??Use a production WSGI server instead.
* Debug mode: off        

Note that the signing service cannot be reached outside the TEE if you do not authorize port 5000 to connect the enclave where you only allow connection on port 9732 (Tezos port).

You can then import your created key using the following command.

$ ./tezos-client import secret key hsmremote? "https://localhost:5000/tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd"        

On your node you can then list the address and register it as a baker using the following commands:

$ ./tezos-client show address? hsmremote
Hash: tz3gvQDEZfYakYEcqaL3ESQGbbNhwVHXAQmd
Public Key: p2pk66EnR4uFKpTmZj9z5Fg9J2J8KmLb7LCwZe5QPySs8JurcTP1PXM



$ ./tezos-client register key hsmremote as delegate?        

If you look at the log of your grep11 service, you should see something similar to

{"grpc_method_name":"/grep11.Crypto/SignSingle","grpc_response_time_ms":23.138927,"grpc_ts":"2022-10-31T15:02:54Z","grpc_has_err":false,"grpc_err_msg":"","grpc_c
ient_ip":"172.19.0.1:54454"}        

Then, you can start up your baker using your protected address with the following command:

$ nohup ./tezos-baker-014-PtKathma run with local node ~/.tezos-node hsmremote --liquidity-baking-toggle-vote pass > baker.log? &        

Conclusion

IBM Confidential Computing technology is designed to protect mission critical applications with no impact on application performance and no application code change.

HSM are supported through API implementing EP11 over gRPCs. Mutual TLS is used for authentication. It will require some minor code change to an existing application using a PKCS11 library.

IBM Hyper Protect technology can be leveraged in two ways:

• Running a Tezos baker, Tezos node and remote signer in a TEE secure enclave to provide technical assurance that your signing service cannot be tampered with if access to the infrastructure is compromised.
• Having a secure remote signer signs operations while by protecting the Tezos private key as an non-extractable HSM key that provides a technical assurance to the custodian controlling the execution of operations.

IBM Trusted Execution Environment relies on the Secure Execution technology made available on IBM zSystems, IBM LinuxONE servers and on the IBM Cloud Hyper Protect services.