Production Readiness of the PAB and Chain Index

Summary

At Loxe Inc., we’d been waiting for the release of the PAB and the capacity to make Plutus scripts interactive and automated for some time. When the release occurred that made interactions on-chain, with real wallets, feasible, we were overjoyed. We were primarily interested in getting our scripts functioning outside the traces and testing wallets we were using.

However, we have found that creating production-ready systems with the PAB and Chain index are no small feat. Especially ones running on mainnet. We attempted to run a modularized/containerized system on kubernetes, see details below, but found the result very unstable because of the node socket communication. We found keeping the PAB container alive was impossible as it routinely ran out of memory, resynced its database, and intermittently stopped responding (see mitigating PAB memory issues with cron)?.

The Chain index sync is a gargantuan task. We had been waiting for weeks for the system to sync completely with mainnet on a 6-core CPU with 32GB of RAM in the cloud. If you know something about costs, this is not a cheap instance (see cost mitigation below). The mainnet sync finally completed with an on-disk size of 215GB, much larger than the ~21GB for testnet. Be prepared.

Emurgo’s libraries are not ready to interact well with the PAB. Plutus does not interpret UTXO information formed through Emurgo serialization of scripts parameterized by some datum. The configuration of transactions is not in line with how you would build a transaction in Plutus. Additionally neither are particularly in line with how you build transactions with the CLI.

Requirements of production readiness (What we consider a production environment)

We consider a minimum production environment to be:

• Cardano node. We are using the loxeinc/cardano-node docker container that we have produced (see below for link).
• Cardano wallet server. We are using the loxeinc/cardano-wallet docker container.
• Chain-index synced on mainnet (size 215 GB at time of writing). We have now been running plutus-apps’ chain-index directly on a VM due to simplicity and because PAB issues in communication with Chain-index made us walk back using a more modern containerized approach (currently in the process of re-attempting a kubernetes solution).
• PAB service running co-located on VM, using the same plutus-apps commit hash used by plutus-apps repo above. There was some instability when using a chain index that had a more recent commit than the PAB’s dependency.

Desired production environment:

• Fully containerized workload that does not require NFS to support file-system based IPC. Modern networking like TCP sockets via IP would be preferred over UNIX sockets for a system like Kubernetes or even docker stack to allow processes to communicate in a normal fashion.?

Time taken to complete all the synchronization for node, wallet, chain-index was 2 weeks.

Mitigating OOM and Out of Disk Space problems with?PAB

We found that keeping the PAB running was an infeasible task. You cannot rely on the PAB a source of data for long-running instances of a contract.?

This is because the PAB constantly eats up RAM as it stays alive. It will exhaust normal limits in docker-compose or stack. It will do the same for kubernetes. If you have the pods restart, you’ll get a never ending loop of unusable PABs.

So, instead, you have to think of the PAB as an instantaneous resource. Even though this is not really the ideal condition of the PAB, it’s what works. What we found is running a script which kills the running image of the PAB (or process if running not via container), updates the config file with the most recent tip of the node, deletes the PAB database (or databases if you’re like us and must run multiple PABs — i.e. PAB for user-related contracts and PAB for oracle related contracts), and restarts the PABs. One might also consider having a monitoring solution like prometheus/grafana and have a trigger for a certain memory usage that kills the pod, modifies the config, and restarts the PAB container with the new config.?

The reason this is undesirable is that it leads to downtime. Nobody likes downtime, and having to routinely kill a process is a failure to think about production environments.

Limitations of the PAB and Serialization for parameterized scripts with Emurgo’s serialization library

We are using the preferred method of writing scripts which offer high parallelizability (parameterized scripts). Script addresses are parameterized by certain data, chiefly public key hashes of wallets making the transactions to the script address. We attempted to provide the script (with the parameterized data lifted into place) in CBOR to the serialization library. We then attempted to provide the script data hash as required in a CLI transaction. However, when Haskell/Plutus read transactions using utxosAt for the script address, no data was verifiable. This means that either Emurgo’s serialization library is flawed, or Plutus is flawed. We assume it’s the prior, due to the fact that manually written transactions to script addresses seem to be read correctly at the script addresses.

Lack of documentation

For Emurgo’s serialization library, there is a significant lack of documentation. Especially related to the use of the library for paying to script addresses and parameterized script addresses. The issues consist of better documentation than examples, but most have no resolution within the scope of the contained repository for successful transactions related to our script use-case.?

The use of 3rd party modifications to the serialization library.

Most people try and fail to use Emurgo’s serialization library and then switch to the Spacebudz (Berry Pool) modified library. However, this library is tempermental, does not seem to contain all recently updated code from Emurgo and did not yield usable results in our case.?

Links: