Path to Running First Hyperledger Fabric App On Multiple Nodes

This post is not so special, there is nothing here you cannot find on internet if you spend hours and hours like me and my team did to find out how to run a Hyperledger Application with Peers/Orgs running on different hosts/servers in a cloud based setup such as AWS.

As of the time of writing this post the information on the internet to build a hyperledger app is very scattered and there are so many ways in which you can build a hyperledger based blockchain that one can easily get lost in the information. This post summaries the path I took to successfully create my first multi host hyperledger based blockchain and I hope that others won't have to go through the tedious process of exploration again.

Objective:

1. We are not going to discuss basic theory of blockchain here, rather we will get our hands dirty creating a blockchain network in its purest form
2. We will discuss dont's of building your first blockchain network
3. By the end you would have a blockchain network running on multiple nodes on AWS EC2 instances (you can use any cloud provider you like)

Pre-requisite:

You need to understand at least some programming language and have basic experience with managing servers. If you are a non tech person, this post is not for you (it will leave you more frustrated and discouraged). Also, it is assumed that you understand blockchain as a concept already. If you do not then you have some pre-work to do and you may start here : https://hyperledger-fabric.readthedocs.io/en/latest/whatis.html

Lets get started: Why Hyperledger Fabric?

There are several blockchain providers in the market but the most popular enterprise blockchain today is Hyperledger which can be used to create a closed blockchain network. Hyperledger is an open source community and their official website is https://www.hyperledger.org

Due to the complexity involved in getting a blockchain network up and running several blockchain management agents have popped up which include big names such as IBM, Microsoft etc. These agents give you tools to deploy your blockchain network on their cloud or any other cloud and they also claim to make deployment easier for you.

So should you use these agents/wrappers or should you use the plain vanilla core component which is called as the Hyperledger Fabric? It is always good if you build using the purest form such as Fabric as it gives you a deep understanding of the technology and better control for debugging in production environment later on. Once you have a good understanding, adopting any wrappers or tools later on makes more sense. The other perspective is if you do not have good technology expertise in your team who can build and manage a blockchain then it is always better to go with an agent managed blockchain or what we call as BAAS (Blockchain as a service)

So lets get started building our network with Hyperledger fabric.

Step 1: Follow the official documentation and get all the required software

It is best to start with the Hyperledger official documentation. There are so many documents and guides available on internet but stick to the official link as follows:

Follow the step by step instructions as mentioned in the tutorial. Install all the pre-requisites which include docker, node, go etc and download all the samples as mentioned. The samples mentioned here are not merely the samples but these also act as templates to build most of the type of blockchain networks thus understanding them in detail is very important if you want to develop something in future from scratch.

When you download the samples and libraries it downloads all the necessary fabric binaries and docker images.

Hyperledger Fabric uses docker extensively and to understand everything deeper you need to have basic knowledge of docker to start atleast.

Don't know anything about docker? Watch the video below, it would get you started.

Once you have all the libraries and sample files you run your first network : the BYFN tutorial. Try to grasp and understand the flow as much as you can from the following files (don't worry if you do not understand a lot of things just now) :

/fabric/fabric-samples/first-network/byfn.sh (This is the main script so try to follow the flow)

/fabric/fabric-samples/first-network/crypto-config.yaml

/fabric/fabric-samples/first-network/docker-compose-cli.yaml

I wish I could say here that the hyperledger official documentation alone will help you build your network and understand everything but this is not the case. The official documentation is not very explanatory, does not talk about a lot of things in detail but we do have some resources to our rescue.

Step 2: Deep dive into official documentation and run on a single node

Watch this 7 video series created by one of the hyperledger contributor which explains the official BYFN network in a bit more detailed fashion and will give you some more clarity around the network basics.

Once you have watched these videos, go through the script and the docker files in the hyperledger samples once again. By now you would be much more comfortable with the flow and would understand the build-your-first-network example a lot more.

Once you complete step 2, what you now have is a network which consists of multiple orgs with multiple peers in respective orgs who all reside on a single host.

But if you think about a practical scenario these orgs and peers in
most of cases would never reside on the same host and you would want 
to now run your orgs on different servers/hosts. 

This is where you would feel that the official documentation becomes completely silent. Move to Step 3 to know how do you do it.

Step 3 : Moving your network to multiple hosts

As I mentioned in step 2 that official documentation does not give you any clue onto how to build a network on multiple hosts. If you have gone through the scripts and docker files, the major hurdle being how will the docker container names be resolved once the blockchain runs on multiple hosts? There is a one word hint that would solve your problem: SWARM or more specifically "docker swarms".

To know what is a docker swarm read here:

I also came across this medium post where the author gives step by step process about running a swarm, bringing multiple hosts on that swarm and then running the Basic network example as in the hyperledger samples on these hosts. Here is the link below:

The above post is very informative and if you follow this step by step you will definitely succeed. The commands that the author uses might look a bit daunting but these commands are nothing but the exploded version of the docker-compose files which you have already seen in our BYFN network above.

Note: please place the project folder that you download from this post under the fabric's samples directory as follows

/fabric/fabric-samples/Build-Multi-Host-Network-Hyperledger/

Tips:

1. If you are running the multi host project on AWS servers then you need to open following ports on your hosts/nodes for the swarm network to work

• TCP port 2377 for cluster management communications
• TCP and UDP port 7946 for communication among nodes
• UDP port 4789 for overlay network traffic

2. If you are feeling lazy you can create your EC2 instance using this public AWS server image (ami-6a81c615) I have published with all the pre-requisites installed, including the samples, binaries and multi-host example files. You can use the same image to start both the nodes (PC1 & PC2) as mentioned in step 3 tutorial.

3. Use the official hyperledger chat channel you would learn a lot from questions people are asking and you can also ask any questions that you might have.

Outcome / Success Criteria:

Finally if you have followed all the steps above correctly and have had some patience then you should see an output as in image below which basically shows the processes that are running for CA, Orderer, Peer1, Peer2, Couch-dbs & CLI (as per step 3 post). The queries and updates run on the CLI do reflect in both the hosts. So if you triggered a transfer of an amount from A->B on host 1 and if you query the balances on host 2 you should see the updated balances. At this point you know that your blockchain network is working fine.

Way Forward: Exploring blockchain tools and BAAS

On completing Steps 1 to 3 you should start feeling confident about building a blockchain network for a practical use case. From this point onwards if you want, you can also pick up any tool like hyperledger-composer to build a network or use any of the wrappers over hyperledger such as IBM or Microsoft's cloud. Since you already know how fabric works these tools will be pretty easy for you to understand. It is always beneficial to understand the core concept first before you dive into using the tools which promise to make things easier or faster for you and that is the reason I think working with hyperledger-fabric first is the right way to start your development journey in blockchain.