Auto Syncing configuration files in Consul

The objective of this article is to demonstrate an approach for auto-syncing contents of files stored in Git repositories to Consul.

Use case

There are many applications where the runtime configurations such as environment variables are stored in Consul's key-value store. By doing so, the application can be configured to dynamically fetch the configurations from Consul during start.

A tool by Hashicorp called 'Consul-Template' makes this possible.

Additionally, Consul-template also keeps watching for changes in Consul. Whenever it detects a change - it will direct those changes to the application as well. The application will be restarted with the new configurations.

If you are interested in getting to know the above approach deeply - I wrote a step-by-step guide for it. Find it here - https://www.dhirubhai.net/pulse/securing-managing-environment-variables-using-tools-khandelwal/

Need for version control

When using the above approach for taking key-value pairs from Consul, it will be wise to have key-value pairs under version control before it gets into Consul as it will give you better audit trails, secure access control mechanisms, and configuration storage.

Without some kind of versioning control, it can get confusing, messy, and problematic.

Challenge

Key-value pairs can be stored in .json, .yaml, or .conf files inside Github. But this will introduce an additional step of syncing Github files with the Consul. In the case of a big application - it would mean syncing hundreds of files to Consul.

Automating this step is crucial!

Solution

Git2Consul is a tool that can sync files from Github to Consul. It's totally automated.

The below image shows the complete flow of what we want to achieve. We will add the corn job towards the end.

Find it here - https://github.com/breser/git2consul

Let's try it out! You can use the files here to follow along - https://github.com/shishirkh/git2consul

Setup

Create a consul server for trying out the tool.

cd consul/
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install consul hashicorp/consul --values consul-custom-values.yaml
        

Create a Docker file with git2consul installed.

FROM node:14
RUN npm install -g git2consul
RUN apt-get update
RUN apt-get install vim -y
        

Create Image

cd docker/
docker build -t shishirkhandelwal/git2consul:0.2        

Write the configuration.json file

We will discuss this file in the next section.

{
? "version": "1.0",
? "no_daemon": true,
? "repos" : [{
? ? "name" : "test1",
? ? "url" : "https://github.com/shishirkh/git2consul.git",
? ? "branches" : ["main"],
? ? "source_root": "samples/",
? ? "expand_keys": true,
? ? "mountpoint": "mount1",
? ? "include_branch_name" : true
? },
? {
? ? "name" : "test2",
? ? "url" : "https://github.com/shishirkh/git2consul.git",
? ? "branches" : ["dev"],
? ? "source_root": "samples2/",
? ? "expand_keys": true,
? ? "mountpoint": "mount2",
? ? "include_branch_name" : true
? }]
}
        

Create config maps & deployments.

Be sure to change the consul host and port according to your namespace!

cd k8s_yamls/
kubectl create cm configuration-file --from-file=configuration.json
kubectl create -f deployment-git2consul.yaml

        

The screenshot below shows the log generated by git2consul.

The screenshot below shows the key-value pairs synced into consul.

The screenshot below shows the tree structure created inside consul.

Configurations

The configuration.json being used above illustrates a 2 repo git2consul setup: Both are taking files from the same repo but are pointed to different branches. Some of the important settings for git2consul are explained below.

source_root: It tells git2consul to navigate into a subdirectory inside the repo before mapping files to KVs.

expand_keys: It tells git2consul to treat any valid JSON file as if it contains a subtree of Consul KVs.

include_branch_name: It tells git2consul to prefix the branch name in the key.

mountpoint: It tells git2consul to prepend a string in the key.

For the rest, pls refer to the original project repo.

Auto Syncing

By creating a Kubernetes cron job, git2consul can be run at regular intervals. In this way and will be able to sync new changes.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
? name: cronjob-git2consul
spec:
? schedule: "* * * * *"
? jobTemplate:
? ? spec:
? ? ? template:
? ? ? ? spec:
? ? ? ? ? restartPolicy: OnFailure
? ? ? ? ? containers:
? ? ? ? ? - image: shishirkhandelwal/git2consul:0.2
? ? ? ? ? ? name: git2consul
? ? ? ? ? ? command: ["/bin/sh","-c","git2consul --config-file $(CONFIGURATION_FILE)"]
? ? ? ? ? ? resources: {}
? ? ? ? ? ? env:
? ? ? ? ? ? - name: CONSUL_ENDPOINT
? ? ? ? ? ? ? value: "192.168.49.2"
? ? ? ? ? ? - name: CONSUL_PORT
? ? ? ? ? ? ? value: "32458"
? ? ? ? ? ? - name: CONFIGURATION_FILE
? ? ? ? ? ? ? value: "/tmp/configuration.json"
? ? ? ? ? ? volumeMounts:
? ? ? ? ? ? - name: configuration
? ? ? ? ? ? ? mountPath: /tmp/configuration.json
? ? ? ? ? ? ? subPath: configuration.json
? ? ? ? ? volumes:
? ? ? ? ? - name: configuration
? ? ? ? ? ? configMap:
? ? ? ? ? ? ? name: configuration-file
        

Apply the cronjob

cd k8s_yamls/
kubectl apply -f cronjob-git2consul.yaml        

This is how Auto Syncing can be achieved from version controlling Github to key-value distributor Consul.

Do leave a comment with your feedback or Queries! Share this in your network!