To bash or not to bash - yq

With so many options at a programmer's disposal, one may wonder is it still worth using the Bourne Again Shell or simply bash? After all one may use Go, Python, Node, or some other fancy tooling. So why bash?

Simple shell programs are extremely easy and natural to write and allow you to effortlessly glue together any command line tools available in your system. Hence, a shell program allows you to rapidly prototype ideas.

This series introduces a couple of bash concepts and shows how to apply them to a few command line tools useful to finish some everyday tasks.

Today, we take a look at yq.

yq - sed for YAML files

yq [1] allows you to perform all kinds of operations on YAML files. Although it is not as powerful as jq, it is still a valuable addition in many toolboxes. Just like jq, you can use yq to do some simple processing on yams files.

yq allows you to perform a couple of operations, such as read (r), write (w) and merge (m) YAML files. It also provides some nice path expressions that help you to navigate through the files.

To illustrate its usage, let me show you some examples.

Example 1: Adding a value to a map

Given the following YAML file

# platform.yml

platform: fancyworld
services_to_install:
  smart/world: "1.0.1"
  bright/star: "2.0.4"

We want to add our service from the previous post to this simplified platform configuration.

All we need to do is to run the following command:

(1) file="./platform.yml"
(2) name="smart/my-fancy-service"
(3) version="1.0.0"
(4) yq w -i --style=double ./"$file" services_to_install."$name" "$version"

And we get:

platform: fancyworld
services_to_install:
  smart/world: "1.0.1"
  bright/star: "2.0.4"
  smart/my-fancy-service: "1.0.1"

So let's break down what happens in line (4).

We want to write to a file (yq w) from ./$file and we want to make the changes in place (-i). We also change the style of how to persist the value by enforcing the usage of double quotes ("value"). Furthermore, we need to state a path expression and value. If the path does not exist, it will be created on the fly. So, we state to create a new entry under 'services_to_install' and assign a value to it.

Path expressions are quite useful especially when you need to just add new entries to files and performance is not a concern.

Example 2: Adding items to lists

Unfortunately, our fancyworld knows some black sheep that date back to some legacy days. One of those is the way proxies are managed.

# proxy_conf.yaml

---
- name: "ensure private key is present"
  copy:
    src: ~/keys/slave-key.pem
    dest: "/home/{{ ansible_ssh_user }}/.ssh/slave-key.pem"
    mode: 0600

- name: "ensure that python present"
  apt:
    name: "{{ item }}"
    state: present
  with_items:
    - python-dev
    - python-pip
  become: True

- name: "ensure that python dependency for Ansible is present"
  pip:
    name: docker
    state: present
  become: True

- block:
  - name: "bright-star repo"
    import_role:
      name: common.internal-proxy-pass
    vars:
      env: "{{ platform }}"
      service_name: bright-star
      service_fqdn: bright-star{{ env_domain_sufix }}.example.com
      target_service_port: 8090
      graylog_enabled: false

We want to append our service 'my-fancy-service' to the list property block. If all you care about is get something working quickly and performance is not a concern you may use the shell again:

file="./proxy_conf.yml"
name="smart-my-fancy-world"
service_port=8080

yq w --style=double -i "$file" '[-1].block[+].name' "$name repo"
yq w -i "$file" '[-1].block.[-1].import_role.name' "common.internal-proxy_pass"
yq w -i --style=double "$file" '[-1].block.[-1].vars.env' "{{ platform }}"
yq w -i "$file" '[-1].block.[-1].vars.service_name' "$name"
yq w -i "$file" '[-1].block.[-1].vars.service_fqdn' "$name{{ env_domain_sufix }}.example.com"
yq w -i "$file" '[-1].block.[-1].vars.target_service_port' "$service_port"
yq w -i "$file" '[-1].block.[-1].vars.graylog_enabled' "false"

This is certainly not an ideal example, but it shows how to use path expressions to get the job done. The tricky part here is that 'proxy_conf.yml' is a list, which contains sublists as well. To get the job done, we need to find the last item in the top level list, look for the name 'block' in order to append a new entry to this sublist. This is done using [-1].block[+].name "your string". Fortunately, the block item is the last one of this file. Once we created a new entry, we can simply navigate to the last item of this list in order to add our properties, using [-1].block.[-1].property.

And we get

# for the sake of readability I ommitted the previous lines
- block:
  - name: "bright-star repo"
    import_role:
      name: common.internal-proxy-pass
    vars:
      env: "{{ platform }}"
      service_name: bright-star
      service_fqdn: bright-star{{ env_domain_sufix }}.example.com
      target_service_port: 8090
      graylog_enabled: false

  - name: "smart-my-fancy-world repo"
    import_role:
      name: common.internal-proxy-pass
    vars:
      env: "{{ platform }}"
      service_name: my-fancy-world
      service_fqdn: my-fancy-world{{ env_domain_suffix }}.example.com
      target_service_port: 8080
      graylog_enabled: false

As you can see, this solution is merely good enough to get the job done as you need a couple of file reads and writes. However, if all you want is to simply get such a job done and performance is not a concern, you may try it using a shell script before writing something more sophisticated.

Therefore, little shell scripts are once again ideal for rapid prototyping.

In a previous article I presented jq, a quite popular command line tool [2]

Resources

1. https://mikefarah.gitbook.io/yq/
2. https://www.dhirubhai.net/pulse/bash-jq-sebastian-kaiser/