Dealing with Complex Nested JSON in Real-World Network Automation: Python and Ansible in Action

In our previous article, Supercharging Network Automation: How to Use REST APIs with JSON for Smarter Networks, we explored how to use REST APIs with JSON to automate network configurations. However, as network infrastructures scale, you will for sure encounter deeply nested JSON structures that represent multi-level configurations across different network devices. Today, we’ll dive deeper into how to handle these complex, multi-vendor configurations in live production environments using both Python and Ansible.

Imagine this scenario: you're managing the automation of a large-scale network for a multinational company with multiple data centers. You need to make configuration changes across various regions, handle different tenants, bridge domains, subnets, security policies, and even deploy firewall rules. After querying the network devices, you receive deeply nested JSON structures that represent not just one device, but an entire configuration tree. The complexity can quickly become overwhelming!

The Problem: Managing Large-Scale, Nested Configurations in Production

Here’s an example of what the JSON response might look like after querying your network devices:

{
  "dc1": {
    "tenants": [
      {
        "name": "Tenant1",
        "bridgeDomains": [
          {
            "name": "BD1",
            "subnets": [
              {
                "ip": "192.168.1.0/24",
                "securityPolicy": {
                  "firewallRules": [
                    {
                      "id": "Rule1",
                      "action": "allow",
                      "source": "192.168.1.10",
                      "destination": "10.10.10.10",
                      "protocol": "TCP"
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      {
        "name": "Tenant2",
        "bridgeDomains": [
          {
            "name": "BD2",
            "subnets": [
              {
                "ip": "192.168.2.0/24",
                "securityPolicy": {
                  "firewallRules": [
                    {
                      "id": "Rule2",
                      "action": "deny",
                      "source": "192.168.2.20",
                      "destination": "10.20.20.20",
                      "protocol": "UDP"
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }
}        

Challenge: Now, imagine needing to update firewall rules or subnet configurations across hundreds of tenants and bridge domains. It’s a multi-layered configuration tree where a simple manual process would take hours or days. Automation is the only answer here!

Why Use Third-Party Libraries Like "jmespath" Instead of the Built-in JSON Module?

At first glance, you might wonder, “Why not just use Python’s built-in json module to handle this?” While Python's native json module is perfect for loading and parsing basic JSON data, it lacks advanced querying capabilities for complex, deeply nested structures.

Here’s why using third-party libraries like jmespath is more efficient for these tasks:

1. Simplified Queries: The json module requires you to manually navigate through each layer of the JSON structure. This can be tedious and prone to errors, especially when working with large, complex JSON objects. In contrast, jmespath allows you to extract data with concise, query-like expressions, reducing code complexity and improving readability.
2. Flexibility and Power: With json, you would need nested loops or conditionals to traverse and filter through multiple layers which would be very expensive from Time and Space complexity perspective. jmespath provides a more powerful and flexible way to access, filter, update, delete data from any depth.
3. Efficiency: In production environments where performance matters, third-party libraries like jmespath are optimized for traversing large and complex JSON structures, allowing you to extract and update specific values with far fewer lines of code.

Solution 1: Parsing and Automating with Python Using jmespath

Installing jmespath in Python is quick and easy. First, make sure you have Python installed on your system. Then, open your terminal or command prompt and run the following command:

pip install jmespath        

Python Example: Updating Security Policies Across Multiple Bridge Domains with jmespath

import json
import jmespath

# Sample complex nested JSON structure (as shown above)
json_data = '''
{
  "dc1": {
    "tenants": [
      {
        "name": "Tenant1",
        "bridgeDomains": [
          {
            "name": "BD1",
            "subnets": [
              {
                "ip": "192.168.1.0/24",
                "securityPolicy": {
                  "firewallRules": [
                    {
                      "id": "Rule1",
                      "action": "allow",
                      "source": "192.168.1.10",
                      "destination": "10.10.10.10",
                      "protocol": "TCP"
                    }
                  ]
                }
              }
            ]
          }
        ]
      },
      {
        "name": "Tenant2",
        "bridgeDomains": [
          {
            "name": "BD2",
            "subnets": [
              {
                "ip": "192.168.2.0/24",
                "securityPolicy": {
                  "firewallRules": [
                    {
                      "id": "Rule2",
                      "action": "deny",
                      "source": "192.168.2.20",
                      "destination": "10.20.20.20",
                      "protocol": "UDP"
                    }
                  ]
                }
              }
            ]
          }
        ]
      }
    ]
  }
}
'''

# Load the JSON data
data = json.loads(json_data)

# Use jmespath to query and update firewall rules
expression = "dc1.tenants[].bridgeDomains[].subnets[].securityPolicy.firewallRules[]"
rules = jmespath.search(expression, data)

# Update firewall rules to 'deny' across all tenants
for rule in rules:
    print(f"Updating rule {rule['id']} from action {rule['action']} to 'deny'")
    rule['action'] = 'deny'

# Output the updated JSON structure
updated_json = json.dumps(data, indent=4)
print(updated_json)        

• This code query nested fields across all tenants, bridge domains, and subnets using jmespath.search() function.
• Then loop through all the firewall rules and update them as needed, such as changing the action from allow to deny.

Why This Is More Efficient: Instead of manually traversing multiple layers of the JSON object using basic Python loops, jmespath allows us to extract and manipulate specific data points in one simple query. This reduces the amount of code, makes it easier to read, and improves maintainability in production environments.

Solution 2: Automating with Ansible

Ansible excels at automating network tasks across multiple devices and configurations. It can handle JSON structures and modify them in real-time. Let’s see how to apply this to nested JSON.

Ansible Example: Updating Firewall Policies in Nested JSON

---
- name: Update Nested JSON for Firewall Policies
  hosts: localhost
  tasks:
    - name: Load the JSON structure
      set_fact:
        json_data: {
          "dc1": {
            "tenants": [
              {
                "name": "Tenant1",
                "bridgeDomains": [
                  {
                    "name": "BD1",
                    "subnets": [
                      {
                        "ip": "192.168.1.0/24",
                        "securityPolicy": {
                          "firewallRules": [
                            {
                              "id": "Rule1",
                              "action": "allow",
                              "source": "192.168.1.10",
                              "destination": "10.10.10.10",
                              "protocol": "TCP"
                            }
                          ]
                        }
                      }
                    ]
                  }
                ]
              }
            ]
          }
        }

    - name: Update firewall rules across all tenants and bridge domains
      set_fact:
        updated_json: "{{ json_data | json_query('dc1.tenants[].bridgeDomains[].subnets[].securityPolicy.firewallRules[]') | map('replace', 'allow', 'deny') }}"

    - name: Print updated JSON structure
      debug:
        msg: "{{ updated_json }}"        

• We used Ansible’s templating engine to query the JSON structure and replace all "allow" actions with "deny" using the json_query filter.
• This method can be scaled across hundreds of tenants, bridge domains, and firewall rules in a real production environment.

Teaser for the Next Article

Dealing with complex nested JSON in network automation can be a challenging task, but with the right tools, you can automate even the most sophisticated and complicated configurations efficiently. While Python and Ansible are excellent tools for handling these tasks, several powerful third-party libraries in Python are specifically designed to handle and manipulate complex JSON structures.

In the next article, we will deep dive into a comparison of third-party libraries, including jmespath, jsonpath-ng, Pelecanus, and JSONnata. We'll test these tools against real-world production network configurations like what we did here and explore their strengths and weaknesses. You’ll get hands-on into which library works best for large-scale network automation scenarios.

Stay tuned for this practical, side-by-side evaluation of the most powerful tools available for handling nested JSON in production!