Autocon 1 - A Deeper Dive

Preface

My first pass at summarizing the Autocon 1 conference left me feeling flat, as if there were not many actionable takeaways from what I provided. The confluence of my work on providing more context for simulated panel discussions I am working on and that feeling of discontent led to a much more powerful approach to extracting takeaways from Autocon 1 and, in the future, other texts. The journey continues, but significant progress has been made. I now have over 600 extracted statements from the Autocon 1 conference on successes, lessons learned, and challenges as I ponder further ways to transform that wealth into value for myself and readers/subscribers.

Most of what you will read below is AI-generated. On a purely human note, the community is creating something special here. Programming in any domain can be overwhelming today. To paraphrase networking legends, the great thing about programming today is that there is so much choice in tools, libraries, and platforms. The bad thing about programming today is that there is so much choice. Look around the industry; you will also notice a growing number of grumbles about documentation and accurate search results - I have a few myself.

On top of all that, while some aspects of IT have managed to land themselves in a walled garden of infrastructure abstractions (don't get me wrong - lots of complexity there), networking practitioners are still dealing with a diverse industry of actual hardware and a few more layers of protocol interactions to manage.

Autocon 1 highlighted the many challenges practitioners face regarding network automation and the successes that have occurred. Perhaps most importantly, it shared the wealth of lessons learned.

I am not actively involved in this community. Still, as a guy who started his IT career in a network operations center, I could not be more excited to see how this community evolves.

And now, back to AI ;-)

Introduction

The AutoCon 1 conference showcased significant progress in network automation across various domains. Key successes included large-scale implementation of automation in network configuration, device staging, and infrastructure management, reducing repetitive work, improved scalability, and cost savings. The conference highlighted the development of innovative tools and technologies, such as custom workflow engines and container labs, and the importance of data management and API development. Organizational shifts accompanying successful automation initiatives were emphasized, including upskilling network engineers, creating new roles, and fostering knowledge-sharing communities. The business value of automation was evident in reduced project delays, improved service delivery, and enhanced customer experiences.

The conference also revealed important lessons learned and challenges in network automation efforts. Attendees stressed the need for a strategic approach to automation, starting small with impactful use cases and evolving solutions over time. The importance of community, knowledge sharing, and cross-functional collaboration was highlighted. Technical challenges included the complexity of automating diverse network environments, integrating legacy systems, and managing data effectively. Organizational and cultural challenges were significant, including obtaining management buy-in, overcoming resistance to change, and bridging the skills gap between traditional network engineering and software development. Emerging challenges related to AI and machine learning in network automation were also discussed, reflecting the ongoing evolution and the complex nature of implementing and scaling network automation in enterprise environments.

Successes and highlights

The AutoCon 1 conference highlighted significant advancements and successes in network automation across various domains. A major theme was the large-scale implementation of automation in network configuration, device staging, and infrastructure management. Participants reported impressive achievements such as automating most network infrastructure in under a year, successfully staging thousands of devices, and migrating large numbers of customers to new platforms. The conference showcased how automation has reduced repetitive work, improved scalability, and significant cost savings. Another key theme was developing and adopting innovative tools and technologies, including custom workflow engines, container labs, and Kubernetes for network automation. Participants also emphasized the importance of data management, with many reporting successes in consolidating scattered data, improving data quality, and developing APIs for better access and integration.

The conference also underscored the organizational and cultural shifts accompanying successful network automation initiatives. Many speakers highlighted the importance of upskilling network engineers, creating new roles like DevOps engineers, and fostering a knowledge-sharing and collaboration community. Implementing agile practices, automated testing, and continuous improvement processes were frequently mentioned as crucial to success. Additionally, there was a strong focus on the business value of automation, with participants reporting reduced project delays, improved service delivery, and enhanced customer experiences. The conference also touched on future trends, including promising experiments with network telemetry data and the potential of AI in network automation, indicating an exciting and evolving landscape in the field.

Lessons learned

The AutoCon 1 conference revealed several key themes in lessons learned from network automation efforts. A significant focus was on the importance of community and knowledge sharing, with attendees emphasizing the value of dedicated forums, learning from experienced professionals, and building cross-organizational connections. Speakers stressed the need for a strategic approach to automation, advocating for starting small with impactful use cases, focusing on business value, and evolving solutions over time. The importance of good network design alongside automation was mentioned, as was the need to empower and upskill network teams rather than replace them. Technical considerations were also prominent, with discussions on tool selection, data management, and the implementation of testing strategies. Speakers emphasized the importance of clean, structured data and the need for a unified operational model across different automation tools.

Organizational and cultural aspects of network automation were another major theme. Attendees discussed the challenges of transitioning from traditional network engineering to software-defined approaches, the importance of addressing resistance to change, and the need to adapt language and approaches when persuading executives and stakeholders. The conference highlighted the value of cross-functional collaboration, particularly between network engineers and software developers. Looking to the future, there was discussion about the potential of AI in network automation, with speakers advocating for starting small with AI initiatives while aiming high. The need for vendor-agnostic, open-source solutions was emphasized, as was the importance of standardization across vendors to enable more effective automation. Overall, the lessons learned reflected a field in transition, grappling with technical, organizational, and strategic challenges while pushing toward more advanced, AI-driven automation solutions.

Challenges

The AutoCon 1 conference highlighted a multitude of challenges facing network automation efforts. A prominent theme was the technical complexity of automation in diverse network environments. Attendees discussed the need for uniform data models across vendors, difficulties integrating legacy systems, and challenges in scaling automation from lab to production. The complexity of supporting various hardware platforms, operating systems, and configurations was mentioned. Data management emerged as a critical challenge, ranging from scattered and low-quality data to the need for a trusted source of truth and effective data synchronization across systems.

Organizational and cultural challenges were equally significant. Speakers highlighted the difficulty in obtaining management buy-in for automation initiatives, overcoming resistance to change, and bridging the gap between traditional network engineering and software development skills. The need for upskilling network engineers in programming and DevOps practices and the challenges of changing mindsets and processes were emphasized. Time constraints were consistently mentioned, with network teams needing help finding time to learn new skills and implement automation alongside daily operations. The conference also touched on emerging challenges related to AI and machine learning in network automation, including ensuring AI systems remain under human control, defining clear use cases, and developing vendor-agnostic solutions for heterogeneous network environments. The challenges discussed reflected the complex, multifaceted nature of implementing and scaling network automation in enterprise environments.

Conclusion

In conclusion, the AutoCon 1 conference illuminated the current state of network automation, showcasing impressive successes while acknowledging persistent challenges. The event underscored the transformative potential of automation in improving efficiency, scalability, and service delivery across network operations. However, it highlighted the complexity of implementing these solutions, particularly in diverse and legacy environments. The conference emphasized the critical role of organizational culture, strategic planning, and continuous learning in driving successful automation initiatives. Integrating AI and machine learning presents exciting opportunities and new challenges as the field evolves. The network automation community must continue fostering collaboration, prioritizing knowledge sharing, and developing innovative solutions to address the remaining technical and organizational hurdles. By doing so, the industry can further unlock the full potential of network automation, driving greater efficiency and innovation in network management and operations.

Appendix A - A Portion of the Statements Made at Autocon 1 by Speakers

Success with Automation and other highlights

1. Community and Knowledge Sharing

• Dedicated forum for network automation
• Exchange of ideas and knowledge sharing
• Learning from experienced people
• Building connections across organizations and continents
• Fostering community and sharing of automation efforts

2. Network Automation Achievements

• Automation of network configuration tasks
• Reduced repetitive work for network engineers
• Improved network scalability through automation
• Successful implementation of zero-touch provisioning solution
• Automation of the majority of network infrastructure in under a year
• Automated customer provisioning process
• Automated core switch configurations

3. Tools and Technologies

• Development of tools for various stages of network automation
• Linux CLI enabling scripting for network automation
• Cloud providers successfully automating large-scale networks
• Implementation of TCL scripting to support various iOS versions
• Development of a custom workflow engine
• Creation of Terrero [sp?], a new community-based product for network automation
• Container lab and images open-sourced
• Kubernetes used as a platform for network automation

4. Data Management and Integration

• Populating Netbox with infrastructure data
• Creating device inventory in Netbox
• Consolidated scattered data into a new repository
• Improved data quality and structure
• Developed GraphQL API for data access

5. Workflow and Process Improvements

• Development of event-driven architecture for network device staging
• Implementation of orchestration for automation processes
• Introduced end-to-end process modeling for infrastructure management
• Streamlining change management processes

6. Project Successes

• The successful staging of nearly 10,000 devices for the first customer
• Completion of 5,000 PnP OS upgrades
• Staging of around 700 layer 3 access devices for a new customer
• Delivered over 14,000 infrastructure patterns to customers
• Upgraded 100 firewalls in a single maintenance window
• Successfully migrated 10,000 B2B customers from Cisco 7600 to ASR 9k routers

7. Efficiency and Cost Savings

• Significant engineer time savings for each pattern delivered
• Reduced project time delays
• Retrieved thousands of unused public IPv4 addresses
• Achieved over 500k cost savings using zero-touch provisioning

8. Testing and Quality Assurance

• Maintenance of high test coverage (98%) for easier long-term maintenance
• Implemented automated testing of automation and infrastructure
• Pre-testing and post-testing strategies
• Integrated testing pipeline combining various tools and approaches

9. Skill Development and Organizational Change

• Teaching network engineers programming skills
• Upskilled network engineers in automation, agile, and software practices
• Created new roles like DevOps engineers within network engineering
• Initiated organizational change to embrace new processes

10. Innovation and New Technologies

• Progress in standardizing network APIs and data models
• Promising early experiments with network telemetry data
• Rapid prototyping and deployment using Terraform
• Implementation of DNS load balancing for VPN concentrators

Lessons Learned Summary

1. Community and Knowledge Sharing

• Importance of dedicated forums for network automation
• Value of knowledge exchange and idea sharing
• Learning from experienced professionals in the field
• Building connections for future support and collaboration
• Benefits of being part of a global, cross-organizational community

2. Automation Strategy and Approach

• Start with impactful use cases that have meaning to the business
• Start small but allow for growth
• Focus on use cases and value, not just the platform itself
• Begin with device inventory, programmatic access, and config pushing
• Automate most time-consuming daily tasks first
• Start with small steps and adjust as you go
• Define your end goal before starting
• Focus on the next immediate step rather than planning everything upfront

3. Technical Considerations

• Good network design needed alongside automation
• Automation is a cycle, not just config generation
• Consider multi-vendor tools to avoid vendor lock-in
• Use version pinning to avoid breaking changes
• Centralize configuration to avoid scattered settings
• Implement a layered architecture with clear demarcation between network and IT systems
• Use tools like Netbox for device inventory management
• Learn basic programming skills, starting with Python
• Utilize libraries like Netmiko for device interactions
• Use Jinja for creating configuration templates

4. Data Management

• Importance of clean, structured data
• Consolidating scattered data into a single repository
• Understanding data in-depth before migration
• Implementing forms for consistent data input
• Using tags to differentiate and categorize resources
• Connecting business and network data through service objects

5. Testing and Quality Assurance

• Write comprehensive tests for easier future maintenance
• Implement pre-testing and post-testing
• Conduct unit, integration, and system-level tests
• Use modeled, virtualized/containerized, and physical environments
• Leverage Batfish for offline impact analysis
• Utilize Suzieq for time-based network analysis
• Employ PATS as a complete testing framework
• Use Pytest for easy test writing and parameterization

6. Organizational and Cultural Aspects

• Empower network teams, don't replace them
• Address cultural challenges in transitioning from traditional network engineering to software-defined approaches
• Upskill network engineers in agile practices, automation principles, and testing
• Evolve network architect roles to understand product ownership and business needs
• Introduce new roles like DevOps engineers and design leads within network engineering
• Adapt language and approach to persuade executives and stakeholders

7. Best Practices

• Consider long-term maintainability when writing code
• Avoid shortcuts that can lead to technical debt
• Maintain up-to-date documentation
• Spread knowledge across team members to avoid single points of failure
• Prioritize thorough documentation for complex automation projects
• Implement version control for better collaboration and tracking
• Use agile and DevOps practices, including CI/CD pipelines

8. Challenges and Pitfalls

• Lack of time is a major barrier for teams
• Myth of uniform data model across vendors
• Users may not use the solution as intended
• Be cautious about adding complexity to the system
• Avoid using forks of open-source projects
• Be aware of API rate limits when working with wireless management platforms

9. AI and Future Trends

• ChatGPT can potentially generate network automation scripts with high accuracy
• AI-assisted automation can improve productivity for network engineers
• AI Ops combines big data and machine learning to enable and help engineers
• Start small with AI and aim high
• AI solutions should be vendor-agnostic and open-source
• Clear use cases and well-trained algorithms are required for AI Ops

10. Vendor and Industry Collaboration

• Communicate openly with vendors about specific automation needs and challenges
• Partner with vendors and development partners to supplement internal skills
• Open-source initiatives can drive education and adoption
• Collaboration across the industry is important for advancement
• Consider using open standards like IETF models for network services

Challenges summary

The text mentions numerous challenges related to network automation. Here's a summary of the key challenges:

1. Technical Challenges:

• Lack of uniform data models across vendors
• Shifting standards and technologies
• Integrating legacy devices and systems
• Debugging high-level abstractions
• Scaling automation from lab to production
• Managing complex configurations and diverse platforms
• Implementing proper testing and rollback procedures
• Handling inconsistent behaviors across different devices

2. Skill and Knowledge Gaps:

• Need for advanced programming skills
• Learning new technologies and tools
• Balancing traditional networking skills with automation
• Upskilling network engineers in software development

3. Organizational and Cultural Challenges:

• Obtaining management approval and buy-in
• Overcoming resistance to change
• Bridging the gap between programmers and non-programmers
• Managing team dynamics and collaboration
• Articulating the value of automation to leadership

4. Tool and Vendor-related Challenges:

• Choosing and evaluating appropriate tools
• Integrating multiple tools and services
• Managing open-source dependencies
• Dealing with vendor-specific limitations

5. Data Management Challenges:

• Ensuring data quality and consistency
• Implementing a reliable source of truth
• Synchronizing data between systems
• Handling large volumes of telemetry data

6. Operational Challenges:

• Balancing automation with manual intervention
• Managing frequent changes to network infrastructure
• Implementing proper monitoring and alerting
• Ensuring security and compliance

7. Strategic Challenges:

• Defining clear automation strategies and goals
• Balancing short-term objectives with long-term improvements
• Making automation solutions consumable by the business
• Adapting to cloud and multi-vendor environments

8. Resource Constraints:

• Time limitations for learning and implementation
• Budget constraints
• Shortage of skilled personnel

9. AI and Advanced Automation Challenges:

• Implementing AI-assisted network automation
• Ensuring AI systems remain under human control
• Developing standardized data models for AI algorithms