Automated Refreshing of Infrastructure Documentation

In a world full of technological advancements and everchanging IT environments, many organisations still struggle to maintain and update infrastructure documentation. In some cases they don't trust documentation or have any.... So what do they do? A site survey of course! Conducting site surveys and extensive DC audits are painful, but not as time consuming as baselining the IT environment properly and verifying change processes are working. Both cause delays to projects, stress out already overworked staff and cost a lot of time and money in the process (especially when contractors are involved). So assuming we don't want to repeat surveys or baseline audits every project (or every year) then refreshing infrastructure documentation using automated methods will directly reduce engineering effort and make information more trusted.

Automation Is A Journey, Not A Project!

If you change the way that infrastructure documentation is created, presented, and maintained then there are many savings to be had in - wasted effort, additional costs and time to complete tasks and projects. It also makes working as a team easier as site surveys are often only understood by those who were there, with recording based on what was important at the time, using the closest tools at the time - sometimes a scrap of paper. To change this behaviour we follow some basic steps.

Step 1 - Simplify and deduplicate

Step 2 - Enhance consistency by using shared data sets

Step 3 - Automate project and operational documentation where appropriate

If up to date documentation isn't in your inventory, then a baseline audit will be required to obtain the gaps in data as well as how information is consolidated and presented. Having a list of every connection in a building is essential for understanding the availability of switch ports, identifying areas with insufficient inter-room connectivity, and addressing challenges associated with relocating user groups within the building. Getting such a list is not so easy however!

Step 1 looks to reduce the problem of multiple naming conventions, symbol types, Excel formats, etc. to fewer data sets that are easier to maintain and process.

I'll give you a hint, discovery tools are partially useful - but what do you put their data into?

Step 2 provides data for integration and processing made possible by common naming and data formats. Data may be useful and trusted, but it still needs to be processed to help understanding and decision making. Using auto-discovery tools sounds good, but if you are designing or upgrading new infrastructure, then your documentation will need to incorporate planned as well as existing components and connectivity. If you want to manage forward capacity allocation then knowing what is used is half the story, you need to know what is still available or planned like power, cabling, switch ports, disk, CPU etc.

Step 3 addresses how we automate infrastructure information, as there is no magic silver bullet to buy. The answer? I'll give you a hint, discovery tools are partially useful - but what do you put their data into? In practice, without addressing the basics of Step 1 and Step 2, Step 3 documentation automation will be limited in impact.

What Does Automation Mean For Me?

Here's a few reasons for adopting automated methods in case you or colleagues need convincing.

• A whole lot of time consuming, very detailed work is removed from individuals and teams.
• There is less need to communicate, chase and confirm updates from colleagues/suppliers.
• Updates and accuracy of information can be predictable and timely - daily, weekly, etc.
• Consistency is assured if there is less manual effort required and the use of standardised formats and data. Compare to current information variations.
• Less updates are needed if data is consolidated and produces multiple outputs, rather than have lots of separate data sets and separate documents/diagrams/web pages. A simple update on a server can propagate to multiple spreadsheets, multiple Visio diagrams and various capacity reports.
• Having confidence in documentation can reduce the necessity for site surveys, travel, and project delays.. Untrusted information creates risks to project delivery and personal reputation, so make it trusted with clear ownership and process.
• With improved update processes there is less need for repeated baseline audits, which can often be reduced to sample verification exercises. Saving everyone time and money.
• Less hassle from difficult gatekeeper colleagues who want more detail than they originally requested. Often detail is removed from information for easy comprehension, so drill downs from dashboards to detail with web links or low level views are possible with automated systems.

To expand in more detail..

Step 1 - Preparation For Automation

As an organisation you need to establish your "why" and start the journey collaboratively, to ensure people across the company are on board and committed to change. Committing to an initial assessment or proof of concept will quickly give us the starting point on the path to automated documentation. How much wasted time is there currently? How many different formats? Which of the 6 ways to label a patch cable should we use going forward - rather than all 6. It is inevitable that when presented clearly with the current documentation state, you'll see many opportunities to improve.

For many in physical infrastructure, the first steps are just to get lists that reflect the locations, rooms, racks, devices, ports and connectivity.

For logical infrastructure it is more conceptual with virtual components, systems, groupings, dependencies, etc. Don't be surprised how many lists you'll find in databases and spreadsheets. Just look for the following to see how many separate lists/tabs there are currently for these two types?

• Inventory List - hardware, software and network components
• Connectivity - port allocations and end devices

Step 2 - Reuse of Data by Integration and Processing

A spreadsheet is a two dimensional view of data, fixed into rows and columns. If we need different data for different devices we add tabs - switches, routers, cabinets, etc. For summaries and consolidation we may add more tabs, or separate spreadsheets for reporting and it all quickly becomes unmanageable. Hence the reason why most will progress quickly to a relational database, rather than try and integrate multiple spreadsheets which is easily broken by changes. Integrating databases is simpler as they have to be more structured, and make reporting from a large dataset simpler.

Here is the approach that we take with the AssetGen System from data collection/update to creating information sets (Excel, Visio, Text, PDF, Web page) for use by project and operational teams. As expected it uses an industry SQL database for single or multi-user use.

Rather than coping with many different documents and formats, the database driven approach simplifies the creation and update of different information sets. This has the added benefit of also making it easy to update both data and diagrams where there be duplication of devices across the many types of documentation. From a time perspective - the automation time taken to create or refresh something like a rack diagram will be 5-10 seconds. Much faster than the hours, days or weeks often experienced with manual methods (or in one extreme case 6 months). So refreshing 100 rack diagrams could be 8-15 minutes, making daily updating a possibility. A complex network diagram may take a minutes, so if 100 network diagrams takes an hour or two, that is still better than diagrams that are over 6 years old. The same approach also creates spreadsheets, reports and other outputs to meet project, work orders, operational reporting and other needs.

Step 3 - Automation For Infrastructure Information Needs

In the diagram above we showed the windows task scheduler on every PC and server, as it can be used to automate creation and updating of Excel and Visio outputs. For instance, every Friday we update all network diagrams for every location. Every day we update rack diagrams for each building into a building specific file share, accessible only by those in the AD group. A project manager updates the new data centre planning pack and BOM (bill of materials) after a review meeting, or when he reaches a design milestone. Automation benefits can be applied to both project and operations - so by designing and recording changes in an operational system it saves effort and also helps maintain consistency of naming and formats.

Unfortunately, unattended updating of infrastructure information is never going to be completely possible due to practical issues, toolset limitations, restrictions on information sharing, etc. There will always be a need for ownership and process, especially as IT systems evolve and teams restructure. In the Visio corner at the end of the newsletter, you'll see that automated updating of Visio diagrams is not so simple when changes occur.

I hope you have found this newsletter useful in understanding that a combination of methods and the AssetGen toolsets can make a real difference in the human resources needed to keep IT documentation up to date. Repeatedly discovering technical data from the same devices doesn't help understand context, dependencies, risks, connectivity, etc. Its why asset databases and CMDBs by themselves aren't trusted or even useful, so project teams create their own spreadsheets and diagrams. But, if the asset or CMDB data was also put into AssetGen then it provides more value instantly. Talk with us and we'll show you how easy it is to automate the refreshing of documentation.

Dave's Pet Peeves

When we explain AssetGen to delegates at tradeshows and conferences there seems to be a common question I am always asked. "What if an engineer just goes rogue and installs something without updating the system?" expecting us to say "Yep, that's what happens and what can you do about it!". However, often my response is "so they ignore instructions and compromise critical infrastructure management processes? If you can't trust them in your data centre, why do you let them in? What's the point of having secure access rules and approving work requests."

In reality it is a bit deeper than I've explained and often the individual is not to blame, but the system they have to work in. This pet peeve is about persisting with bad practice and situations which lead to frustrations and inconsistent work quality.

Years ago I gave up the exciting world of network troubleshooting and designing complex IT networks to focus on IT documentation methods. Why? because it frustrated me then (and still now) that infrastructure design and operational use are compromised by lack of good documentation and day 2 management. Square Mile Systems is now 20 years old, with the root causes still much the same with project focused, busy people coping with imperfect situations. Maturity in our industry takes a long time to become common or best practice. Those of you who follow us know we are working within the US based TIA trade association to develop a life cycle management framework called TIA TR-60 and its getting close to releasing the first standard on guess what - infrastucture documentation!

So my hope from this pet peeve is that more people speak out when they have to do a site survey because others are not doing their job, or there has been no management policy or QA enforcement on project handover. Be difficult, be careful of how you spend your time, make sure your handover information will help the next guy or project. Educate all that your time is precious, so why waste it re-documenting the same infrastructure with every project.

A lesson I have learnt to my personal benefit is that once you lose confidence and trust in IT documentation and controls, it can be very expensive to regain it as specialist skills and resource are rare. Hence why Square Mile today develops software, training and support services. All it takes is to be a bit more professional and think longer term as a team player. Challenge situations where you have to do unnecessary documentation!

Visio Corner - Automated Refreshing Of Visio Diagrams

A great Visio capability is the ability to work with data from spreadsheets and databases, but the native Visio features don't support the functionality we need for multiple sets of diagrams, showing devices in different perspectives. Visio can refresh shape data from supporting data sources, but while useful it is a manual process and has its limitations. As devices will appear in many diagrams - racks, floor plans, network, system, application, data flow, etc. it makes sense to automate updating if you have 10s, 100s or 1000s of diagrams.

The ability to refresh embedded data is really useful and saves a lot of time, but we need more capability when dealing with updates to minimise manual effort. Here are a few things that prevent Visio giving the level of automation that we need.

• Only existing shapes on a diagram are refreshed - no additions or deletions are handled
• Connections between devices have to be added and changed manually
• The shape used for any device has to be manually selected
• No error logs or confirmation of changes
• Overwriting of good diagrams due to data problems
• Manual archiving of updates and exports into different formats

The automation features we've incorporated into the AssetGen system have successfully addressed these issues. Suppose you have 20 racks each averaging around 25 devices in them, so you need to track the lifecycle changes of 500 equipment items with names, serial numbers, configurations, ports, locations etc. Now add on top of that the data for all the connectivity, the cabling involved and the power configurations. It is likely that some data sets will be consolidated, so you might need to start by unpicking or filtering out the important data that is rack focused. Visio diagrams will need this so a rack diagram can show the location of devices in a specific rack, with embedded data like equipment names, serial numbers and IP addresses.

However, a network diagram will show connectivity much better and still have the same device data. Think of the challenges of maintaining accurate diagrams if there are 100 racks, spread across 30 buildings, with multiple connectivity types and with changes made by different teams! The number of spreadsheets and work involved is considerable, which is why documentation is often out of date and not maintained. Basically, too much engineering time is needed (and often better Visio skills).

The AssetGen Approach To Diagram Automation

AssetGen maintains one set of infrastructure data in a consistent format - making it ideal to refresh the multiple diagram types with changes. When do you want to refresh? - immediately, at the end of each week, after a major design change, just before a contract handover, or regular feeding into other toolsets such as portals or content management systems. The picture below shows the basic process. Data changes can be a mix of manual updates, bulk feeds in Excel formats or API updates from databases.

Note that AssetGen refreshes the diagrams for you, so they are not linked back to the database like the Visio / excel linking. Layout, resizing, comments, notes, etc. are not changed so you do not have to layout every time there is a refresh. In most cases we don't delete shapes or connections in case of data errors, so we highlight them in red to prevent mistakes.

The AssetGen system filters out the data needed for a diagram, checks the existing diagrams for content, makes changes and then moves onto the next diagram in a batch process. The only manual work is to check where layout changes are highlighted in a log file (typically with additions or deletions).

What is AssetGen doing as part of automated refreshing?

Here is a sample list of what is needed for automated refreshing of diagrams. Its much better done in software for consistency and scalability compared to manual effort.

• Archiving original diagrams according to type and location, adding a latest refresh date.
• Shapes checking, adding new shapes, assigning symbols, marking shapes for deletion.
• Connector checking, adding new connections, marking connectors for deletion.
• Updating embedded data for all devices and connections.
• Update embedded device hyperlinks (back into Connect / SysMap applications).
• Detailed logging of each update progress, highlighting warnings and errors to admins.
• Publishing updated Visio diagrams in Visio, pdf and web versions for portals and file shares.

Alternatively you could try and do all this manually for your enterprise, project, customer, boss, colleagues and auditors!

Now you can make an informed decision - tedious checking or automation?

Check out this video on network diagramming creation and updating to see how the automation features in the AssetGen system make a difficult job easy for any size of infrastructure.

Website: Network Diagram Automation Video

Upcoming Events - 2024

Sep 19th?? – Visio Tips and Tricks - Part 5 Building, Campus and Site Infrastructure Details & Registration 13:00 UK 08:00 US EDT

I hope you enjoyed this edition of this newsletter, make sure you subscribe by clicking the button at the top of the page to keep updated on future articles and events.