CMDB Integrations and IntegrationHub ETL

What is actually CMDB and why do we need it?

IT Organizations are tasked with overseeing a myriad of devices, applications, and services to ensure not only effective operation but also compliance with regulatory requirements and alignment with overarching business objectives. This is exactly the reason why Configuration Management Database (CMDB), a robust tool designed to empower companies in the management of their IT assets, infrastructure, and services, has become increasingly important - especially in the context of ServiceNow software.

The ServiceNow CMDB is essentially a database housing detailed information about the applications and devices that provide services within an organization. Each service is associated with a unique Configuration Item (CI) in the CMDB, allowing for a granular and organized representation of an enterprise's infrastructure. It provides the necessary data to support service management practices, offering insights into Event and Incident Management. When issues arise, the CMDB can pinpoint the root cause by providing information about the relevant CIs and shed light on services that might be affected.

How to populate the CMDB data?

Several methods can be employed to populate a CMDB, each catering to specific needs and environments:

1. Data can be entered manually, far from ideal. It sometimes might be necessary for attributes that cannot be discovered or imported using automated methods.
2. Use discovery tools designed to automatically identify and populate the CMDB by scanning your network and identifing present devices. (few examples: ServiceNow Discovery, BMC Discovery, Qualys, Solarwinds, Nagios). Service Graph Connectors offer prebuilt integrations to seamlessly connect and import data from such systems.
3. Load data into your CMDB from local xlsx, csv files
4. Build custom integrations with external systems holding important configuration management related information that are necessary to efficiently run the operations. This would usually mean using technologies like for example REST, SOAP, JDBC etc.

What is IntegrationHub ETL and why you should use it for your CMDB integrations?

IntegrationHub ETL is a powerful tool offering robust functionality for creating and managing Extract, Transform, Load (ETL) transform maps. Primarily designed for integrating third-party data into the Configuration Management Database (CMDB), IntegrationHub ETL ensures a seamless integration process without compromising data integrity. The tool boasts a simplified user interface that guides users through the entire integration process, from setup to execution, including a test run with sample data for validation.

What you need to know:

• All other out of the box Service Graph Connectors like Intune Service Graph Connector, SCCM Graph Connector, Azure Service Graph Connector, Qualys Service Graph Connector are based on the IntegrationHub ETL and therefore using it for your custom integrations will make you comply with the standards
• IntegrationHub ETL is 100% compliant with the CMDB IRE (Identification and Reconcilliation) Engine that will help you to ensure you don't get duplicates and support you in case of having a multi-source CMDB (almost always the case)
• Starting with the Vancouver Release IntegrationHub ETL as a visual tool for building transform maps will also become usable for non-CMDB integrations.

Building own Transform Maps with IntegrationHub ETL

1. Specify Basic Details: IntegrationHub ETL empowers users to kickstart the integration process by providing essential information for the ETL Transform Map. This includes specifying basic details such as the name and description of the transformation, defining the data source, and specifing the CMDB Application name.

2. Prepare Source Data for Mapping: Before mapping third-party source data to CMDB classes and attributes, IntegrationHub ETL enables users to preview the source data. This step allows for a comprehensive understanding of the data's structure, quality, and relevance. Users can then prepare the data for mapping by transforming some of the values before mapping them. This proactive approach ensures that the integration process is built on a solid foundation, minimizing errors and enhancing the accuracy of the mapped data. There are plenty of predefined transform actions, but it is also possible to use scripts easily to provide you 100% flexibility.

3. Map Data to CMDB and Add Relationships: IntegrationHub ETL offers a user-friendly interface for mapping third-party source data to target classes within the Configuration Management Database (CMDB). Users can effortlessly choose the target classes and attributes, facilitating a seamless translation of data from external sources to the standardized format within the CMDB. Additionally, the tool allows users to establish relationships among these classes, enhancing the depth and richness of the integrated data.

4. Preview Sample Integration Results and Schedule Import: IntegrationHub ETL provides a practical approach to integration by allowing users to run the ETL Transform Map and preview sample integration results. This preview enables users to assess the outcome before committing to a full import, ensuring the accuracy of the mapped data. If adjustments are needed, users can refine the mapping or configuration as necessary. Once satisfied with the preview, users can schedule the import to run at specified intervals, automating the integration process. Importantly, IntegrationHub ETL incorporates a rollback mechanism, providing a safety net in case any issues arise during or after the integration.

Few Best Practices for IntegrationHub ETL and CMDB Integrations

1. Distinct Discovery Sources - Best Practice: Always set the discovery source to a new source to facilitate clear differentiation between various sources. This ensures that you can easily identify and manage data from different origins within the CMDB. A distinct discovery source simplifies troubleshooting and enhances traceability in case of integration-related issues and multi-source CMDB
2. Leverage IntegrationHub ETL Automation - Best Practice: IntegrationHub ETL streamlines the integration process by automatically managing critical aspects, including setting the configuration item most recent discovery date and updating some other relevant system fields. Embrace this automation to reduce manual efforts, minimize errors, and maintain accurate and up-to-date data within the CMDB.
3. Consider Multiple Sources and Reconciliation Engine - Best Practice: Anticipate scenarios where different sources may send data related to same configuration item. When configuring CMDB integrations, think about the reconciliation engine and establish precedence rules to handle potential conflicts. This proactive approach ensures that the CMDB maintains data accuracy and consistency, even in situations with multiple data sources.
4. Review Containment and Hosting Rules in Case of Import Errors - Best Practice: In the event of import errors, specifically investigate the CMDB class-related containment and hosting rules. Ensure that these rules align with the structure of the incoming data. Addressing containment and hosting issues promptly can resolve errors and maintain the integrity of relationships between configuration items.
5. Utilize Source Recency Timestamp - Best Practice: Leverage the source recency timestamp as a valuable tool to optimize data updates in ServiceNow. By using this timestamp, you can selectively update records in the CMDB that have been modified in the source system since the last data load. This practice minimizes unnecessary updates, enhances efficiency, and ensures that only relevant changes are reflected in the CMDB.
6. Populate the Correlation ID - Best Practice: Ensure that you always map the external unique ID as the configuration item Correlation ID. This might help you in case you might need to quickly find the information about a particular configuration item in the source system
7. Ensuring Data Completeness- Best Practice: Prioritize data completeness to maximize the value and effectiveness of your CMDB. For instance, when dealing with hardware configuration items, strive to include all relevant information. Hardware entries should encompass comprehensive details such as CPU specifications, operating system details, installed software, network adapter information, serial numbers, and other pertinent attributes. By consistently maintaining comprehensive data, you create a more accurate and robust representation of your IT environment within the CMDB.
8. Regularly Review and Update Data- Best Practice: Establish a routine for regular reviews and updates of data within the CMDB to ensure its ongoing accuracy and alignment with the current state of your IT assets. Technology landscapes evolve, and keeping the CMDB current is essential for maintaining its effectiveness as a reliable source of truth. Regular reviews allow you to identify and address any discrepancies, outdated information, or changes in the IT environment promptly. This proactive approach enhances the overall quality of the CMDB, supporting precise decision-making, streamlined operations, and effective IT asset management. Regular data maintenance is not just a one-time effort but an ongoing commitment to keeping the CMDB a valuable and up-to-date resource for your organization.

In summary, these practices collectively contribute to the effectiveness of CMDB, promoting data accuracy, efficiency, and informed decision-making in the realm of configuration management and service operations. IntegrationHub ETL emerges as a powerful tool, facilitating the seamless integration of third-party data into the CMDB and non-CMDB tables while ensuring data integrity.

When talking about CMDB we should also mention the CSDM framework that is truly essential. To be continued...