What are the best techniques for testing RPA bots that use unstructured data?

RPA bots against unstructured data requires a mature business to service, e.g., the business area wanting automation must have a very clear definition of what good looks like and how to achieve this. Where this is not the case, the organisation should launch a cross-functional project with the business area, automation team, and business intelligence or similar function to ensure this occurs. The primary thing to remember is that automation provides a service, and the business is responsible for the outcomes of consuming that service, if the business does not know what outcomes they want - you cannot start.

When testing RPA bots using unstructured data, focus on exposing the bot to diverse, messy examples to validate accuracy in interpreting randomness - unusual date formats, irregular naming patterns, poor scan quality. Specifically assess data extraction parts of the process and confirm the bot handles good data vs exceptions properly in downstream logic. Finally, do volume testing with large data batches to reveal stability/performance issues that surface more often at scale. Getting creative with curveballs in test data is key to thoroughly vetting real-world robustness.

In case of unstructured data, RPA bots have a high probability of failure. In most of the cases, the target of the Bot is to ensure straight-through processing without human intervention. However, in this case, it's imperative to have human-in-loop in the testing process. Human can identify the failure of bot due to unstructured data and pass that as feedback to the development team. This can help refine the development process and come up with improved bots. The error rate, depending on the type of unstructured input, should be defined in advance. An AI based extraction solution integrated with bots can also be considered. However, that may be beyond this discussion.

Testing RPA bots handling unstructured data involves: 1. Data Validation: Verify data extraction and formatting. 2. Mock Data: Create mock unstructured data for testing. 3. OCR Validation: Confirm OCR accuracy. 4. Regression Testing: Test bots across various unstructured data samples. 5. Error Handling: Ensure bots handle exceptions gracefully for robust unstructured data processing.

It’s important to define not just success and failure, but also gray areas. When siphoning up unstructured financial data where 1000+ sources all used their own format, we applied incremental parsing logic to scrape and harmonize the format. Based on the results, we could flag the status of records pulled from a data source - when parsed items returned more entries than were expected, failed altogether or returned a different amount, etc., we applied a status. By triaging the results of this, an admin could monitor changes and know what cases needed verification. This approach turned out very effective. By identifying different levels of success, we were able to pull from countless sources with very minimal human intervention.

Don't feed your bot the same old salad! Design test data sets that mirror the real world's messy variety. Think typos, incomplete information, and outlier formats. Throw in edge cases and outliers to see how your bot handles the heat.

Testing isn't a one-time event. Monitor your bot in production, using logs and analytics to identify potential issues and refine its data handling based on real-world experiences. Think of it as continuous learning for your bot.

Beyond logging the activities performed by the bot, I find if impersonating to capture the stages of data processing on each record. E.g., if you’re pulling data from an email and trying to format the extracted data into numeric records, those individual steps should be preserved. If you’re working with many records, setting up a custom output where each stage of the data processing can be viewed side-by-side for each record is invaluable. This makes it possible to drill down on oddities and quickly review what’s working and isn’t, as opposed to having to dig through logs and stitch things together for each individual entry. With external, unstructured data it’s a given that things will fail sooner or later. Log with that in mind.

RPA bots dealing with unstructured data introduce unique challenges due to the inherent variability of data sources. While the article wonderfully covers testing techniques, I'd like to emphasize the importance of feedback loops. As bots evolve, continuous feedback from end-users and regular updates based on real-world anomalies will ensure their robustness. Also, integrating Machine Learning could assist in pattern recognition and prediction, further enhancing the bot's efficiency with unstructured data. Leveraging both automated and manual insights can make bots more adaptive to dynamic data environments.