Systems Engineering in Building Complex AI Systems
An extended abstract of the invited presentation at the workshop Leveraging Systems Engineering to Realize Synergistic AI/ML Capabilities, part of 2021 AAAI Spring Symposium Series, Palo Alto, CA, https://aaai.org/Symposia/Spring/sss21.php:
Subrata Das, Zainab Ali, Sai-Nishant Bandit, Ankush Bhagat, Nithya Chandrakumar, Pritesh Kucheria, Marielle Pariente, Anand Singh, Blake Tipping
AdventHealth Data Science
The process of building a complex AI system must address two issues to guarantee its correct functioning: 1) the software engineering errors that are inadvertently introduced during building the system; and 2) the inherent uncertainty in finding a solution that generates actionable insights due to the application of symbolic, machine and deep learning-based AI techniques in building complex and interdependent modules. The first issue is addressed with the help of both a declarative approach to programming and the sound modularization of many complex algorithms, all in the form of libraries. However, the second issue calls for an incremental agile systems engineering approach without fully committing to a development path that is uncertain.
A traditional software engineering project starts with a full specification of system design before the actual implementation, and then the progress is subsequently tracked. However, both the specification and the expected outcome in a computational AI system are often vague. This is due the accuracy being driven by noisy and incomplete data as well as the performance of the available algorithms to run against the data. With this being the case, an AI project would have to be agile in nature with an evolving expectation. Our focus in this short note is this aspect of systems engineering.
At AdventHealth we pursue a scrum-based agile approach to building data products. Most of our NLP related products so far involve machine and deep learning-based document classification techniques, including SVC, sequential LSTM, and attention-based BERT models. A potential project kicks off with a high-level specification from the product owner. The development team is cross-functional with analysts, developers, and data scientists. A big part of most of our projects is to prepare the data, labelled by our analysts. We emphasize on supervised learning to achieve higher accuracy but at the high cost of labeling upfront. Both active and semi-supervised learning are augmented to enhance agility in our process. We monitor and adjust the course of a project plan on a daily basis, incorporating feedback of various subjective and objective evaluations of actionable insights.
The feedback from stakeholders via our product owner is a vital part of the process. At the end of every sprint, we incorporate any such feedback, resulting either increasing or lowering the expectations for the data science product, keeping the rates of false positives and missed detections in mind. We have realized that the earlier in the lifecycle a design decision is corrected, the less costly the project becomes in terms of total effort. We often abandon using deeper complex models and settle for slightly lower performing traditional ML models to save the time for further development and evaluation.
An agile systems engineering approach fundamentally changes the mindset of a data scientist working in an industrial environment. The approach prevents the scientist from getting bogged down in achieving perfection, like a researcher in academia would do. In many cases within an industrial environment, an acceptable solution would suffice. A full-scale industrial data science project (take for example a consumer recommender system), involves developing a system of interconnected systems, including customer and product databases, web-based front-end, back-end processing, ML-based predictive recommender, and reporting. Targeting a high accuracy of the recommending algorithm may or may not be achievable and may align with the pace of the rest of the components where the outcome is deterministic. Another example is in the case of topic classification, where we often adopted simple NBC-based bag-of-words approach, favoring acceptable performance over more complex semantic and attention-based approaches.
In conclusion, industrial data science projects will continue to thrive on agile systems engineering approaches. In predictive analytics, if accuracy is the most important measuring stick then it needs to be broken down into successive milestones to avoid allowing substantial time to have passed by with no working model in sight.
Corresponding Author: [email protected]