Get Ready for the Software 2.0 Era

We are just entering the era of convergence - convergence of data, software, and AI/ML models. Andrej Karpathy introduced the term Software 2.0 to refer to machine learning models that help us solve a number of interesting problems of classification and recognition without any human writing a single line of code in the traditional sense. In contrast, he refers to traditional programming as Software 1.0. While Software 1.0 was primarily about data and software; Software 2.0 is about data and AI/ML models. For the foreseeable future both 1.0 and 2.0 need to co-exist. We are building software that have AI/ML models embedded within them.

In a series of articles published in Toward Data Science I explore a number of themes around the era of Software 2.0. I provide a synopsis for each of these articles below.

Software vs Models

Software Engineers and Data Scientists come from two different worlds — one from Venus and the other from Mars. They have different backgrounds mindsets, and deal with different sets of issues. They have a number of things in common too. Software and models differ by five key dimensions.

1. Output: The output from traditional software is certain, while model output is typically uncertain;
2. Decision space: The decision space for traditional software is static, while for models it is dynamic. The data used to train the model might be constantly changing requiring frequent retraining of the model.
3. Inference: The inference process for traditional software is deduction implemented through code; while for models the inference is inductive based on generalization of data.
4. Development process: The development process for traditional software is linear and iterative; while for models the process is based on experimentation.
5. Mindset: Software developers typically have an engineering mindset; while model developers or data scientists typically have a scientific mindset.

These fundamental differences between software and models cannot be ignored. I have often seen these two being conflated as one, by business executives and sadly even by IT professionals. We ignore the differences at our own peril.

Twelve traps to avoid

The challenges of building models is exasperated when we conflate models and software. The standard methods of scoping, designing, building, testing, deploying, and maintaining traditional software and ML models are not the same. In particular, there are three key areas of challenges that can lead to the twelve traps.

• Data Traps: When it comes to building models the data has to be of good quality (i.e., garbage in, garbage out), available in sufficient quantity, and for supervised learning models also labeled (i.e., a label is a response variable that is being predicted by the model). The data also needs to be fit for purpose - not be a skewed sample or biased. We call these data traps, the data quality trap, data volume trap, data labeling trap, and data bias trap. A company can suffer from one or more of these traps. Getting a realistic sense of the data trap is critical to help ensure you don’t go down the wrong path and spend millions on your modeling effort and not realizing the expected returns.
• Scoping Traps: Assuming that we want our models to satisfy certain performance criteria (e.g., accuracy, precision, recall, etc), it is hard to estimate the effort and duration it will take to achieve the results. Worse, we may not be able to tell a priori if we can, in fact, succeed in satisfying the performance criteria. We call this difficulty to scope the model to achieve a certain performance before the model is built as the pre-build scoping trap. Similar scoping traps can occur at different stages of the model giving rise to training scoping, deployment scoping, and drift scoping traps.
• Return Traps: The benefits of AI/ML models in companies typically fall under two broad categories — efficiency and effectiveness. The benefits accrue from being faster and better. The question we need to ask is — faster and better relative to what baseline? It is in estimating this baseline that companies often fall short. We call the difficulty of estimating the efficiency and effectiveness of models as return efficiency estimation and return efficacy estimation traps. In addition to the estimation traps we also have the equivalent realization traps resulting in return efficiency realization and return efficacy realization traps.

Model Evolution

As we have tracked and helped enterprises evolve from building simple analytical models to more sophisticated, continuously learning models embedded within larger transactional applications we have seen three distinct phases.

• Standalone Model Phase: In this phase, companies have typically deployed models on a standalone basis using command line interfaces and Jupyter notebooks.
• Prediction as a Service Phase: Once companies were able to demonstrate the value of these models there was a demand to make the predictions from these models available to other groups within the enterprise. 
• Model Factory Phase: More advanced companies are moving to a factory model where hundreds (if not thousands) of models are deployed. Automated CD/CI (Continuous Deployment/Continuous Integration) pipelines are being combined with CL (Continuous Learning) pipelines.

The increasing scale and sophistication across the combination of software, models, and data has seen the emergence of new roles. Software development has had roles such as business analyst, systems analyst, architect, developer, tester, development-operations (DevOps), etc. When you examine more closely, these roles reflect the scoping, design, development, operations, and maintenance phases of the software lifecyle. With the emergence of Machine Learning models and the paradigm of Software 2.0 we see a number of new skills and roles, including Data Scientists, Data Engineers, ML Engineers, and MLOps. These roles also reflect the coming together of multiple disciplines — software engineering, software operations, statistics and machine learning, and data management.

Model Lifecycle: From ideas to value

While there are many similarities between lifecycle of an AI/ML model and a typical software lifecycle, there are significant differences as well, stemming from the differences between software and models. We breakdown the entire lifecycle of models into nine steps grouped into four major phases — scoping, discovery, delivery, and stewardship. The model lifecycle is iterative with three specific feedback loops.

• The first feedback loop is in the value discovery phase. The three steps in the process (i.e., data extraction, pre-processing, and model building) are iterative and could also result in the re-examination of the business objectives from the value scoping phase.
• The second feedback process is in the value delivery phase. This phase could result in going back to the value discovery phase or could even trigger a value scoping phase.
• The final feedback comes through the value stewardship process. This can result in going back to the value delivery, value discovery or value scoping process. The rich interplay between all these steps in the four phases results in a very complex management process for models, data, and software.

Agile Software 2.0

While Andrej Karpathy uses the word Software 2.0 only for systems developed using machine learning and deep learning, it is clear that we would need both 1.0 and 2.0 versions of software development to co-exist for the foreseeable future. As a result, we really should be combining the key values of agile software development with agile model development. We see the objective of this combined manifesto, what I call Agile Software 2.0 Manifesto call as follows:

We are uncovering better ways of developing software powered by (AI/ML) models by facilitating continuous integration (of data, software, and models), delivery, and (machine) learning.

Extending the agile software development manifesto into the Agile Software 2.0 Manifesto, we have these four basic values:

?        Multi-disciplinary teams AND individuals & interactions;

?        Insightful actions/decisions AND working software;

?        Data & model exploration AND customer collaboration;

?        Being innovative & disruptive AND responding to change.

Given the fundamental differences between software and models, and the reasons mentioned above in the Agile Software 2.0 Manifesto we should separate the traditional agile software sprints from the model lifecycle in terms of the timing. At the same time, we do not want to run them as two separate and distinct agile cycles either as that could prevent us from realizing the full benefits of traditional software and models. Interleaving the sprint cycles and having a separate clock for software sprints and model sprints is one solution that has been successful in practice. The software, data, and modeling teams should come together at specific interaction points and work together in a couple of phases to exploit the full advantage of AI/ML enabled software deployment.

The convergence of data, software, and AI/ML models is just beginning. Extensive software engineering, software development and maintenance methodologies don’t exist for AI/ML models. Best practices are just emerging and the next decade will see more academic and industry advances in this area. We started this series with a claim that data scientists are from Mars and software developers are from Venus. These disciplines are coming together over the past couple of years. This five part series is just the beginning of what is likely to be a major area of focus for software developers and AI/ML modelers.