Design Goals of AI/ML Model Serving Software

In this post, I would like to share our learning so far in developing a software - Picominer- aiming to serve AI/ML models.?

Let's start with a few essential goals of the platform.

1. Heterogeneous model serving.
2. Separation of model development.
3. Standards based model representation.
4. Pre-production model verification and validation.
5. Continuous feedback on model performance.

Models (knowledge) represent the systems to make "future" decisions. Broadly, we classified the AI/ML models into the following categories.?

1. Rule based (Logic programming)
2. Numerical
3. Causal (Statistical)
4. Algebraic (Polynomial)
5. Acausal (System of differential equations)
6. Neural Network
7. Information based (Data mining)
8. Affinity/Cluster based (Data mining)
9. Association based (Data mining)
10. Probabilistic

Of course, the list above will be refined as we gain knowledge about the applications.?

Goal #1: Heterogeneous model serving platform.

The problem we started to solve was how to deploy models in a cost effective way. It has been a challenging journey in conducting the software R&D and understanding the essential subs-systems. Why should we consider heterogeneous model deployment was a tough question. Not all applications need a full spectrum of models. However, in many cases, the applications need more that one type of models. For example, a home automation system may need Rule based and Acausal models and a precision farming application may need Rule based, Numerical and Acausal models. So, in the design of modern digital run-time infrastructure, it is important to provide an environment to run heterogeneous models.?

Goal #2: Separation of model development.

There are several mature open source tools to develop models. For example, OpenModelica produces Acausal models; REDUCE and Maxima produce Algebraic models; Data Mining tools produce Information based, Cluster based and Association based models. Furthermore, model developers may get quality datasets through various sources and tune to the application and the context.?

Goal #3: Standards based model representation.

It is very important to use a standard based representation of the models.?

For example,

1. PMML (Predictive Model Markup Language)
2. FMU (Functional Mock-up Unit)
3. Gluon (Neural Network)
4. ONNX (Neural Network)
5. FuzzyLite Language (fll)
6. Horn clauses (Rules, Prolog and CLIPS)
7. REDUCE Language (LISP dialectic) (Algebraic, Polynomial based)
8. Maxima Language (LISP dialectic) (Algebraic, Polynomial based)
9. COOL (CLIPS Object Oriented Language)

Goal 4: Pre-production model verification and validation.

It is a very challenging to integrate various model formats. So, Picominer has a process check-point to run the models in an environment that is close to the target. The objective of this step is to gain insights of target runtime and to calibrate the model parameters.

Goal 5: Continuous feedback on model performance.

Every context and model demand special metrics to collect. Picominer software provides a metric collection hooks.?

Here is an important leaning! We first attempted to develop a super software with tightly coupled sub-systems. We realised progressively that flexibility is more important than feature-rich and bulky software. We tend to offer custom development (solution) rather than a fully packaged software product.?

I believe open source is a revolution and even a small companies (startups) are able develop cutting edge software. Furthermore, if we look at the open source libraries, we can find well designed and mature software to realise the above goals.?

I will discuss about some of the open source libraries in my subsequent posts.