Why didn't I make my own FeatureStore?

Let me describe in the six steps, why in my case making my own FeatureStore (as part of MLOps) didn't make sense.

BTW: I expect that you know what FeatureStore means (briefly, it is the data management layer for machine learning that allows to share & discover features and create more effective machine learning pipelines, etc.).

The whole story began with a collection of requirements ...

Phase 1 – Collecting the requirements

At the beginning, it was necessary to solve the existing problems of the data science team. These were problems such as constant parsing of training data, missing data history for models/predictors, performance-unsatisfactory real-time solution and then also new business requirements, etc.

When designing the solution, we focused on the design, which we only later identified as very close to the FeatureStore concept.

Phase 2 - Designing my own solution

I designed the architecture of an internal solution for online part (I considered creating an API layer and using storage such as Cassandra, MongoDB, Redis, etc.), off-line part (focus on data lake, HDFS, etc.), including meta-repository. From a minimalist point of view, this solution would take 7 months (including analysis, development, integration and testing) in a team of about 7-10 people, including the adaptation of new technologies, the expected effort would be about 1400 MD.

Subsequently, we would have to have a team, of at least 4 people, who would take care of the solution mentioned above (rollout to x countries, take care of infrastructure, operation across multiple time zones). Annual costs in the amount of another approximately 1000 MD.

The total costs for the first year would be ~2000 MD: initial delivery ~1400 MD and maintenance for the first 7 months ~600 MD (with a small overlap of the delivery phase).

It may sound like a great investment, but we only focused on the basic functions such as FeatureStore for on-prem, DevOps-level automation, SSO/IAM, operations, monitoring plus integration into the company's existing environment, including perf. test and SLA, etc. It is without a rich UI, without major topics like MLOps/ML pipelines and their scale up/out, without continues training, continues monitoring, without the support of cloud operators, etc.

Phase 3 - Price of my own solution

In case that business would like to open other requirements (increase maturity), the development team (increase 2x) with the impact to management and maintenance team would have to be increased. The annual effort would be about 6000 MD, and the development would not take one year, but 2+ years.

Moreover, since the development would be purely for internal needs, the solution would not be as mature as it could be if the development were through more companies with a focus on other industries. We would gradually gain experience during development and catch up with companies that have been dealing with MLOps with FeatureStore for 3 years or more.

Developing solutions from the scratch would not pay off for our internal needs, although I'm sure it would be an interesting technical act.

Phase 4 - Finding an existing solution

After leaving the idea of our own solution, we increased the priority of the topics. We started looking for existing MLOps solutions with FeatureStore (with focus on open source or commercial solutions).

We have defined a lot of new requirements, such as MLOps, better scaling, better python parallelization, focused on API quality, future development direction to the cloud also and compliance with our business and data visions, including the possibility of operation by an external supplier over multiple time zones and in modes better than 8x5.

We did a market scan where we focused on about 20 solutions from which we selected five solutions which best fit to our needs.

Then we launched RFP officially and began to focus on more details (limits, blockers, enterprise security, solution complexity in terms of reporting, maintenance, technology stack, roadmap, vision, etc.).

We reduced these five solutions, we chose three, one open source without SaaS as a backup (in case we didn't meet the price) and with the other two we launched PoC.

During the PoC, we received training, including explanations of terms, concepts, APIs. We tested the solution ourselves, we defined 73 use cases for testing and the external supplier helped us primarily with installing the solution (to our infrastructure), consultancy and development support.

Within Use cases, we checked the functionality of the solution (it was the main part), the quality of documentation, the quality of support from the vendor, the stability of the solution, we made performance tests, ability of integration, work with larger data sets, including the ability of the solution to recover from a crash (further details, what we also evaluated, see here or whole post about it ). PoC took us nearly 5 months with two parallel suppliers.

Phase 5 - Evaluation

We evaluated both candidates and chose the winner. I must state that both solutions have had more than 3 years of development. Both solutions offered on-prem/cloud and SaaS (i.e. support solutions as a delivered service) and references covered a large number of areas (financial service, anti-fraud, technical industries, etc.).

I honestly have to say that if we developed the solution by ourselves, we would not be able to meet our requirements in less than 2 years (happy path: if we could build a capable team). In addition, our solution would certainly not be in such a maturity and tested by so many customers and industries.

And most importantly, the final annual costs (for high matured solution with full support as SaaS) in one country (include the production and non-production environments) did not reach even a quarter of the costs of the maintenance for our own developed solution. In addition, we saved 2 years of our own development (i.e. time and money for a medium-sized team).

I am therefore very glad that I did not go my own way, suppressed my ego and that we can now focus on using the chosen solution. I consider the practical experience from PoC from both vendors to be really beneficial.

Phase 6 – Conclusion

If you are building a solution just for yourself and not as a product to sell – or do not want to bring a major innovation, consider whether it is worth to develop your own solution (principle “buy before build”).

It's also less expensive to buy a new car than to build a car by yourself.

Thank for reading the article and have a nice day.

#ai #ml #mlops #featurestore #iguazio #hopsworks #tecton #rasgo #feast #kaskada #molecula #h2o #databricks #abacusai #sagemaker #vertexai #zipline #featureform #featurebase #continual #metarank #feathr #accenture #ibm #deloitte #mckinsey #bcg #pwc #gartner #capgemini #kpmg #ey