AI with Elastic Machine Learning for Cheap

AI is a buzzword nowaday and everyone is jumping on the AI bandwagon, from startups building AI product for IoT, over vendors adding AI features to their existing products and services, to enterprises eager to employ AI to transform their IT infrastructure and operations to AIOps. It’s easier said than done for those companies who don’t have root in AI, given the hot job market where companies are fighting for the best talent in software engineering, especially good data scientists and data engineers are very hard to find because most of them got snapped up by deep pocket companies like the FAANG already. So what can we do to stay fashionable? Use third-party machine learning framework and API. There are plenty of it out there, but I think Elastic Machine Learning is a good choice if you want to quickly make your infrastructure and services “smart” and you only have “average” software engineers (please don’t feel offended my fellow software developers, I know you all are very smart, but there is always someone who is better than you), and I am sure that you have plenty of them. However, there is a dilemma. You need to buy X-Pack Platinum license which is 6k a year per node. Imagine that you have a 100 node Elasticsearch cluster, that will set you back 6 millions in license fee alone per year. You can negotiate for a discount but the annual license cost is still in millions. So here is a way to help you pay much less but still fully use Elastic ML in your cluster with 100 or more nodes:

• Use free open-source version of Elasticsearch for you data cluster(s)
• Set up a cluster with X-Pack Platinum with only 3 beefy nodes each has lots of CPU and memory. This cluster doesn't need a lot of disk storage because it stores only aggregated metrics with related meta data.
• Write your own aggregators to aggregate your relevant metrics into ES Machine Learning cluster, you can use Elasticsearch watchers (part of X-Pack Platinum) to aggregate your data if the aggregation logic is not very complex and resource-intensive.
• Elasticsearch X-Pack supports nice and easy integration with many tools (PagerDuty, Jira, email, chats...), it's handy for you to manage your alerts.

Why pay the license fee for the nodes (dedicated master nodes, data nodes, ingest node, tribe node) that don't need machine learning features? With a powerful 3-node machine learning cluster, you can serve many license-free (free Apache 2.0 license I mean) Elasticsearch clusters as you want to make your operations smarter.

If you like my idea and use it, please donate a small token here for my daily Starbucks.

If you don’t like this idea, please also donate here so I can hire a therapist to fix my stupidity.

If you like it and not sure how to implement it, please contact me and I am more than happy to help.

Elastic might not like this article at first, but if you see your sales goes up after this article then I'd love to accept your donation as well ;-D. 100 deals with 3 node license are far better than 1 deal with 100 node license though.