The Machine Learner’s Cookbook: Why Companies Fail to Implement Machine Learning Effectively

Imagine you’re a chef determined to serve the most exquisite dishes to your customers. You’ve heard whispers about a new kitchen appliance called “the Machine Learner” that can revolutionize your cooking.

Excited, you purchase the latest model, expecting it to magically whip up culinary masterpieces with minimal effort. However, to your dismay, the Machine Learner only provides you with a complex set of blueprints and instructions on how to build its own components.

Frustrated and confused, you realize you’ve made a crucial mistake: you’ve focused on acquiring the tools of a machine learning researcher instead of the skills of a machine learning chef.

This is the predicament of many companies who fail to implement machine learning effectively. They confuse the oven builders (researchers) with the bread bakers (appliers). They spend valuable time and resources trying to build their own machine learning tools from scratch when there are already pre-built ovens available.

As a result, their kitchens remain empty, their customers remain dissatisfied, and their vision of culinary excellence remains unfulfilled.

The Two Faces of Machine Learning

Research: This field focuses on developing the general-purpose tools for others to use. Researchers primarily build algorithms and methodologies that form the foundation of ML applications.

Examples

• Natural language processing (NLP): Researchers are developing new NLP algorithms that can understand and generate human language more effectively. This has the potential to revolutionize a wide range of industries, such as customer service, marketing, and healthcare. For example, NLP can be used to automatically generate customer service responses, personalize marketing campaigns, and diagnose medical conditions.
• Computer vision: Researchers are developing new computer vision algorithms that can see and understand the world more like humans. This has the potential to improve a wide range of applications, such as self-driving cars, facial recognition, and medical imaging. For example, computer vision can be used to detect objects in the road, identify people, and diagnose diseases.
• Machine learning for healthcare: Researchers are developing new machine learning algorithms that can help doctors diagnose diseases, develop new treatments, and personalize patient care. This has the potential to revolutionize healthcare and improve the lives of millions of people. For example, machine learning can be used to identify patients at risk for developing certain diseases, predict the effectiveness of new treatments, and personalize treatment plans.

Application: This domain involves using existing tools to solve specific business problems. Applied ML professionals leverage research findings to design, implement, and manage ML solutions that address real-world challenges.

Examples

• Customer segmentation: Machine learning can be used to segment customers into groups based on their characteristics and behaviors. This information can be used to develop targeted marketing campaigns and improve customer service.
• Fraud detection: Machine learning can be used to identify fraudulent transactions. This can help businesses reduce their losses and protect their customers.
• Risk assessment: Machine learning can be used to assess the risk of lending to a particular borrower. This can help lenders make more informed decisions and reduce their credit losses.
• Inventory management: Machine learning can be used to forecast demand and optimize inventory levels. This can help businesses reduce their costs and improve customer service.
• Pricing optimization: Machine learning can be used to optimize prices for products and services. This can help businesses increase their revenue and profitability.

Reasons for Failure

Misunderstanding the difference: Many companies mistakenly believe there’s only one type of ML, leading them to hire researchers expecting them to solve practical problems. This is akin to hiring a chef to build an oven instead of cooking.

Focusing on building instead of using: Organizations often fall into the trap of trying to build their own ML tools from scratch, neglecting the vast array of existing resources and platforms readily available.

Building ML tools from scratch is a time-consuming and expensive process. It requires specialized skills and knowledge, as well as access to large amounts of data. For many organizations, it simply isn’t feasible to build their own ML tools. Even if an organization has the resources to build its own ML tools, it is likely to be reinventing the wheel. There are already many excellent ML tools available, both open source and commercial. By using these existing tools, organizations can save time and money, and they can be confident that they are using tools that have been tested and proven to work.

Using existing ML tools can help organizations avoid common pitfalls. For example, many ML tools include features that help to prevent overfitting and ensure that models are interpretable. By using these tools, organizations can reduce the risk of building models that are inaccurate or that cannot be used to make informed decisions.

Incomplete teams: Successful ML implementation requires an interdisciplinary team with expertise in data engineering, statistics, software engineering, and domain knowledge.

Data engineering is responsible for preparing the data for modeling, which includes cleaning, transforming, and enriching the data. Statistics is responsible for developing and evaluating models, and software engineering is responsible for building and deploying the models. Domain knowledge is essential for understanding the business problem and ensuring that the models are relevant and accurate.

Ignoring decision intelligence: This emerging discipline bridges the gap between research and application, providing a framework for using ML effectively to achieve strategic objectives.

However, despite the potential benefits of decision intelligence, many organizations are still struggling to adopt it. One of the main reasons for this is that decision intelligence is a complex discipline that requires a deep understanding of both data science and business operations. Decision intelligence can be expensive to implement, and it can take time for organizations to see a return on their investment.

Simple Solutions for Success

Clearly define your goals: Understand what problem you want to solve and how ML can help you achieve that. Don’t reinvent the wheel, focus on applying existing tools effectively instead of building everything from scratch. Always keep your business goals at the forefront and ensure your ML efforts contribute to tangible outcomes.

Use existing tools: Leverage the vast array of open-source libraries, cloud platforms, and pre-trained models available. Begin with small, manageable projects and learn from your successes and failures.

Hire the right team: Build a diverse team with expertise in research, application, and relevant domain knowledge. Invest in the right talent and build a team with diverse skills and expertise to navigate the complexities of applied ML

Embrace decision intelligence: Utilize decision intelligence frameworks to guide your ML strategy and ensure effective implementation.