Machine Learning in Research vs Production
Machine Learning in Research vs Production
Research often offers a broader scope of freedom, where individuals or teams work with a singular objective—to advance knowledge or develop a state-of-the-art system focused on a specific, well-defined goal. The constraints in research are typically limited to this one objective, allowing for creative exploration and experimentation, with less immediate concern for real-world implementation or operational challenges.
However, when moving towards product development, the complexity increases significantly. The objectives expand to include multiple considerations such as computational efficiency, cost, end-user experience, data availability, fairness, interpretability, scalability, and deliverability. Each of these factors imposes additional constraints, and balancing them becomes essential for creating a viable, market-ready product. Unlike research, where innovation is the primary focus, product development requires the integration of various practical concerns to meet real-world demands and ensure the system functions reliably at scale.
Objective:
Research:
The primary objective in research is typically narrowly focused and highly specialized, often centering on advancing knowledge or improving a specific aspect of technology. For example, the "Attention is All You Need" paper focused on replacing the sequential nature of Recurrent Neural Networks (RNNs) with a parallelizable architecture, leading to faster training and inference. The objective here was clearly defined—improving performance with parallel computation, particularly in contrast to RNNs.
However, in research, this focused objective doesn't cover all aspects of a system, such as interpretability or scaling. These issues may be explored later in basic research (e.g., studies on model interpretability) or applied research (e.g., research on scaling laws or upgrades like multi-query attention to reduce memory usage and latency). These improvements often build on the original work but with new objectives.
Productional Machine Learning (ML):
In contrast, productionizing an ML system involves multiple, sometimes conflicting, objectives. Different teams and collaborators bring in diverse priorities, and these must be balanced to build a successful product. For instance:
Sales Team:
Objective: Increase revenue through improved model predictions.
The focus is on maximising the business value that the model generates, which might push for models that predict high-revenue actions or outcomes.
Customer Relationship Management (CRM) Team:
Objective: Enhance customer satisfaction through more accurate or personalised predictions.
This objective may require balancing revenue-focused predictions with ones that keep customers satisfied and engaged, requiring a trade-off between profit and user experience.
Product Team:
Objective: Ensure the model is fast and minimises latency.
领英推荐
As user attention spans decrease, the product team may prioritise reducing model inference times to keep the system responsive, ensuring better user retention.
ML Platform Team:
Objective: Build a scalable model that can handle high traffic.
The platform team needs a model that can operate reliably at scale, handling surges in demand without crashing or slowing down, which requires engineering for efficiency and robustness.
Data:
In Research:
Researchers often work with publicly available datasets or smaller, curated datasets specific to their research question. These datasets are typically static and do not change over time, meaning researchers don't have to worry about issues like data drift (changes in the data distribution over time) or content drift (shifts in the meaning of data). The focus is on conducting controlled experiments to evaluate algorithms or models based on specific metrics like accuracy, F1-score, etc.
The data is usually well-defined and pre-processed for the sake of simplicity, with a generic view that may not be tailored to any specific real-world task. This allows researchers to isolate and test the effectiveness of their models in controlled, static environments without the complexity of ever-changing data. Since the data is less volatile, there’s no need to worry about ongoing maintenance of data pipelines or continuous monitoring for shifts in the underlying data.
Key Focus: Experiment design, evaluation in controlled settings, and less emphasis on real-time data issues.
In Productional Machine Learning:
In production, ML systems operate in a completely different environment, dealing with large volumes of volatile, real-world data from diverse sources. The data is often messy, unstructured, and constantly evolving. This introduces challenges like data drift and concept drift, where the underlying data distributions or relationships change over time, potentially degrading model performance.
Production ML requires robust data pipelines to handle the entire lifecycle of data, including ingestion, cleaning, transformation, and feature engineering. These pipelines must ensure that the system can continuously process and transform raw data into usable input for the model. Ensuring data quality is paramount, as bad data can easily lead to inaccurate or biased predictions. Production systems must also address data biases and build mechanisms to detect and correct shifts in data distribution over time.
Additionally, every sample in production matters—unlike in research, where a model can be evaluated on a fixed test set. In production, models interact with real users, so errors can have direct consequences on user experience, business outcomes, or system performance. Maintaining and monitoring the system over time is critical to ensure that the model adapts to new data without degrading in performance.