How do you manage big data and AI in industrial competition?

In the world of bioinformatics, effective data management is crucial. I learned to understand the importance of during PhD and postdoc training. I organized my data meticulously, simplifying sorting and filtering. Version control, metadata documentation, and data security were top priorities, while automation ensured data quality. Ethical considerations and continuous learning are key. This dedication led to reliable research, emphasizing that solid data management forms the foundation for groundbreaking insights into complex systems like the immune system or cancer research.

Understanding Data assets is the most crucial step in the AI journey. Many firms jump on the AI hype without setting up effective Data pipelines and understanding data quality. I've learned that the first step is to know how the data is generated and assess the quality and reliability. This allows us to calculate better, more reliable models and metrics. Otherwise, you're left chasing a moving target.

Data management and data documentation in particular is key to know the organization data patrimony. Implementing and documenting data catalogs with qualitative meta data is key to share and leverage data value.

Data gathering and data validation are the steps that usually take most of the time in AI related projects. If these steps are proactively, taken care of, then the AI journey will be much smoother for the entire team.

Having said that understanding of data assets is crucial. However this should be the first point to start any AI/ML workflow. We should always start with understanding the problem statement we are solving for, and writing down the hypothesis or steps or perspective variables we can use to solve our model. The features may or may not be available to us in our data sources but it would initiate a analytical thinking. Now, here comes the assessment of Data sources we have, we can map how many variables we though of were available and what others are there in the data sources we have which could be helpful for the project.

Choosing the right AI strategy, also should align with the customer goals. Breakdown the customer requirements by having workshops with your customers to analyze, what kind of pain points are they currently facing. A lot of times they are not aware of what exactly they want, so you might have to pitch in the right solution for the problems. These tasks need to be streamlined effectively before planning the AI strategy.

In a world of costly and limited resources, AI strategy must include a robust and well documented prioritization procedure to ensure the implementation of high-valued use cases.

Mostly in industry, the AI strategy is dictated by the problem statement. If the problem statement is like "whom we should target" nothing other than classification would be helpful here. If the problem statement is "classify the wiki articles" computer vision cannot help you there.

Select an AI strategy that aligns with your business goals and industry dynamics. Decide whether you'll use AI for process optimization, predictive maintenance, product innovation, or customer experience enhancement. A well-defined strategy guides resource allocation and technology choices.

Infrastructure is crucial step here. For example as if you are working with LLMs which have billions of parameters, there would be a need to use GPU clusters. There could be ways to do the same using CPU ones but that could be very inefficient ways of doing the same.

Mutualized capabilities is key to leverage AI power. Also considering public cloud services help scalability and flexibility while avoiding physical infrastructure management complexity.

Invest in the infrastructure, talent, and tools needed to harness big data and AI effectively. This includes data storage and processing systems, data scientists, machine learning engineers, and AI platforms. Building a data-centric culture within your organization is equally crucial.

The most widely used metrics in Machine Learning are Mean Absolute Error (MAE) and Mean Squared Error (MSE) to evaluate regression models, while the ROC curve (Receiver Operating Characteristic Curve) displays the performance of classification models across classification thresholds and AOC (Area Under the ROC Curve) nicely complements it as it measures the rank and quality of the predictions, irrespective of classification thresholds. It is important to understand how to apply the aforementioned statistical error metrics, however, it is also equally necessary to use Real-World metrics, that are understandable for stakeholders on how the applied AI is improving or would improve the process that it is been applied for.

Implement robust monitoring and analytics to assess the performance of your data and AI systems continually. Track key metrics like accuracy, efficiency, and ROI. Regular audits can uncover areas for improvement and ensure that your systems remain competitive.

Data monitoring and performance monitoring are the main drivers of operational tracking in MLOps. AI performance can deterioate by time, which is possible due to Data Drift - changes in data distributions - or Performance drift - drifts in baseline metrices of the AI performance - or both. Tracking both of these would be helpful for the DS team for retrain and redefining the AI models.

The industrial landscape is dynamic. Be agile and adaptive in response to changes in data sources, regulations, or technology advancements. Flexibility ensures your AI solutions remain relevant and compliant.

Use your data and AI capabilities as a springboard for innovation. Explore new business models, product offerings, and customer experiences that leverage your data assets. Collaboration with partners and ecosystems can spark novel ideas.

Measuring a concrete ROI (Return of Investment) is a KPI that will always help stakeholders to understand how effective the AI solution has been or will be. If we measure how much time is reduced when a process takes place with the help of AI vs how much time is needed before AI intervened, we can have an effective sense of an improved ROI for that process. An example of this is customer support. If we deploy a chatbot that can help resolve the most common, first level support cases, cost are reduced and the human specialist will only need to get involved for the resolution of more complex cases (and AI can help at that level too!)

Data privacy and security: Prioritize robust data protection measures to maintain trust with customers and partners. Ethical AI: Ensure your AI systems adhere to ethical guidelines to prevent unintended consequences or biases. Compliance: Stay abreast of industry regulations and standards related to data and AI to avoid legal pitfalls.

Key success factor is to manage and steer the quality of data. This is the starting point to ensure the quality of the analysis, output and then insights to consider to take decision. Also, data documentation, especially in a big data environment is key to ensure the right level of sharing and data reuse within the entire organization. Data products, mixing end users autonomy, high level of quality and detailed description help leveraging data value.