How do you choose between simple and complex data analysis methods?

We've always been hearing about "big data” but it can be misleading, as a large amount of low-quality data actually harms your model's performance. What we need is a good amount of high-quality data that shows variations, diversity and well represents the underlying population. While more recent generative AI relies on self-supervision and unlabeled data, the majority of ML/DL models still depend on supervised learning approaches and labeled data. One key area for improvement during data collection is to improve annotation guidelines. This can help reduce issues of inter-annotator variability (when different annotators interpret the same guidelines differently) and intra-annotator variability (variations in the same annotator’s labels).

when deciding between simple and complex analysis methods, think about how intricate your problem is, the quality of your data, and the resources you have. Keep in mind that simpler methods often lead to more understandable results, but more complex methods can provide higher accuracy. It's all about striking the right balance for your specific situation.

My take on this is to create a clean and representative data set, even if small. The quality of the data in most cases trumps the quantity that may come later. Before feeding the data to complex ML algorithms, I would suggest spending a good amount of time on exploratory analysis. Can we quickly discover some interesting insights, correlations, data bugs, etc? Feed the results of analysis to domain experts and see if they make sense before digging deeper into modeling. Usually, more data issues are identified at this stage so that they can be fixed before feeding "garbage in" into a complex modeling process.

I think from my perspective, When I am dealing with straightforward questions, where the answers lie on the surface, I prefer to stick with simple analysis methods. These methods are generally quicker and require less computational resources, allowing for a swift and efficient analysis process. On the other hand, when the questions are more intricate, and the data is rich with potential insights, I lean towards complex analysis methods. These methods, while requiring a deeper understanding of data science, can unearth nuanced patterns and relationships that simple methods might overlook.

In my view, the decision between simple and complex data analysis methods depends on understanding the trade-off between the added complexity of the model and the incremental value derived from it. In situations where the impact or risk is high, or where accuracy is of utmost importance, opting for a more complex model makes sense. Conversely, in other scenarios, a simpler model might suffice.

Furthermore, the context in which the data was collected plays a crucial role. Consider whether the data was obtained from controlled experiments, observational studies, or real-world scenarios. The nature of data collection can influence the validity and generalizability of results obtained from simple or complex analysis techniques. Additionally, the potential for interactions and relationships between different data types should be evaluated. If your data contains a mix of numerical, categorical, and textual elements, exploring hybrid methods that can capture intricate interactions might be necessary for a comprehensive analysis. Lastly, don't overlook the potential for data evolution.

My take on this is to go for relevance and simplicity. On one end, we want to have the data relevant to the problem at hand. It usually helps to get advice of domain experts on what that may be. In other cases, the relevant data need to be generated, e.g. via adding logging to a software system. On the simplification side, we want the initial set to be manageable so it is easier to come up with insights and modeling approaches. For example, consider resolution of images, length of videos or text, etc. In my experience, it is more important to iterate quickly end establish model performance quickly. The follow up becomes a tradeoff, e.g. how much accuracy improves with more complex or rich inputs and what is associated cost.

Understand the different types of data and how they are typically analyzed. For example, numerical data is typically analyzed using statistical methods, while textual data is typically analyzed using natural language processing methods. Identify the specific questions that you are trying to answer with your data analysis. This will help you choose the most appropriate methods. Clean and preprocess your data before analyzing it. This will help to ensure that your results are accurate and reliable. Be aware of the potential biases in your data and how they could impact your results. By following these tips, you can choose the right data analysis methods to extract valuable insights from your data and make informed decisions.

Returning to the fundamental step of data comprehension, which entails grasping metrics, KPIs, data origins, and the structure of facts and dimensions tables. This comprehensive understanding provides a foundation for determining the appropriate analysis approach. It's crucial, even if one is only acquainted with a limited set of tools and methods, as broader awareness can lead to more effective analyses and better opportunities for insight discovery.

This is a key factor. If the particular use-case is a big revenue generator, then you can invest the time to build the most sophisticated model to squeeze out that extra 0.1% lift in model accuracy. On the other hand, if this is not some mission-critical project, then good-enough-is-good-enough. Get it to 80%, ship-it, and move on to other projects. There is always more work to do than we have time for. Dont over-invest in something that's not that important. Data scientists often struggle with this element.

In my opinion this should be first factor to consider to choose the complexity of analysis methods because often times the Data Cleaning Process is dependent on the kind of analysis technique or method that we’re going to choose. Having said that, the end goal, the insights we are trying to uncover are also major factors to be considered.

The CRISP-DM framework offers comprehensive guidance. Emphasizing the understanding of the business is important, as our actions are geared towards improving the business. Once the business objectives are clear, aligning them with the appropriate analysis method becomes straightforward—a simple lookup from objective to analysis method.

Consider the potential for iterative analysis. If the initial objective involves basic insights, starting with simple methods can provide quick preliminary results, which can then inform the need for more complex techniques as your questions evolve and deepen. Another thing to keep in mind is the impact of uncertainty and risk associated with your analysis. If the questions involve critical decisions, opting for simpler methods might reduce uncertainty and provide a more robust foundation for decision-making, even if sacrificing some intricacies.

This section is quite important indeed. We might end up creating dashboards or models that nobody really needs in the end. It is easy to believe that that new "cool" feature will be useful for everyone, but that is often not the case. Ultimately, establishing clear analysis objectives is crucial.

The potential for model overfitting in complex methods should be carefully thought of. While some methods provide high accuracy on training data, they could struggle with generalization to new data. Striking a balance between complexity and generalizability is essential for reliable insights. Another aspect to consider is collaboration. Complex methods might be harder to communicate and share among team members or stakeholders, potentially hindering collaboration. Simple methods could facilitate clearer communication and alignment among project contributors.

Always run simple sanity check models first before doing something overly fancy. Besides potentially providing quick insights, it can also help to reveal biases and issues with the data that you can check, fix and account for before you throw it into a black box and trust the result.

An important factor to consider is how easy or expensive is to deploy a model. The fancier a model is the more expensive it is to run. We need to balance performance, execution time, and deployment costs.

I will reiterate: start simple and iteratively improve until diminishing returns are reached. Focus on the simplest possible way to find a solution at first -- that becomes your base line. Then establish your measure and criteria of success (which can be multi-dimensional, e.g. accuracy, cost, speed, explainability, schedule, etc.).

Indeed, the approach can vary depending on the specific business and client. If the client lacks technical expertise, opting for simpler solutions becomes prudent, as it mitigates deployment and maintenance challenges that might otherwise arise. Other factor can be cost too which can be taken into consideration while thinking about this.

One of the most important point to consider is the potential for bias and fairness in the analysis. Complex methods might inadvertently amplify biases present in your data, making thorough validation and fairness assessments crucial. Simpler methods, while less prone to overfitting, may also carry biases that need careful evaluation. At the same time the reliability of your evaluation metrics. Complex methods might require more advanced and context-specific evaluation criteria, while simple methods often rely on widely accepted and standardized metrics. The reliability of these metrics can impact the confidence you have in the validity of your analysis outcomes.

I would focus on the domain-specific criteria first: what would the business, customers, investors, etc. want? Then one can develop more technical metrics that can be tracked internally, without losing the ultimate focus on the "top line" metrics. In practice, no one will benefit from a model that looks good in terms of an obscure metric only mathematicians can understand but not making a dent on customer experience or business performance.

I personally will be replicating the entire solution and then play it around with real data to check the results but won't be publishing until I am satisfied with the evaluation.

In my experience when you present your audience is managers, engineers and sales. Therefore your presentantion needs to have value for all of them. You make presentantion for managers and sales and add slide with technical details for Engineers. For more information you will send after presentantion document with more information. Most Data Scientist don't invest a lot of time in the presentantion. If you want to be one of the best Data Scientist you need to learn to present and communicate excellent.

The tech savviness of audience/end-users can influence the decision around the choice of simple vs complex analysis methods. If the solution is going to be used by non-technical users, then simpler is better in many B2B products. I recall a project earlier on in my professional career. We built a highly complex staffing model for a financial services organization. I threw all my doctoral knowledge into that work (I know better now). A statistician from the organization (also a PhD), when presented with our solution, responded "This will never work for us because we will not be able to explain how this works to our managers. If they dont understand it, they wont use it."

This is the most crucial step. It helps if you practice your storytelling skills, and give a clear explanation of the step-by-step that you took to achieve the final result for your analysis. If the presentation is for a nontechnical audience, focus on the main business goal behind this analysis and always come back to it multiple times during the explanation.

It is important to tell a story and to take a step back from all the tiny details. We need to present the findings that have the highest consequences for a business. Furthermore, if you can connect your model metrics with cost savings then you will definitely get some attention.

Choosing between simple and complex data analysis methods relies on factors like data quality and quantity, data type and structure, analysis objectives, complexity considerations, evaluation and validation, and effective communication. Assess your data's quality, type, and objectives to decide between simplicity and complexity. Balance interpretability and performance and thoroughly validate results, paying attention to biases. Adapt your communication style to your audience. Ultimately, your choice should align with your data, goals, and audience preferences.

I would not frame the question in this way. From a business perspective, the simpler the better (less to go wrong essentially). From a data science pratictioner perspective, the more complex the better. In any case , starting a project it is always good have a benchmark with the simpler model possible even if the complex one is better from the start. Comparing the two (or more) models allow the data scientist to understand 'why' the complex model is actually better. And I almost guarantee that even the simple model, for certain cases, it will have similar (sometimes better) performances (understanding why is the key!).