How can you align a big data project with the needs of data scientists?

Defining the problem is the bedrock of any data project, setting the stage for data scientists to apply their analytical acumen effectively. This involves crystallizing the project’s objectives, identifying key performance indicators, and establishing clear, measurable goals. The definition must be sufficiently granular to guide algorithmic development yet broad enough to allow for explorative data science. It’s about balancing specificity with the flexibility to pivot as insights surface. This foundation facilitates a targeted yet agile approach, ensuring that the big data initiative remains aligned with the evolving landscapes of both business exigencies and data science innovation.

In my experience, I totally agree with the point the understanding of any domain for any project like based out in service based or product based , if small understanding of domain could help to drive more insights and can help to understanding the forecasting projects and same it can help in different other use cases!!

At this stage before any development planning or design is started, I look for the "5 Ws": who, what, when, where, and why. Once these details are clear, we can have a discussion with the appropriate parties on how it will be done. Without excluding any of the "Ws", I will highlight the importance of everyone's alignment on what is to be accomplished/desired outcome and why it is needed. From this point, more questions can be answered through productive conversations between stakeholders.

One more thing to define along with the problem, scope and deliverables is the definition of done/success criteria. Many a times what Data engineers deliver has gaps with what Data Scientists (DS) /Stakeholders (SH) consider as done. For example things like, the data format, update cadence/frequency might not be very consequential from an engineers point of view as they might think few tweaks can give DS/SH the desired results although those tweaks becomes painful and sometimes time consuming for them. Definition of done will restrict the deliverables variability and make sure interested parties are aligned.

Em um cenário orientado por dados, é crucial alinhar projetos de Big Data às demandas dos cientistas de dados. Isso envolve compreender profundamente suas necessidades, promover colabora??o contínua, garantir flexibilidade na infraestrutura, investir em capacita??o e estabelecer monitoramento constante. Ao adotar essas práticas, construímos uma base sólida para impulsionar inova??o e alcan?ar resultados significativos, criando um ambiente adaptável que excede as expectativas dos profissionais de ciência de dados.

Providing data to data scientists isn’t merely about access; it’s about delivering curated datasets that ignite analytical exploration. To align with their needs, ensure data provisioning includes rich metadata, clear data lineage, and robust feature sets conducive to deep learning algorithms. The datasets should be comprehensive and of high fidelity, facilitating nuanced analysis and predictive modeling. A big data project must furnish a data ecosystem that is both expansive and precise, allowing data scientists to harness machine learning techniques fully and uncover transformative insights. This precision provisioning is key to powering data-driven decisions that propel business objectives forward.

Providing clean data is good and is always seen as a good Data engineering work however, this is not always recomended unless you are very sure on the cleaning and have agreed upon type of cleaning to be done. For example, a simple one is replacing the missing values with average or any other metrics or be it removing data which has more than x% of records missing. Data Engineer might clean up these and make substitution on the name of data cleaning and prep however, DS might have made some other arrangements for the missing values or it could have been used for any other feature development. Being in sync and performing agreed upon cleaning is recommended. In come cases, better to have a log exposed to DS on types of cleaning performed.

In practice, on initial steps of project, a data scientist knows more than a data engineer regarding data itself, BUT you have to make ALL to improve leveraging of this data, by optimising data storing, scale a data treating in your Data Lake layers to decrease job times and accelerate data scientist work. After initial steps, you can include auto-descriptive tools like the DataHub to track dependency between data objects and see fields description.

Clean data, enriched with business metadata, provides the ideal platform for data scientists to start developing insights. The metadata might reflect several key aspects of data, including freshness (e.g. last updated time), quality (e.g. endorsements), and availability (e.g. accessibility). It is essential that the metadata be maintained by data stewards in collaboration with the owners of the dataset. Access restrictions due to policies such as GDPR and 7216 must be foreseen and paved paths must exist to unblock impacted data scientists. The time taken for data scientists to discover relevant data is a popular measure of the efficiency of a big data ecosystem.

Acknowledging the varied preferences of data scientists concerning data formats, structures, and technologies is more than crucial in the field of data science. Effective collaboration and consultation are essential, emphasizing active engagement with data scientists to tailor solutions to their specific requirements. In summary, establishing the requisite data infrastructure and promoting transparent communication with data scientists are pivotal actions, this level of maturity must be reached. It is instrumental in realizing the complete potential of data-driven insights and decision-making within the framework of large-scale data projects.

In my experience, a data scientist mostly pay attention to a good predict result of a model, but don't have a time to adjust whole data pipeline for treating data effectively. Therefore, time-to-time a data engineer need to review a data scientist pipeline, challenge it by stress tests and propose good data patterns or even refactoring some part of code by themself, after discussion with a data scientist.

It involves implementing a multifaceted approach to empower data scientists throughout the complex and iterative process, that is a challenge, not to interfer. Agility is the key, understand the creative mind. Begin with the necessary computational resources needed, the software tools, and datasets that are required. Delve deeper into the subject and extend it to other experts to enhance the depth and accuracy of the analysis, create a constructive feedback based upon topic you have made a bit more detailed. But always, remember to respect autonomy and creativity.

Supporting data scientists in their work is essential for the success of any data-driven organization. By providing them with the necessary resources, guidance, and feedback, you can help them conduct their analysis effectively and achieve their goals.

Aligning a big data project with data scientists' needs transcends data delivery it necessitates a rigorous review mechanism for the results they produce. Establish a structured process for iterative feedback, incorporating cross-validation and model tuning based on performance metrics. Engage stakeholders in evaluating the results through the lens of business relevance and scalability.This critical review phase should facilitate a dialogue where data scientists interpret model outputs, validate hypotheses, and refine algorithms for predictive accuracy.By cultivating a collaborative environment that prioritizes empirical evidence and statistical rigor,you cement the symbiosis between big data projects and the scientific acumen driving them.

The different styles used to presenting results as an data scientist must of course vary because of the sometimes complex topic (when to add a dimension?) I would always recommend the data scientist to frame the presentation as a narrative to guide the audience through the data analysis journey. Develop a compelling story that connects the data insights to real-world implications and solutions as well as the choice of style. If abstract make it concrete/metaphorical via these real-world implications. And a hard lesson to learn (myself included): Avoid unnecessary jargon and technical language that might alienate non-technical stakeholders, give this as an hint to your aspiring data scientist in the preview of the presentation.

What we are considered less good at in bigger as well as in smaller companies. Recognizing and celebrating the accomplishments of not only data scientists is integral to fostering a positive and conducive work atmosphere. I would start already in the achievement of project milestones, celebrate the successful delivery of impactful outcomes, adept resolution of challenges, continuous learning and skill enhancement, and effective collaboration within the team. The customization of celebrations based on individual preferences is nothing more than paramount in the long run.

In my opinion it is equally important for a data scientist to protect sensitive data within the bounds of regulatory frameworks.