How can you ensure that your big data project meets the needs of end-users?

If you are starting something like an analytics internal product, but you have no idea of where to begin, try to be aware of the customer journey in your company. It will lead you to where you should work first and give you a good basis for problem understanding.

Here are some key steps to achieve this: Define Clear Objectives: Start by clearly defining the objectives of your big data project. Understand what specific problems or questions you aim to address and how they align with the needs of end-users. Involve End-Users: Involve end-users from the beginning of the project. Gather input from them to understand their pain points, requirements, and expectations. Data Quality: Ensure that the data you collect and analyze is of high quality and relevant to the needs of end-users. Data quality issues can undermine the usefulness of insights. Data Governance: Implement robust data governance practices to maintain data integrity, security, and compliance.

It is determinant the step of refinement like you do working in agile project management framework where the team (tech lead, data engineers, project manager and product owner) discuss the requirements and needs of stakeholders to make possible to define the tasks needed until the deliver. All the projects begins this way or something like this independently of the team organization.

When determining the ideal tool, in addition to the considerations outlined in the article, it's essential to assess the cost/budget of the project (as these tools can range from expensive code-based solutions to no-code alternatives) and evaluate the skillset of your team.

Yes but knowing purchase software is expensive and its use implies people effort. But you don't need to have all the tools you just need to find the right tool for specific need for example you need software for ETL, scheduling or batching, data visualisation, data storage, etc. Having tools that respond this needs is the goal.

The right tools can be effective for a long running business. It should not be some legacy tool which cost more but came with lots of limitations. The tool should be easier to learn by the new members who join the team and it should be having a vast community support as well. The tool should have compatibility to integrate with already existing applications to ensure the scalability , security etc.

It is important the BI team is composed of data scientists and data architects actively in continuous research in best practices and better ways to work. The teamwork it is determinant for good standards of documentations and tasks like fail controls, data exploration or data purge.

Effective communication is extremely important, but REGULAR conversation is critical. Large data science projects can sometimes take months to complete, and leaving the end user in the dark throughout this process is a recipe for failure. The end user's needs likely will change over the duration of the project, so a scheduled series of check-ins to keep everyone aligned can really go a long way.

The art of arguing is a very important skill on organizational context. The organizations are made of people and off course when I know what the stakeholder need and I can speak with same vocabulary I'm definitely near to reach success.

It is called the stabilization project phase. To the improvements I like the continuous follow-up between the product owner and the rest of the team because the feedback never is a one shot one. It is continuous.