What's your process for choosing data platforms and tools?

In my experience it's best to work backwards, and actually start with the business objectives. Or, more specifically, start by understanding who the data consumers will be, and what their requirements are. While talking with the data consumers for the project, you also want to understand what types of tools they plan to use to consume the data (SQL, BI tools like QuickSight or PowerBI, or Machine Learning platforms like SageMaker). Once you understand the business objectives and the requirements of the data consumers, you can then figure out which data sources can be used to meet their requirements.

Business Requirements, Staffing, SDLC and available data. Once business requirements are known, I also check to see what the staffing plan is, and what process is in place to achieve the goals set out by the business. In the last 6 years or so, management has asked me to mentor junior staff and even senior staff on how to access, process and use data. Some data might be in files - such as photographs, maps or sound files, but most data is pure business data - conforming to any SQL database, such as Redshift or Snowflake. I've had great success only in the last few years teaching and mentoring staff to do self serve using dBt and its fully integrated support of SQL models. For data scientists we use Python models - but sparingly.

It is important to perform rigorous business and data analysis and document your business and data requirements in any format which will give you sufficient information to further select your data architecture. Map the requirements with capabilities what you would like to achieve.

Create an architectural decision and design decision document with PROS , CONS and Recommended option and perform peer review to reach certain decision. This will give you confidence that you are going in right direction.

When deciding on a solution, make sure to keep flexibility as a top priority, as things change very quickly in this industry. I suggest that you always ingest to a popular object-storage solution, rather than directly into an end-product. If you use AWS, then make sure you bring all your raw data into Amazon S3 and make that your foundation. Just about every product out there is able to ingest data from popular cloud object stores (such as S3), so ingesting into S3 gives you flexibility in future. If instead you ingest directly into a specific product (such as Snowflake or Redshift), it may be more difficult to change your ingest pipeline down the line. But once data is in S3, you can ingest from there into just about anything.

Try to select your data solution in a way where it maps to your requirements and capabilities. Always remember to avoid any Vendor Lock In Scenario. Your architecture should be loosely coupled so that in case you change any tool or technology it should not impact much and avoid the technical debt etc.

In the name of agile people have stopped doing proper documentation for the requirements and design. I have seen in agile people are forwarding the tasks one sprint to sprint. Lot of rework because people do not do proper planning and design. They say its waterfall approach and unfortunately it is not . In agile as well you have lot of opportunity to improve the design and requirement nd backtrack.

Here in this section the most important aspect is missing which is evaluating your data solution in terms of non-functional requirements like schedule , frequency , data format , performance of data pipeline which is crucial for the data centric approach.

Section 5 and Section 3 go hand-in-hand. You need to look at least 18 months down the road to determine what you should do. You need a solution that can scale 'as expected' not 'when we get there'. What you should not do is "we'll implement solution 'X', because it's cheaper and fix it later if we grow.".

To optimize data solutions, define clear requirements, evaluate options based on performance and scalability, ensure compatibility, prioritize security, analyze total cost of ownership, prioritize usability, consider community support, conduct proof of concept, choose reputable vendors, and assess future roadmap.

Big Picture IT decision-making checklist: Inventory Existing Tools: - Identify all current tools, including those from “shadow IT.” - Involve actual decision-makers in the process. Assess Needs: - Evaluate integrations, governance, security, scalability, and features. - Prioritize alignment with organizational goals. Formalize Processes: - Establish clear procedures and SLAs. Benefits: cost management, productivity, accountability, and risk reduction. Avoid Common Pitfalls: - Seek diverse information. - Be willing to change course. - Allocate sufficient decision time. - Address biases and conflicts. Remember, decisions are like code—well-structured and thoughtful choices lead to better outcomes! ????