You’re starting a new project automation initiative. What data warehousing tools should you consider?

There are multiple things to take into account while choosing the correct data warehousing tool: 1. Compatability with data sources/formats 2. Ability to perform complex transformations, robust schema support 3. Scalability, efficient data processing and query performance 4. The tool should comply with your data governance policies 5. Cost-effectiveness 6. How well it can be integrated with your current ecosystem 7. Good technical support It's important to note that there's no one-size-fits-all solution. Key priorities revolve around integration, scalability, and cost-effectiveness.

I would not choose tools on your own in the complexities of corporate infrastructure. Work with your IT/OT department(s) to solution to your needs and fits into the companies IT/OT strategy.

Multiple things to be taken into consideration. 1) requirement to be on cloud or in house. 2) usage expected. 3) technical expertise of the teams 4) need for disaster recovery. 5) complexity of the task to be fulfilled. 6) cost associated with the solution. 7) platform independent.

Define data strategy in terms of Enterprise data vs functional verticals and then warehouse Enterprise data. Functional data may be needed in some cases to enrich (to-be) warehouse data. Define and adopt to open architecture as far as data format is concern as your iPaaS(Integration platform) solution should be scalable enough to handle different sources and types of data ingestion.

There are many popular data warehousing tools but when you start a new project automation initiative, selecting the right data warehousing tools is crucial for ensuring efficiency, scalability, and compatibility with your project's goals like Google BigQuery, Amazon Redshift, Snowflake etc.

To ensure data quality in processes that involve dealing directly with the customer, it's crucial to standardize inputs as much as possible. That practice makes it easier to maintain a higher level of data quality by source and also deliver better and cleaner data for analysts to consume and do their job.

Data quality is the key. Need to have rigorous ETL (Extract, Transform & Load) strategy to ensure data quality for your respective models.

When automating processes involving data, ensuring quality and governance is paramount. Avoiding inaccuracies, incomplete, or duplicated data is essential to preserve the integrity of analyses and automations. Look for tools that validate, clean, and standardize data before use. Additionally, it's important to choose solutions that manage security policies, access, and data compliance, ensuring the reliability of the data manipulated during automation.

- Define schema contracts with SORs for higher quality data, let's not be victim of "Garbage in, garbage out" - Define & follow data quality rules during data in motion - Setup monitoring on modeled data - Avoid custom coding to monitor and perform data quality checks. Utilize low-code /zero-code platforms as much possible . Reach out to me to find out more details on tools/processes to achieve effective Data Governance & Data Quality

I'm not an expert in Data Warehousing, but I would not make that decision dependant on the Process Automation initiative, unless the DWH only holds data related to your robotic workforce.

A careful selection of a tool became essential to enable efficient querying, exploration, and reporting through various techniques and tools. The automation of analysis and visualization tasks using scripts and workflows required a thoughtful choice, considering the diverse capabilities and integrations offered by different available solutions. The experience emphasized the need for a flexible tool that supports specific requirements, providing a more effective and seamless data management and analysis.

Virtualize your data required for real time analytics and visualization. Effective data consumption model is a must and I prefer self-service/data product model approach

Na atualidade devido ao crescimento massivo de informa??es a serem disponibilizadas e aplicadas para análise, tem cada vez mais a necessidade de planejamento e arquitetura de dados aplicada ao negócio. Logo se faz necessário a pesquisa aprofudada de ferramentas que ser?o aplicadas ao projeto e a vida útil da mesma. Nowadays, due to the massive growth of information to be made available and applied for analysis, there is an increasing need for planning and data architecture applied to the business. Therefore, in-depth research into tools that will be applied to the project and life is necessary.

I feel this is a major challenge for any ML tools deployed in the industry. It is not scalable to new incoming data without a lot of cost and effort of retraining or a lot of compute resources. i think we should aim for a pipeline which is scalable and gives us a good performance on the incoming data over time.

Define strategy around Cold, Warm and Hot data to manage scalability & performance at optimal cost. Effective modeling is the key to successful warehouse implementation that offers scalability with performance at right cost.

When assessing data storage tools, scalability and performance are crucial considerations. It's essential to handle large volumes and varieties of data without compromising speed, reliability, or cost. Adaptability to changes in business requirements and user demands without disrupting data warehouse processes is also key. Look for a tool that provides the optimal balance between scalability and performance to meet the needs of your data warehouse project.

I feel both are different perspectives and not tightly coupled with each other. Data warehouse tools needs to be added if really required considering volume, output expectations and ROI that justifies the investment in the tools. For many a cases, you might not need these tools

Data Compatibility: Tools that work with multiple data formats and sources are essential. Choose scalable tools to suit the company's future growth and expansion ambitions. Consider the tools' flexibility and interoperability to avoid getting locked into one source. Adoption and Training: Determine the length of training and steep learning curve for the selected technologies to enable project automation team integration. Choose solutions with strong security measures to prevent unauthorized access to sensitive data. Integration: Consider how difficult it will be to integrate the selected solutions with your existing systems and apps to allow data to flow freely during automation.

Em meus projetos atuais estou aplicando técnicas de aprendizado de máquina, com uso de webscraping para coleta de dados, fa?o o tratamento e aplico no modelo para análises e estou tendo ótimos resultados. Logo, uma recomenda??o particular é o uso de IA e ML para conseguir automatizar tarefas complexas que exigem muito tempo de m?o de obra humana. In my current projects I am applying machine learning techniques, using webscraping to collect data, I process it and apply it to the model for analysis and I am having great results. Therefore, a particular recommendation is the use of AI and ML to automate complex tasks that require a lot of human labor time.