Dot Points on Successfully Implementing Cloud Data Warehouse, Business Intelligence, and AI Solutions

Strategy

• Understand the current state, pain points, and desired future state.
• Start small, do things correctly from the outset.
• Pilot as a starting point and build upon it.

Platform Selection

• Engage third party consultants for cloud & AI platforms evaluation.
• Scan the environment (e.g., Vic gov partners with Snowflake, Federal gov with Microsoft Fabric, Legal sector on Azure) to align with industry trends and sector standards.
• Consider internal preferences and skill sets.

Data Architecture

• Use the Kimball approach for data modeling during the ETL process.

Data flow: Staging --> Dim Tables --> Fact Tables --> Business Views --> Excel/Power BI

• Leverage Inmon's principles exclusively for conformed dimensions and enterprise data marts.

Focus: who (e.g., clients + service provider, or customers + product supplier), what (e.g., services or products), how (e.g., programs or channel + finance + operations), and why (e.g., key performance indicators for the board)

• Include a layer to bridge technical terms and business terms to enhance the AI model's natural language processing capabilities.
• Implement role-based access control (RBAC, role/row-level security) and attribute-based access control (ABAC, column-level security) to prevent data leaks; start small and passive (e.g., no sensitive data exposure without RBAC in place) and scale gradually.
• Utilise Power Apps to maintain reference data.

High-Performing Team

• Dedicated time for team learning (individual members focus on different topics and then share knowledge) to enhance team capabilities.
• Revamp Agile processes.

Conduct peer reviews for quality assurance and knowledge dissemination

Hold retrospectives for lessons learned and performance appraisal

• Use DevOps boards and Power BI dashboards to enhance task visibility and promote team accountability.

Data Engineering & BI Practice

• Data Engineering

DEV for ETL development

UAT for ETL testing

INT for ETL execution; copy it as PRD once ETL succeeds

PRD for data extracts, reports and dashboards

• Data Visualisation

DEV, UAT, PRD

• Automate repository management:

Script out database structure

Push scripts, including Jupyter Notebooks and Power BI projects, into Git repository

• Implement CI/CD pipelines for code promotion.

Data Ingestion (ETL)

• OK to use Azure Data Factory (ADF) Data pipeline and Dataflow Gen2 for simple tasks, but -

• Prefer Jupyter Notebooks (Python) for data ingestion:

Shared skills for Data science

Easy to integrate Git version control

• Prefer Power BI + Power Automate for data ingestion as well:

Shared skills for Data Visualisation

Easy to integrate Git version control

• Both methods work well for on-premises and cloud platforms:

Files

Databases

Web APIs

• Define coding style standards for Python, SQL, and DAX and apply automated check.

• Two-stage data quality checks:

Before ingestion, ensure no garbage data in

After ETL: ensure outputs is of high quality

Use Power Automate to monitor data quality check logs and notify data stewards

• Implement parallel ETL processes.

Data Visualisation

• Use Power BI with the preview features (pbip, TMDL, PBIR) for easy Git version control integration.
• Create templates per data mart with Direct Query and Data Modeling.
• Develop a generic template (with configurable URL parameters) to explore tables and business views.
• Avoid creating measures directly in Power BI. Instead, embed business logic and counting rules in the data warehouse to ensure consistency across dashboards and eliminate the need to recreate the same measures in multiple reports.
• If role-based access control and attribute-based access control are implemented in database layer, permission control in Power BI dashboards becomes effortless, allowing you to grant everyone read access without the risk of data leaks.

AI Adoption

• Integrate AI to enhance customer experience, operational efficiency, and data-driven decision-making.
• Create your own AI models using open-source platforms that remembers context and utilises natural language processing.
• Run large language models (LLMs) within the data warehouse, eliminating the need to transfer data to external AI systems.
• Start with data champions to drive adoption.

Data Governance

• Apply sensitivity labels.
• Enforce permission controls (in database layer whenever possible).
• Develop data quality dashboards.
• Implement monitoring and auditing.
• Introduce active data retention: archive unused datasets and dashboards.

Data Community of Practice

• Host short but frequent sessions.
• Provide hands-on training, workshops, and mentoring to business areas to uplift organisation data and AI literacy.
• Invite business areas to present their data and operating procedures to uplift data team business literacy.

GordonData is your trusted partner for maximising data potential and enhancing operational efficiency. With expertise spanning full-stack development, data architecture, and business intelligence consultancy, we deliver tailored solutions that simplify data management, streamline integration, and automate processes. Whether it's building robust applications, designing scalable databases, or creating actionable dashboards, GordonData empowers your business to make informed decisions, save time, and reduce costs.

Feel free to ask questions or request more detailed information on topics of interest. Feel free to reach out for consultancy on re-platforming your data infrastructure and modernising your data platforms.