From Fragmentation to Focus: A Data-First, Team-First Framework for Platform-Driven Organisations

Success in today's complex, engineering-led enterprise organisations, where autonomy and scalability are paramount, hinges on more than adopting the latest tools or methodologies. The real challenge lies in aligning decentralised teams with shared goals while embedding governance without stifling innovation, creating a framework where teams can innovate freely, stay aligned, and ensure data is no longer treated as a second-class citizen.

While CI/CD revolutionised software development, it overlooked the unique challenges of managing and governing data at scale. Data pipelines, quality, and compliance often remain fragmented, manual, or inconsistent, creating bottlenecks and risks. Enter Continuous Governance (CG): the evolution that puts data on equal footing with software, embedding compliance, quality, and automation directly into workflows without stifling creativity.

This article introduces a Data-First, Team-First approach, a blueprint for organisations to elevate data to a first-class citizen while empowering teams to innovate with autonomy. Guided by principles like Conway's Law, Team Topologies, Golden Paths, and Recipes, it discusses transforming fragmented workflows into a seamless, high-performing ecosystem.

“However, achieving this balance of innovation and governance is no small feat in today’s decentralised organisations.”

“In previous articles, we explored the importance of aligning team autonomy with data governance and introduced foundational concepts like Golden Paths and Continuous Governance. This article builds on those principles to present a comprehensive Data-First, Team-First framework.”

The Cost of Fragmentation in Decentralised Organisations

Decentralised teams, each managing their own workflows, tools, and data pipelines, often operate like independent islands within an organisation. This autonomy fosters innovation but frequently comes at a cost: fragmented systems, duplicated efforts, and governance gaps.

Traditional governance frameworks built around static, document-based policies struggle to keep up with the dynamic needs of such environments. Reliance on manual oversight slows teams down and introduces friction and internal politics, turning governance into a roadblock rather than an enabler. Teams are left with a choice: bypass cumbersome processes or become paralysed by bureaucratic delays.

• Inconsistent Governance: Teams may interpret compliance or quality standards differently, leading to risks and inefficiencies.
• Duplication of Effort: Similar pipelines, tools, and workflows are built repeatedly, wasting time and resources.
• High Cognitive Load: Developers and engineers grapple with complex tooling and siloed processes, detracting from their core tasks.
• Lack of Interoperability: Cross-team collaboration becomes a bottleneck without shared language or standards.

How can such an organisation balance the freedom of individual teams with the need for alignment and governance at scale?

Shifting from Fragmentation to Alignment

Organisations can address these challenges by introducing Recipes modular, parameterised scripts that automate and operationalise workflows for deploying resources, updating metadata, and enforcing compliance. Unlike static, document-based policies, Recipes embed governance and quality directly into workflows, ensuring consistent implementation across decentralised teams.

By standardising key tasks such as pipeline deployment and metadata validation, Recipes eliminates duplication of effort, reduces cognitive load, and ensures interoperability through vendor-neutral syntax like Score. Teams can work faster and more confidently, knowing compliance is built into the process without adding friction or delays.

Imagine a decentralised organisation where each team leverages Recipes to enforce repository standards, maintain lineage tracking, and inject metadata into shared catalogues. The result? Fragmentation gives way to alignment, enabling teams to innovate autonomously while remaining connected through scalable, repeatable workflows.

Data-First: Embedding Governance and Quality at the Core

1. Metadata Inflow: Ensure that data products are enriched with lineage, quality metrics, and compliance attributes as they are created. This shared metadata forms the foundation for interoperability and cross-team collaboration, ensuring data products can integrate seamlessly across the organisation.
2. Data Products as Code (DPaC): Define data products declaratively, applying the principles of Everything as Code to data workflows. Governance, quality, and compliance rules are consistently and automatically enforced through Policy as Code (PaC). By eliminating reliance on manual governance processes, this approach reduces errors and enhances scalability.
3. Golden Paths: (Templates) Predefined, opinionated workflows within the IDP guide teams through compliant and optimised practices. These workflows embed Policy as Code (PaC) into every step, ensuring governance and validation are automated and consistent. By abstracting complexity, Golden Paths enable teams to execute workflows with confidence and focus on innovation.
4. Recipes: Parameterised scripts automate key tasks like resource deployment, metadata validation, and repository configuration. Acting as the building blocks of Templates, Recipes ensure repeatability and reduce duplication of effort. By adopting vendor-neutral syntax (e.g., Score), Recipes provide portability and scalability across tools and teams. Recipes also integrate Policy as Code, ensuring compliance and governance are enforced directly within the scripts.

Team-First: Empowering Teams with Autonomy

1. Stream-Aligned Teams: Assign ownership of specific data products to domain-focused teams. This ensures that teams deliver value independently while adhering to organisational governance standards.
2. Self-Service Platforms: Provide intuitive tools, templates, and workflows via an Internal Developer Platform (IDP). These platforms abstract complexity and enable teams to work autonomously without deep governance expertise.
3. Golden Paths for Teams: Empower teams with simplified, pre-configured workflows that embed compliance and governance directly into their pipelines. Golden Paths allows teams to focus on innovation while aligning with organisational goals.

Continuous Governance (CG): The Engine of Computational Governance

A New Service in the Integration & Delivery Plane

At the heart of Continuous Governance (CG) lies a dedicated service within the Integration & Delivery Plane of the IDP. This service performs computational governance checks as a critical step before workflows proceed to the Continuous Delivery (CD) function. By embedding these automated checks (PaC) directly into the pipeline, CG ensures that governance, compliance, and quality standards are met without introducing manual bottlenecks.

(For information about the Internal Data Marketplace, see Reimagining the Data Marketplace: The Gateway to AI Innovation.)

Golden Paths and Computational Governance

Golden Paths are opinionated workflows designed to streamline compliance and optimise practices. These paths integrate directly with the CG service, ensuring that every action adheres to organisational governance policies. Before deployment, the CG service validates key governance attributes, such as:

• Metadata Compliance: Ensures lineage, quality metrics, and regulatory attributes are included and adhere to organisational standards.
• Policy Enforcement: Validates that pipelines and resources align with access control, naming conventions, and compliance requirements.
• Quality Assurance: Checks for predefined thresholds in data quality, ensuring the output meets expected standards.

Recipes as Enablers of Computational Governance

Recipes are parameterised scripts that operationalise specific tasks within the Golden Path. While Recipes help enforce governance during pipeline execution, their integration with the CG service ensures:

?? Important

Embedding Brownfield Services with Recipes: Recipes can help integrate existing brownfield services into new workflows by automating tasks such as resource discovery, compliance validation, and metadata enrichment. By standardising these processes with vendor-neutral syntax, Recipes enable organisations to utilise legacy systems while aligning them with modern governance and operational standards. This approach simplifies migration challenges and supports incremental modernisation without disrupting current operations.

• Automated Validation: Recipes are executed only if they pass CG checks, embedding compliance into every process step.
• Seamless Workflows: Teams can focus on delivering value while the CG service consistently applies governance standards.

The Impact of Computational Governance

By introducing a CG service within the Integration & Delivery Plane, organisations gain:

1. Automated Compliance: Governance checks are built into the pipeline, removing manual oversight and reducing delays.
2. Improved Quality Assurance: Issues are detected and addressed before the CD function, ensuring reliable and compliant outputs.
3. Frictionless Innovation: Teams can innovate freely, knowing compliance is embedded in workflows without requiring manual intervention.

The Central Enabling Team: The Glue That Holds It All Together

In a Data-First, Team-First framework, decentralised stream-aligned teams focus on delivering domain-specific data products, and platform teams provide self-service infrastructure. However, a Central Enabling Team is necessary to ensure cohesion and scalability across the organisation. This team acts as the glue, enabling alignment between decentralised teams while maintaining organisational standards, governance, and platform capabilities.

The mission of the Central Enabling Team

The Central Enabling Team's mission is to:

1. Scale and Maintain Platform Capabilities: Develop and sustain crosscutting services such as the IDP, Continuous Governance (CG), and the DevEx Plane. These services ensure compliance by embedding governance and quality into workflows without burdening individual teams.
2. Support, Not Control: Act as an enabler by facilitating the adoption of templates, Golden Paths, and Recipes, reducing the complexity teams face without constraining their autonomy.
3. Reduce Cognitive Load Across Teams: Provide automated validation and feedback mechanisms to simplify governance, compliance, and operational tasks. This allows teams to focus on innovation while the platform seamlessly handles governance.
4. Standardise for Interoperability: Establish and enforce consistent standards for metadata, Recipes, and governance policies, enabling teams to collaborate effectively and scale efficiently.

Key Responsibilities

The Central Enabling Team is a custodian and guide for key organisational resources, ensuring consistency, compliance, and usability. However, ownership of these resources remains with the teams that create and maintain them. The Central Enabling Team supports by:

1.??Building and Managing the IDP: Develop core features like the DevEx Plane, Golden Paths, and self-service tools. These tools abstract complexity and provide developers with guided workflows, enabling them to work autonomously while adhering to governance requirements.

2.? Operationalising Governance: Automate compliance checks and policy enforcement through Policy as Code (PaC) and CG services. These validations ensure that governance is seamlessly embedded into workflows, reducing manual intervention and delays.

3. ?Reducing Friction with Continuous Governance: Implement CG services within the IDP's Integration & Delivery Plane to validate compliance, metadata quality, and organisational standards before deployment. This reduces teams' cognitive overhead, allowing them to focus on delivering domain-specific value.

4. Maintaining Shared Resources: ?

·????? Guiding Compliance: Collaborating with stream-aligned teams to ensure that recipes, templates, and metadata resources meet organisational standards. The Central Enabling Team provides validation and guidance rather than enforcing direct control.

·????? Facilitating Shared Standards: Establishing and maintaining organisational standards for metadata, templates, and repositories, ensuring that all teams can easily align their resources with these guidelines.

·????? Supporting Maintenance: Assisting teams in improving or validating their scripts, templates, and workflows to ensure long-term interoperability and scalability. The Central Enabling Team offers support to refine and adapt these resources without taking ownership.

5. Facilitating Team Adoption: Collaborate with stream-aligned teams to onboard them to platform tools, troubleshoot challenges, and improve the adoption of governance and best practices.

6. Collaborative and Automation-Focused:

·????? Enabler, Not a Gatekeeper: Unlike traditional centralised teams, the Central Enabling Team empowers stream-aligned teams by providing resources and guidance rather than controlling workflows.

· Automation-driven: Automates repetitive tasks like compliance validation and resource provisioning, enabling teams to focus on innovation.

·????? Collaborative: Works closely with teams to co-create solutions that align with their specific domain needs while adhering to organisational standards.

Practical Example

In a retail organisation:

• The Central Enabling Team provides pre-built Golden Paths for workflows such as data ingestion or transformation, ensuring governance and metadata standards are embedded by design.
• It helps the Sales Data Team implement Recipes for deploying resources and updating metadata, ensuring compliance with organisational policies.
• It maintains the metadata catalogue, enabling all teams to discover and integrate data products seamlessly.

By serving as enablers rather than gatekeepers, the Central Enabling Team ensures that decentralised teams have the tools, resources, and support they need to innovate while maintaining alignment with organisational goals.

Conway's Law: Structuring for Success

Craig Larman's presentation, "Myths of Software Management: Conway's Law," provided an invaluable gateway into revisiting Melvin Conway's original insights. Larman draws attention to a critical section at the end of Conway's 1968 paper, where Conway makes a striking recommendation that directly challenges the common interpretation of his law.

Conway's Law:

"Organisations which design systems … are constrained to produce designs which are copies of the communication structures of these organisations."

This is frequently interpreted as a call to align team structures with the desired architecture. However, as Larman highlights, Conway's concluding thoughts suggest a far more nuanced view:

• The architecture you choose first is likely to be wrong.
• Organisations should ensure their teams are structured to adapt to evolving architectures, not constrained by them.

This foundational idea, highlighted but often overlooked in Conway's writing, reframes the relationship between teams and architecture. Conway advises focusing on flexibility in systems and structures instead of forcing teams to stick to a set design. This approach allows for ongoing improvements and adaptations as needs change.

The True Insight: Adaptability Over Alignment

This recommendation aligns with the principles of the Inverse Conway Manoeuvre, which flips the common interpretation of Conway's Law. Instead of forcing teams to fit a predetermined architecture, it advocates for architectures that adapt to autonomous teams' natural interactions and workflows.

By acknowledging that initial architectural designs are rarely perfect, Conway emphasises the need for:

1. Teams capable of adapting as architectural requirements evolve.
2. Architectures designed to emerge from iterative collaboration rather than rigid, upfront definitions.

Embedding Conway's Insight in Practice

In my article, From Fragmentation to Focus, I advocate for a Data-First, Team-First Framework that operationalises these principles through tools like Golden Paths, Recipes, and a Central Enabling Team. This approach ensures adaptability while maintaining governance at scale:

1. Golden Paths for Consistent Flexibility: Golden Paths serve as predefined workflows that standardise best practices without limiting innovation. By embedding governance directly into these workflows, teams can adapt them to domain-specific needs while ensuring compliance.

2. Recipes as Modular Building Blocks: Recipes (parameterised scripts) for tasks like resource provisioning or metadata validation allow teams to evolve workflows dynamically. This modular approach ensures that changes in architecture can be reflected in team processes without disrupting governance or alignment.

3. The Central Enabling Team as a Catalyst for Change: The Central Enabling Team facilitates rather than enforces rigid controls. It supports teams with templates, tools, and guidance, enabling them to adjust workflows and architectures as needs change. This ensures cohesion and scalability across the organisation without compromising autonomy.

A Practical Example: Retail Transformation

Consider a retail organisation struggling with fragmented pipelines and siloed teams. Applying Conway's recommendation for flexibility, the organisation:

• Redesigns its Internal Developer Platform (IDP) around domain boundaries, enabling teams to structure their workflows around specific needs.
• Deploys Golden Paths for standardised, compliant data ingestion, ensuring governance and adaptability.
• Uses Recipes to automate governance checks, metadata updates, and resource provisioning, reducing manual effort and duplication.

This allows teams to innovate within a framework, ensuring consistency and alignment and reflecting Conway's vision of adaptable systems and team structures.

From Fragmentation to Focus: The Impact

Conway's recommendations highlight a critical challenge for modern organisations: the systems we design today will evolve, and teams and architectures must be prepared to grow with them. This vision underpins the Data-First, Team-First Framework, where adaptability is embedded through:

• Continuous Governance (CG): Automating compliance and quality assurance to minimise friction as systems change.
• Self-Service Platforms: Enabling teams to innovate autonomously while staying aligned with organisational standards.

Conway's insights remain as relevant as ever as we navigate the complexities of decentralised organisations. The question isn't whether your teams align with your architecture; it's whether your teams and systems are flexible enough to adapt to the architecture of tomorrow.

1. Interoperability at Scale: Shared metadata and governance policies are implemented consistently through Golden Paths and recipes.
2. Faster Innovation: Teams can leverage pre-built Golden Paths and recipes, accelerating the deployment of compliant and scalable services.
3. Stronger Governance: Automated workflows ensure all resources, metadata updates, and repositories meet organisational standards.
4. Aligned Autonomy: Teams innovate independently while adhering to shared standards, supported by workflows that embed governance and quality at every stage.

The DevEx Plane: Empowering Developers

The Developer Experience (DevEx) Plane is the linchpin of the Data-First, Team-First framework, providing a streamlined interface for interacting with the platform. By reducing cognitive load, enabling autonomy, and seamlessly embedding governance, the DevEx Plane empowers developers to innovate without sacrificing compliance or efficiency.

Simplifying Complexity Through Self-Service

The DevEx Plane abstracts away the complexities of infrastructure, governance, and compliance, providing developers with intuitive, self-service workflows that streamline their efforts. Key features include:

• Self-Service Workflows: Developers can select predefined Golden Paths to implement compliant, scalable services like data ingestion or transformation pipelines, ensuring they adhere to organisational standards without needing deep governance expertise.
• Direct Feedback: Continuous validation ensures developers comply with metadata, lineage, and governance rules as they work, catching issues early and preventing costly downstream errors.
• Unified Tools: The DevEx Plane eliminates tool sprawl and provides a cohesive user experience by centralising activities such as deploying pipelines, updating metadata catalogues, and managing resources within a single interface.

A data engineer creating a streaming ingestion pipeline can use the DevEx Plane to select a Golden Path, deploy resources through parameterised recipes, and automatically inject metadata into the organisational catalogue(s) without involving additional teams. They are also free to build their own template, which utilises existing recipes, or even develop their own.

To clarify: Golden Paths and Templates: Balancing Best Practices and Autonomy

Golden Paths and Templates allow teams to choose between following pre-approved patterns or defining their own workflows. Both approaches rely on Recipes, which are the foundational building blocks that automate compliance and governance.

• Golden Paths: Opinionated, best-practice workflows that guide teams through compliant, optimised processes. Golden Paths embeds governance and validation into every step, ensuring teams follow organisational standards with minimal friction.
• Templates: Customisable workflows created by teams to address specific domain requirements. Templates give teams autonomy to innovate while ensuring that all processes adhere to governance rules through embedded Recipes.
• Recipes: Parameterised scripts that automate individual tasks such as resource provisioning, compliance validation, and metadata updates. Recipes ensure consistency and repeatability in both Golden Paths and Templates.

This Recipe-driven framework allows teams to innovate while maintaining organisational alignment, enabling autonomy and governance at scale.

Embedding Governance Without Friction

Governance, often seen as a barrier to speed, becomes a seamless part of development workflows within the DevEx Plane. By automating governance, the platform removes human bottlenecks and ensures consistent enforcement at every step.

• Recipes as Operational Scripts: Recipes, accessible via the DevEx Plane, execute tasks such as provisioning resources, updating metadata, configuring repositories, and embedding governance rules directly into workflows.
• Continuous Compliance Checks: Developers receive real-time feedback if a pipeline or resource violates organisational standards, enabling them to address issues immediately without slowing development.

Scaling Autonomy and Collaboration

In large, decentralised organisations, the DevEx Plane is the connective tissue that ensures consistency across teams while empowering them to operate autonomously. It achieves this by embedding governance and compliance into automated workflows and providing tools that streamline development processes. Key benefits include:

• Aligned Autonomy: Teams can innovate within their domains while relying on the DevEx Plane to enforce governance, compliance, and organisational standards. By offering predefined workflows, templates, and recipes, the platform enables teams to focus on their core objectives without worrying about the complexities of compliance.
• Transparency and Traceability: The DevEx Plane supports automating governance features, enabling lineage tracking, compliance validation, and quality metric integration into workflows. It doesn’t directly provide these attributes but ensures that all automated processes incorporate them. By embedding these features into workflows, the platform fosters accountability and trust across teams, ensuring that governance and compliance are achieved by design.

From Fragmentation to Focus

Aligning autonomy with governance is a delicate balance in large, decentralised organisations. The Data-First, Team-First framework enabled by IDPs, guided by Inverse Conway Maneuver, and operationalised through Golden Paths and Recipes provides a clear path forward.

Organisations can unlock their data's and people's full potential by embedding governance through Continuous Governance (CG), empowering teams with self-service tools, and reducing cognitive load through the DevEx Plane. The result? Accelerated innovation at scale, supported by a foundation of trust, compliance, and collaboration.

This article references concepts found in

Skelton, M., & Pais, M. (2019). Team Topologies: Organising Business and Technology Teams for Fast Flow. IT Revolution Press.

Melvin E. Conway, How Do Committees Invent?

Copyright 1968, F. D. Thompson Publications, Inc. Reprinted by permission of Datamation magazine, where it appeared April, 1968. https://web.archive.org/web/20190919111512/https://melconway.com/Home/Committees_Paper.html

Fowler, M. (n.d.). Conway's Law. Retrieved from https://martinfowler.com/bliki/ConwaysLaw.html

Larman, Craig. Myths of Software Management: Conway's Law https://www.youtube.com/watch?v=NTG6w0u8K_g