Flix Scales Data-Driven Transport Logistics With Conduktor

Flix scales Kafka data-streaming operations from 25 to 50 teams with Conduktor. Optimizing fleet tracking, route efficiency, and AI-driven customer experiences across Europe and North America.

"Conduktor easily integrated with our CI/CD pipelines to enhance data governance. It's given us the tools to centralize compliance standards across the business, while allowing team-specific autonomy to speed up our processes. It's been central to our unified data platform strategy ."

Taras Slipets, Staff Data Engineer, Flix

About Flix

Flix is at the forefront of green passenger mobility, connecting millions to over 5,500 destinations across 40+ countries. Managing over 400,000 daily passenger connections, Flix relies heavily on real-time data to make critical operational decisions.

Digitization of Traditional Bus Travel with Kafka

Operating at the crossroads of a tech startup, e-commerce platform, and transportation company, data is the lifeblood of Flix. Kafka plays a critical role in managing vast streams of real-time data, from vehicle locations to customer feedback, enabling Flix to:

• Dynamically modify bus schedules on-the-fly
• Optimize routes by analyzing traffic patterns and vehicle data
• Rapidly respond to customer feedback, enhancing service quality and customer satisfaction

Kafka Growing Pains

As Flix expanded, so did the complexity of its Kafka infrastructure. Within five years, Kafka utilization at Flix expanded to over 2,300 topics and 9,300 partitions.

Failing to consider ownership during Kafka implementation can lead to future complications when teams need to discover, access, and share existing data streams. With varied access methods and complex permission setups, managing a data operation of this size presents multiple challenges:

• Managing permissions and access controls across global teams, while adhering to local data governance and privacy regulations became complex.
• The lack of a single interface for technical and non-technical users to independently access Kafka data created a bottleneck in handling access requests.
• Discovering available data, viewing schemas and filtering streams were difficult for analysts without a platform for observability and simplified data exploration.
• Without a unified view of data access, proactively consolidating data pipelines and storage to prevent runaway costs was challenging.

Data Management and Security in a Scalable Ecosystem Using Kafka

At Flix, robust data security is paramount. The sensitive nature of payment transactions and personal information required for travel bookings made data protection a critical concern, especially when developing new access control processes. The challenge was to streamline these processes without the complexity and overhead of custom synchronization tools.

To address this, Flix adopted a resource-centric approach, shifting away from traditional user-centric models. This transition not only optimized security protocols but also improved the efficiency of data management.

The new strategy is grounded in the principles of the data mesh concept, which gives teams ownership of their data and projects within a federated control framework, eliminating operational bottlenecks and enabling teams to deliver value faster.

This approach is delivered through:

• Resource-Focused Permissions: Permissions are specifically tied to resources, centralizing control over crucial assets rather than dispersing it across extensive user access.
• Distributed Governance: Decision-making is decentralized, allowing resource owners—such as individual teams or departments—to manage access to their respective resources.
• Centralized Framework: Despite the decentralized governance of resources, all oversight remains under a cohesive framework that ensures consistency and maintains comprehensive security and compliance across the entire organization.

This structured yet flexible approach allows Flix to maintain high security and governance standards while empowering teams with the autonomy to manage their data effectively.

Managing a Modular Data Architecture With Conduktor

Flix has implemented a modular data architecture that leverages individual YAML files for each teams Kafka resources. This modularity provides significant advantages:

• Clear Ownership: Clearly identifies the team responsible for resources within a specified namespace.
• Contained Leakage: Limits problems with overly lenient access settings to the affected resource, thereby preventing broader security issues.
• Auditable Changes: Utilizes CI/CD tools like GitLab to track all changes, complete with a merge history, enhancing both accountability and transparency in data management.
• Promotes collaboration: Teams can request access to another teams resources through pull requests. Team owners have the autonomy to approve/reject, without needing to involve a centralized authority.

How it works:

Flix's federated Kafka security management system operates through a combination of open-source technologies and structured CI/CD workflows. This approach ensures efficient, scalable, and secure management of Kafka resources and permissions across multiple teams.

Step by Step:

1. Each team has a Kafka resource-oriented YAML file that lives in the namespace for each team. The file defines the owner, and specifies permissions granted to other teams.
2. Changes to those files trigger CI/CD workflows that validate the configurations through federated governance
3. These files are aggregated and transformed into a format that is readily compatible with Kafka Security Manager (KSM), a Conduktor open-source project that turns the external source of truth into Kafka ACLs.
4. The same files are used to propagate permissions into Conduktor, so that people can interact with, and troubleshoot Kafka.

This streamlined workflow ensures that each team is responsible for the resources they own, while permissions are uniformly applied across the entire data platform. This includes:

• Kafka Access Control Lists (ACLs): Control application access to resources, ensuring that only authorized applications can interact with specific Kafka topics and clusters.
• Conduktor Role-Based Access Control (RBAC): Provide team members with the necessary permissions to manage, monitor, and troubleshoot Kafka resources and data they have access to.

Introducing Conduktor Self-Service

Through our collaboration with Flix, we've developed a new framework to enhance collaboration between Platform and Application teams through the Conduktor platform, optimizing Kafka resource lifecycle management and access requests.

By allowing users to manage both their data and applications independently, Conduktor reduces reliance on central teams and promotes responsible data use with intuitive controls. Automated permissions validation ensures security and compliance, while configurable approval workflows align with your security model. This setup allows platform teams to enforce best practices on Kafka configurations, maintaining consistency and minimizing errors.

Book a demo to learn how Conduktor can help your organization to maximize data use, while keeping control.