The Convergence of Data & Software Engineering in the Age of AI

The patois of data teams has become a dialect of modern engineering teams because the commonalities in the stack.

This convergence signals how far data teams have evolved into core engineering teams. Machine learning’s demand for data has accelerated this movement because AI needs data to function.

Data teams receive tickets from their internal customers & develop data products that serve both internal & external users, much like a classic product management & engineering team.

Data teams architect their systems in a modular way, paralleling the microservices movement in software design.

Data contracts express the commitments data teams make to others in the company about data freshness, format, & consistency - again drawing parallels to the service-level agreements in core engineering.

Security systems govern access to databases akin to secrets management & identity access management solutions do in the cloud.

To identify issues in production systems, both types of engineers leverage observability tools for anomaly detection & responding to incidents.

Twenty years ago, the data team meant managing centralized BI & producing analysis in Excel.

But today, data teams are engineering teams in their own right, with specialized tools for their particular domain.

They are central to product development & operations in technology companies. Their evolution into full-fledged engineering teams enables more seamless collaboration with software developers.