Data Team Prioritization: Balancing Foundational vs. Urgent Work

Balancing long-term projects with urgent tasks is an important capability for data teams. I have found that this muscle is built over a period of sustained operation. Initially, a new data team’s work may consist solely of ad-hoc tasks, but over time, it naturally separates out into long-term vs. short-term work. There is often a struggle to prioritize long-term foundational work, especially when it doesn't yield immediate, visible results for business stakeholders. Having a deliberate approach to prioritization ensures that the team can spend the right amount of time on each type of work. Here’s how to achieve this balance.

Understanding Foundational Work

Foundational work includes a data team’s behind-the-scenes efforts such as infrastructure development, standardizing KPI definitions, and improving data quality (see my article on data quality here). While less visible to business stakeholders, foundational work is important to the data team’s long-term success. Simplifying infrastructure improves team efficiency and reduces maintenance burdens associated with tech debt. Standardizing KPI definitions minimizes the risk of human errors and confusion, especially among business leaders who might misinterpret similar but distinct metrics. Data quality, well, you just can’t avoid it, but it works out better for everyone involved if you face it head-on rather than in a reactive fashion.

Necessities and Challenges of Urgent Work

If you do nothing to control it, short-term tasks can dominate your team’s capacity. Handling “quick questions” from stakeholders (which are rarely quick - see my article on this topic here) and addressing truly urgent requests, such as exec inquiries, deadline-driven decisions, or analysis of unexpected trends, is inevitable. However, if you allow short-term work to take over, it disrupts planned work and diverts resources. The challenge is to limit the extent to which urgent work cuts into your ability to do foundational work.

Using Time Management to Balance Priorities

Set firm boundaries on the number of hours devoted to urgent work. Running sprints to manage short-term tasks will allow you to reassess and adjust priorities every few weeks. Reserve a certain agreed-upon block of hours in each sprint for foundational work to prevent it from being overtaken by urgent tasks. In larger teams, an “on-call” rotation can help manage distractions like investigating and fixing broken pipelines, allowing other team members to focus on planned work.

Communication and Stakeholder Management

Stakeholders may resist the idea of foundational work taking time away from producing results that are more immediately visible to them. However, setting clear expectations about the data team’s bandwidth for foundational work can help them understand its importance. Include foundational work in your strategic plans, represent it in team goals, and celebrate foundational achievements alongside business-facing successes whenever you review your team’s progress with execs.

Conclusion

Balancing foundational work with short-term tasks takes deliberate planning, effective time management, and clear communication with stakeholders. By prioritizing both types of work appropriately, data teams can ensure long-term success while meeting immediate needs. This balance leads to a more efficient and effective team capable of delivering sustained value to the business.

Credits

This article was inspired by a recent conversation that I had with my friend and former colleague Rawi Nanakul . These days Rawi runs a coaching business for tech professionals with ADHD called Tech Atypically (his newsletter is great). He also happens to be on the leadership team of a whiskey business. Here's to interesting people!