How Data Folks Can Ace Work Effort Estimates

What do I mean by “data folks?” That’s my shorthand for “data analysts, data scientists, and data engineers.” I didn’t want to put all those words in the title, so I’ll lump you all together as data folks. Hope that’s okay.

Most of the writing I’ve done so far has been meant for leaders of data teams. This article is a bit different. It’s meant for individual contributors, especially people who are early in their careers as data folks. If you’re in grad school now or in the first few years of your professional career, this one’s for you!

When I was just starting my own career, I wish someone had helped me understand the importance of accurate work effort estimates and given me ideas for ways to overcome common challenges. That’s what this article is about.

Why It’s Hard to Get Right, and Extra-Hard for Data Folks

Key factors that impact estimation accuracy include task complexity, pre-existing experience, and the clarity (or ambiguity) of the work you’re setting out to tackle. Data folks face unique challenges with work effort estimation due to various issues, including:

• Uncertainty in data quality and availability: Your project may involve significant cleaning, transforming, or even sourcing of new data, all of which complicate estimation.
• Complexity of data pipelines and infra: Your work may require getting up to speed on new tools and tech, which takes time to learn and implement. Unexpected tech issues can also crop up.
• Iterative nature of data work: Science-minded data folks will often take work that involves multiple rounds of analysis, model building, and refinement, making it hard to pin down a due date.
• Evolving requirements: Stakeholders may agree to certain requirements and then make refinements later on based on new insights. When scope flexes, it may throw off your initial estimates.
• Tech debt: This nuisance is often overlooked during work estimation. Your work may involve clearing old tech debt or avoiding creating new tech debt. This is key for ensuring long-term sustainability but often hard to quantify upfront.

Common Pitfalls

Data folks often want to please others by promising quick turnarounds, but this can backfire if we overlook factors that could slow us down. Viewing a project as a single large task can cause us to miss important details that might extend our estimates. Especially early in our careers or in a new job, we may lack the historical knowledge to anticipate problems. Also, we might give overly optimistic estimates without accounting for unforeseen issues. If our estimates turn out to be flawed, we can get stressed out when we realize that the deadline is upon us and we’re nowhere close to done.

Ready for some advice? Here are five tips to avoid common pitfalls:

Tip #1: Break Down Tasks into Smaller Parts

Improve estimation accuracy by breaking down tasks into manageable pieces. Estimates are more accurate when you have clear requirements and a well-defined scope. Assess this at the project's start, and gather any missing information to make an informed estimate. It will serve you well later. Data folks often underestimate the time needed for testing and QA, so consider these steps individually and be realistic about how long they will take.

Tip #2: Use Historical Data and Past Experiences

Reflect on past projects to inform your current estimates. While estimating new tasks is always challenging, building a repertoire of past estimates helps. Recognize common patterns and use them to your advantage.

Tip #3: Add Buffer Time

Don’t shy away from adding buffer to your estimates. Factor in time for unexpected issues and troubleshooting. Consider the availability of teammates, as their support might impact the timeline. A trick I still use is to have two due dates: one shared with stakeholders (including buffer time) and another more aggressive one that I keep to myself (shhh!).

Tip #4: Seek Input from Others

If you’re uncertain about your estimate, gather input from a peer, mentor, or manager. Different perspectives can refine your estimates. It’s especially helpful to get input from someone with experience in similar projects.

Tip #5: Make Adjustments if You Must

Monitor your due dates and don’t hesitate to adjust them based on progress and new information. Revising a due date, adjusting scope, or asking for help isn’t a failure. It’s proactive management. It's better to address issues early rather than ask for an extension at the last possible minute. Be realistic about your capacity to take on new work, and communicate clearly with stakeholders about how new tasks will impact existing commitments.

Conclusion

No one gets work effort estimates right 100% of the time. We make mistakes, learn from them, and use what we learn to improve our future estimates. Developing strong estimation skills is important for all data folks. Your leaders will appreciate it, your stakeholders will appreciate it, and you can build trust and influence by approaching these exercises with confidence.

Credits

This article was inspired by a recent conversation that I had with my friend and former colleague Rawi Nanakul . He and I had a far-ranging chat on the topic of prioritization. My last article, Data Team Prioritization: Balancing Foundational vs. Urgent Work , was also a product of our discussion. These days Rawi runs a coaching business for tech professionals with ADHD called Tech Atypically (his newsletter is great).