Agile Analytics

Introduction

We are quick to consider Agile when we think of front-end development (e.g., web/mobile applications) but less likely when it's data work. After all, the data work is just a necessary part of the front-end development under the covers, right? Well, kind of.

If you have a pure data project with little to no front-end work, then Agile is still relevant (and preferred, in my opinion). You can structure the team and work using Agile principles to get the data work done in an iterative manner just as well as any other software development project. (Keep in mind that data work is another form of software development too. ??)

This article is not intended to be an exhaustive description of all Agile principles, processes, and techniques. That's enough for an entire book (and they already exist). The purpose of this article is to show how Agile is as applicable to data analytics as it is for other software development work. So, let's begin...

What does Agile mean?

You can find a variety of definitions on the Internet, but all those definitions will have the same general theme. And that general theme is that Agile uses iteration to apply the software development lifecycle to smaller, more manageable, chunks of work. Agile methodologies allow teams to deliver increasingly better increments of a working solution that facilitates better feedback loops, refinement, and adjustment. Agile mitigates rework and waste by focusing on several key concepts:

• Minimum Viable Product (MVP) – A version of the product that meets enough of the requirements to be considered usable. MVP is typically used to determine if the team is on the right track and helps to determine next steps, which may be continuing as-is or adjusting things going forward. And ultimately, an MVP is released to the users.
• Product Increment – The improvements made during a Sprint which makes a feature better and potentially shippable. Each Sprint should iterate on and refine features to improve them towards the vision of the final product. Although a product increment may not be deployed, it should be capable of doing so.
• Definition of Done (DoD) – A phrase used to describe the conditions that must be true for an epic, story, product increment, etc. to be considered complete. These conditions are agreed upon across the team, including the client and especially the Product Owner.

There are a few areas of confusion with these concepts and the following information should help to make things clearer:

• MVP is not the same as a Product Increment. A Product Increment is a subset of an MVP. A collection of Product Increments will eventually lead to an MVP.
• MVPs are deployed/used whereas Product Increments may or may not be deployed/used.
• DoD can apply to anything as it is simply a way for the team to define the conditions that determine when something is complete. It is gaining agreement from everyone involved to clearly know when something is done or not.

How does Agile apply to analytics?

By now, you likely see how Agile applies to analytics for a few reasons. Analytics require software development and Agile Scrum is, in my opinion, the best methodology to develop software. And in cases where analytics solutions are in maintenance mode, Agile Kanban is the best methodology. For the purposes of this article, we will focus on Scrum and not worry about the distinction between Scrum and Kanban because they are both Agile methodologies. Besides, you can find lots of resources that compare and contrast each one elsewhere.

Agile everything, not just the code

Agile can be applied to the early parts of the project, before you even get to the coding part. For example, the requirements and design stages can also benefit from Agile. We can iterate through requirements with increasingly greater levels of detail to gain more clarification on what the client needs. And the same is true for design where we can iterate from sketches to mockups to functional products. The power of iteration is fueled by feedback cycles that keep things calibrated between you and the client. The idea is to close the gap between what is needed and what is built, and that is best accomplished by building, showing, and refining something tangible early and often.

The key to success is breaking down the work into bite-sized chunks that can be worked, and completed, in each Sprint (as much as possible). The less work (and technical debt) that carries over into subsequent Sprints, the better. All too often development begins with very little understanding of what needs to be built in the name of “progress,” and developers start slinging code with vague (or missing) requirements. That leads to bad implementations, poor software/data quality, and re-work that shatters expectations with clients, making it harder to recover much less succeed.

Let's take an example ... building a dashboard

In general, the following steps are needed to execute an Agile analytics project:

1. Identify requirements for the solution at the highest level and work your way down from there, decomposing bigger ones into smaller ones. These requirements should resonate with the stakeholders/users.
2. Create designs that begin with basic shapes that evolve into high-fidelity mockups. These designs should clearly connect to the requirements and give an idea of how things will look.
3. Build a prototype that demonstrates how the solution will work. This prototype should bring the design to life, showing how the requirements are met.
4. Develop the prototype into a fully-functional solution that is capable of being deployed as an MVP (see MVP definition above). This solution is what will go live once tested and approved.
5. Test the solution to fix bugs and refine features. This testing should include users that can vouch for its value and accuracy.
6. Deploy the solution into a Production environment that makes it available to the user community.
7. Sustain the solution based on feedback from operational use.

You can easily see how these steps also apply to things like data integration, database development, reports, data warehousing, etc. Here are some thoughts when building dashboards as an example:

1. Requirements: Meet with stakeholders and users to determine business questions that can be answered with metrics, quickly and visually.
2. Design: Create mockups that show which visuals will be used to answer which questions and how metrics will be displayed. These mockups can be done using sticky notes, paper/pencil sketches, whiteboards, or mockup tools (like Balsamiq or even PowerPoint). The idea is to communicate the expected interface of the dashboard quickly and adjust at the speed of thought (before developing anything).
3. Prototype: Build a semi-functional dashboard using the tool of choice (e.g., Power BI, Tableau, Qlik) to show how the dashboard will look and work. Be sure to focus on the UI/UX (user interface/user experience) aspects since that’s what will attract and retain a user’s attention. Also feel free to use a representative dataset in a spreadsheet to produce something quicker. (The real data integration work can be done later.)
4. Develop: Using the prototype as a starting point, build out the full functionality that is expected to be deployed to the user community. This stage is where the Agile Scrum methodology shines because that’s where you and the Product Owner/users are able to refine the actual dashboard.
5. Test: Release the dashboard to select users to perform user acceptance testing (UAT), making sure it performs as expected. That includes visual appeal, performance, and above all … accuracy of the data!
6. Deploy: Publish the dashboard for operational use so users can make business decisions based on well-integrated, and accurate, data.
7. Sustain: Maintain the dashboard in terms of software upgrades, feature enhancements, and bug fixes to keep it operational.

And keep in mind that although these steps (and dashboard example) are presented as a list in this article, they are done in an iterative fashion. Your team will iterate through each step (1-7) with increasingly more refinement each Sprint, getting closer and closer to what is needed for the prototype/solution with little to no re-work.

The image below is a common graphic depicting the Agile Scrum process and is available at https://www.scrum.org/learning-series/what-is-scrum.

Sprint Cadence

The big question that everyone asks is “How long should a Sprint be?” Well, it depends, but generally anywhere from 1-4 weeks is an acceptable range. Smaller or newer solutions may benefit from 1–2-week Sprints whereas bigger or mature solutions may benefit more from 3–4-week Sprints. The decision depends on the team’s capabilities, Sprint velocity, and client expectations.

For newer projects, I recommend starting with 1-week Sprints to get through the prototyping stage. Once the prototype is built and you are ready to build it out into a fully-functional solution, I recommend using 2-week Sprints. Then after the solution goes live, I would re-evaluate the 2-week Sprints and stick to them if they’re working; otherwise, consider 3-week Sprints. Although 4-week Sprints are acceptable, you start getting into the territory of allowing stories to get too big (vague) and not finishing committed work. Never, ever go longer than 4 weeks for Sprints! (I came across one project that used 6-month Sprints! Yes, you heard me -- 6 months; that’s a very slow walk, not a Sprint. ??)

Scrum Ceremonies

Agile Scrum “ceremonies” include a variety of events that keep the team moving forward. The most common ceremonial events are shown in the table below.

The Sprint Review/Planning/Retrospective can be done on the same day and the time needed will vary depending upon team size, attendees, Sprint readiness, backlog health, and size/complexity of the solution. Generally speaking, you’ll want to carve out up to a half day for these ceremonies. Less time is needed if you regularly complete all work in the Sprints with high confidence and have a well-groomed backlog, more time if you don’t.

Sound Familiar?

After getting this far, you may be thinking, "Wait a minute, that just sounds like Agile for software development." If so, you are correct! Oftentimes, people don't think of applying Agile methodologies to data work but it's equally useful. You can apply Agile methodologies to anything that requires developing solutions to solve business problems, including predictive algorithms and artificial intelligence/machine learning (AI/ML) models.

I've seen, and used, Agile Scrum with great success on many different projects ranging from a single dashboard as a proof of concept (using 1-week Sprints) to a mission-critical data mining project (with no UI/UX work) to a full-scale enterprise reporting and analytics platform with many [complex] integration points (using 3-week Sprints).

Final Thoughts

Some people have mixed feelings about Agile methodologies, especially Scrum. Everyone has a different experience and opinion but from what I’ve seen over the years, a lot of projects say they are using Agile when they are not; they are either using Waterfall or WAgile (i.e., Waterfall/Agile hybrid). And those most critical of Agile are not grooming backlogs, treating daily Scrums as long status meetings, using story points as an absolute measure of time (e.g., hours or days) instead of relative level of effort, and other misuses of Agile principles.

If you've tried Agile Scrum in the past and had a bad experience, I encourage you to learn more and try again. If you haven't used Agile Scrum yet, learn more and try it out on a smaller project with a smaller team. In either case, I recommend having an experienced Scrum Master run your project so you can see what it looks like and how it works.

Helpful Resources

Here are some resources that may help you get you started (or re-started) in the right direction:

• Mountain Goat Software
• Scrum Alliance
• Scrum.org