Maximizing Data Opportunities through Data Sprints: An Analytics Framework
In earlier newsletters I present a six-stage framework to structure the work a data science team performs and five techniques for performing the work in intense, two-week cycles called "sprints." These techniques go a long way to making the data science team productive.
In this newsletter, you'll see several pitfalls that commonly undermine the data science team's efforts, and I provide guidance on how to avoid these pitfalls. Generally, your data science team needs to squash anything that limits their mission to something other than exploration and discovery.
Change the Organization's Mindset
Many organizations create data science teams and then essentially tie their hands, preventing them from truly exploring the data. Much less frequently, organizations provide their data science teams with too much freedom, so the teams end up chasing data and questions that are irrelevant to the organization's success or getting so wrapped up in routine chores, such as managing the data warehouse, that they fail to produce anything of value. In most organizations, though, the problem involves a strict hierarchy that tries to control what the data science team does, and that is a formula for failure.
Prior to installing a data science team, an organization often must change its mindset and values. It must embrace a spirit of creativity and innovation, especially in respect to its data science team. When the team is doing what it should be doing, it is learning and helping the organization learn. It is discovering what the organization doesn’t know. Attempts to micro-manage the team run counter to its mission.
However, the data science team does need to deliver value. It should serve the needs of the organization. Data science teams can achieve that goal by being highly service-oriented and by collaborating with everyone across the organization to get their questions answered, help them overcome any challenges they face, and inform their decisions.
Work without Objectives
Most organizations still view work as a series of goals and objectives. They invest a great deal of time, money, and effort on planning, management, and compliance. Teams are expected to set goals in advance, formulate plans to meet those goals, execute their plans, and deliver the promised outcomes. While that approach works well for most teams, it is counterproductive for data science teams whose mission it is to explore and innovate. Data science teams need to follow the data and the questions, and they cannot shift direction if their path is carved in stone.
If you're on a data science team, you may feel as though your team is trying to hit a constantly moving target. Every sprint introduces new questions that may lead the team in a different direction. Sometimes, the team may not even know what the moving target is. The team may be looking for patterns in the data that reveal new targets. By working without objectives, the team has the flexibility it needs to let its curiosity and the data determine the outcomes.
Take Advantage of Serendipity
Serendipity is a happy happenstance, such as striking up a conversation with the CEO of Microsoft at a Mariners game and having him offer you a job on the spot. It is an odd concept in the world of business, where strategy, goals, objectives, and planning are enshrined as the essential components of success.
However, more and more evidence points to the advantages of serendipity over goal setting and planning. One of the best books on the topic is Why Greatness Cannot Be Planned: The Myth of the Objective, by Ken Stanley and Joel Lehman. According to the authors, “Objectives actually become obstacles towards more exciting achievements, like those involving discovery, creativity, invention, or innovation.”
Data science teams are wise to capitalize on serendipity. For example, if a team member sees something unexpected and intriguing in the data the team is analyzing, the team needs to follow up on that discovery. You don't want your team focused on objectives at the expense of overlooking a groundbreaking discovery. Professor Stanley calls these “stepping-stones” — interesting things that eventually lead to insights. If you ignore them, you are likely to miss key discoveries.
Deliver Practical Knowledge and Insights
When you're working on a data science team, it's easy to get so caught up in the data, analysis, exploration, and discovery that you lose sight of the organization's needs. Driven by innate curiosity to follow wherever the data leads, the team forgets that others in the organization are relying on it to deliver knowledge and insight that guide strategy and inform decision-making. Every couple weeks, the team delivers its reports or presentations, which the team finds fascinating but which leave everyone else in the organization wondering "So what?" or "Who cares?"
To avoid this pitfall, the data science team must engage, to some degree, in guided exploration. Three tools in particular are helpful for structuring and guiding the data team's work:
Focus on Exploration over Routine Work
By its very nature, routine is repetitive, and it can become hypnotic, lulling you into a complacency that prevents you from noticing the wonderful world that surrounds you. The same is true for a data science team. It can become so wrapped up in capturing, cleaning, and consolidating data and creating data visualizations that it loses its sense of adventure. It falls into a rut and stops asking interesting questions. When looking at the data, it may not even notice an intriguing fact that's staring right back at them.
To avoid this pitfall, try the following techniques:
Keep in mind that your data science team should be committed to exploration, discovery, and innovation that's relevant to the organization's needs. If the team works toward achieving that mission, it will be less susceptible to the most common pitfalls.
Frequently Asked Questions
What is a data sprint in the context of maximizing data opportunities?
A data sprint is an intense, focused period, typically lasting around 10 days, where a team of data scientists, analysts, and business experts collaborate on a data project. The goal is to develop a prototype of a data solution that addresses a specific business need or opportunity.
How does a data sprint aid in digital and data transformation?
Data sprints drive digital and data transformation by rapidly identifying and testing data solutions, which helps organizations adapt to new technologies and methodologies more swiftly. This approach allows for timely feedback and iterative improvement, ensuring the data solutions developed are effective and aligned with business goals.
What are the key components of a successful data sprint?
The key components of a successful data sprint include a well-defined project scope, a diverse and skilled data team, access to relevant data sets, clear goals, and a collaborative work environment. Additionally, utilizing suitable data science and AI technologies is essential for creating effective solutions.
How do you ensure data privacy during a data sprint?
Data privacy is ensured during a data sprint by adhering to strict privacy policies, implementing secure data handling practices, and using anonymized or synthetic data when necessary. Compliance with regulatory standards and thorough training of all team members on privacy best practices are also critical.
Can a data sprint be beneficial for startups?
Yes, data sprints can be particularly beneficial for startups. They provide a structured yet flexible approach to rapidly develop and prototype data solutions, helping startups quickly validate their business ideas, improve their data products, and gain a competitive edge in the market.
What role do data scientists play in a data sprint?
Data scientists are integral to a data sprint. They use their expertise in data science and AI to analyze data, build predictive models, and develop solutions. Their skills in handling large datasets, programming in languages like SQL, and applying machine learning techniques are crucial for the success of the sprint.
How can data sprints foster innovation within an organization?
Data sprints foster innovation by bringing together cross-functional teams to focus on solving specific problems in a condensed timeframe. This intense collaboration and quick iteration surface new insights and ideas, leading to the development of innovative data-driven solutions that can transform business processes and models.
What types of projects are suitable for data sprints?
Data sprints are suitable for a wide range of projects, including developing new data products, improving existing data solutions, testing the feasibility of digital transformation initiatives, and exploring new AI and machine learning applications. The key is that the project should have a specific goal and be feasible to prototype within the sprint period.
How does a team prepare for a data sprint?
Preparing for a data sprint involves several steps: defining clear objectives, gathering the necessary data, assembling a skilled team, setting up the technology infrastructure, and planning the sprint timeline. Holding a kickoff workshop to align on goals and expectations is also beneficial for setting the stage for success.
What is the impact of data sprints on long-term data strategy?
Data sprints have a significant impact on long-term data strategy by accelerating the development and validation of data solutions. They enable organizations to quickly iterate and refine their approaches, informing more comprehensive and effective long-term strategies that are data-driven and adaptable to changing business needs.
This is my weekly newsletter that I call The Deep End because I want to go deeper than results you’ll see from searches or AI, incorporating insights from the history of data and data science. Each week I’ll go deep to explain a topic that’s relevant to people who work with technology. I’ll be posting about artificial intelligence, data science, and data ethics.?
This newsletter is 100% human written ?? (* aside from a quick run through grammar and spell check).
More Sources