Scrum project management in Data Science: a review and pitfalls

In a?previous Article, I shared a hybrid?CRISP-DM?and?Agile-Scrum?methodology for data science project management.?Scrum is the most popular Agile methodology that nowadays used for product development projects in various industries and beyond its origin in software industry. A review of Scrum and the pitfalls is explained in the current article.

An overview of the Scrum's empirical?process is shown in the above picture and described below:

• A Scrum Master (SM) is facilitating the Scrum project management and coaches the Scrum team, to ensure the team lives Agile values and principles and follows the Scrum processes and practices including but not limited to effective communication, self-organizing, reporting, and eliminating externa blockers.
• A Product Owner (PO), who represents the stakeholders and users, creates a product backlog which is a list of prioritized requirements and features from the user. PO works with the Scrum development team, maintains, grooms the backlog, and promotes the product during and beyond the development phase.
• The Scrum team is a group of multi-disciplinary collaborators, typically less than 10 individuals, who works with PO and SM toward completing project and delivering the product. They deliver a part of the product, called increment, in time-boxed iterations called Sprint. The Sprint cycle involves four events (aka ceremonies): Sprint Planning, Daily Standup, Sprint Review, and Sprint Retrospective.
• The Scrum team conducts Sprint Planning with PO facilitated by SM, in which, creates a sprint backlog for the high-priority subset of items in the backlog selected based on the team capacity and chosen sprint duration (usually 3 weeks), breakdowns the items into more details and tasks (aka Story Points), and estimates the effort required for completion of tasks.
• The SM arranges and conducts a brief daily Scrum meeting (aka Daily Standup), where each team member shares the progress of the last day, impediments (if any) and plan for today.
• The Scrum team together with SM, PO, and stakeholders of the product, conduct a meeting (aka Sprint Review) at the end of each sprint, to present and discuss the progress and completion of the sprint backlog, and assess the increment achieved in the sprint is fulfilling the success criteria and requirements.
• Sprint Retrospective is a meeting between SM and Scrum team on process and team. The area of strength and improvements regarding people and collaborations, processes, and practices in the sprint are discussed and necessary actions are taken to reflect on the team performance, find deviations, and suggest scope for improvement.

While Scrum process has been successfully adopted and implemented by many product development teams around the world and across industries who are undergoing Agile transformation. Scrum is not the best choice for project development methodology everywhere; it is more suitable for projects that are not predictable. The requirements and design of data science and advanced analytics projects are changing constantly and that makes Scrum a desirable choice for such projects.

Despite the suitability of Scrum for data science and advanced analytics projects, proper adoption, and implantation of Scrum in larger teams and organizations is challenging and can face common pitfalls as below:

• Resistance to change in team and organization's culture to adopt and implement a new process. Scrum is easy to explain and understand, but sometimes can be difficult to implement, especially in process bound organization and teams which cannot focus on the project. The resistance could be due to lack of knowledge, dislike of change, fearing the accountability by the members, etc. These lead to misunderstanding, miscommunication, mistrust and sometimes no desire to follow the process by the members.
• There are many project management methodologies and processes, which might fit various kinds of projects. A general rule of thumb is that Scrum works best on projects that have a prominent level of uncertainty that are well-suited for an iterative and incremental development approach based on an empirical process control model. That means Scrum is not a good choice for projects with a relatively low level of uncertainty.?
• Since sprints are time-boxed in Scrum, the Scrum team is expected to take on only so many tasks and story points that match its capacity and it can deliver in a sprint. Although in Agile methodologies (including Scrum) the change is inevitable and they welcome change in the requirements even in later stage of the project, however, if the unplanned work is not managed appropriately, it poses a significant risk to the deliverables of the sprint. New requirements and unplanned work need to be estimated, then swapped with user stories in current sprint, or postponed to the next sprint, or even the new phase (MVP), depending on the priority, complexity, and size of the work.
• With the lack of technical or Agile maturity in the Scrum team, the backlog items and story points may not be sized properly. Then the team either commits to more than it can deliver resulting in unmet sprint goal or under-commits resulting in lost productivity. In addition, if the team is not self-organized and doesn't have cross-trained members, not all the members are contributing to Scrum ceremonies and there will be risk of work waiting on individuals.
• The lack of a proactive and focused product owner to create and groom product backlog and review the increment at the end of the sprint could potentially result in re-work downstream which is both expensive and increases overall delivery time. The success criteria (aka Definition of Done) are required to be defined and agreed between PO and the Scrum team properly.

Scrum is popular and an excellent choice for AI and data science projects, however we always need to remind ourselves of the fact that Scrum is easy to learn but extremely difficult to master. Some of the Scrum pitfalls are unavoidable to new adaptors and adopting Scrum is a never-ending process, especially in the larger teams. However, Scrum is all about being consistent with the framework while, at the same time, support, learning, and flexibility is provided to the Scrum teams.