Why Agile does not work for you?

Agile has gained immense popularity which is proven by the fact that many organizations across the globe have already adopted or are in the process of adopting it. The goal is to reap the benefits of agile - Speed (time to market), Improved customer satisfaction, Better quality, Reduced risk, Higher return on investment (ROI), Continuous improvement, Continuous delivery of value, Quick response to business needs, and so on. The reality is that the success rate of these Agile transformations is disappointing and the frustration is only increasing making people think that Agile does not work and that they are not getting the bang for the buck. In some companies, the success is seen only on paper - reports show that their Agile maturity has improved a lot but they are facing product quality issues or their releases are not meeting customer expectations. Why Agile is not working??

There are many agile methodologies that are being adopted by companies. Scrum and Kanban are prominent among others like XP, SAFe (Scaled Agile Framework), LeSS (Large Scale Scrum), FDD (Feature Driven Development) etc. In this article, let us try to understand the factors and see why Agile doesn't work for your benefit. There are several factors that influence Agile adoption and maturity. These factors can be broadly classified into 3 groups -

1. Foundational factors - These are MUST items. Without having these fundamental elements in place you are bound to fail irrespective of Agile or any other methodology you follow. Ex: Employee engagement, Good organization culture, Having valid requirements, Finalizing a design approach that works both in the near term and long term, Writing quality code by following proper coding standards, Conducting code reviews, Writing proper test cases covering all the possible scenarios, Having good test automation, Maintaining traceability, Taking care of NFRs (Non-Functional Requirements), Having the right skills & resources available in the team, Reducing throw away work and so on. Apart from these general elements, the foundation of any agile framework is the Agile Manifesto and its 12 principles . Also note that Agile is based on an Empirical process for which the 3 pillars are Transparency, Inspection, and Adaptation.
2. Framework factors - Based on the Agile framework you choose to adopt, including the selection of a suitable Agile framework itself, it recommends some mandatory practices and allows some customizations. It also comes with a set of values. While you need to follow the mandatory practices, you need to take a calculated and cautious approach to adopting guidelines and customizations. For example, Scrum defines Roles, Events, and Artifacts and requires you to deliver a product increment (PI) every sprint. Kanban mandates the use of WIP (work in progress) limits and a continuous flow of work. Other items you want to consider are Team size, Estimations, Agile metrics to measure and monitor, Breaking down bigger requirements into smaller chunks, etc.
3. Best practices - These may be defined by the Agile framework or generally considered best practices as per the industry you operate in. To continuously improve, it is strongly recommended to adopt the best practices and every agile framework allows you to embrace them. For example, writing User stories is generally followed by all the Agile teams while it is actually defined by the Agile XP (eXtreme Programming) framework. Similarly, other best practices like DoR (Definition of Ready), DoD (Definition of Done), Acceptance Criteria, 4Cs technique (Card, Conversation, Confirmation, and Context), INVEST principle (Independent, Negotiable, Valuable, Estimable, Small, and Testable), DevOps, DevSecOps, NoOps, FinOps (Financial operations - used to drive financial accountability and to promote a culture of financial responsibility), SRE (Site Reliability Engineering), Pair programming, TDD (Shift left approach with Test Driven Development), BDD (Behavior Driven Development), Conducting RCA (Root Cause Analysis), Following LSD (Lean Software Development) principles, Pair programming, Logging standards, Work estimates etc. should be looked at and adopted as possible.

As we can see, there are a number of factors that can influence and impact your agile adoption. While going through each one of them is beyond the scope of this article, let us look at some of the common pitfalls (in no particular order).

• Agile framework selection: This is where your agile execution begins. You need to look at the nature of your work, business scenario, technologies, team capacity, etc., and choose the correct agile framework that is suitable for you. It takes a good amount of effort, about a few months to start getting the desired results. Don't try to switch to a different framework or select the wrong framework just to look good in the near term. Instead, you should make use of the mandatory practices of the framework to help identify and address the issues. For example, I've seen software development teams adopting Kanban just because they are unable to achieve product increments every sprint with Scrum. They simply switched to Kanban as it doesn't have any iterations or sprints defined to deliver regularly. Though some of the metrics were looking good in the near term, they could not get many benefits of Agile.
• Story points: Probably the most misused and misunderstood concept across all agile teams. Everyone has their own opinion and every team or management understands in their own way, in terms of where and how to use them. I don't want to go into the details here, but you can find many articles online (scrumalliance , industriallogic , betterprogramming, and many more). The common issues I've observed are - teams converting story points into time (e.g. 1 story point = 3 hours), comparing teams' performance and efficiency using their velocity (e.g. Team1 is doing better compared to Team2 as their velocity is more), completely relying on and using story point estimates for actual development even after achieving DoR, leadership forcing the teams to improve their velocity (Number of story points completed in a given sprint or release) indefinitely.
• Estimations: These are essential for planning, tracking, and continuous improvement. Many questions arise - Are you doing work estimates as part of your planning? How you are doing estimates - hourly or story point or both? Are you tracking estimated time vs. actual time? At what requirement level(s) (aka theme, epic, feature, story, task, bug) you are doing? Who is doing estimates? How do you know whether they are realistic or skewed? What is the overall process followed? Few agile teams simply avoid estimates and think it is a waste of time without realizing that "well beginning is half done". Some of the issues I've noticed in my experience are - Estimates provided by non-engineering members like the Scrum Master or Product owner or manager, Random numbers assigned without any reasoning or justification (e.g. User stories assigned story points 48, 76, 112, etc.), Not using estimates and/or not tracking them, Providing wrong estimates just to look good (e.g. Better velocity, improve cycle time, etc.)

Vague plans deliver Vague results.

• Agile Tool: Implementing any Agile framework is not a simple exercise. It involves adopting many framework factors as discussed above, including the foundational factors and best practices. You definitely need one or more tools to simplify, automate, and measure along the way for continuous improvement. Having the right tool to simplify and automate your job is essential. I've come across seasoned Agile coaches and trainers, who just make fun of Jira and other tools but fail to provide a better alternative. Manual processes using Excel files, Word documents, Confluence, Sharepoint, etc. are not going to work for sure, though they can be used as supplements. Luckily, most of the Agile tools available in the market are good at a certain level and allow you to customize further as per your requirements. Go for it.

It is essential to have good tools, but it is also essential that the tools should be used in the right way. - Wallace D Wattles

• Cross-functional skills: This is another misunderstood concept after Story points. Agile recommends having all the required skills to successfully deliver within the same team. The "functional" here merely refers to different functional areas you have in your business (for software development teams, this could mean requirement analysis, coding, testing, documentation, deployment, and support). The commonly used jargon here is T-shaped skills or V-shaped skills. Unfortunately, this is understood in unrealistic ways where everyone is expected to know/learn/work on everything, like Multi-Technical skills (team expected to work on all programming languages, frameworks, libraries, frontend technologies, databases - RDBMS and NoSQL, cloud - any vendor, AI/ML, Big Data, etc. based on the requirements) OR Multi-Application/Product skills (team expected to work on any product(s)/application(s) based on the requirements - think of different applications you have currently in your organization) OR Multi-Domain/Business skills (team expected to work on any domain - think of different line of businesses you have in your company including domains, sub-domains). As we know, it is not possible for everyone to know everything and everyone in a team is not equal in terms of their skills & and experience levels. Trying to force this approach will only lead to quality issues, missing timelines, increased attrition, etc. In fact, Agile recommends having dedicated and long-term teams focused on specific features or applications.
• Casual approach: As agile methodologies have been in practice for a few years now, some of the principles, and practices are familiar to people. Some people just use jargon like stand-up, story point, retrospective, velocity, cycle time, self-organized teams, etc. without a deep understanding of the principles behind the same. Agile is not just a set of jargon and events. You need a focused and serious approach for its adoption at all levels to be successful.

Agile is simple to understand but difficult to master.

• Continue to follow the old methods: Having just basics in place is not enough to adopt agile unless you follow them religiously and improve continuously. Some teams just schedule the required agile ceremonies and even have the required roles in place e.g.: Scrum Master, Product Owner, Kanban lead, etc. But they will continue to follow the same old practices and manual processes. We can’t expect to get new and improved results by doing the same things and just by chanting “We are doing scrum, We are agile”. We will still have the same issues/problems we had earlier. We need to understand and then adopt. Practicing anything without understanding doesn’t help, no matter how long you practice it.

Agile is just not an old wine in a new bottle.

• Team sizing: The recommended Agile Team size is 7 +/- 2. This is true irrespective of the Agile methodology you follow to reduce operational overhead and to help in better planning, tracking, collaboration, and delivery. Just because the framework, does not recommend or mandate a team size, you should not misuse it. I've seen teams intentionally adopting Kanban to have a flexible team size to implement their own agenda - one team has 20 resources and the other team with only 3 resources.
• Metrics and Reporting: Metrics really help you to understand how your agile teams are doing - what is working and what needs to be improved. You need to focus on relevant metrics, measure them with the right data, and use them in meaningful ways for continuous improvement. You should have a clear purpose and understanding of what/why/how you are measuring and should be able to clearly articulate the same at all levels. Ensuring data quality is very important and you should also have governance practices to avoid/detect data manipulation. Remember, what gets measured gets managed. Some of the issues I've observed are - Leadership focus on metrics making the team take shortcuts or manipulate them, Wrong usage of metrics data, Inconsistent understanding of metrics at different levels, too many metrics even at a micro level, manual processes to calculate metrics and generate reports, Reports showing high-level agile maturity while the release quality is sub-par, Cycle time (the time it takes for the work from start to finish) is 8 weeks and the team not doing any estimates (imagine breaking down the work and planning for 8 weeks all in your brain), Work items where more than 1 month effort is spent to implement and later got discorded are no where shown/visible in the reports, Agile maturity is not measured holistically, Teams not following some Agile events are not highlighted in the report, etc.

Do not use Agile metrics for performance management. They should be used for learning, innovation, to support and for continuous improvement.

• Organization politics: We cannot ignore this while implementing any initiative - It is not about executing one or few individual's agenda but working on a common goal and continuous improvement. Leadership should proactively address any issues, gaps, or concerns as and when they arise, instead of avoiding or compromising. In some cases, the time will make matters worse. So, it is important to act quickly. Providing visibility to all the stakeholders, and setting up regular review and feedback calls will help to make sure everyone is aligned well. I've worked with teams where - all stakeholders had visibility to the backlog, everyone had access to the agile tools, all were invited for planning and demo events, and everyone was encouraged to provide continuous feedback in terms of Jira updates or emails or in-person discussions, retrospective items were maintained centrally, metrics & reports were visible to all. They were able to achieve success and continuously improve. There are other cases where the leadership simply kept quiet or acted in an unfair manner without any justification. For example, in one of the Agile transformation projects I've handled, I heard that all the stakeholders were not invited to the sprint review (demo) calls and in some cases, the demo calls were not even scheduled. Agile promotes transparency and visibility, and demo calls are an excellent opportunity for the stakeholders to review the work, and provide early feedback. This will also help the teams to fix any issues before planning the actual deployment to production. When this issue was raised, we got different responses - It will take time or we will start as a POC (Proof of concept) with one team and later expand to others when it works or the management team should not join the demo calls. Not entertaining any discussions or action when a few metrics were looking bad for some teams was also clearly visible in some of my Agile engagements. Another classic example is when the employee opposes a decision and cautions in the beginning regarding a decision/initiative. However, the leadership uses their authority to overwrite and mandate it. Later when they don't get the desired results, they penalize the employee. No wonder, why agile does not work for them.

Management is doing things right; Leadership is doing the right things. - Peter Drucker

• Self-organized teams: This should not be taken in literal meaning and is a myth until your team becomes really mature. Self-organized or self-managed teams are expected to plan their own work, and choose how best to accomplish their work, rather than being directed by others outside the team. It takes at least a few months to years for any Agile team to achieve this state. Until then, agile coaches, and management teams should closely work with the team and guide as required till they become self-organized teams. Someone might argue - let the team manage on their own from the beginning, they may fail, learn from their mistakes, and improve. This approach will not work and is like planning for your failure. Also, you will not be making use of the experience and knowledge of your management team for proactive risk management. Agile coaches cannot be everywhere and will not work at all levels. Sometimes, I hear statements like - Team has decided not to break down the work and estimate, Team has decided to manage their test cases in Excel files independently, Team has decided not to document DoD, Team has decided not to perform code reviews, It is the team decision not to conduct demo calls, etc. When questioned, they simply say that the management should not get into team decisions as per the "self-organized" teams concept of Agile, and surprisingly all this is approved by the Scrum master or Kanban lead and the product owner. What a pity??
• Decision making: With small and dedicated teams, Agile should allow you to make decisions quickly. Instead of taking this opportunity, sometimes, we hear that "Agile transformation is a journey and will take time" when you ask questions about the lack of continuous progress. While there is some truth to it as with any other new initiative, several things are in our control and can be adopted immediately. For example, using a consistent format for user stories, writing acceptance criteria, backlog maintenance, estimations, DoR, DoD, sprint or release commitment, regular updates in the tool, capturing and tracking retrospective notes, approach to breaking down bigger requirements into smaller ones, agreeing on a standard cadence for agile events, test case management, automation planning, etc. should not take months. Also, this can be dynamic - the team should continuously revisit to understand what is working and what is not and make the required adjustments until they are satisfied. This will help the team to get into the rhythm quickly. The leadership should be aware of these decisions and provide their input as required. If we are not making these decisions quickly and simply justify that "it is a journey and it takes time" then we lose the essence of agile iterative development.
• Work assignments: Many agile frameworks recommend the pull method, where the team members assign the work to themselves from the release or sprint backlog based on their skills and availability. This will help to improve cross-functional skills in the team, have fair work assignments, and drive towards a self-organized approach. Management should monitor on a monthly basis to make sure everything is going fine. For example, a team member working only on testing activities for more than 4 months, a team member not pulling the work from the backlog even after completing the current work, etc.
• Having qualified people in the role(s): We don't need to discuss much about this as we all know why every company in the world has a recruitment process for all hiring. It is important to make sure you have the right people in the roles, especially in the context of Agile for leadership roles like Agile Coach, Scrum Master, Product Owner, Kanban lead, etc. Don't try to force-fit or mismatch the current employees without proper training or before they become qualified for the role. For example, if you have an existing employee working in the company for more than 10 years and you would like to consider this employee for a Scrum Master role in the recent Agile transformation initiative, ask the employee to achieve CSM, PMI-ACP, PSM, SSM or a similar credential. For new employees, their prior experience working in Agile environments, and Agile certifications can be considered. I've seen companies simply moving their Quality, Business Analyst, or Project manager employees into these Agile leadership roles, without mandating proper training or certification. Moreover, they make this employee responsible for 2 or more Agile teams. How you can expect to get the desired results?

The tool is only as good as the practitioner.

• Co-located teams: Agile promotes the concept of colocated teams to encourage in-person communication, support and to enable better, quicker decision-making. The idea is to have the entire team present in the same physical location. Companies adopt different flavors based on their need - The entire team is present in the same campus or location (not in the same office building), Team is distributed across but in the same region (working in the same time zone), Only the development team is colocated, Only development team + Scrum master or Kaban lead is colocated, etc. While you adopt one of these, you should always work towards having a fully colocated team which is possible with some customization. When planning to form colocated teams, you should ensure that work can be fairly distributed across all the teams, teams are collaborating at a certain level so that the business knowledge is not retained by one team and best practices can be shared.

Colocated teams do not mean teams working in silos.

• Rewards and Recognitions: This is one of the foundation factors within Employee Engagement. You need to make your Agile journey, a fun and engaging experience for your employees. Create opportunities for innovation and career growth. You can do this in multiple ways - conducting training, sponsoring Agile certifications, Agile games, Using storytelling and/or quotations for better understanding, rewarding teams for adopting new practices (e.g.: TDD, BDD, Feature mapping, using a tool plug-in to simplify something), recognizing individuals who find gaps/issues to help bring in improvements (never punish them for reporting problems/issues), organizing hackathons and other events, planning team outings/lunches and so on.

CFO asks CEO: What happens if we invest in developing our people and then they leave us? CEO: What happens if we don't and they stay?

In addition to the above, the other factors you want to look out for are - having long-term teams without frequent changes, practices being followed in the intended ways, tools being used in the right manner and Agile maturity improving in genuine ways.

To conclude, Agile methodologies naturally foster a culture of continuous improvement in all aspects (time-to-market, quality, customer satisfaction, stakeholder experience, etc.), a way to adopt new technologies or frameworks, help to identify gaps/issues quickly, reduce risk, encourage diversity, equity, and inclusion and to scale your business as required. To realize these, you need to consider the factors mentioned above and handle them carefully.

Que: When to use iterative development? Ans: You should use iterative (Agile) development only on projects/releases that you want to succeed - Martin Fowler

Please tell me about your experience. What are the other key factors having a major impact on your Agile transformation journey? How do you deal with them? What are the example scenarios you like to share? Do you agree with the below?

When Agile implementations are adulterated we will begin to believe that Agile does not work when the real problem is that we are not applying it correctly.

Disclaimer: This article portrays my personal views, opinions, knowledge, and skills acquired from my vast experience and learning. Any resemblance to your knowledge and/or experience is a mere coincidence and is not intentional. Please let me know if I've missed any credits. This is related to the Agile transformations as part of the Enterprise Transformation Framework .