Is agile the right choice for your project? How agile has evolved and how to evaluate its effectiveness for your business needs

As of 2022, 71% of companies are adopting agile and 60% of companies experience growth in profits after adopting an agile approach. While the waterfall failure rate for projects is 21%, the agile failure rate is just 8% (ideamatics, n.d.).

How does agile work?

Agile approaches were born out of the need to solve issues associated with high rates of change, uncertainty, and complexity on projects. In 2001, several software development professionals defined the Manifesto for Agile Software Development, which encompasses four main values: “individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan” (ideamatics, n.d.). It also includes twelve principles.

Twelve principles of agile

In addition to the core values, agile has twelve principles. The first is to “satisfy the customer through early and continuous delivery” (Kanban Software, n.d.). By releasing early and often, agile teams increase the likelihood that products will meet customer needs and expectations, resulting in higher customer satisfaction. Customers not only get the benefit of continually improving products, but they can provide feedback throughout the process.

The second principle is to “welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage” (Kanban Software, n.d.). Being able to adapt to change, such as to new information, shifting priorities, and customer feedback, shows the customer that their voices are heard and their needs valued, even though that can lead to scope creep, thereby increasing costs.

The third principle is to “deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” (Kanban Software, n.d.). This was prompted by the extensive amounts of documentation traditional project planning requires, and is intended to keep teams working on the product more often than the planning and processes.

The fourth principle is that “business people and developers must work together daily throughout the project” (Kanban Software, n.d.). Because agile teams are cross-functional — here, “developers” can be substituted for designers or any other team members — the team members who create value need to synchronize with groups like marketing, sales, and services, who need to attract buyers, close deals, and implement projects.

The fifth principle is to “build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done” (Kanban Software, n.d.). This aims to reduce micromanagement and empower teams, leading to better quality products that are completed more quickly.

The sixth principle is “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation” (Kanban Software, n.d.). In today’s global teams, this can be interpreted as synchronous or direct communication where team members can quickly reach each other and make decisions.

The seventh principle is that working software is the primary measure of progressm while the eighth principle is to maintain a sustainable working pace, or “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely” (Kanban Software, n.d.). Team members should have a manageable workload and should be taught how to optimize the time they spend working, without relying on superheroes to deliver quality product updates frequently.

The ninth principle states that “continuous attention to technical excellence and good design enhances agility” (Kanban Software, n.d.). This allows teams to create a stable, high-quality working product, wherein code changes are less likely to have a negative impact. If operations are optimized, adapting to change will be easier.

The tenth principle is that “simplicity — the art of maximizing the amount of work not done — is essential. This empowers teams to find a simpler, less time-intensive way to optimize their work” (Kanban Software, n.d.).

The eleventh principle is that self-organizing teams generate most value, “the best architectures, requirements, and designs emerge from self-organizing teams” (Kanban Software, n.d.). Teams should be intrinsically motivated to deliver quality products.

The final principle is “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly” (Kanban Software, n.d.). Teams must evaluate their performance and identify what is working, what isn’t, and what they should start doing to further improve performance.

Agile shifts power to the team

With agile, power has shifted from the project managers to the product team. Agile suggests minimizing documentation, does not use extensive up-front planning, but relies on ongoing feedback and change, and engages the customer throughout the project lifecycle, thus does not need extensive customer involvement at the start of the project (Cram, 2019).

Rather than a commanding, controlling management style, agile encourages a participative, collaborative environment (Cram, 2019).

Team works in sprints with constant communication and feedback

The team works in iterations, or sprints, that last from two to six weeks. Because teams are self-organizing, they are more likely to collaborate, waste less time multitasking and context-switching, and deliver work faster (Project Management Institute, 2021). The team manages both how they work and what they’re working on, estimating what they think they can complete in a sprint (Project Management Institute, 2021). Team members work together frequently, known as pairing, swarming, or mobbing (Project Management Institute, 2021). All team members should ideally be 100% dedicated to the project. They have daily stand-up meetings to discuss progress and problems, engage in pair programming where two developers work together on coding, and conduct retrospectives, wherein the team discusses what went well, what to improve, and what to stop (Cram, 2019).

Instead of building for a long time to get to the value, as with a traditional project, agile projects can get to value more quickly, as the team builds in short sprints, getting increasingly more value and increasing velocity as the project progresses. By building more components for every release, the project is less risky (Kukhnavets, 2016).

More collaboration mitigates risk of not building the right solution

Because teams collaborate more effectively, this enhanced communication and sharing of information helps the team ensure that they’re building the right solution. Because everyone is testing the product continually, bugs are identified and prioritized more quickly. Team daily standups and weekly project leads meetings help remove blockers, mitigating the risk of missing sprint and release deadlines (Kukhnavets, 2016).

Team members must prioritize their interpersonal and technical skills

In an agile team, members must prioritize their interpersonal and emotional intelligence skills, as well as their technical skills (Project Management Institute, 2017). Each person must work to “exhibit more initiative, integrity, emotional intelligence, honesty, collaboration, humility, and willingness to communicate in various ways so that the entire team can work together well and respond more easily to change (Project Management Institute, 2017).

Agile team members often become generalists with a wide breadth of skillsets combined with one or more areas of deep specialization, also known as T-shaped people. Agile also has I-shaped people who have one area of deep specialization, but not much breadth of skills (Project Management Institute, 2021).

Make sure to celebrate wins

The team should celebrate their wins, while project managers should support the team and act as connectors to engage other groups, thereby improving collaboration (Project Management Institute, 2021).

Project managers coach and grow team members

Agile advocates servant leadership that promotes self-awareness, serves team members, helps people grow, coaches instead of controls, promotes safety, respect and trust, as well as embracing a growth mindset that promotes the energy and intelligence of others (Project Management Institute, 2017). In addition to coaching and growing the team, servant leaders need to support the team by building relationships to communicate and coordinate across the company, remove blockers, and optimize processes (Project Management Institute, 2017). They must also decide whether to develop particular functionality in-house or use third-party widgets or external agencies.

Agile provides greater visibility

Agile projects benefit from greater visibility so that project teams know exactly where they are. With waterfall, there are long periods where the team and other stakeholders cannot tell how much progress they’ve made and can’t get feedback, so it is easy to make big, costly mistakes that are hard to fix so late in the process. The team also cannot pivot quickly if they find that they are moving in the wrong direction.

On the other hand, with agile, the team and stakeholders are continually assessing requirements and have visibility at many points along the path, so it is easier to fix smaller mistakes earlier in the process and also allows the team to pivot quickly if needed (Kukhnavets, 2016).

Define requirements, break work down, create definition of done

Agile project planning includes defining the requirements, breaking work into user stories, and creating a definition of done. This definition is determined for each feature, so that the team understands what the requirements are. Requirements need to be clear, concise, verifiable, consistent, complete, and traceable. They should not leave room for misinterpretation and requirements should not conflict with each other. It is important that it is clear how the team can track that each requirement has been satisfied (JadeALM, 2021).

The team needs to break work down into user stories and acceptance criteria. User stories are units of work that make up a project and can ideally be done within 1 to 2 sprints. Sprints are usually 2 to 4 weeks. At the end of every sprint, the team has a feature that is complete or partially complete, but that is functional (Agile Pain Relief, n.d.).

Negotiate and adapt quickly

Agile project managers must work with stakeholders to determine requirements, manage expectations, resolve issues, negotiate conflicts, prioritize functionalities and features, and make decisions with stakeholder buy-in.

An agile approach enables the team to quickly and effectively adapt to the changing of requirements, new competitor advances or market shifts due to a light planning phase and iterative learning phases to evolve the product (Project Management Institute, 2021).

In software companies that need to deal with complex systems interactions, an adaptive agile development approach will reduce conflict delays by negotiating and making decisions quickly. By keeping team members and stakeholders engaged and continuously collecting feedback, the company reduces employee turnover, risk, time to implementation, and costs, while increasing customer satisfaction, retention and referrals (Project Management Institute, 2021).

Measure success

Teams can measure success via stakeholder engagement, feature adoption, task success during usability testing, user sentiment and satisfaction surveys, user retention, user referrals, and employee retention. They should also provide a seamless support system, training and documentation to ensure that their users are successful.

How agile development has changed system design

The agile design methodology was developed in the early 2000s by software developers in the United States to combat software projects that were consistently late, which in the fast-moving world of software rendered many products obsolete before they ever hit the market (Sofeast, n.d.).

Get products to users quickly via DevOps tools

To get products out more quickly in the age of cloud, software development now has DevOps. This enables teams to get their product in front of users quickly, mitigating risk and automating the build process to free up engineering time (Keery et al., 2019). Development teams now have a number of tools, including source control tools, such as Git, Mercurial, Subversion, and CVS, and continuous integration tools, such as Hudson, Jenkins, Travis CI, and Strider (ideamatics, n.d.). There are also team management tools, such as HP’s Agile Manager, Active Collab, JIRA Agile, and Agile Bench (ideamatics, n.d.).

Many software companies now operate either entirely on the cloud or have a hybrid model of on-premise and cloud services. That means that these companies are often on the public cloud, using systems such as Amazon Web Services (AWS) or Google Cloud Platform. Due to Continual Iteration/Continual Deployment (CI/CD), these cloud platforms are continually rolling out new features (Keery et al., 2019). This creates a catch-22 situation, since in order to increase developer productivity, the company has to allow them to experiment with new features, but AWS, for example, turns out new features at a rate faster than companies’ security personnel can thoroughly review (Keery et al., 2019). This means that the company must trust their developers in order to help them grow and reduce attrition (Keery et al., 2019).

Provide sandboxes and undo functionality to rollback changes

To further mitigate risk, complex business-critical systems like ERPs or middleware such as integration Platform-as-a-Service (iPaaS) products now often provide a sandbox for users to test changes they plan to make to their system, such as modifying a data integration flow or making a major change to the system that houses customer records. Often, software providers provide an undo function to undo more recent changes, a rollback function and version control system to rollback global changes. These mitigate the risk to customers by enabling them to back out a change that has an unintended consequence.

Make data-driven decisions on what to build and why

Because stakeholders are engaged throughout the project and the product is tested with users, companies no longer have to rely so much on team member or stakeholder opinion, and can make data-driven decisions to add the right features and functionality, and to prioritize some features over others. Agile has made team members develop stronger bonds and be more empowered, productive, and satisfied. Many developers appreciate agile due to increased project speed, value creation, knowledge enhancement, and improved autonomy (Cram, 2019).

What project types work best with agile?

There are three factors that determine whether a project or application is best suited for agile principles: urgency, complexity, and novelty. Software projects, due to their inherent complexity and the urgency of releasing a product quickly and updating it frequently, are a good fit (Cohn, 2011). However, the lean planning and documentation of agile can result in poor product quality, exceeding expected budget and time allocations, and security vulnerabilities.

Projects well-suited for agile

Agile techniques and approaches effectively manage disruptive technologies (Project Management Institute, 2017). Projects that have a high degree of uncertainty, are exploratory in nature, and have a goal that is difficult to describe lend themselves well to agile. These projects have high rates of change, unclear or unidentified requirements, complexity, and risk, which often leads to wasted work and rework (Project Management Institute, 2017). Some teams reduce this risk via short feedback loops, frequently optimizing the process, reprioritizing, continually updating plans as more feedback is collected, and delivering frequently.

Projects not well-suited for agile

Agile is not well-suited to projects where the team cannot provide interim deliverables or prototypes to benefit from continuous feedback. If a project has uncertainty around both the technology and the requirements, it is considered chaotic, which is ill-suited for agile methodologies. Projects where companies require sign-offs are not well-suited to agile, as sitting in a queue loses all of the effectiveness and momentum agile brings. Agile teams need to make decisions quickly, so if developers working on building any application are inexperienced and are not used to responsibility, the whole project might fail (Kukhnavets, 2018).

When security requirements are misunderstood due to lack of security expertise or focus on security early in a project, they are either not specified or inaccurately so, which can have severe consequences, a problem which is further exacerbated with agile, where companies do not often have a security expert on staff. Security requirements, also referred to as Non-Functional Requirements (NFRs) are particularly important for business-critical web applications that manage and protect sensitive, valuable data and processes, thus are a favorite target for hackers. To mitigate this, it is essential that the team collaborate closely and often and that team members trust each other and the company. In a study of 16 software companies using agile, many neglected NFRs, had sparse documentation and no requirements specifications (Villamizar et al., 2020), thus making agile’s definition of done impossible to ascertain. It is difficult to measure security requirements, as they are something to prevent instead of create; however, the tolerance for security gaps is very low to none (Villamizar et al., 2020).

Adopting agile requires a mindset shift

Even though many organizations have adopted agile, 62% diverge from its guidelines some of the time (Cram, 2019). Today, many companies tailor and hybridize multiple agile-oriented methodologies together, such as Scrum and Extreme Programming (XP) or put together traditional and agile methods, such as Waterfall and Scrum (Cram, 2019). Due to this variability, it is difficult to gauge the true effectiveness of pure agile (Cram, 2019).

In order to get employees to work in an agile environment, companies will either need to force agile or will need to tailor agile to get their teams to embrace agile (Cram, 2019). One core agile practice is for the team to work in open, shared environments. However, per Lindstrom and Jeffries, open workspaces should be considered optional, as forced participation can lead to staff turnover (Cram, 2019).

Shared code ownership

Agile principles encourage shared code ownership, meaning team members can contribute to any areas of the project, and more general skillsets (Cram, 2019). Collective code ownership is more effective when the developers do not have highly specialized skillsets (Cram, 2019). Pair programming is another agile principle, but some developers prefer to work alone, thus will be resistant to this (Cram, 2019).

Teams should be autonomous

In agile projects, teams should be self-directed and autonomous, and should be able to decide on the roles and responsibilities of each member during the project. The team will need to be trained on social skills and communication to develop interpersonal skills, responsibility and trust, which are imperative for agile success (Cram, 2019). While agile deems up-front planning and written documentation to be less important, the team needs to encourage flexibility by relying on interactions with each other (Cram, 2019). Stakeholder design suggestions should be encouraged and accepted, and the product should be refined based on emerging requirements (Cram, 2019).

Determine if agile is appropriate and how to tailor it

The agile development work system consists of agile adoption, agile tailoring, and agile use. In adoption, the team will need to determine whether to use agile, what type of agile to use, and why. In tailoring, teams will need to determine how closely to adhere to agile and how much to diverge. In use, they determine how they will apply agile, when, and whether it is worth it in different situations (Cram, 2019). Teams will often need to tailor methodology to include or exclude agile practices that do not suit their needs or create a hybrid approach of multiple methodologies.

Organizational culture and progress roadblocks can impact agile

Organizational culture, especially in a global organization, can make agile difficult, due to differing cultural expectations, time zones that make synchronous meetings difficult and impede work/life balance of team members. Several studies have found that it is imperative to have high collaboration, communication, trust, and the ability to deal with uncertainty (Cram, 2019). Companies highlighted the importance of being aware of culture conflicts, and the risk of diluting agile benefits (Cram, 2019).

If the company requires approvals for each phase of development, these bottlenecks will make agile lose its momentum (Kukhnavets, 2018); non-agile techniques such as extensive documentation, formal signoffs, and stage gates will decrease the benefits of agile (Cram, 2019). Combining agile and non-agile techniques can cause confusion for employees, as well as creating bottlenecks and diluting the effectiveness of agile. To incorporate agile successfully, the company needs strong advocates at the top and bottom, such as a CIO as an agile sponsor and developers as agile champions (Cram, 2019). Team members should earn certifications as Scrum masters and agile practitioners. Companies must be strongly committed to agile for it to be successful (Cram, 2019). Employees need to understand the benefits of agile principles ahead of time (Cram, 2019).

Measure performance effectively

Another key challenge is to measure performance, such as ability to make requirements changes and product visibility, although product delivery time and software quality can be measured (Cram, 2019). Managers can have difficulty measuring outcome, such as whether it resulted in a product faster that was of a better quality, and whether stakeholders were more satisfied with the results (Cram, 2019). It is recommended that managers implement techniques to measure speed, quality, and employee satisfaction to validate the agile impact (Cram, 2019), including a metrics strategy that measures on-time delivery, product quality, team productivity, and customer satisfaction (Cram, 2019).

References

Bartlett, A. (2020, November 11). What Is Agile Design When Developing New Products? Sofeast. Retrieved October 17, 2022, from?https://www.sofeast.com/knowledgebase/what-is-agile-design-when-developing-new-products/

Cohn, M. (2018, March 1). Deciding What Kind of Projects are Most Suited for Agile. Retrieved October 17, 2022, from?https://www.mountaingoatsoftware.com/blog/deciding-what-kind-of-projects-are-most-suited-for-agile

Cram, W. (2019). Agile Development in Practice: Lessons from the Trenches. Retrieved October 15, 2022, from?https://eds-s-ebscohost-com.ccco.idm.oclc.org/eds/pdfviewer/pdfviewer?vid=6&sid=5f30c259-d7d9-4bf7-bb98-983767a8d8ce%40redis

ideamatics, Inc. (2022, March 24). How Agile Development has changed the Software Development World. Ideamatics. Retrieved October 17, 2022, from?https://ideamatics.com/2022/03/how-agile-development-has-changed-the-software-development-world/

Kanban Software. (n.d.). The 12 Agile Project Management Principles Explained. Kanban Software for Agile Project Management. Retrieved October 17, 2022, from?https://kanbanize.com/agile/project-management/principles

Keery, S., Harber, C., & Young, M. (2019). Implementing cloud design patterns for AWS: Solutions and design ideas for solving system design problem: Vol 2nd ed. Packt Publishing. Retrieved on October 16, 2022, from?https://eds-s-ebscohost-com.ccco.idm.oclc.org/eds/ebookviewer/ebook/bmxlYmtfXzIxMTY0NTNfX0FO0?sid=971632b2-c3e6-4ed3-b096-0e161d2cc9d6%40redis&vid=0&format=EB&rid=20 .

Keng Siau, Roger Chiang, & Bill C. Hardgrave. (2011). Systems Analysis and Design: People, Processes, and Projects. Routledge. Retrieved October 17, 2022, from?https://eds-s-ebscohost-com.ccco.idm.oclc.org/eds/ebookviewer/ebook/bmxlYmtfXzM0MzQyN19fQU41?sid=971632b2-c3e6-4ed3-b096-0e161d2cc9d6@redis&vid=3&format=EB&rid=21

Kukhnavets, P. (2021, February 10). When Not to Use Agile. Hygger: Project Management Software & Tools for Companies. Retrieved October 17, 2022, from?https://hygger.io/blog/when-not-to-use-agile/

Project Management Institute. (2021). A Guide to the Project Management Body of Knowledge (PMBOK? Guide) — Seventh Edition and The Standard for Project Management (ENGLISH): Vol. Seventh edition. Project Management Institute. Retrieved on September 24, 2022, from?https://eds-p-ebscohost-com.ccco.idm.oclc.org/eds/detail/detail?vid=1&sid=03b98785-86e3-43ec-b3f9-f15e0a5b62c0%40redis&bdata=JnNpdGU9ZWRzLWxpdmUmc2NvcGU9c2l0ZQ%3d%3d#db=nlebk&AN=2942429

Project Management Institute. (2017). Agile practice guide. Project Management Institute. Retrieved October 17, 2022 from?https://eds-s-ebscohost-com.ccco.idm.oclc.org/eds/ebookviewer/ebook/bmxlYmtfXzE1OTUzMThfX0FO0?sid=5f30c259-d7d9-4bf7-bb98-983767a8d8ce@redis&vid=1&format=EB&rid=13

Schenker, R. (2018, March 9). The Application of Agile Project Management Methodologies (APMM) in Engineering Fields. Retrieved October 17, 2022, from?https://www.dhirubhai.net/pulse/application-agile-project-management-methodologies-rudi/

Villamizar, H., Kalinowski, M., Garcia, A., & Fernandez, D. (2020). An efficient approach for reviewing security-related aspects in agile requirements specifications of web applications. Retrieved on October 16, 2022, from?https://eds-s-ebscohost-com.ccco.idm.oclc.org/eds/detail/detail?vid=9&sid=971632b2-c3e6-4ed3-b096-0e161d2cc9d6%40redis&bdata=JnNpdGU9ZWRzLWxpdmUmc2NvcGU9c2l0ZQ%3d%3d#AN=edsarx.2009.03678&db=edsarx