Agile will improve your software development teams or will it?

With over 25 years of experience in developing software for commercial use, I've observed significant changes in some approaches to building software, while others have remained largely the same. Software developed solely for use within a single business, the process used will depend on the team and management in terms of formality. If the software supports the company and is not the primary business focus, it often serves to boost the efficiency of the business. Therefore, getting fixes and bespoke changes out quickly is often the focus; this often will come at the expense of formal processes, but adopts a mindset of delivering software quickly. A friend of mine worked at a Formula 1 team in a team that supported the design of the car, so speed of getting quality working software was more important than a formalised complicated test and release process.

My experience has been that if the software is being created for another party, either as a service or a product, that has changed. Twenty-five years ago, projects and products were delivered using linear stepped processes described now as waterfall projects. The rise of Prince and Prince 2 in the UK became the trend for training project managers in the 80s through to the early 2000s. It was also great for outsourcing projects and paying on the milestones. But in most cases, it locked scope early and started expensive time and money change control processes. It also governed the release cadence of many enterprise software releases. The introduction of the Agile Manifesto in 2001 marked the beginning of a 20-year process transition.

Agile is not a single process; it's a set of principles and approaches that emphasise values, driving a mindset for more effective delivery. Whilst the philosophy does not mandate an implementation, there have been many frameworks and approaches that capture the agile. These have included RAD, Unified Process (UP), Extreme Programming (XP) to name a few, but the clear winner for notoriety today is Scrum. With the backing of organisations like the Scrum Alliance it has become the de facto destination for companies with a huge industry supporting transformation, certifications and training.

Change the process, to Scrum or not to Scrum?

Changing the fundamental project delivery method is a pivotal decision-making process organisations encounter when considering the adoption of new Agile methodologies, particularly Scrum. This choice isn't merely about selecting a project management framework; it's a deeper question of aligning with the core values of the Agile Manifesto, which emphasises 'Individuals and Interactions over Processes and Tools.' The consideration here is whether the structured nature of Scrum, with its defined roles, sprints, and ceremonies, genuinely resonates with this Agile principle or detracts from it.

At first glance, Scrum's structured approach might seem contrary to the Agile ethos, potentially prioritising processes over people. Yet, this view overlooks Scrum's inherent flexibility and its capacity to foster a collaborative environment where individual contributions are valued within a clear, supportive framework. The real challenge for organisations lies in implementing Scrum in a way that it becomes a facilitator for innovation and collaboration, rather than a rigid procedural boundary. I've seen over and over again Scrum implemented as a process rather than adopting the ethos it represents as well. Choosing to adopt Scrum, therefore, is more than just a process shift; it's a commitment to an Agile mindset where the methodology serves as a tool to enhance human interactions and productivity, not as an end in itself. The decision to 'Scrum or not to Scrum' is thus a strategic one, requiring a thoughtful balance between maintaining the agility of team dynamics and the structure provided by the process.

Often the commitment to an agile mindset can be challenging for some team members and business leaders. Often it will require external assistance to work through the mindset shifts and difficulties. Scrum helps adapt mindsets but progressing to other agile methods such as Scrum ban or Kanban might work better in many cases. These are less regimented processes but if you can't make Scrum work, you may have other difficulties with these. If the individuals haven't grasped the principles of agile the process will likely fail to given the benefits you look for.

Improve productivity

Productivity in software development is a multifaceted concept, often challenging to quantify due to the creative and iterative nature of the work. Broadly, it can be defined as the efficiency and effectiveness with which software is developed and delivered. Productivity encompasses not just the speed of coding, but also the quality of the output, the relevance and usability of the software, and the degree to which it meets user needs and business objectives. It includes factors like the amount of functional code produced, the reduction of bugs, and the agility of the development process. However, measuring productivity must go beyond mere lines of code or tasks completed; it should also consider the innovation, maintainability, and long-term value of the software developed. In essence, productivity in software development is about creating high-quality software in a time-efficient manner while continuously adapting to changing requirements and environments.

The ecosystem that surrounds agile is what brings the productivity, small cycles of feedback from stakeholders, test systems, build systems and other developers. Consider whether the team has embraced continuous integration systems like Jenkins, Team City, GitHub Actions, etc. Does that extend to continuous deployment and delivery? This helps frame the requirements around the process and the expectations for releasing more often to get feedback. This can be challenging for teams to make the mindset shift that have shipped enterprise software with long upgrade cycles and long release cycles. You can't have a month long regression cycle on a daily release of software.

The Estimation Problem

Software time estimation is a tricky endeavour, posing similar challenges in both Agile and Waterfall methodologies. The core issue lies in the inherent inaccuracy of human predictions. In Agile, while the size of estimations tends to be smaller due to breaking projects into smaller chunks, and activities like planning poker to reduce variance the problem of accuracy remains. In fact, it can be exacerbated as teams often lean towards overestimation to account for uncertainties. Whether you're meticulously planning every detail in Waterfall or adapting to changes in Agile, the common thread is that if a team can't estimate accurately, the chosen process won't mitigate this challenge. It underscores the need for continuous improvement in estimation techniques and the recognition that software development estimation will always have a degree of unpredictability.

When work is being sold to a timeline especially if it is for a specific contract this is where stress with business leaders can occur. Leaders want a timeline they can rely on, and delays or extra efforts can lead to problems. From a business perspective the relationship with a customer can often be sorted with fixed price work if the scope is well known but the internal cost still occurs. As a technology leader I care about both parts of the delivery and have to find solutions. One of the characteristics of agile is the ability to course correct if new information comes along and changes the aspects of the work with huge change control process. If managed correctly this benefits everyone, however if badly managed can lead to the same place as bad estimation. Breaking projects down into manageable chunks that can be estimated and those estimations reflected on retrospectively over time to improve accuracy.

People over process remember that?

Agile methodologies represent a fundamental shift in workplace culture, emphasising the importance of individuals and their approach to work over rigid adherence to processes. Central to this philosophy is the recognition that even the most meticulously crafted processes, such as Scrum, cannot compensate for the shortcomings of a poorly performing engineer. While Agile is not a cure-all for productivity issues, it's a framework that thrives on the skills, commitment, and collaboration of team members. Thus, if an engineer is underperforming, relying solely on Agile processes for improvement is a misplaced hope. Many embark on agile transformation to gain performance installing tools like Jira and following scrum processes but ignoring the people underlying performance and interactions.

Such situations necessitate direct performance management, addressing the root causes of underperformance and helping individual succeed. The power of Agile lies in its ability to adapt and respond to change, facilitated by a team of competent and engaged individuals, where people are valued more than the processes they follow. This focus on people underscores the need for a cultural shift within organisations, one that nurtures talent and fosters a collaborative, responsive, and innovative work environment. However, delivering measurable value should be the exchange for this environment.

The SaaS Drive

Over the past decade, one significant transformation in the software industry has been the shift towards Software as a Service (SaaS). This is the logical next step for many software deliveries in terms of efficiency of deployment and issuing updates. Many enterprise software companies through 2000's moved from perpetual licencing with maintenance to subscription based licencing with multi-year commitments. SaaS is the next evolution for many but some pure SaaS software vendors become disrupters to the enterprise software companies.

SaaS software is not just hosted versions of the enterprise software, it should deliver a seamless service with continuous updates for functionality and security, transparently to the customer. To do this it requires the agile mindset and processes to support the deployment. Can the team release to live several times a day and as a business leader you are confident a client is going to call? In some ways moving from 6 monthly releases to multiple a day is the same as moving from waterfall to agile.

On challenge to the daily release mindset can be the agile process itself such as Scrum. As a team or leader the iteration size is decided say 1 week or 2 weeks but that shouldn't align the completion of the work and often does. This is why Kanban or Scrum ban limiting the work in progress may be more appropriate in SaaS delivery. If you add the additional dimension of multi-tenancy supporting multiple clients on unified platforms this requires a mindset that doesn't align with waterfall thinking.

Well what does this all mean?

So, what does this all mean? Why discuss Agile now when so many, as highlighted in the "16 AMAZING AGILE STATISTICS" article, have already made the shift? Notably, 71% of companies in the US have adopted Agile, with 61% using some form of Scrum. Although 61% are using a form of Scrum, it's worth questioning whether this reflects a true mindset shift or merely a procedural change

In summary, choosing to implement Scrum or any Agile methodology is more than a procedural shift; it's a strategic commitment to a culture that prioritises individuals and collaboration over rigid processes. While Scrum provides structure, its effectiveness relies on its application as a tool for fostering innovation and team synergy, rather than as a strict set of rules. Begin with Scrum, master its nuances, and then, through a pragmatic approach, adapt to what best suits your team's needs.

Agile methodologies align effectively with modern software development, particularly within the SaaS model context. It promotes a feedback-driven development cycle and frequent releases, essential for high-quality, relevant software. However, challenges like accurate time estimation and balancing client expectations remain. Agile offers a framework for better estimation and adaptability, but this requires continuous improvement and a recognition of inherent unpredictability in software development.

At its core, Agile and Scrum emphasise the importance of valuing people over processes. Successful Agile transformation transcends the adoption of new tools; it requires a cultural shift that enhances talent, collaboration, and responsiveness. Therefore, the decision 'to Scrum or not to Scrum' fundamentally revolves around embracing a philosophy that prioritises human interaction and adaptability in project management.

Disclaimer: The opinions and views expressed in this blog are entirely my own and do not reflect the views, opinions, or policies of my employer, its affiliates, or any other organisation I may be associated with. The content provided here is for informational purposes only and is not intended to represent professional advice or the official stance of any company or organisation. Any references to specific entities or products are used for illustrative purposes only and do not imply endorsement.