Agile from the perspective of 25 years in IT

This article has been originally posted on Agile/Scrum Community onsite in Polish - please click here to see the Polish version.

Agile project management was born at the end of the 20th century, so it is not new, and yet it is constantly changing. I have decided to share my insights and reflections from the perspective of someone who has not only observed these changes, but also experienced them personally.

Why do I do what I do? Why is it important to me?

I originate from a technology path. At the very beginning of my career, I was a network administrator, a database administrator, a developer, and developed for a while in the direction of systems architecture. Entering the project management path was sort of automatic - the moment I started my own company. I started with a few people employed, to reach a team of more than thirty people after some time.

The experience I gained allowed me to start the next stage - moving from projects carried out for small and medium-sized, mainly local organizations, to projects carried out for corporations or large, very often international clients. This also required some standardization of the knowledge I had gained so far on my own through training - at that time in the methodologies used in most projects, i.e. based on Prince2, PMP, ITIL or related.

Later came Agile - today's Scrum, AgilePM, Kanban and their various variants or modifications evolved from methodologies such as Lean, Kaizen or XP. The organizations responsible for them strongly promoted the new approach to software development proving that projects implemented using agile methods are more effective than previous approaches.

The methods they resorted to, by the way, were not always completely "clean". On several occasions I saw slides proving how many classically run projects ended in failure, and how many agile projects were crowned with success. The problem with this comparison, however, comes down to what is defined as project success in either approach. In the case of classical methods, the success of a project is defined at the very beginning and the "failure to deliver" any of its elements automatically results in the failure of the project. In the case of agile methods, the conditions for project success are very fluid and can change during the course of the work - so it is natural that changing them "in the course of the game" allows for success even without meeting the expectations set at the very beginning. Of course, here we can discuss what project success is and what is not (I mean delivering value to the customer and defining that value) - both in the context of agile and classic projects. However, this is a broad topic hooked also to the issue of return on investment, which I may not touch on here (examples of projects that were successful, but unfortunately did not meet expectations - "the operation was successful, unfortunately the patient died").

But going back to the genesis of the creation of agile methods, one thing must be acknowledged - they addressed one of the most difficult areas of waterfall, which is change management. Agile was therefore built, as it were, on the foundation of change and rapid feedback, which was lame in waterfall (the process of change management was extraordinarily complex, and thus did not defile with special speed or flexibility). Resources became flesh-and-blood people, and managing them, while maintaining an important level of motivation, brought about a great humanization of the manufacturing process. Not insignificant to the success of this approach was also the generational change and the evolution of people's approach to their work-life balance. Earlier generations were oriented toward working to satisfy their, often quite basic, needs; their children began to see life as more than just work. In the case of a small company, this direct contact with people was easier than in the case of a corporation, where until recently people were just another cell in Excel. Agile made it possible to see real people behind those cells - and this is something that drives me in my daily work: allowing people to spread their professional wings, which many times they don't even expect themselves. This motivates me and truly makes me happy.

Do I have any regrets and why?

In 25 years of work, I have made many mistakes that I might not have made, that I might regret. However, I cannot say with full responsibility that I regret them - because I have learned from each one. I try to adhere to the principle that mistakes can be made, provided you make them only once. And it has happened to me that I have made the same mistake a second and a third time - and I may regret it. I used to pay less attention to reflecting on the mistake I made, which led me to make it again. And that's something I'm not necessarily proud of, because I know it was avoidable.

This is why the retro "institution" that exists in agile methods is so important - it allows you to reflect on what you did wrong and avoid it in the future. It also allows you to capture the things you did get right, which leads to improving the process by repeating only the actions that had positive results. Nevertheless, it is an art that takes time to master - and perhaps if I could accept the lessons learned from other people's mistakes, I would make fewer of my own and regret them less.

If I could meet myself 5, 10 years ago, I would advise myself…

Many of the management methods or techniques I use today were actually developed by me. Only later did I find out that I was kicking off an open door - because similar ideas (and often better ones - which is difficult to admit) had been thought up by someone much earlier and had even been described for posterity. Unfortunately, in the belief of my own revelation, I happened to ignore the knowledge of others and thought that I would produce a better solution myself.

Unfortunately, the path to the optimal solution was often littered with mistakes and failures that I could have avoided if I had spent more time looking for solutions that had already been described or discussed somewhere. In other words, instead of wrestling with a problem on your own, in a sense of your own greatness, it is better to approach the problem more humbly and try to find a solution by the method of exploration, and not necessarily by trial and fail. In this way, you can avoid making many mistakes that you will later regret - especially if you find a ready description of the solution to a problem somewhere after the fact. Unfortunately, people often learn better from their own mistakes than from those of others. So, it is possible that the advice to get humbler and listen to others would not have found fertile ground 5-10 or even 15 years ago - that kind of maturity comes with age. And how do you explain to someone who doesn't want to listen to others that they should?

What skills and experience have been most useful in the role of Project/Programme Manager?

The roles I am currently working in certainly require a certain level of maturity. This in turn comes from the variety of projects or programmes I have had the opportunity to work on. Getting to know people, cultures, observing and analyzing how they go about their work enriches and "opens your head" to different behaviours or situations, but most importantly, at least in my case, it teaches you that the so-called "root cause" is usually much deeper than where we expect it to be at first glance. And this is where the fundamentals of business analysis come in handy - the ability to get to the root of a particular problem, rather than making decisions or forming opinions based solely on what we see at the "user interface" level.

I also have the comfort of having worked in technical roles before. This gives me a much deeper understanding of what my people, or even individual teams, are dealing with. And that, in turn, helps to anticipate the consequences of one decision or another at the technical level as well.

Interesting challenges along the way

An interesting experience was my first purely business project - when IT was not a domain I was directly involved in, but just a supplier of certain elements to make it all work. I was building a contact center for one of the largest companies in the direct sales industry. I remember the first few months when I had serious doubts about whether I was in the right place. On top of that, I was working with people who were not only incomparably more competent in the given processes, but also much older, some of whom could have been my parents' age (at least with a little effort). It was difficult to manage these people, to give them instructions, to see if I could control them - until one of them, a colleague at least 20 years older than me (which may be important - Dutch, so he says what he thinks very directly), who knew the company I was working for practically inside out (he had probably been working there longer than I had set foot on this earth), noticed this and told me that he didn't have the slightest problem with the fact that I didn't quite understand what he was doing. What's more - he doesn't expect me to understand his tasks, but he has a big problem with organizing and prioritizing them. And he expects me to help him - because he knows I can do it better than he can. Each of us has certain knowledge and skills - and we work in a team not to duplicate each other, but to complement each other.

This was a kind of epiphany for me - a hard blow to my world of wanting to know and understand everything that goes on in a project. I realized (finally?) that a project is not my knowledge + cogs doing specific tasks. It's a team that only has certain knowledge and skills in common if it's structured properly - and the removal of one of these elements doesn't just lead to a drop in productivity, but to a loss of operational capability in a specific area.

In any case, multidisciplinary teams made up of people who complement each other's competencies are not exclusively the domain of agile methodologies - this model was not even invented with agile methodologies but was "hijacked" from the military. In fact, in the case of classic methodologies, we had an approach similar to the now obsolete doctrines of large armies with specialized whole divisions composed of people with similar competencies. In the case of Agile, the approach is modelled on Special Forces - small but self-sufficient and highly multidisciplinary squads, made up of soldiers with different skills, able to carry out even extraordinarily complex tasks comprehensively.

The future of Agile/Scrum/Kanban - how do I see it?

Earlier I mentioned how I saw the development of agile methodologies so far - what dictated it and how it became so popular. I am now very curious to see where these methodologies go from here - especially as the form we saw in the beginning is quite different from what we see now.

It seems to me that to fully understand the direction in which agile software development methods are heading, we need to recognize one more fact. It is the burden of responsibility for the final product - or, more precisely, the entity that carries this burden. Regardless of whether the product is developed by a software house for an external customer or by a product team for the business part of the organization, we always have a supplier-customer relationship. In traditional (waterfall) methodologies, the vendor is responsible for the product from A to Z - it is the vendor who provides the analysts who "cull" the functional specification from the customer and then, based on that, delivers the product within the specified framework of the project triangle (time - cost - scope, sometimes with quality thrown in). In the case of agile, this responsibility for the final product has in fact been transferred to the customer - it is the customer's representative, who should be the product owner, who delivers the work specification in the form of an epic or user stories (at this level, sometimes with the help of an analyst), sometimes limited to several sprints ahead. He also decides whether a particular feature is really ready or whether it needs to be reworked, which will "eat up" some of the time of a particular team, and sometimes even a tribe.

Initially, this created a rather comfortable situation for the supplier - as his work was almost entirely based on the time'n'material model - colloquially translated as "we do until we're done", and whether we're done or not - the customer decides. At the same time, in this case we do not bear the slightest budget risk - after all, it is the customer who decides how long we must work and who spends the corresponding funds.

However, this model could not be considered advantageous for the customer. Of course, it also had its advantages for this stakeholder group, such as: quick response to changes, no need to think (carefully) about everything at the beginning. This led to greater flexibility and the ability to respond to market changes during the course of the project, but it made it difficult, if not impossible, to budget appropriately and to plan for the release of the solution (along with all the activities associated with this process - such as timely preparation of the operations or marketing teams).

What were the results? Well - in product-type companies it worked well because the supplier, which was internal IT from the start, was empowered as a cost center. And as such it had to have a job, otherwise it would be unnecessary. However, the cost of running it was usually an operating expense and was offset against the company's general income. However, when a specific project was an investment that was handed over to a more or less external provider (including, for example, SSC-type development centers), the project required specific budgeting as well as planning of the entire environment around it - something that was quite difficult in the basic version of agile methodologies.

Looking at where scalable agile methodologies such as SAFE, LESS or related methodologies are going these days, I see that while at the squad level Scrum or Kanban itself still works very much as it used to, at the multi-squad level (whether we call it a tribe, a programme or something else) you can see a return to design phases. The construction of a roadmap is often preceded by a series of workshops in which a blueprint is created - perhaps much simplified, but still like a functional specification. In contrast, the previously very independent, squad-to-squad defined velocity and estimation process is standardized by the creation of reference user stories or an additional estimation level using T-shirt sizes. Alternatively, more or less complex mechanisms are created to convert story points into time and thus financial measures. Roadmaps are then created with appropriate resource allocation, and the freedom to modify and change is restricted (i.e. we introduce an increasingly complex mechanism for accepting change requests, which de facto kills any agility in the process) to avoid shifts in Epic's delivery - especially when they are interdependent.

It seems to me that the development of agile methodologies will continue in this direction - we will rediscover more and more principles and rules that we used to follow in waterfall and classic methodologies and try to "smuggle" them into agile. In extreme situations, we will create projects that are sometimes called "Scrumfall" - i.e. with strongly marked project phases, but still divided into sprints, and we will "rigidly" convert story points into working days (I saw such a situation in one of the "agile" projects, when it was decided that 1SP would be equal to 1 Manday - and that rule applied to all 8 teams in the same way).

Nevertheless, I do not expect to see a complete return to classical methods at a general level. Rather, it will be a kind of mix of agile and classical methods that, on the one hand, retains a certain degree of the humanization of the process I mentioned earlier (resulting in a sustained elevated level of motivation) and, on the other hand, meets the needs of planning at the strategic level. Something along the lines of Prince4Agile or Agile ASAP, perhaps, but with a greater penetration of methods at higher project levels as well. At the same time, I hope that in this entire process of merging the "old" with the "new", the best of both is not lost, to paraphrase Winston Churchill: "Agile is the worst form of project management, except for all the others that have been tried".