Agile in Business: Why We Learn More from Open Source than Commercial Projects

Agile, the modern approach to project management, has been in use for decades now. In the field of information technology, Agile is a well-established concept. This term is frequently used across the board, from developers to management, and even at the executive level with CEOs, CTOs, or CIOs. It is a mindset supposedly created to address the uncertainty that inherently exists in (mostly) software development projects, often in commercial sectors. You may have come across terms like Scrum or Extreme Programming (XP), two of the most renowned Agile implementations in the field. You can work at the banks, fintech companies, e-commerce platforms, social media platforms, and many others and you will see all of them work with Scrum or other Agile methodologies somewhere.

Agile was born out of the need of new software development methodology which provides improved time-to-market, greater flexibility, and, better tolerance for failure. The original waterfall method, a well-established project management approach, has been around for decades and has brought engineering to what it is today. The waterfall project management methodology has been responsible for all the iconic buildings and landmarks that we see. It gives us space shuttles, power plants, and so on. However, as the world evolves, and we entered the information era, we were initially oblivious to see that software development projects do not, at all, resemble civil engineering projects. We started encountering a surge of uncertainties, unreasonable demands form marketing departments for quicker time-to-market, and volatility in customer experience requirements. It soon became apparent that the waterfall method was unfit for this novel landscape.

The VUCA in Information Technology

In this VUCA (Volatility, Uncertainty, Complexity, Ambiguity) era, clearly the use of Agile becomes crucial for companies in order to win their shares in the market, essentially the digital segment. Information technology is now intertwined with every modern business process we know. Unleashing the power of information technology is not without consequences, because we are now in concert with the natural VUCA existing already in information technology.

For starters, and for some of you reading this that might not come from information technology that might be interested in Agile (maybe you are a manager or a director or CEO), it is important to understand some reasons why the information technology field is riddled with VUCA.

• Technology moves fast: The information technology field moves incredibly fast as it is very flexible. We have new programming languages every now and then. New hardware components are introduced, new frameworks are introduced, and thus new software technologies get introduced. The incredible demand for newer and better technologies introduce volatility in information technology.
• Software development is a creative process, not procedural: It is common to see software development, as part of engineering, as a very procedural process. This is to certain extent, not true. In fact, software development is one of the most creativity-demanding field of engineering out there. There are many ways of writing codes, and different people write different codes. We have code reviews every time someone pushes an update to our central code repositories and we can see which one is from which not by looking at the authors, but by looking at the delicate details that are often not included as part of our clean coding conventions and designs, such as the code naming style (the wordings), line breaks, even the architecture of the codes itself speaks a lot about the authors themselves.
• Murphy's Law: The Murphy's Law states that anything that can go wrong will go wrong. An extension to this is the thing that will go wrong will always be the thing that we least expect. Murphy's Law has been around for ages but it is made famous thanks to software engineering. It highlights what is existent in every single software development project: uncertainty. No one knows for sure if the codes that we write will work in the future, or in different environment, or even in different laptops. There are simply too many factors to consider when writing codes, we simply choose to assume or ignore many of them. This contradicts the waterfall project management approach, in which everything must be planned and designed correctly from the start. It is clear to see that the existence of Murphy's Law warrants the need of fault tolerance in software development.

Free and Open Source Software

Now, allow me to introduce you to FOSS, which stands for Free and Open Source Software. Open source projects are software development endeavors that are generally open for public, allowing everyone to contribute free of charge. The software developed is typically made freely available for anyone to use. Wait, so what do we gain from working in FOSS? Usually nothing, at least not anything monetary in value. I have told you that software engineering is a very creativity-demanding process, and creative people are more often than not quite idealistic. We engage in FOSS just because maybe we want to provide assistance. Or maybe we have been using this particular piece of software and it is riddled with bugs and we wish to help fixing them. Or maybe just because we are too idealistic and have a lot of free time. One of the most significant FOSS projects is the Linux kernel, spearheaded by Linus Torvalds himself, still until this day. More than 29 years in the making, Linux has accepted more than 1 million commits (a singular piece of code update from a particular contributor) from thousands of contributors hailing from thousands of companies, including the major players such as Intel, AMD, Google, Meta, and even Microsoft, among others.

FOSS represents the epitome of VUCA that already exists already in information technology. Those involved in FOSS often operate under no clear deadlines or targets. Features are never guaranteed to be delivered in time. Bug fixes are never assured, and certain issues may languish for years without anyone addressing them, even though everybody knows (even the users) that there are some issues in the codes. Some FOSS projects find themselves abandoned without any maintainers, effectively halting the development of new features or bug fixes for the project until a new, benevolent maintainer is willing to take over the project.

Some FOSS projects do operate under clear Agile principles, often utilizing Scrum and conducting daily standups among its developers. Most of the time however, only a rough guideline is presented to new contributors. Most of contributors are therefore required to work under a set of conventions based on their experiences. However, a lot of successful and important FOSS projects have made it out today with clear guidelines and principles for contributors without considering any Agile or Scrum principles at all.

One thing for sure in FOSS projects is everyone working on it must do so voluntarily. This makes the Scrum principles or other similar development structures inadequate or unnecessary for FOSS projects. Each FOSS project must then develop its own guidelines and conventions. These guidelines are developed to address VUCA among all of its present and future contributors. They more often driven by coding best practices rather than business practices and are usually very technical.

The Agile Manifesto

So what can we learn from FOSS projects? Many of commercial software development projects follow the Agile principles but have failed to address VUCA issues and maybe even have slower time to market than many FOSS projects. A lot of commercial software development projects have less developers but have more management issues than FOSS with many more developers. Many FOSS projects without a single project manager have clearer and more transparent software update schedule than commercial Agile projects. This makes you wonder about the effectiveness of high level project management theories and principles compared to the low level coding guidelines.

In fact, there is nothing wrong with Agile. It's just that sometimes we don't fully grasp it, and as a result, we become overwhelmed by the process of selecting the correct project methodologies, often forgetting the essential task of executing them correctly. To refresh your memory about Agile, here are the principles of Agile, also known as the Agile manifesto:

• Individuals and Interactions over Processes and Tools
• Working Product over Comprehensive Documentation
• Customer Collaboration over Contract Negotiation
• Responding to Change over Following a Plan

These are the original principles of Agile. We will soon see why FOSS projects unwittingly and unsurprisingly follow these principles to be able to deliver features and fixes in time with amazing quality.

Learning Agile from FOSS

Transparency in Communication

FOSS projects do not usually fuss over the tools they use. They make use of available tools, such as Github issues, some free-to-use CI servers, and others. They do not use tools just to jump in the bandwagon, but they consider which tools provide the best experience for the majority of their contributors. Most of FOSS contributors also communicate with no direct channels with each other. Majority of interactions happen usually over some Github issues, whether a bug or a feature request, in which everyone in the world can see. The key point to see here is contributor interactions with each other are very transparent and tool-independent. Noises in communications (misunderstanding) are most likely corrected. Communication tools are very simple and easy to use, and everyone uses the same communication channel, including the users of the software itself. Users make comments and reports issues independently to the same issue board, contrary to some commercial projects in which users need to go through a lengthy process for their issues to reach the original team of developers. Some issues are added to the issue board just to spark discussions about the future of the project, which might last up to months or even years. Take a look at these discussions from the Nvidia Linux Open Kernel Modules and the Wayland support for Steam. Some commercial projects have failed to even utilize an issue board effectively and makes use of the method of "just talking in Slack and finding anyone who is to blame for a particular bug".

Working Products AND Documentation

Many FOSS projects mandate clear documentation and guides to be written by all contributors, even in the event of approaching deadlines. How come a FOSS project have feature deadlines? They are usually not deadlines, just a schedule of next releases which contains forecasts of features or bug fixes that need to be implemented. Some notable projects like the GNOME project have clear release schedules, even describing the dates in which no more changes can be published for a particular to-be-released version (freezing). These schedules are more often than not derived from work culture, and not just some company rules. This work culture in FOSS is also based on contributor interactions, which require transparent communications. It is important to remember that code documentation is also part of communication.

Although working features/fixes are always the top priorities, documentation is also forced to be prioritized higher due to its importance in encouraging interactions among contributors and users. We understand how a code work by seeing documentation in the code. After failing to understand how it works, only then we would look at the issues related to the code, if exists. This makes documentation very important as it is actually the first layer of communication from developers. The code documentation conveys information from the original developer to other developer, while user guide documentation conveys information to the users, in which both are usually equally important.

The need for working products is usually driven by demand. Less famous or demanded FOSS projects are usually less active, as why would anyone fix something that is already working, or something that is not used by anyone. This is in contrast to some commercial projects, which often fail to address the real user demands and force quick time to the market pushing unnecessary features/fixes that might be half-baked, to follow deadlines set by management. Users have all the authority to report issues directly to contributors, which clearly bring up the real user stories and demands. Users don't report to product managers that have little or no experiences with software, but to contributors that are working by hand directly on their products.

Productive and Happy Employees

Ask yourself, are happy employees productive or productive employees happy? The latter might also be true. Being productive is not always for the benefit of the company. Being productive simply means that employees learn something over time and work according to their purposes, unrelated to the company. Productivity does lead to happiness by the fulfillment of one's purpose.

I have previously stated that FOSS projects have to be voluntary. This means that FOSS stands on the opposite end of commercial projects, in which contract negotiations are important. Most of the contributors in the Linux kernel or other FOSS projects are never obligated to work under the FOSS license. However, most people in FOSS are very idealistic, apart from the fact that they are wicked smart. They work just because they are needed. Direct customer (user) collaborations provide goals directly to the original contributors. Even without any contracts, they know they have to do it.

If you ask why would anyone work on a FOSS project, well, fulfilling user demands, even without incentives, can be productive for some people. Of course there is no one in the world that wants to work in commercial projects without incentives, but being productive is not always (not only) about monetization. Many companies omit the fact that sometimes people work not just because they want incentives, but because they want something to learn according to their purposes. Some even give up higher paying jobs for a more laid-back job which offer better working conditions and cultures that are more aligned with their personal purposes.

Contributors to FOSS projects are never under any contracts with their users, so FOSS projects are never guaranteed to deliver working products in time. However, enabling direct contributions with users introduce genuine demands to contributors without confining them to work in a shell without a glimpse of the reality of their products. Of course, as contributors, we can just choose to disregard user demands, which often happen in overlooked issues in FOSS. Nonetheless, direct user collaborations eliminates inefficiencies that can be present in commercial projects, where developers are constrained by rigid contracts and business processes, which can hinder productivity. Stifling creativity is the least you want in allowing employees to be productive, although achieving a balance between creativity and structure in a team can be challenging, as the creative process is inherently difficult to scale. Trade-offs must be made to strike a balance between these elements to maintain productivity. The ultimate goal is to avoid introducing inefficiencies to this balance. Because no matter what, even with or without incentives, nobody wants to work in a team that is unproductive.

The "contract negotiation" defined in the Agile manifesto often refers to any part of the contract or project that introduces inefficiencies to the team working on the project. These inefficiencies can stem from various factors, such as unclear objectives, undefined deadlines, or vague customer requirements. In some cases, contracts demand that we place an intricate structure of people to report to. Some other contracts hinder or slow the way the users can collaborate with developers. Some contracts also fail to mention the importance of customer collaborations and so fail to remind the real users to consistently provide feedback to the projects.

Responding to Change

In FOSS projects, it is often found that even without contracts contributors work under voluntary motives to respond to demands from users. An important consideration here is contributors typically work independently, but they are all focused on addressing issues that are reported to the project by the real users. They focus on one thing: responsiveness.

Most FOSS projects are never started with a plan. It is usually just some people working on a product on their free time, in which at some point in their life they choose to publish their product and source codes freely for anyone to use and contribute. More often there was never a plan in doing so. The plans usually come after, typically in the form of release schedules. Although commercial projects cannot live without a plan, the lack of plan in FOSS projects emphasizes the need for responsiveness often outweighs the importance of plans.

Plans are essential. Failing to plan is equivalent to planning to fail. But how far are you going to plan something in a commercial project? Estimating the most common approach. As in practice, meticulously detailed plans seldom align with the realities of software development projects. Software development is a highly creative process and plans often fall outside the spectrum of creativity. Nevertheless, customers require estimates, which necessitate the creation of a plan.

More often than not we are so engrossed in creating the right S-curves or Gantt charts that we eventually realize none of them make any sense. Without a conducive working culture and effective collaboration, uncertainty can never be addressed. Hence, project managers should strive to foster a healthy working culture between team members and users, and then with other members of the company. This is a fundamental prerequisite as, there is no point in devising a plan in which the majority of the elements are uncertain.

In FOSS, simplicity of tools, transparent contributor interactions, and straightforward collaborations with users promote responsiveness significantly. Learning from FOSS, this principle requires us to invest our time to develop the right convention and choosing the right tools that actually promote these aspects immediately following the start of a project. Due to the fact that demands are always volatile and uncertain it is always required that companies be responsive with demands. Those that are stuck in following a rigid plan are always left behind.

In Agile, development is split into iterations. Iterations should not be seen as "no-plan", but as "mini-plans". This means that iterations should never be started without a plan. In fact, iterations introduce rigidity in a small scope enough for responsiveness to not be sacrificed greatly. Think about this as the release schedule in FOSS projects. Most contributors and even users are aware of these release schedules. Features are developed and estimated to be released in predictable manner according to release schedules. Contributors need to discuss together what features and bug fixes to be released for the next release window. Thanks to alpha and beta versions that are usually released before the official release, we can also get a taste of working products immediately after updates have been pushed by contributors, even far prior the release schedules. This flexibility is important as they provide both stability and responsiveness at the same time.

Finally, FOSS projects do have their own flaws, so we do not need to learn everything from FOSS projects alone. Majority of FOSS projects that involve user interfaces (UI) often suffer from poor user experiences (UX), due to the absence of UI/UX designers and coordinated design plans. Feature requests or bug fixes are frequently overlooked, and the absence of contracts eliminates any certainty regarding the project lifetime itself. Some less active FOSS projects have bug issues that remain unresolved even months after they are reported. This instead underscores that Tools, Documentations, Contracts, and Plans are still essential in software development projects. Seeking the right balance among these principles are what managers are for.

If you are a developer, consider exploring some FOSS projects out there. They offer valuable learning experiences without the obligation to adhere to a specific work schedule. If you are a manager, take a closer look at some prominent FOSS projects and learn from how their contributors collaborate. It is through continuous learning that we can navigate through the VUCA inherent in information technology.