Tranformation

Software Evolution

Software goes through an evolutionary process as expressed by conway’s law;  with mutations overtime, one gets various italian flavors - Spaghetti,  Ravioli, Lasagna which unlike the healthy italian diet is not as palatable for software organizations; this evolution builds up constraints to the value creation process. Dynamic business landscape requires rapid changes to software to support the evolving business process; however, in reality the expected pace is not met by the software organizations; question like - “why does it take so long to make such a simple change and why is it so expensive” - is not uncommon.  So not only there are challenges to the time taken to develop new features but also with the increasing marginal cost of adding new features; however, with pressure to add features faster, teams often ends up prioritizing short term convenience over long term correctness, resulting in more accumulation of technical debts, which then, over time, makes it time consuming and expensive - a vicious catch-22 cycle;  Breaking this cycle is the “holy grail” of many software engineering organizations. Breaking the cycle or, better yet, not even getting into the cycle, requires conscious efforts to (1) simplify the architecture or keeping it simple (2) eliminate hobby initiatives and (3) carve out slivers of investment (time) to keep the architecture current and technical debts in check.

Transformation

As soon as a working piece of code is in production, the software is legacy; once in production, changes gets harder and expensive; in addition, risk to changes needs to be factored in, which means bold changes are no longer possible;  overtime, without conscious or concerted effort to keep things clean, the code base becomes unwieldy and gets the organization to a place, where some change has to be done. Viewing from a simplistic prism, there are two way to go about the change (a) too late, greenfield approach (b) too late, brownfield approach; a green field approach to swapping out the problematical applications with a ground up rewrite is tempting; often times rationality gets clouded by outcome bias (an excellent HBR read here) with confirmation bias providing the necessary support for “rational” justifications; however, in reality, in a greenfield the challenges to achieving the outcome are many; of the many, the two key ones that causes even the most experienced folks to trip up are (a) accounting for non-happy path flows made harder due pervasiveness and tribal nature of knowledge capital (and remember there is no such thing as centralized or current documation) and (b) keeping the greenfield code current with functionality of the production application in parallel (since the greenfield could span months or even years); of course, besides these reasons there are others like talent, skills, budget etc; many who go through the greenfield experience live to tell the tale of many iterations to wrap up the long tail, and the last 5% drag coming down as the last nail to the coffin.

This leaves us with the best option of “inside out” brownfield transformation. With this approach, the software is refactored overtime time to comply with key architectural principles of the organization; achieving the goal  of - simplicity - without compromising on security, scale, reliability etc. However, to pivot from what we have to something “better”, is hard and is akin to changing wheels on a running car; also to note, this is not as exciting as greenfield.  But then, doing this way has better chance for success with reasonably fewer risk (risk exists but can be mitigated by taking appropriate mitigation steps). However, this activity requires stamina, and unwavering commitment to keep at it. Multiple challenges exists and some of them are loss of focus, loss of budget; vague outcomes, technical challenges, limited immediate business value and an activity spanning possibly years; so keeping at it, requires good old fashioned grit. However, this type of transformative work, when done right,  will yield not benefits of better architecture but also set the tone for a culture that helps to keep things current and clean...

At Hyla, we started the “inside out” modernization two years ago. This pivot was centered on three core areas (1) people (2) process and (3) technology;  On the people side, some of the work involved better balancing of talent across three geographies where we do work, focusing on building up knowledge capital and hiring the right people who can help “find the true north”; on the process side we continue to fine tune our modified agile process that best aligns with our business strategy; on the technology side, which is the main focus of this article, the work being done falls under these three broad themes

1. Incremental and ongoing clean up
2. Cloud pivot
3. Simplification

Incremental and Ongoing Cleanup

This involves going deeper than what is typically is done as part of feature development; the focus areas span: development, qa and production; the top areas that we are focused on;  

1. Basics - better logging, exception handling and code documentation - Some developers are better than others in this regard;  our ongoing focus here is keeping pushing towards adequate logging that helps to quickly triangulate production issues; better exception handling and documenting tricky parts of the code;  the holy grail we are after is make this activity part of the cultural fabric of the development team across all geos which will enable to keep the software entropy in check.
2. Reducing bogus alerts - Constant fine tuning the alerts and underlying code to either ignore bogus alerts or to eliminate bogus alerts;  working hard to transition from the an alarm fatigue state to “actionable alerts” state.
3. Code refactoring and kill dead code - Active focus on refactoring code to leveraging newer styles of programming paradigms (functional languages bits) and by eliminating dead code every opportune moment.
4. More and more test automation: -  Continued focus on unit testing and automated functional testing; ensuring that focus and investment stays in place to keep the automation suites current.
5. Environment consistency and infrastructure automation:  Consciously make an effort to reduce the Snowflake effects; also putting efforts automating infrastructure areas; while at it, closing security gaps in the environment aggressively.

Cloud Pivot

1. Move the entire application stack to cloud (AWS); use this opportunity to reengineer (simplify) by taking advantage of the services available in the cloud environment.

Simplification

1. Standardize on few key technologies:  Having many “framework” components that provides similar functionality increases complexity and opportunity for errors;  we believe sticking to fewer frameworks helps to keeps things simple, easy and manageable.
2. Kill hobby projects:  Oftentimes, a new technology is introduced, in a whim, to solve the immediate and current problem substantiated by claims of “future” benefits; however, if not followed through, this becomes yet another technology that needs babysitting;  keeping these impulses in check and weeding out the loners, help keeps things clean and simple.
3. Data needs: Data needs are typically fulfilled through batch or application integration; one type does not fit all the needs all the time;  however, minimizing batch and keeping the focus on application integrations (through API) is the path we are moving forward with.
4. Decomposing the monolith:  As Hyla grows, some of the issues with monolithic is starting to become a concern;  we are cautiously optimistic that microservices will help solve some of the growing pains and dipping our toes in a careful, controlled steps.

In next series of articles we will explore in depth some of the key initiatives starting with cloud pivot and other exciting work done at Hyla…

Additional Reading

• Exploring the Duality between Product and Organizational Architectures: A Test of the “Mirroring” Hypothesis
• Why Reengineering Projects Fail - Carnegie Mellon University
• Big ball of mud (absolute classic)