CLean Principle Of Engineering
One man's "magic" is another man's engineering. This principle will describe a software engineering mindset, engineering a software core and how to cultivate an engineering team.
A Software Engineering Mindset
Good engineering process includes the following phases: Requirement Analysis, Design, Development, Testing, Release, and Maintenance. Software development using Strawman Waterfall Model is a myth as most teams do follow a Software Development Life Cycle (SDLC) that is iterative in nature, and the difference is how iterative of this feedback loop. To succeed in software development is no different to any other activities in life, the faster the feedback, the quicker one can observe and adapt. Similarly, any long-term activity needs a roadmap to guide to its desired destination.
As software development becomes more customer focused, developers have begun to associate their discipline with Software Craftsmanship and not Engineering. If one thinks these are just pretentious titles, these two articles on engineering and craftsmanship do provide some insights of the difference and that this is mostly down to the individual's personality and experiences. A common criticism of engineering is creating gold plated solutions that end up impacting real deliveries. However, this problem is due to lack of real understanding how to achieve a flexible design to deliver more with less.
With millions of developers and we still have a skills crisis means as an industry, we are collectively inefficient and there is massive scope for improvement in productivity by the way we develop software. We need to re-engage an engineering mindset that promotes common design and architecture patterns, embraces standardisation, values reusability and believes the scalability of productivity is just as important as the scalability of the product.
Software delivery is more than developing features for customers, it's also about on-going maintenance and support. Time spent in initial core development is just a fraction of a product lifetime. Time to Market (TTM) is not an excuse to craft solutions that totally blown out the Total Cost Of Ownership (TCO). A software development team trying to code as fast as possible for a large system without analysis and a design will not deliver as quickly as one who develops software with a core design that evolves by delivering with less code, does more, highly structured, and provide flexibility to embrace changes.
Software craftsman who embraces an engineering mindset will standardise their well-crafted masterpieces to deliver better quality, consistency, and reusability to increase the productivity of others. Sometimes we are so used to delivering what the customer wants that we lose sight of the best way to deliver long-term value is to create flexible solutions which are quicker, cheaper, and with quality to satisfy customer needs
A Software Engineer believes the scalability of productivity is just as important as the scalability of the product.
Engineering A Software Core
One way to help encourage developing software with an engineering mindset is to ask a fundamental question of what does most programs actually do? A large part of programming is to create abstractions to model, process, and moving data around. The real commodity is the data, not the software. The software is just the medium used to deliver data from A to B. It really doesn't matter to the users of an application how 'beautiful' the software is written underneath the hood, it is the data which get processed into information that is interesting to them. A feature is a conductor of this data flow to do something useful for someone or something.
We can view the whole system as an onion having a core with successive layers to create an overall application. An application with a rotten core is difficult to add new features because it doesn't insulate the 'application plumbing' of its data flow, meaning a change can cause ripple effects. A well-engineered application, in contrast, has a core that rarely changes because it adheres to the open/close principle, allowing new code and configuration to extend the application without modifying the existing code, making it easier to change without worrying about breaking something else.
An application's core spans the whole system and contains logical and physical layers of interconnected modules and processes to form an architecture that supports the data processing and data flow, and we can imagine this as an information highway. To embrace changes, you need to develop this information highway that is resilient to change. There are many well-defined architecture patterns that software engineers have used as a foundation to build an application core, with Layered, Event-Driven, Plugin, Service Oriented, and Microservices being some of the common architectures used in today's enterprise software development.
One way to develop an application that embraces change is to build an application framework which grows the core to provides structure and extensibility. This is the engineering mindset of evolving towards the solution (b) in the Hofstadter & Sander example above. The development of this core will start as a skeleton framework based on a chosen architecture pattern, focusing on the non-functional requirements of 'data plumbing', connecting the application core that spans different logical and physical layers. As more features are built, the written code could be generalized if possible to encapsulate what varies, and the common code should be pushed into the different application layers' core to enrich the software abstraction. The application core will become more stable and rich in reusability as the product evolves, enabling faster ongoing features development.
An alternative software incremental approach is to create specific to what the customer wants and use software refactoring techniques to generalised once common patterns emerge. This is the mindset of solution (a), spending the efforts creating a number of shapes before arriving at an abstraction of solution (b). Both approaches work, but the first approach is more efficient as the application framework provide structure and extensibility to guide the emerging design process. This means fewer reworks, and team members have a vision how to grow the application. The second approach also works in theory, but in practice, the temptation to move on with new features or the refactoring efforts is so costly that it is tempting to add to the pile of technical debts and the application becomes harder to develop, supports, and maintains as the product evolves.
The purpose of software is to process and move data around. Understanding the core to create software will facilitate real agility.
Cultivating An Engineering Team
The software development skills crisis is real. Organisations accept this reality as technologies keep changing, and it is easier to pay a premium to acquire new skills. It may surprise to many that most of the popular fundamental skills haven't changed that much in the last twenty years, and likely to remain that way for a foreseeable future.
Take Javascript as an example, a programming language developed in ten days to ride the marketing hype of Java with built-in core fundamentals of Procedural, Object Oriented, and Functional Programming style has played a significant part in the last two decades of web innovations. It retired Java Applet from the web browsers; open up real-time data update with AJAX; discovered JSON to become a de-facto messaging format, and has become a viable high-performance server-side solution with NodeJS.
An organisation can educate an engineering team without incurring massive training costs. Many of these skills are timeless and technologies tend to grow with it. An organisation can start cultivating an engineering team with a few 'Engineering Champions' to inspire others through their professionalism. The efforts of a few, writing structured, consistent, adaptable and reusable code will immediately lift the productivity of many. This will have knock-on effects to cultivate more software developers to embrace engineering excellence to deliver lasting customer values.
Cultivating an engineering team is not expensive as a few engineers will improve productivity of many.
How This Principle Increase Productivity?
The CLean principle of Engineering focuses scaling productivity through engineering practices. A software engineer is a software developer who follows a well defined SDLC and embraces an engineering mindset that believes scaling productivity is one of the best ways to deliver lasting customer values.
A well-engineered application is a software that provides a medium to process and pushes data to present useful information to someone or something. Software engineers exploit this fact with an architecture that encapsulates this core, building an application framework resulting in a software with a stable core that can continuously evolve and grow.
The software development skills crisis is real, but by cultivating an engineering team using professionalism of 'Engineering Champions' will allow a few to increase the productivity of many.
Disclaimer: These are the author's own thoughts on the importance of software engineering. The objective of this article is to highlight the importance of an engineering mindset, engineering a software core, and how to cultivate an engineering team. This is just one way, and there is certainly no claim that this is applicable to all contexts.
(C)ustomer: https://www.dhirubhai.net/pulse/clean-core-principle-customer-andy-ng
(O)rganisation: https://www.dhirubhai.net/pulse/clean-principle-organisation-andy-ng
(R)eusability: https://www.dhirubhai.net/pulse/clean-principle-reusability-andy-ng
(E)ngineering: https://www.dhirubhai.net/pulse/clean-principle-engineering-andy-ng
FSO and ESG Architecture Portfolio Lead at EY
7 年in my experience the best exponents of Agile are software engineers - people who are at ease with the approach you mention above. Not all organisations can practice 'fundamentalist' agile but all organisations should employ at least some software engineers. Agile practiced without knowledge of key solutioning skills which have been proven over decades, as mentioned in this article, will result in an almighty mess.
I remember when in 2007 Martin Fowler told me at a conference, noticing my lack of experience "if your iteration looks like a mini waterfall, you are not agile". It took me years to understand what he means, and today I teach people on what that actually means. I've told you this number of times, your methodology mathematically, I repeat mathematically, does not work. It has nothing to do with people, or their skills, it doesn't work for other reasons. I will bet you a nice dinner that your best UI, best BL, best DB, best testers and best deployment experts, if organised the way your methodology is recommending, will produce the work slower and of lesser quality than my average T shaped developer. You are literally like one of those people who are saying theory of relativity is a fraud and promote theory of ether. Your theories are not new, they have been around for yonks and have been debunked millions of times, millions. I have told you how to, in 10 minutes, prove to yourself that your ideas are wrong. What you and your pal Simon did was to delete my comments to your posts. I wonder if this one will be deleted as well. Anyhow, your ideas mathematically do not work. 2+2=4, and you cannot make it equal seven, no matter what.
LLM Engineer, Team Lead (Flutter & Native iOS/Android) - iOS & Flutter Developer IOT (Bluetooth) - - Loc: Melbourne - Designing & building Large Language Models on edge devices to understand LLM architecture.
7 年Andy, you are dealing with a 'religion' and its zealots. Enough said!
Engineering Chapter Lead @ UBS
7 年For some reason I am not allowed to comment on my own post in the Agile group: https://www.dhirubhai.net/groups/81780/81780-6288745831378014212 So, here is my answer to some of the comments: Criticize good engineering practice required analysis, design, dev, test, and maintenance and use 'strawman argument' to discredit this article. I have already asserted most teams is iterative in nature as it is unlikely, waterfall or not, to deliver in one shot. The core of software development is to deliver values - not data. Except I said only information that people care about, not software or data. Another strawman argument. The reason I made this statement is to explain how we can build a system with a core if we understood the data flow. With information hopefully, users can draw values from it. I encourage a 'frozen core' and just patch externally will create a fragile system. Another strawman argument. Not only I did not say that my thinking is that every well-engineer systems should have multiple cores and you extend the core by adding configurations and extensions. You just simply do not open the core for every change. However, the core also needs to evolve in time as technologies changes. For example, rework a well engineered layered architecture to microservices may give more deployment agility, for example. Then having a core mean you know how to reapply the logic which is not scattered everywhere. Few example successful open sources, Maven, Eclipse, and Linux build this way.
Problem Solver
7 年There is no silver bullet. The devil is in the details. All I read here are theoretical generalities that sound nice, but will not do much better (if any) in practice than people are already doing, because nobody knows what they do not know until they run into it. The fact is that everyone makes mistakes no matter whether they call what they do coding, programming, craftsmanship, agile software development or engineering. The key to succeeding despite those inevitable mistakes is not to double down on theory or process or formalism. The key is to maximizing feedback loops, minimizing the latency time of those feedback loops, paying attention to what the feedback is telling you and committing to a solution at the last responsible moment.