Recipe for CI-CD — Chapter Two: Development Practices

How we develop, decides what is developed.

Hope you enjoyed the?Chapter One?of the recipe, that talked about the Culture and Team, which is necessarily the first and foremost important aspect of moving ahead with CI/CD. Once you got that sorted, what matters the most next is to setup disciplines of development practices in your engineers’ team.

One may challenge the word ‘development practices’ to be called as ‘delivery practices’ instead, given we are talking about the whole delivery. I respect the view, but it’s also to be understood and accepted that, with CI-CD in place it will be only the engineers of your team who will be delivering the product in the hands of consumers day-in and day-out. So, setting principles and discipline among the engineers, and hence their?development?practices, will play a vital role towards the success of continuous delivery of your product.

Development practices can be as simple as when and how to make a branch, to when to call a story under progress as complete.

Let’s talk about some of the key aspects to consider and to be made part as pillars of development practices for your team.

Deliver small incremental chunks

This is not a new phrase. Agile methodologies taught us this as the first thing to follow. Keep work small and continuous to have lower time to market, faster feedbacks and delivering constant value stream to end-consumers.

The base of this philosophy starts somewhere before the actual development beings, and hence this one sits quite on the fences from development perspective. Engineers are morally bound to deliver what’s defined as the acceptance in the story. If the story is not broken into small enough chunks to deliver a minimal value, the whole phenomena of CI-CD crumbles.

Engineers need to discuss, debate and be brutal to decrease the scope of any story to an extent that delivers value into customers hand but in the shortest possible span of time. This does not mean to change the requirements but rather to fit it in the situation such that we can deliver it fast and incrementally if needed. This starts with educating the business about your current state, technological estate and challenges, and any other strengths or limitations that can impact the delivery of requirements in good or bad ways.

Getting involve in such discussions upfront and keeping everyone informed is vital in making right informed decisions which are time appropriate so that any upcoming work can be refined, and if required, redefined to make it most impactful in shorter span of time.

In simpler words, it’s the responsibility of development team and engineers to help business locate the shortest possible deliverable value out of the huge requirements, derive smaller stories out of it, such that the whole requirement can be delivered continuously in smaller chunks. That bring us to another very important aspect of stories preparation i.e.?definition of ready.

Definition of ready

Define when the story is ready to be worked upon by the development team.?Development work on a story will only start when it is ready as per the definition.

A story is considered ready when,

• It has a gone through a detailed refinement session
• Have clear acceptance criteria written on it, has?example-mapping?exercise done for it
• All ancillary or non-technical artifacts and information, applicable for the story, are ready for reference. E.g. UI-UX designs, 3rd party integration details etc.
• Or anything else that your team decides to have ready.

These looks obvious but believe me these are ones mostly skipped due to the effort and time required for it. That is when one staggers to deliver fast and defeats the first pillar of the practice —?deliver smaller chunks at constant pace. In fact, many times a lot of time is lost in back and forth communications to get that missing information during development, and results in over-all longer time to market.

Knowing your definition of ready and ensure all story you accept to work for development has met the criteria will thrust your goal, of delivering constant value stream, even further.

While the definition of ready is important to commence development, another guided principle known as?definition of done, to call the work complete, is also equally important.

Definition of Done

A superhero has well said, “With great power comes great responsibility”.

When you have got the power of CI-CD at your disposal, it brings along the responsibility of completing the work with operations in mind.

That’s all?

Yes, that’s all, but I hope you understand what it takes to make a product production or operations ready. Nothing much, just ensure to

• Deploy a bug and regression free application to production
• With right KPI’s being always monitored
• Having appropriate logging and tracing capabilities for debugging of an issue.
• Leave no security gap in the application
• Implement non-functional-requirements of the product or ones specific to the story.

Congratulations! We just got our definition of done. Of course, besides this all one also needs to meet the acceptance criteria of the story ??

So far with me?

All this looks obvious but in the traditional way of development, where you get a separate testing, security review, production support and many such different teams to carry out their roles in delivery of work to production, continuous delivery cannot be achieved.

So, we got to solve this problem. Let’s do that by?shifting left.

Start shifting left

You may have heard a lot of time of the phenomena of shift-left. This was first used to bring testing cycle closer to development cycle by shifting your teams left.

In any SDLC diagram the development phases are always showing flowing from left to right. In the shift-left concept, many of these phases were advised to be shifted and merged with development phase, and hence shifted left.

Though the concept started with shifting of testing but today for empowering your team with DevOps and CI-CD, many more phases now need to be shifted left and folded into development phase.

Let’s go through those quickly to build an understanding.

Test before code is a new normal

Of course, testing is the first phase to consider. Right test coverage and automated testing is the game changer in the days of CI-CD. This topic needs more time and devotion, and I will talk about it in-depth in the next blog in the series. For now, you need to understand that, every code merge to trunk (of your SVN repo) will potentially take the code to the hands of consumers.

Hence its quality must be ensured. That would mean, to have right test coverage at the time you merge the code, implies to get the test code written right when the development happens. In fact, if you take it to next level and care about?TDD, the test code comes into existence even before the functional code.

The whole takeaway is that, writing tests is not left for later for a testing team, who barely knows the logic and code. Instead it’s written by the development team right when the producing code. It must be remembered that, no one better than the engineer, writing the code, knows the code better and hence can produce the right test coverage. So, shift your testing left.

App security is your concern

Just like testing, security has also always been differed for quite late in SDLC. That could mean, the security team finding a high severity issue in the code at a phase its development has completed, testing has concluded, and code is ready to be shipped. Such last moment surprises, either forces one to release with exceptions and hence along with security loopholes or differ the release to fix it. In either of the approaches, there is an impact to customer. Yes, the customer, for whom all the work was done and released just in time.

Here as well, if implementing security concerns and testing becomes development practice of the team, the feedback cycle can be tremendously small to save the pain, and deliver it to customers in time, without security issues. So, shift your security left.

You developed it, you release it

Releasing of code in any traditional working style, is a ceremony to watch. Multiple teams, many people and some sort of high echelon project managers all working round the clock on the release day, and some time days, to make it happen. You can only realize the reason of my sarcasm above once you have witnessed the power of CI-CD for yourself.

By now you must know where I’m going to. Yes, release also is your development team’s responsibility. But, will not that our continuous deliver pipeline does for us? Absolutely, it will. But remember CD pipelines only deliver what is coded.

You as the development team, need to consider all aspect of releasing a software. It includes code for the infrastructure required by your written piece of code, configuration management for different environments, and possibly feature flags. So, yes, shift your releases left.

Is the application support-ready?

Just to reiterate myself, any merge to master has the potential to reach to production when you start using CI-CD pipelines. And, any code running in production must always be monitored and observed. That brings us to the next aspect to consider for continuous releases i.e.?observability.

Observability is a concept of operating an application in production with right sets of tools and system in place that enables support engineer to debug runtime issues. It includes 3 things in particular:?monitoring of metrics,?logging?and?distributed tracing.

No need to repeat but with CI-CD, you don’t have luxury for some support team to work on this at a later point in time, but instead it becomes another responsibility of the development team to ensure all these pillars of observability are implemented when developing. And again, who better than the development team would know the right logging statements and monitoring KPIs for the service. So, start shifting operation readiness towards left.

Full-stack team

Above four bullets may have confused you a bit and in fact, may look too demanding from a person aka?the developer. Some may even call it impractical. But believe me, and I am telling you from my current work experience where we as a team do this day-in day-out that, it’s possible. All needed is the like-minded people and the zeal to do it. Of course, right culture and team is must, and that is why it was the?chapter one?of this series.

If you have noticed in the blog, I’ve very carefully used the word?engineers?and not?developers. People start carrying a virtual crown as soon they are called developers, which can become the reason of mind-blocks that stop them from accepting any work, other than writing functional code, as their responsibility. And, they may indeed start seeing some non-developer clans accountable for testing, app-security, releases or any other work item. The day it happens, take it for granted that, your ambition for CI-CD is sunk.

So, if you build a team that has people of different technical expertise and work experience, and open to experience, learn and adopt new skills, you will soon have a team which is multi-talented by its sheer nature, self-sufficient by its cumulative skill set and highly capable by its strengths. Such teams are also called, full-stack team.

So, in order to shift a lot of phases to left, you need to first build a team of engineers and not just developers, who has the strength and attitude to achieve and maintain what is needed for CI-CD.

DevOps is a mentality, not a role

What we discussed so far are the key pillars that a development team need to adhere to in order to adopt, implement and make CI-CD a success for their product. But all these talks are incomplete till the time DevOps is not discussed.

DevOps is known to be the one of the key bases for achieving CI-CD. However, people still define DevOps in a wide variance. The simplest and most crisp definition comes from Wikipedia where it says,?DevOps?is a set of practices and principles to achieve continuous delivery with high software quality. And, that absolutely it is.

It is for sure?not a role?or a title one can carry to show-off. DevOps simply demands that, if you have built it, you must run it. ‘Run’ here indicates operate the application in production. And hence the work?Dev+Ops=DevOps.

This word comes into focus here, due to fact that, all the points we discussed so far above are originated from the concept of DevOps. Where the development team knows that, they are going to be the one to run and support the product in production environment. That sheer realization makes one to ensure, they delivery highest quality product (hence have good test coverage as quality gates), with possibly no security loopholes (hence implement all security considerations) and with adequate cover of observability (hence upfront configuration for monitoring and logging). All these aspects can be considered at once only when you deliver smaller chunks of work, and you know what does done look like.

So, you see, a single para above gave us the excerpt of all the pillars of development practices discussed so far.

It’s vital and unavoidable that, everyone in the development team accepts and develop the mentality around the principles of DevOps to make it happen.

Don’t strangle yourself with branches

Along with all the key principles mentioned so far, there is rather a small but equally crucial development aspect that is mostly left out, i.e.?branching strategy.

How your team takes out branches for working on a story, from which branch you deploy to production, which branch is used for hot-fixes and which ones for running new features, answers to all these points that are necessary in order to establish DevOps practices in your team.

I may not go too deep on this subject, but would like to emphasis on the fact that, unless you start working directly on trunk/master branch of your code repo, you will not only fail at CI-CD but also make it suicidal for your product, given getting lost and strangled in branches.

So, ensure your integration and delivery pipeline runs only from master. Developers either work directly on master or take out feature branches, which lives only for the work of one story. Keeping deliverable chunks and hence the stories short will ensure your branches merge back into master at very short interval and keeps you away from hassles of integration with other engineers’ work.

This principle enforces discipline in the team, reduces the time of work from getting complete to merge to trunk, and hence faster value delivery in hands of consumers.

Summry

It’s time we conclude this rather long sermon. I hope enough testimonies are mentioned above to help you get on the path of setting development practices that empower and enable your team to embrace CI-CD with success.

In next chapter?I will bring to your notice the importance of Testing Automation and Testing Strategy plays in the success of delivering quality product via CI-CD.

Happy reading!