Practical DevOps Learning Path — from where should i start ?

The article is available aslo in medium ( click above)

I spent more than 8 years retenlessly working between software development, Cloud Operations, Release Engineering then Architecting. People are thirsty to see DevOps at the ground. I let our directors / VPs breathe with me “DevOps”, not just to talk about it… i am here to share the whole experience practically

1. Thu Shalt be a software developer guided by software engineering principles

• Learn GIT , sorry , Don’t learn it ! but Master it ! Otherwise, Don’t proceed.
• Know what’s README, why, and how to write it concisely.
• Learn one of frontend Technologies ( VueJS, ReactJS or AngularJS) in webdev & Practice that → Recommend React+Redux
• Learn/Practice how to write unit-tests for your frontend components
• Learn one of backend Web Dev Technologies ( Gin of Go,Express of NodeJS, Flask of Python, Laravel of Php, Grails of groovy, Spring Boot, ..… ) & Practice that! → Recommend Go or NodeJS
• if by mistake ?? you met something called “Spring”. Do the following: Give it a Respect ?????♂? !Stop there, Go and learn about it ! Learn AOP vs OOP ! Learn IoC Design Pattern and its implementation which is DI … Indeed, you just met the Software Architect of JAVA apps.
• Learn how to build a fullstack web application with these technologies & BUILD it!
• Get Basic Knowledge about OWASP.
• Scan your app(s) against the top 10 OWASP ( Try Sonarqube as tool)
• Learn how to enrich this fullstack application with enterprise features ( Oauth/OpenID Connect, Maching Learning, … ) → Recommend OAuth with Google
• Optionally, Learn/practice Superficially mobile development → Recommended (React Native)
• Establish reusable components as per need.
• Do not re-invent the wheel
• Do not repeat your self
• Composition with Go , OOP with JAVA .. No OOP in Go.
• Make it The Clean Code even if you have to practice “the Art of Destroying a Software” ??

It was a hands-on mission which took 6 Years of My career (2012–2018). Still applying that now, but my in-house software is now built to ship operational knowledges more than being an end-user apps

2. Thu Shalt use Linux Systems as a BOSS

• Learn Basics of Linux Administration ( RHCSA is a good path ) & Apply that in your daily work!
• Focus on process management, file management, Systemd, Log rotation.
• Learn how to prepare a Linux server for deploying app(s) that you built in the first part.
• Learn Vagrant to be able to spin off VM Linux as Code with no time — Basic Knowledge is enough
• Deploy your application to linux server
• Conclude all used commands and add them to your Vagrantfile.
• Git commit your Vagrantfile and do not forget gitignore, otherwise, you will end up with a fat git repo ?? . Congratulations! you just touch an area from GitOps areas.
• Have the skills to troubleshoot why your app goes down just after one hour ?? . May be “nohup” is a solution, or may be systemd ,… or … but Let’s not think about kubernetes now.

3. Thu Shalt jump into Public Cloud

• Go into AWS Solution Architect-Associate Path & Catch two birds ?? by one stone
• The first bird is getting the required knowledge that expands your vision & introduces to you all main Cloud Architecture & Cloud services : Events ( Cloudwatch), Network as a Service (VPC), Computing as Service (EC2, Lambdda) , Storage as a Service(EBS, S3, ..).. so on.
• The second bird is to help you getting your hands dirty with Cloud services.
• Unfortunately, Do not think that Getting AWS Solution Architect-Associate will qualify you as a Solutions Architect unless you have architecture skills in prior.
• Instead & fortunately, this path puts you on the Cloud road by giving overview about main cloud services, how can they interconnect as well as introducing you smoothly into Infra/Ops world ( Resilience, Failure & DR , Loadbalancing, Scalability, Subnetting, Supernnetting, Routing, Firewalls, WAF, … ) even if you were 100% pure developer previously.
• Use AWS Services (EC2 Linux instance) to deploy app(s) mentioned in part 1. Actually, do not appear to the exam without getting these hands-on in place.
• I assume you’ve already learnt cost effectiveness. So don’t forget to terminate the EC2 instance ??

4. Thu Shalt eat ?? Container Technologies

• Get the basic knowledge about containers from two perspectives: SysOps perspective (linux process) and Developer perspective (app package)
• Learn Docker then Docker Compose
• Dockerize app(s) that we mentioned in the 1st part. ( Dockerfile,.. so on)
• Translate “docker run/build” commands into Code called “docker-compose.yaml”, then update README
• Git Commit your docker-compose file → Congrats! You touch GitOps again but not the full cycle.
• Provision EC2 instance from AWS, Install Docker depencies, Deploy your app(s).
• Absorb the Architecture of Kubernetes, how it works and how the whole infrastructure is exposed thru a REST API !!
• Deploy the container image of your application to a public registry (Docker hub).
• Write the required Kubernetes resources, in YAML format, in order to deploy app(s) mentioned in the first part.
• Deploy your app with kubectl
• Scale your app

5. Thu Shalt develop Software with CBD in mind

• CBD stands for “Cloud By Design” (My own Abbrv)
• Folks call it also “Cloud Native”.. Sorry! I just call it CBD.
• Learn 12 factors & refactor app(s) mentioned in the 1st part accordingly.
• Learn about Serverless Architecture
• Learn Serverless implementation using FaaS (Function as Service Offering Model) which fall into learning Lambda +one of DB backend (dynamo) + API Gateway + may be more (SQS,… )
• Run your first AWS Lambda, add an HTTP request trigger to it , let it responds accordingly.
• Refactor app(s) mentioned in the first part ! Break down it into functions ! Deploy it to AWS Lambda.
• Do not use directly AWS Console to develop Lambda! Use a framework that helps you to control the Code with Git → Recommend serverless Framework.
• Preview your app locally (“serverless offline”) even if it requires remote services in prod runtime.

6. Thu Shalt Codify Everything

• Treat everything as Configuration Item.
• In one of previous part, you codified your Linux administration using Vagrant
• In the other previous part, you codified your container image definitions using Dockerfile
• In that same part, you codified your container runtime definitions using docker-compose.
• → You did it. But you just need to know why ? so you keep doing it.
• Code vs Adhoc Scripts = huge difference
• — — more practice — —
• Delete EC2 that you created previously to deploy yout app(s)
• Learn AWS Cloudfmration, then implement a Cloudformation template that ensures the creation of the EC2 instance with a startup script that installs all dependencies & Deploy the application. ( startup script = user metadata)
• Deploy the Cloudformation template as a new Stack
• Git commit the template inside the git repository of app(s) mentioned in the first part.

7. Thu Shalt breathe the 10 Commandments of Release Engineering (RELENG)

• — — theories — —
• Release Engineering is Speciality & you have to be SME here.
• RELENG is about communicating Code Change to Production in a reliable manner.
• Google’s Engineer (Dinah McNutt) defined 10 Commandments of RELENG. Just understand then, then, practice it with real use cases.
• It’s at least 2 phases in the software delivery : Build , then Deploy
• Binary Artifacts are the single source of truth for deployment phases
• Binary Artifacts must be scanned against security issues
• Binary Artifacts must be signed for integrity concerns
• From RELENG perspective, releasing non-containerized app is the same as releasing containerized app.
• Secrets must be versioned/managed outside Code repositories
• Semantic version is the single source of truth for the release.
• Semantic version must be used everywhere ( git tag, artifact naming, jira release name, sonar tag, vault secret path,… etc )
• A Release has a scope : Add feature(s) OR/AND Fix a bug(s)
• Releases Scopes can be managed with Task Management Systems ( e.g Github Issues, Jira )
• Use Package Managers ( “mvn/gradle” is a must comparing with “javac”, “Helm” is a must comparing with “kubernetes hard-coded manifests”, .. )
• — — practices — —
• Assuming your code is hosted in Github, ….
• Create a release scope by creating a Github issue ( feature to add, or bug to fix ) .. Take note of the Github issue ID ( ie.g #12)
• Implement the issue #12 as per its acceptance criteria.
• Use #12 in the git commit message.
• Make a meaningful git commit message with Semantic Commit Message. That time, #12 will be in the footer of the commit message.
• Once it’s ready, create Pull-request from the feature branch to the integration branch (dev)
• Install Artifact management system (Nexus Sonatype ), Configure Docker Registry to be served by it.
• Regarding app(s) mentioned at the first part, try to change to way of deployment : Build → Push Artifact to Nexus → In deployment server, Pull it → Run it.
• Install Secret Management system (Vault Hashicorp)
• Regarding app(s) mentioned at the first part, externalize sensitive configuration from the source code of the application(s) → push the sensitive configuration to a Vault secret path → refactor your application to read this configuration externally using environment variable or directly call Vault ( Hint: update README & .gitignore at the end)
• Once the new feature (#12) is deployed, Go back to the Github issue (#12) and close it. .. ( Better, to have bots do that your behalf )

8. Thu Shalt adopt Pipeline Theory for your software delivery:

• Pipeline are invented to deliver things from source x to destination(s) Y.
• A Software Delivery Pipeline (CI/CD pipeline) orchestrates the process of the delivery.
• CI/CD pipeline is a set of quality gates, namely: Build → Deploy.
• Adding quality gates before build/deployment helps to establish more confidence in the pipeline.
• Pipeline is just an orchestrator and it must communicate with the right systems thru APIs until getting things done.
• — — Practices _____
• Install Jenkins
• Configure a declarative pipeline using Jenkinsfile inside the git repo of app(s) mentioned above.
• Implement 2 stages: build , deploy.
• Embed other stages to establish confidence : run unit-tests before build stage.. run static code analysis before the build stage also.
• I used to write unit-tests for my smart contracts in blockchain dapp(s) using ethereum framework.
• Deploy stage requires a communication protocol between CI/CD system and target deployment(s): If it’s VM based deployment, SSH/WINRM is the protocol.. If it’s Kubernetes-based deployment, HTTP protocol (REsT of kubeapi) is the protocol.
• Hint: Deploying containers without kubernetes is classified as VM-based deployment and SSH/WINRM is required.
• Congratulations ! DevOps One-way (from left-to-right) is implemented

9. Thu Shalt apply Pipelines not only for your Apps but also for its Infra

• Infra as Code should give you the impression to treat infra changes as a software.
• Because it’s a software, it must have unit-tests,… and for sure Pipeline.
• Use the right techs/tools (e.g. Terraform, Ansible,… )which comes with main features of infra as code ( modularity, reusability, artifacting, externalizing secret data smoothly).
• — — Practices — —
• Build a pipeline (build / Deploy) for the Cloudformation template mentioned initially in the part (6).

10. Thru Shalt Gather Accurate Feedback about what you deliver

• Measure then Measure the Measure
• 1st Measure = Setup required tools (prometheus, grafana,..) to monitor your system performannce and behavior.
• 2nd Measure= Measure app(s) & fix issues
• 3nd Measure = Keep measuring
• Gather feedback from customer smartly : UX techniques (hitmap,…) can help, Bots also are a good thing to cook.
• Congratulations! DevOps Second-way is implemented

11. Thu Shalt scale by services not by people

• “It’s OK to use our services, but it’s not ok to use our people” said by someone.
• Install tools but expose it as a service
• .e.g Jenkins is a tools.. Users should know it as Pipeline as a Service
• e.g ArgoCD is a tools … Users should know it as CD as a Service.
• e.g …
• Exposing tools to be as-a-service requires building interface between the tools and its users. Interface can be : Rest-API, Configuration File (Jenkinsfile, buildspec.yaml ,… so on )
• System owners must be promoted from System-admins to builder of software operators.
• Cloud is not only the Public Cloud. Build your private Cloud.
• Cloud is architecture before being internet facing or not.
• Some services must be at the level of the API Gateway for AuthN, AuthZ, API economy and other features. (Learn here Kong or 3scale)

12. Thu shalt ship all your Operational Knowledge into a Software

• Migrate the AWS Cloudformation Template to a Terraform Plan.
• Reuse ready-made Terraform modules
• Build your own terraform modules if needed.
• Thru a pipeline, Release all your own Terraform modules as a software library.
• ___ another practice: for kubernetes deployment ___
• Create the whole kubernetes cluster using a ready-made Terraform module.
• Migrate the Kubernetes hard coded Manifests to a Helm Chart.
• All these Code (terraform files, helm chart directory ) must be committed with GIT.
• All sensitive data must be pushed to a secret management system ( e..g: terraform backend = vault) .
• Gitignore must be updated (https://gitignore.io ).

13. Thu Shalt adopt SRE to connect all these principles

• Reliability is the solution of the trade-off between : Stability(Ops) vs Frequent-Change(Dev).
• Practically, SRE foundation is “Code Don’t Lie”; Which means treating everything as code helps to shift from a human operators to a software operators.
• Reliability can built on top of software but it cannot be built on top of people operations.
• A Software can be reliabile by following all GitOps practices: Version the code, Review the Code, Test the Code, Having a Pipeline for that Code.
• The previous principle is a subset of this principle

14. Thu Shalt Go Crazy

• API Maturity Level 4, rfc7807, .. so on.
• Podman, skopeo, sigstore, .. so on
• Kubernetes + ingress + metric-server + grafana + kibana + kafka + … → shipped as a service, used as a service.
• Security in containers: seccomp, SELinux, AppArmor, syscalls, distroless, secret encryption, OIDC with kube-api …, Clair, Trivy, egress router, netnamespace, … etc
• Data pipelines : ingest data, analyze data, …
• Software Defined Storage: Ceph (OSD → RGW, RBD, CephFS) , GlusterFS ( PV → VG → LV thin pool → Brick → volume), …
• Software Defined Everything: SDN, SDDC (datacenter), …. etc
• iso27001 — e.g Calculation of the cooling energy needed for a datacenter by taking into account the heat generated by the bodies of people entering the datacenter ( people bodies are not the same ?? )
• Performance Engineering, I/O , Latency, Throttling, ….
• Kill technical debt
• Hybrid cloud, Direct Connect, Enterprise Architecture,…
• Business Values Driven Architecture
• Kill WIP , Kanban
• TOGAF, Strategies, Principles, Use Cases, Business Goals, Architecture Steering Committee,… so on.
• … so

15. Thu Shalt Practice or figure out ways to Practice

• Sometimes, your company adopted some good technologies, and you got the opportunity to work on them
• Sometimes, you have a good Boss who listened to you and gave you area for experimentation
• But sometimes, you find yourself in the Desert !!
• That time, you need to dig the hole and build the well. I mean you need to generate Labs to yourself by any means. You have to manage it !

16. Thu Shalt Never Stop Learning

• The more you learn/know, the wider your vision become. Do never stop learning !
• Learning is not only studying but also learning from failures.
• Congratulations! DevOps Third-way is implemented

17. Thu Shalt use the right Tech in the Right Context

• In a company where most of teams have only SSH access, Ansible might be better than others.
• In a company where a Private/Public Cloud is enabled, Terraform might be better than others.
• Vendor lock-in is relative , and hard to measure.
• Technology lock-in is inevitable

18. Thu Shalt Think Big

• Be ambassador and not just an implementer
• DevOps Culture is established by practicing the 14 Leadership Principles.

Question : What’s the relationship between the word “DevOps” and these 18 points ? ??????

Answer: Simply! XyzOps means “Operate with Xyz” . DevOps means “Operate with Software Development” Does that make sense ?