10X Engineering: A Series

About the series

This series is an initiative for the engineering community to learn, share and grow together in a journey of professional and personal excellence with the spirit of 10X. We will share this via blogs, references, masterclasses, live streaming and podcasts. It will be done with partnership of friends and community. This is an evolving and live content.

Subscribe to our newsletter for updates.

About me

Find me on Github or Linkedin

Hi you may call me Ayush. I am the author and the facilitator of this series. Here I will take you through a journey of exploration of the wonderful and exciting world of technology.

For me engineering is a craft I enjoy very much. I also wear other hats like a founder, innovator, teacher, mentor, family, friend and of course eventually everything human! :-)

By virtue of intention, effort and destiny's touch, I landed in IIT Kanpur in 2002 to study engineering. And by stroke of destiny, I landed in Computer Science department while I wanted to originally take up aeronautics. That's another story. I will be sharing more of my journey through stories as we progress in this series. But here I want so state that couple of times in my lifetime, I considered moving out of tech to pursue other callings like MBA. I am so glad I choose to stay invested in tech. It continues to be my way of expression, innovation and service to the world. I built or continue to build somethings I feel very satisfied about.

For me tech is my ikigai. It does four things for me: I love to play with it. It earns me livelihood. It helps me grow. It is also my means to serve the society.

I founded Godspeed for the mission to democratise technology for the tech ecosystem in a collaborative manner. This series is as much part of the mission as is the tools that we make for the same purpose.

I am starting this series to share what I have learned with the world. Here I will focus on the path of tech, its purpose and engineering.

I will be maintaining the content of this particular series and run this program called 10X Engineer, aimed primarily for tech professionals, students and innovators, who are equally desirous of learning and growing in their journey.

About the series: 10X Engineering

At Godspeed Systems, we believe in excellence and want every engineer to be a 10X version of himself or herself. For the same every engineer needs to

• Have a certain mindset called the 10X mindset
• Continuously develop understanding, information and knowhow
• Keep on to play, serve, learn and earn

Hence we will be running this series where we share with the world whatever little we know or are making, and also continue to learn and upgrade ourselves. Consider 10X Engineering not as a course but a guide designed to empower developers and teams at every stage, providing a systemic approach to thinking, solution design and application.

The choice of tools for the journey

The path to 10X is independent of tools. Hence you are free to choose tools that suit your purpose.

What makes the difference between excellent and poor engineering (and engineer) is not the tools we know or use. It is design and application using fundamentals and first principles. The evidence of excellence or success lies in the final output that we ship from day to day, as a team. The job of tools is to help humans make stuff excellently. So a good engineer or team is strong on concepts, and can apply those fast and properly with any tools.

A 10X engineer is like a really good athlete who can run well with any shoes. A novice athlete can not run well with even the best shoes.

In this series, there will be a recommendation from our side but you are indeed welcome to pick any toys you wish to play the game with. We have the freedom to choose and switch between the tools that serve our purpose.

Topics every dev should know

These are some of the topics which I believe every engineer must know, from a little bit to expert level knowledge.

The chapters

Chapter 0: Making of a 10X Engineer & team(Published)

What is Software Engineering

Types of Careers and Roles in Software Engineering

Chapter 1 - Touching the fundamentals(Published #wip)

Pre-requisite

Knowing atleast one language

• JS/TS (preferred for full stack aspirants)
• Or Python, Java, Golang etc.

Knowing basic datastructures and algos

Software Engineering Lifecycle and environments

Data

• What is data
• The four types of data
• Software is all about playing with data

Retrieve/receive, compute, transform and send/store data

What is business logic and what is boilerplate

Software Testing and its importance

Single bad UX can be more powerful than 99 good features combined

Unit, integration and functional testing

Basic Unix commands

Best practices

1. Coding guidelines
2. Schema driven development
3. Configure over code
4. First principles based approach
5. Shift left approach
6. Continuous Integration and Continuous Deployment
7. Decoupling
8. Project management and teamwork - best practices
9. Version Control and Git

Chapter 2 - Programming advanced

Paradigm of programming languages

• The four generations of programming languages
• All about P and NP
• Types of programming languages- Procedural, Object Oriented, Functional, Aspect Oriented
• When to use which languages?

Concurrent data access with mutable and immutable data structures.

Programming - best practices

The art of handling errors and debugging

• Eating versus throwing the exceptions
• How to inform the user or how to log (best practices)- Error code, description, context, and tips for resolution

How to debug

1) Locally

• Reading logs & stack trace with eye for detail
• Step-through debugging in the IDE

2) On hosted environments

• Using monitoring tools like Grafana, CloudWatch, ELK, Datadog etc.
• What are distributed traces and spans?
• Alerting
• Pre-emptive and post-error

Let’s write some Unit Tests

Chapter 3 - Modern software architecture - basics

Evolution of software architecture

1. Monolithic
2. 3 tier
3. N tier
4. Service Oriented
5. Microservices
6. Serverless
7. Event driven architecture
8. Server-Client architecture

Three pillars of any microservice/service

• Eventsources, datasources (APIs, datastores) and workflows/functions

Eventsources and events

• Two kinds of events

1) Sync- All kinds of APIs whether REST, Graphql, gRpc, SOAP etc

2) Async - Message Bus, Socket and CRON

API protocols

• What is an API schema
• What is Schema Driven Development and its benefits

Three main modern standards

1) REST

2) Graphql

3) gRpc

Two kinds of Datasources

• Datastores and APIs (other services)

3rd gen Service development frameworks and ecosystem

• Nodejs, Springboot, Gomicro, Django etc.

4th gen Meta-frameworks for service development

• Godspeed

On the UI side:

• Web and native mobile apps
• HTML/CSS/JS
• Frameworks like React & Angular
• Strategies for optimising site load time

Web sockets - what it is, its benefits and real world use cases

Develop some APIs and basic UI

Writing Integration and UI tests for APIs

• Chai/Mocha in JS or Robot in Python etc.
• Selenium versus no code UI test frameworks

Chapter 4 - The world of data and datastores

Types of datastores

• SQL (row based typically used for OLTP, column based popular for OLAP)
• NoSQL (key-value, document, graph)

Important aspects of the datastore domain

• Model (entities, types, relationships), ACID properties, indexes, constraints, querying, joins, materialized views, normalization/denormalization
• Replication and sharding

Popular datastores

• SQL (Postgres, Mysql), Document (Mongodb, Elasticsearch), Search Engine (Elasticsearch), Cache (Redis, Memcached, in Memory), Analytics (Clickhouse, Duckdb, Elasticsearch), Graph (Neo4j, DGraph)?, Vector (Elasticsearch)

Search and filtering: Text based and vector based

CachingSearch and filtering: Text based and vector based

Caching

Types

• OS
• In memory
• Browser
• CDN
• Gateway
• Caching service

Cache eviction policies

Data warehouse and data lakes

Migration

• Model
• Model with data
• Storage (location) migration

Deep dive: Postgres/Mysql (SQL) and/or Mongodb/Elasticsearch (Nosql)

What is Data federation and its use cases

• Popularity of Graphql
• Backend for frontend design pattern (BFF)

Dual writes and consistency (real time versus eventual)

• About Primary and secondary datastores
• When is it needed?
• Syncing mechanisms

Mutation subscriptions (via CDC, Graphql, message bus)

Lazy loading versus write through cache

• Data Pipelines - ETL and ELT
• Subscribing to filtered events: Kafka with KSQL

Chapter 5: Security

Layers of security

• Perimeter, network, endpoint, application, data

User Authentication

• JWT, Oauth2, Simple Auth

Authorization for endpoint and data access

• RBAC and ABAC
• Actor, resource, action, context
• Access Control and Policy engines

Application security

• Secret management and vaults
• Scans (code, image, dependencies)

Data security

• Encryption of data at rest and transit

A brief history

• Deterministic and non-deterministic encryption
• One way and two way encryption
• Encryption algorithms

Redaction of sensitive data from logs

Chapter 6: Performance profiling & debugging

Load Profiling using Jmeter or similar tool

• Measuring latency (99P, 90P, 50P)
• CPU and RAM usage

Debugging performance bottlenecks

• Database, API calls, Code (CPU and mem profiles)

Detecting Memory leaks

DEPLOYMENTS

Chapter 7: OS and networking

Hardware: CPU, RAM, DISK

OS — Unix (Linux, BSD and Mac), Windows

Concepts: Caching, swapping, threading/cores, process, concurrency (locking, counters etc)

Networking (IP address and port), Firewall, Proxy,

Unix basic commands recap

Chapter 8: Containerization and scaling strategies

What is containerization? What is its history and modern trends?

When should it be used?

Tools: Docker, Kubernetes

Chapter 9: Continuous integration & Deployment

What is CI/CD and its benefit in Shift Left Approach of Software Development

Role of the CI process and CD process

CI essentials — automation testing, security scans, containerisation, storing in artifactory

CD — Deployment strategies — Blue Green and Canary

What is infrastructure as code.

Popular tools: Terraform, Ansible, ArgoCD

Tool: ArgoCD: setup argocd and deploy over K8s using ArgoCD

Vertical versus horizontal scaling

Microservice mesh

• Proxy sidecar pattern (for left-right traffic)
• Gateway (for north-south, egress/ingress traffic)

Gateways

• Popular gateways: Istio/Linkerd, Kong, Tyk, and traditional (Nginx, Apache2)
• Load balancing strategies
• Rate limiting strategies

Chapter 10: Observability (Not just monitoring)

Why do we need observability?

What all can we observe?

What is the role of SRE?

Concepts:

• Temple: The six pillars of observability — Traces, Exceptions, metrics, logs, events
• Alerts
• Open Telemetry standard by CNCF
• Advanced: Use of AI/ML

Tools

The FOSS based Grafana stack and ELK, Datadog, Splunk

• Setup Grafana stack. Integrate with a service in Godspeed meta-framework or any 3rd gen framework. Customize a Grafana dashboard.

Chapter 11: ML/AI

When do we need ML? And what is ML?

AI: Its uses and technical overview

• Models
• Langchain
• Vector stores
• Clustering
• Neural Networks
• Semantic analysis and meta-data extraction
• Sentiment analysis
• Popular services, open source models and tools
• Use cases

Chapter 12: Advanced Topics

Serverless functions or Function as a service (FAAS)

Event (Async) driven architecture. Pros and cons.

Distributed workflow management

• Orchestration versus Choreography
• Distributed transactions (Saga pattern) and distributed locking
• Retries with backoff, circuit breaking, “rollback”, retry queue, dead letter queue

What is a rule engine? And its use cases. How can dynamically computed policies and workflows come together for highly agile and fluid product offerings or business use cases.

Chapter 13: Case studies and use cases (study with practical examples and practice)

Food delivery app delivery tracking with mutation subscriptions and notifications

Loan Origination System with rule engine based dynamic workflows

Ecommerce distributed workflows

Real time fraud detection using KSQL, Elasticsearch, Data structures

Movie recommendations: Content based and collaborative filtering based approaches

Shortest path recommendations for a navigation app

(Artificially) Intelligent customer support agents

• Natural language to database or API queries. Dynamically compute workflows. Converting response to natural language.

That’s it for now!

For participating in the learning and sharing journey as a community, you can join us on Discord,Github, Youtube, social channels and subscribe to the blog’s newsletter.

Stay tuned for the series. We have a long way to go.

The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, And miles to go before I sleep.

Original Blogs belongs to https://godspeed.systems/blog/10x-engineering-a-series