ByteByteGo

I’ve been writing the system design newsletter for 12 months. Here are the 5 most popular ones: ?? 1. From 0 to Millions: A Guide to Scaling Your App 2. A Crash Course in Caching 3. API Architectural Styles 4. How does ChatGPT work? 5. 8 Data Structures That Power Your Databases Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/3FEGliw .

Things Every Developer Should Know: Concurrency is ?????? parallelism. . . In system design, it is important to understand the difference between concurrency and parallelism. As Rob Pyke(one of the creators of GoLang) stated:“ Concurrency is about ?????????????? ???????? lots of things at once. Parallelism is about ?????????? lots of things at once." This distinction emphasizes that concurrency is more about the ???????????? of a program, while parallelism is about the ??????????????????. Concurrency is about dealing with multiple things at once. It involves structuring a program to handle multiple tasks simultaneously, where the tasks can start, run, and complete in overlapping time periods, but not necessarily at the same instant. Concurrency is about the composition of independently executing processes and describes a program's ability to manage multiple tasks by making progress on them without necessarily completing one before it starts another. Parallelism, on the other hand, refers to the simultaneous execution of multiple computations. It is the technique of running two or more tasks or computations at the same time, utilizing multiple processors or cores within a computer to perform several operations concurrently. Parallelism requires hardware with multiple processing units, and its primary goal is to increase the throughput and computational speed of a system. In practical terms, concurrency enables a program to remain responsive to input, perform background tasks, and handle multiple operations in a seemingly simultaneous manner, even on a single-core processor. It's particularly useful in I/O-bound and high-latency operations where programs need to wait for external events, such as file, network, or user interactions. Parallelism, with its ability to perform multiple operations at the same time, is crucial in CPU-bound tasks where computational speed and throughput are the bottlenecks. Applications that require heavy mathematical computations, data analysis, image processing, and real-time processing can significantly benefit from parallel execution. --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

Top 5 Kafka Use Cases Originally built for log processing, Kafka now powers a wide range of applications. Its durable message storage and flexible data access enables consumers to pull records at their convenience. Here are some popular Kafka use cases: - ?????? ???????????????????? ?????? ????????????????: Efficiently handles massive volumes of log data for analysis and insight generation. - ???????? ?????????????????? ?????? ??????????????????????????????: Powers real-time data streaming to deliver personalized recommendations. - ???????????? ???????????????????? ?????? ????????????????: Facilitates real-time monitoring and alerting systems for timely responses to system metrics. - ???????????? ???????? ?????????????? (CDC): Captures and processes database changes to keep data in sync across systems. - ???????????? ??????????????????: Supports the seamless migration of systems by ensuring data consistency and availability. Do you have any Kafka use cases to add to this list? --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

What is DevSecOps? . . DevSecOps emerged as a natural evolution of DevOps practices with a focus on integrating security into the software development and deployment process. The term "DevSecOps" represents the convergence of Development (Dev), Security (Sec), and Operations (Ops) practices, emphasizing the importance of security throughout the software development lifecycle. The diagram below shows the important concepts in DevSecOps. 1 . Automated Security Checks 2 . Continuous Monitoring 3 . CI/CD Automation 4 . Infrastructure as Code (IaC) 5 . Container Security 6 . Secret Management 7 . Threat Modeling 8. Quality Assurance (QA) Integration 9 . Collaboration and Communication 10 . Vulnerability Management --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

GET, POST, PUT... Common HTTP “verbs” in one figure. ? 1. HTTP GET? This retrieves a resource from the server. It is idempotent. Multiple identical requests return the same result.? ? 2. HTTP PUT? This updates or Creates a resource. It is idempotent. Multiple identical requests will update the same resource.? ? 3. HTTP POST? This is used to create new resources. It is not idempotent, making two identical POST will duplicate the resource creation.? ? 4. HTTP DELETE? This is used to delete a resource. It is idempotent. Multiple identical requests will delete the same resource.? ? 5. HTTP PATCH? The PATCH method applies partial modifications to a resource.? ? 6. HTTP HEAD? The HEAD method asks for a response identical to a GET request but without the response body.? ? 7. HTTP CONNECT? The CONNECT method establishes a tunnel to the server identified by the target resource.? ? 8. HTTP OPTIONS? This describes the communication options for the target resource.? ? 9. HTTP TRACE? This performs a message loop-back test along the path to the target resource.? ? Over to you: What other HTTP verbs have you used? --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

What is a deadlock?? .? .? A deadlock occurs when two or more transactions are waiting for each other to release locks on resources they need to continue processing. This results in a situation where neither transaction can proceed, and they end up waiting indefinitely.? ? ?? Coffman Conditions? The Coffman conditions, named after Edward G. Coffman, Jr., who first outlined them in 1971, describe four necessary conditions that must be present simultaneously for a deadlock to occur:? ? - Mutual Exclusion? - Hold and Wait? - No Preemption? - Circular Wait? ? ?? Deadlock Prevention? - Resource ordering: impose a total ordering of all resource types, and require that each process requests resources in a strictly increasing order.? ? - Timeouts: A process that holds resources for too long can be rolled back.? ? - Banker’s Algorithm: A deadlock avoidance algorithm that simulates the allocation of resources to processes and helps in deciding whether it is safe to grant a resource request based on the future availability of resources, thus avoiding unsafe states.? ? ?? Deadlock Recovery? - Selecting a victim: Most modern Database Management Systems (DBMS) and Operating Systems implement sophisticated algorithms for detecting deadlocks and selecting victims, often allowing customization of the victim selection criteria via configuration settings. The selection can be based on resource utilization, transaction priority, cost of rollback etc.? ? - Rollback: The database may roll back the entire transaction or just enough of it to break the deadlock. Rolled-back transactions can be restarted automatically by the database management system.? ? Over to you: have you solved any tricky deadlock issues? --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

Ranked right after Nexus and Elon Musk biography. Wish it could stay like this forever. If you are interested, check it out here: https://lnkd.in/eAvT42Eb

25 Papers That Completely Transformed the Computer World.? ? 1. Dynamo - Amazon’s Highly Available Key Value Store? 2. Google File System: Insights into a highly scalable file system? 3. Scaling Memcached at Facebook: A look at the complexities of Caching? 4. BigTable: The design principles behind a distributed storage system? 5. Borg - Large Scale Cluster Management at Google? 6. Cassandra: A look at the design and architecture of a distributed NoSQL database? 7. Attention Is All You Need: Into a new deep learning architecture known as the transformer? 8. Kafka: Internals of the distributed messaging platform? 9. FoundationDB: A look at how a distributed database? 10. Amazon Aurora: To learn how Amazon provides high-availability and performance? 11. Spanner: Design and architecture of Google’s globally distributed databas? 12. MapReduce: A detailed look at how MapReduce enables parallel processing of massive volumes of data? 13. Shard Manager: Understanding the generic shard management framework? 14. Dapper: Insights into Google’s distributed systems tracing infrastructure? 15. Flink: A detailed look at the uni?ed architecture of stream and batch processing? 16. A Comprehensive Survey on Vector Databases? 17. Zanzibar: A look at the design, implementation and deployment of a global system for managing access control lists at Google? 18. Monarch: Architecture of Google’s in-memory time series database? ? 19. Thrift: Explore the design choices behind Facebook’s code-generation tool? 20. Bitcoin: The ground-breaking introduction to the peer-to-peer electronic cash system? 21. WTF - Who to Follow Service at Twitter: Twitter’s (now X) user recommendation system? 22. MyRocks: LSM-Tree Database Storage Engine? 23. GoTo Considered Harmful? 24. Raft Consensus Algorithm: To learn about the more understandable consensus algorithm? 25. Time Clocks and Ordering of Events: The extremely important paper that explains the concept of time and event ordering in a distributed system? ? Over to you: I’m sure we missed many important papers. Which ones do you think should be included? --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

There are over 1,000 engineering blogs. Here are my top 9 favorites:? ? - Netflix TechBlog? - Uber Blog? - Cloudflare Blog? - Engineering at Meta? - LinkedIn Engineering? - Discord Blog? - AWS Architecture? - Slack Engineering? - Stripe Blog? ? Over to you - What are some of your favorite engineering blogs? --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .

Popular interview question: how to diagnose a mysterious process that’s taking too much CPU, memory, IO, etc?? ? The diagram below illustrates helpful tools in a Linux system.? ? ??‘vmstat’ - reports information about processes, memory, paging, block IO, traps, and CPU activity.? ??‘iostat’ - reports CPU and input/output statistics of the system.? ??‘netstat’ - displays statistical data related to IP, TCP, UDP, and ICMP protocols.? ??‘lsof’ - lists open files of the current system.? ??‘pidstat’ - monitors the utilization of system resources by all or specified processes, including CPU, memory, device IO, task switching, threads, etc.? ? Credit: Diagram by Brendan Gregg --? Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://bit.ly/bbg-social #systemdesign #coding #interviewtips? .