Pitch a project, never an idea

Thank you so much for reading this edition of the newsletter ?? If you found it interesting, you will also love my courses

1. System Design Course for Beginners
2. System Design Course for SDE-2, SDE-3, and above
3. Redis Internals Course

Pitch a project, never an idea

Most of us struggle to get our ideas to be prioritized and worked upon, they are awesome, but still, leadership ignores them, why? The answer is simple - we pitched the idea and not the project. Let's understand the difference.

Ideas are abundant and easy to come by, but execution is what separates successful projects from mere concepts. Leadership needs to see a clear execution path, and that's where many engineers fall short.

When you pitch a project, you show that you've thought through the details and are prepared to bring the idea to life. Some of the things you can do are

• estimating the number of engineers required
• list down skill set requirements
• estimate the timeline of the project with a buffer
• list down the risks involved
• predicting potential outcomes and benefits for the org

This not only increases your chances of approval but also demonstrates your capability to see the project through to completion and thus increases the success rate of getting your project approved.

It is all about doing your homework well. Pitch a project, not just an idea.

You can find this post on my LinkedIn and Twitter; do leave a like.

By the way,

Being hands-on is the best way for you to learn. Practice interesting programming challenges like building your own BitTorrent client, Redis, DNS server, and even SQLite from scratch on CodeCrafters.

Sign up, and become a better engineer.

?? Video I posted this week

This week I posted Designing Realtime Gaming Leaderboard with Rockset

This is the 5th video in the Rockset Database Internals Series. In this one, I thought of designing a Realtime Gaming Leaderboard and the architecture is similar to what companies ship to production.

Apart from the actual design, I also shared the thought process behind every design decision helping you build the right intuition and understanding the read/write path separation framework.

?? Paper I read this week

This week I spent reading Predicate Caching: Query-Driven Secondary Indexing for Cloud DataWarehouses

This week I read a research paper by Amazon that talks about how they optimized Redshift and made certain sets of queries 10x faster.

A common way to speed up queries is by caching the 'result' data or creating Materialized Views closer to what is expected in results. Both of these approaches are prone to staleness. Although caching is common, it has an average hit rate of around 20%; while MVs are expensive to build and maintain.

Amazon flipped the approach, and instead of caching the result data (through caching of MV), they went ahead with Predicate Caching which stores and caches the ranges of rows from table scans and joins that match specific query predicates, commonly observed on a Redshift Cluster.

So, for a given query, it evaluates the query predicates, and if the rows are in the cache, they are straight away processed, if not then they are evaluated in a regular way from primary table storage. It minimizes staleness because it now need not recompute the entire cache, given what is being cached are rows and not computed results.

You can download this and other papers I recommend from my papershelf.

A quick introduction to PostgreSQL's amusing feature - Tablespaces

Tablespaces are the storage locations for data files, essentially directories on the filesystem where the actual data is stored. When we create a table, it is made part of a default tablespace, and, all the meta and data files are created are placed in it.

We can also create additional tablespaces to organize the data across different storage locations. For example, we can create one tablespace for hot data on SSD and one tablespace for cold data on HDD.

Now when we keep the frequently accessed tables in hot tablespace, we are optimizing for the performance of queries accessing that data; and by moving infrequently accessed data to cold tablespace, we are saving cost.

By the way, we can provide a tablespace while creating the table and even move the table from one tablespace to another by just firing an alter table command. Apart from performance and cost, tablespaces are also used for moving tables across disks for load distribution, multi-tenancy, and even geo-proximity.

You can find this post on my LinkedIn and Twitter; do leave a like.

?? Interesting articles I read this week

I read a few engineering blogs almost every single day, and here are the three articles I would recommend you to read.

• Why we no longer use LangChain for building our AI agents
• Lessons Learned from Scaling to Multi-Terabyte Datasets
• C can be memory-safe

Thank you so much for reading this edition of the newsletter ?? If you found it interesting, you will also love my courses

1. System Design Course for Beginners
2. System Design Course for SDE-2, SDE-3, and above
3. Redis Internals Course

I keep sharing no fluff stuff across my socials, so, if you resonate do give me a follow on Twitter, LinkedIn, YouTube, and GitHub.