3 ingredients for success in Netflix Engineering

Netflix has a unique engineering culture focused on high talent density in a “freedom and responsibility” environment. What does it take to succeed as a back-end software engineer at Netflix? I’ve interviewed 100s of developer candidates for Netflix and have mentored 10s of developers across various levels of experience. Here are 3 key ingredients for success along with real world examples.

Be Curious

Ask, Challenge, Explore

To scale as an engineering organization, we rely on coarse grain levels of abstraction to reason about the systems we’re developing. Abstractions like eventual consistency, consensus algorithms, load balancing, caching, persistence stores, message queues and more form the building blocks for the distributed systems we create. We add business logic on top of these reusable components to deliver our product offering.

There are hundreds of libraries and services in use in Netflix’s cloud architecture. You can get your job done by just focusing on your own application’s business logic. Conversely, you can get lost in a maze of code if you dive into the layers beneath. But if you never look beneath, you’ll miss the opportunity for deeper understanding and questioning that leads to breakthroughs.?

Tom, a playback systems lead engineer, was curious to understand the end to end flows for how we enable “clicking play”. He analyzed the abstract and concrete aspects of this part of our architecture. One key component was an in-memory metadata solution for movies and shows and their encoded streams that had to be dynamically updated in every server instance responsible for playback logic. He wondered about possible conflicts between keeping gigabytes of memory up to date while also serving real time customer traffic. He proposed inverting the approach - precomputing the metadata transformations needed for playback and caching these in a separate external service. It was a paradigm shift that resulted in big wins in resiliency and efficiency.

Daniela, an API and resiliency expert, wondered what would happen if we sharded traffic to our monolithic API layer as a stepping stone towards a larger migration to microservices. Much of the org was skeptical that this was worth it, instead favoring focusing on the multi-year decomposition of that API monolith. She asked “why not” and her team took a quarter to experiment with this idea. This proved to be one of the largest cost savings wins that year and significantly improved the resiliency for this critical API tier. Daniela continues to challenge the status quo and unlock wins in areas others had missed.

Sunny, a security applications expert, noticed a trend in production incidents related to how we managed identity tokens in our server ecosystem. The logic was spread across too many systems, making it hard to reason about what was happening where and why. Rather than treating symptoms, he focused on solving the root cause. He drove a redesign of our edge authentication solution that simplified identity management, improved operability, and, most importantly, reduced unnecessary logouts for our members.

The strongest engineers I’ve worked with at Netflix bring an intellectual curiosity to their work, combined with refined judgment on which areas to drill into. They ask, challenge, and explore what’s not there already; these key practices have led to many of Netflix’s cloud architecture breakthroughs.

Be Versatile

See a need, meet the need

The most successful engineers at Netflix demonstrate versatility and end to end ownership. In cross functional projects, they identify the gaps between what’s needed and who’s involved. They fill the gaps and encourage others to do so too.

Think of all the roles needed for a software project to succeed. On the technical front, this may include architect, developer, tester, release engineer, support engineer, data analyst, operations engineer, and more. On the organizational front this may include product owner, business analyst, project manager, and more. Every project requires a variety of roles to be successful. But most projects don’t have a 1:1 mapping between function and person. Lack of experience in a role doesn’t mean lack of competence; learning a new role grows the individual and benefits the team. The gaps provide the opportunity to demonstrate versatility.

Dhruv, a distributed systems developer, had built a scalable aggregation solution for device data. He was interested in moving into an adjacent space but didn’t know the domain. He volunteered to spend a quarter redesigning the testing infrastructure for that domain. Through this, he learned the domain from the inside out and spurred adoption of this new test framework across his team.

Robert, a back-end expert, proposed a new architecture leveraging GraphQL federation that had the potential to benefit 100s of engineers. The partner teams were excited about the ideas but worried about the amount of cross functional coordination and dependency management required to make it happen. Despite not having led an effort of this size, Robert volunteered to manage the project. With a growth mindset, he sought out advice, coaching, and feedback on how to do this well. With perseverance, he succeeded in the project manager role while also developing part of the federation solution.

Jennifer, an API systems expert, stepped up as an API lead for our mobile games discovery efforts in 2021. We assembled the technical experts from each area in the stack (e.g., mobile UI, API, back-end), but each person tended to focus on just their part. Jennifer donned her project manager and end-to-end debugging hats and started connecting the dots across the stack. She brought equal parts technical and partnering expertise to ensure we delivered on launching mobile games discovery before the end of the year.

Go far together

If you want to go fast, go alone. If you want to go far, go together.

In some corporate cultures, individual success is rewarded above group achievements. To get ahead you must prioritize “I” ahead of “we”. Inside of Netflix, there are no individualized incentives like performance based bonuses; your impact gets amplified if you deliver results through group efforts. This aligns well with research showing that teams perform better in environments that support and encourage collective efforts.?

In my developer days at Netflix, we had two primary APIs for the consumer streaming experience. One was the API for playback, which was a strictly defined protocol running on a slowly changing but well instrumented service. The other was the API for content discovery, which was a rapidly evolving layer that prioritized flexibility and dynamic protocols.

I was working on the playback side and got frustrated with my colleague on the content discovery side over a perceived lack of operational rigor. I expressed my frustration directly to my colleague and that team’s manager. They listened graciously. They didn’t dismiss my concerns. More importantly, they shifted the conversation to what we could achieve together with our different strengths rather than letting our differences divide us. That grew into a multi-year collaboration to bring the best of each side into our playback and content discovery APIs - clearer protocols, operational rigor, and flexibility to evolve rapidly.

Very few problems require going alone; most meaningful work requires group effort. Time spent pursuing alignment around shared goals, and then pursuing those goals together, is how we’re able to succeed over the long term.

Conclusion

Many Netflix engineers deliver high impact results while also growing their careers by combining the three ingredients of curiosity, versatility, and going far together. For those who identify strongly with Netflix’s values and culture, it is a place where you can spend years solving big problems with stunning colleagues.