Reflecting on QCon SF 2023: Learnings, Insights, and Community in Technology

It has been 3 weeks since QCon Software Development Conferences in San Francisco and it was an amazing experience, not just because I was speaking at the event, but I got to learn so much more than I thought I would.

Shout out to Wesley Reisz, Sid Anand, Hien Luu, Courtney Hemphill, Justin Cormack, and Krys Flores (冯晶) for putting together an amazing set of tracks. The reason I loved the track setup was it focussed on all aspects of Engineering, which involved the latest trends such as Generative AI, scaling systems and Architecture, how to grow as a Staff engineer, Building organisational resilience and more. As an engineering leader, you were more than one hat and when you go to a conference where you get to see a 360-degree view, it hits you differently.

A massive shout out to Morgan Casey , Danny Latimer , the volunteers and the entire crew at InfoQ for the amazing conference organisation.

As speakers, we had the privilege of being in a session with Wesley Reisz, who ran a session on how to present at a conference. Having presented at several conferences over the years, I learned how much I did not know how to present well at a conference.

Of course, I couldn't attend all the talks, but I eagerly anticipate catching up on the ones I missed once they become available online.

One of the highlights of the conference was delving into the thought-provoking technical talks that left a lasting impression. Here are some key takeaways from these insightful sessions:

1. Suhail Patel 's Keynote: The conference kicked off with an exceptional keynote from Suhail Patel, setting the tone for the days ahead. What resonated with me was the idea that organizations are reevaluating architectural patterns based on their unique needs, rather than blindly following trends. In a world where work and products drive architecture, there's no one-size-fits-all solution. It was refreshing to witness Patel's focus on sharing his experiences rather than evangelizing.
2. Surabhi Ahuja 228 Million Subscriptions Management at Netflix : Surabhi Ahuja injected humor into a deep technical discussion, making it not only informative but also entertaining. Managing subscriptions may sound straightforward, but the intricacies of doing so at scale are mind-boggling. The evolution of the solution helped with the understanding of complexities in different geographies and pricing. Solving distributed problems with high accuracy is a pretty hard challenge. Given the scale at which this service was operating, it was amazing to see how auto repair mechanisms were built into the system to provide high availability with high accuracy. The talk also shed light on challenges such as when to create libraries or services, a decision that even large organizations like Netflix find complex. Talks like this make you fall in love with complex distributed system problems all over again.
3. Mike Danilov on Serverless Computing in Amazon Web Services (AWS) : Serverless computing, epitomized by AWS Lambdas, has transformed the tech landscape. Mike Danilov and his team at AWS shared the complexities and brilliant solutions behind this innovative product. Their insights revealed the remarkable work that goes into making serverless computing accessible to all. One needs to pay attention to every little detail, to knock off a few seconds in bringing up a lambda function and the devil lies in the details. After the talk, I realised the thing that has been achieved in the last 2 decades is the evolution of abstraction from all the complexities to an end user, enabling people to focus on value creation. The technical details were astonishing and to be honest some of they were beyond my current understanding of networks and systems.
4. Jules Damji on Distributed Computing with Ray: Jules S. Damji's session on distributed computing with Ray was nothing short of stellar. His eloquent explanation of the necessary changes in distributed computing and their application at Anyscale with Ray provided invaluable insights. There is a difference in taking Machine learning from development to production, especially if you need to run it at scale. These lessons are particularly relevant to us at Scoutbee as we embark on implementing Ray for large-scale machine learning inference and various non-ML workloads. The privilege of catching up with Jules at the speaker dinner added a layer of tech trivia that enriched the experience. Looking forward to catching up with Jules Damji soon.
5. Redis as a Vector Store, Revealed by Sam Partee : Redis took on a new role in my perception, thanks to Sam Partee's engaging presentation. He showcased how Redis can serve as a vector store at scale, especially for Retrieval Augmented Generative work. Production considerations, such as caching, came to the forefront during this talk, offering cost-effective solutions, particularly when interacting with Large Language Models (LLMs). The infectious energy and enlightening discussions at the speaker dinner, fueled by Sam's insights on high-performance computing and infrastructure, were a true highlight.
6. Shreya Rajpal on Building Guardrails for Enterprise Applications with LLMs: Shreya Rajpal's presentation on building guardrails for enterprise applications with Large Language Models (LLMs) struck a chord with me. It emphasized the significance of reliability and verifiability in the LLM journey, a topic we're actively exploring at Scoutbee. Our shared interest led to engaging conversations during the speaker dinner and beyond, reinforcing the importance of these principles in our respective journeys.
7. Adam Schirmacher on Managing Staff Engineers: Adam's session hit home with its authenticity. He shared scenarios and experiences that felt remarkably relatable, as though we were in the same room when these events unfolded. The art of establishing clear responsibilities for staff engineers and supporting those who aspire to reach that level is a formidable challenge for engineering leaders. Adam delved into the dynamics of different personality types within teams and how they influence team members, offering valuable insights into effective team management.
8. Shweta Saraf on Managing Engineering Productivity Amid Constant Change: Shweta's talk was nothing short of incredible. It felt like a therapeutic session for engineering leaders. Her insights into common accidental diminishing tendencies and her own journey through various organizations, including her latest role at Netflix, were enlightening. Building a successful team and organization requires careful consideration, and Shweta's wisdom resonated deeply. This was one of those talks that I would have to listen to time and again to just recalibrate my mental state as an engineering leader.
9. ????♀? Joy Ebertz on Things Every Staff Engineer Should Know: Joy Ebertz brought an entertaining twist to her talk, covering topics like domain-driven design, API design, and service architecture and how they converge to solve complex problems. She shed light on the emergence of design patterns within organizations and how these patterns can inadvertently hinder progress. As I listened, Conway's Law echoed in the background, reminding me of my own journey in dealing with poorly thought-out systems and services.

Participating in a conference isn't just about sharing our own insights; it's an open invitation to embrace the wealth of experiences and knowledge that others bring to the table. QCon SF 2023 was a remarkable journey for me. It provided an opportunity not only to learn and exchange ideas but also to connect with a diverse and generous community of individuals who were eager to share their wisdom. This conference was a vivid reminder that in the ever-evolving world of technology, the power of collaboration and the joy of continuous learning are at the heart of our shared journey.