The Data Engineering Insider - Week 34 2024

Hey everyone!

Let's get into the weekend with some hot stuff from the Data Engineering world:

Don't Just Focus on Tools!

In this episode of the Plumbers of Data Science podcast, I’m diving into what truly matters when building data platforms and pipelines.

As engineers, it’s easy to get caught up in the latest tools, but real success starts with understanding your data sources and defining clear goals. I’ll walk you through the key questions to ask, from data retention to processing speeds and user needs.

Listen now on all major podcast platforms: https://bit.ly/3SWHtGf

Why Apache Spark is Such an Essential Tool for Every Data Engineer

In this Hero Talk podcast episode, we explore the essentials of learning and mastering Apache Spark. Joining me is Philipp Brunenberg, an experienced Spark developer and educator, who shares his expert roadmap for becoming proficient in Spark.

We discuss why Spark is a crucial tool for Data Engineers, how to set it up effectively, and the best approaches to start your Spark journey.

Listen now on all major podcast platforms: https://bit.ly/3ArChnj

I Want You on the Plumbers of Data Science Podcast!

Are you passionate about Data Engineering? Have a cool story to share about your journey, projects, or innovations? I’d love to showcase your unique experience on my Hero Talk series!

Interested in being featured? Fill out our Google form for a chance to join me as a Hero Talk guest: https://bit.ly/3VRYjrA

Data Engineering Explained

Do you already know my Data Engineering Explained video series on YouTube? Here I break down complex concepts & tools for you and give answers on frequently asked questions.

Check out some the latest episodes:

Good Platform Design

Learn what it takes to do solid data platform design: https://bit.ly/4dx53B8

SQL vs. NoSQL

Learn about the difference between SQL and NoSQL and why this can be so confusing for people: https://bit.ly/3SWrUhB

Data Engineering Alphabet 2024 Visual

In one of my livestreams, I was drawing up the Data Engineering Alphabet 2024 together with the chat. Dive into our comprehensive visual to browse through the most important keywords and concepts of the year!

From Civil Engineering in India to Data Engineering in the U.S.

Check out this blog post where I share insights from an inspiring chat with Ayan Tiwari, who transitioned from Civil Engineering to Data Engineering.

Discover the challenges and triumphs Ayan faced during his career transition, the importance of continuous learning, and how he navigated the shift to working with cutting-edge Data Engineering technologies.

Read now on Substack: https://bit.ly/3LZkNkw

Have fun diving into my top Data Engineering content of the week.

Looking forward to reading your thoughts and questions in the comment sections!

??

Best,

Andreas