Seattle Data Guy

Joe Reis recently wrote a piece titled, Playing Not To Lose. The piece referenced the fact that many data teams are simply going through the motions and doing "data stuff" as he put it. Gordon Wong later tagged in and referenced the point of being defensive vs offensive in terms of what work you take on. This made me want to dig even deeper into what Joe Reis meant and how he feels data teams can go on the offensive effectively to take on projects that actually are worth investing in. Or perhaps are there certain companies that don't benefit as much from data teams? So don't miss out on this discussion!

We’re hiring, come work with me! Estuary is building a unified data platform for enterprises and we’re looking for someone with a technical background in data and experience in content creation to help us convey our message to an audience that only settles for the best.

I don't know who needs to hear this, but not every dashboard needs to be real-time. That being said, it is becoming far easier to implement real-time data workflows! In particular, I just saw that Google announced continuous queries for BigQuery. Meaning you can now express complex, real-time data transformations and analysis using the familiar SQL. First off, another +1 for SQL. Second, this continues to make real-time an easier and easier decision. The barriers that once existed in implementing real-time event-driven approaches are slowly dropping. I will provide one warning, I have noticed that real-time can still be expensive on the compute side for one reason or another, but if you need to implement a real-time workflow, the technical friction that once existed continues to be lowered. Also, if you're looking to read more about how you can implement real-time workflows, then you can check out the article written by Jobin George and Daniel Palma from Estuary https://lnkd.in/gE63bg3W

Are you considering becoming a data engineer in 2024 and soon to be 2025? Then here is a free mini crash course with +25 free resources you can use to get up to speed fast. It's broken down by the basics, then goes into higher level concepts like OLTP vs OLAP, data modeling, and finally goes into a few solutions like Airflow, Snowflake, etc. The Basics (Programming, SQL, The Cloud, etc) Data Engineering Vocab 101 1. https://lnkd.in/gRQagRax What Is Data Engineering - Why Is Data Engineering Important? 2. https://lnkd.in/gFmtavpu SQL For Beginners By Alex Freberg 3. https://lnkd.in/gAXcqn8W Python Tutorial - Python Full Course for Beginners Mosh Hamedani 4. https://lnkd.in/gpdCy4mf Python Libraries You Should Know As A Data Engineer - Python For Beginners 5. https://lnkd.in/gzupGDh4 Data Warehouse Tool Kit 6. https://lnkd.in/gVSCTWYi Becoming A Better Data Engineer - Tips On Translating Business Requirements 7. https://lnkd.in/gnQNyZbs Normalization Vs Denormalization 8. https://lnkd.in/gf_ivben Data Modeling - Walking Through How To Data Model As A Data Engineer - Dimensional Modeling 101 9. https://lnkd.in/g4QcjgXy What Is Orchestration by Mage and Matt Palmer 10. https://lnkd.in/g4-rEnWu From Basics to Challenges: A Data Engineer's Journey with APIs 11. https://lnkd.in/gjvesj7z Higher Level Concepts What Is A Data Pipeline 12. https://lnkd.in/ginyTQYM Transactional Databases Vs Data Warehouses Vs Data Lakes 13. https://lnkd.in/gxpbCRPi Why Is Data Modeling So Challenging 14. https://lnkd.in/gd5CQrZs How To Set Up Your Data Analytics Team For Success 15. https://lnkd.in/ey5tbjPQ OLTP vs OLAP 16. https://lnkd.in/gfMqKQHB How to design resilient and large scale data systems by Zach Wilson 17. https://lnkd.in/gWXzeqre Setting Standards For Your Data Team 18. https://lnkd.in/gSpjWpvt How To Come Up With A Data Engineering Project? 19. https://lnkd.in/gZTGK2Cz 7 Data Engineering Projects To Put On Your Resume 20. https://lnkd.in/gPtU5TSm ?? Uber Data Analytics | End-To-End Data Engineering Project by Darshil Parmar 21. https://lnkd.in/gAy7mGjJ Some technologies data engineers might need to know. Using AWS Lambda As A Data Engineering 22. https://lnkd.in/gR3HT6Jh The Realities Of Airflow - The Mistakes New Data Engineers Make Using Apache Airflow(ok this one is not free) 23. https://lnkd.in/gRRnpCjh What Is Docker - Docker Intro And Tutorial On Setting Up Airflow 24. https://lnkd.in/g3fPuz8r Spark vs Polars. Real-life Test Case by Daniel Beach 25. https://lnkd.in/gDEeHn5h Snowflake Vs Databricks - ??♂? A Race To Build THE Cloud Data Platform ??♂? 26. https://lnkd.in/gfy-8gDs What are your favorite free resources to learn data engineering? #dataengineering

I once had an engineer tell me that they essentially didn’t want to consider cost as they were building a solution. I was baffled. Don’t get me wrong, yes, when you’re building, you iterate and aim to improve your solutions cost. But from my perspective, I don’t think completely ignoring costs from day one is a good plan. In fact, many data teams reaching out to our team in the past 6 months have asked about how they can cut their data stack costs. Cost plays a role in all forms of projects, whether you’re building bridges or writing code. How much budget is allocated to build and maintain a solution is important. In the real world, it can change the materials used, the timeline, or the final product’s design. Whereas in the software and data world, it might push other features and decisions that you make. If anything, cost and performance optimization are likely one of the top things I enjoy doing as an engineer. Sure, it’s fun to build new solutions and infrastructure. But it’s often when we are trying to figure out how to run systems more efficiently or cost-effectively that I’ve felt myself solving a real problem. It forces you to consider methods of storing or processing data in more effective ways that are still easy to maintain, which can feel limiting. There are plenty of common issues that drive up data infrastructure costs, but to find them, you likely first need to approach your cost-saving efforts in an organized way. So let’s go over how you can cut your data stack costs in 2024(5)! https://lnkd.in/g9VGcnCq

With HLTH Inc. right around the corner, our team put together a whitepaper that dives into how you can unlock hidden opportunities to improve patient care by harnessing the power of your untapped data. This is a common challenge we’ve seen in many of the healthcare organizations and companies we work with today and we would love to share with you our insights. Explore the whitepaper to learn about: ?? https://lnkd.in/geJk9Ayf ? Barriers to Effective Data Utilization ? Recent AI Technologies Transforming Healthcare ? Opportunities From Untapped Data ? How to Unlock Your AI Advantage #Gradient #GradientAI #Healthcare #UnstructuredData #DataReasoning #DataReasoningPlatform #LLM #AIHealthcare

Going from a data engineer or analyst to leading a data team is hard. You have to pick up a whole new skill set fast, learn a whole new set of terminology and not to mention you likely spend a lot less time coding and writing SQL. Just ask Alex Freberg. Not to mention you're often pressured to deliver results quickly, and have stakeholders who like using the the words "This should be quick" and "just" a lot... This is why I have been talking to dozens of data leaders over the past few months to understand how they made the transition and what skills they felt were crucial to being a successful data leader. But I'd also love to hear from you. What skills, articles or books would you recommend to new heads of data out there? Also, if you'd like to learn more, then you can check out this video. https://lnkd.in/gfbFUewJ

Over the past 5 years I have written well over 200,000 words in articles and newsletters on data engineering, infrastructure and more. Basically, I have written more than a books worth of words. Which is why I have started to put together a book to help share not only my experiences working for and leading data teams but also dozens of other data leaders. It's been a great experience thus far, and I have only written two chapters thus far! Truthfully, it takes a lot more focus, rereading and there isn't any instant gratification like when you write an article or a Linkedin post. And thats partially why I have put out one of the chapters in my newsletter. The two parts of it are listed below! Don’t Lead a Data Team Before Reading This - Aligning The Data Team with Business Objectives https://lnkd.in/gen_uaH5 Building Credibility As A Data Leader https://lnkd.in/e9tMvPwc And if you've got any thoughts you'd like to add, or questions you think need to be answered on the subject, feel free to reach out.

There are a lot of "easy fixes" in data that aren't the right solution. They either add unnecessary tech debt or don't really improve the situation. Here are 3. 1. Let's just fix it in SQL - It can seem really easy to fix business logic in the SQL layer rather than from the source. However, this means, in the long-run, anytime that business logic changes, your team needs to update the SQL as well. It's far more impactful to get the source team to ensure the data is right where they create it rather than in the data warehouse. 2. Let's just add more data quality checks - Data quality checks are necessary. But too many become noisy and coupled with no real change occurring leads to people ignoring them. Less critical data quality checks can be aggregated into a dashboard and reviewed on a normal cadence and connected to larger projects. For example, perhaps your data pipelines keep landing late. Instead of being pinged every morning, this is an opportunity for a larger initiative. 3. Let's just build this real quick - This is great for a POC but POCs and temporary solutions often become permanent. Do you have any other examples? Also, if you'd like to learn more about data engineering and leading data teams, check out my newsletter https://lnkd.in/gNwgxpkk

As a consultant, I have been called in to review and, in many cases, replace dozens of half-finished, abandoned, and sometimes forgotten data infrastructure projects. The data infrastructure in a few cases may just need a little tweaking to operate effectively, but other times the project is either so incomplete or so lacking in a central design that the best thing to do is replace the old system. Trust me, I’d love it if I could come into a project and simply change a few lines of code, and then everything would just work. However, so many projects are filled with unclear design decisions or resume-driven development that were never rooted in good planning. Of course, business stakeholders may have also push to get things done quickly. Forcing data teams to take on tech debt that will never be fixed. Don’t get me wrong, you want to get things done and move projects forward. But taking on technical debt is a decision that needs to be made intentionally. Otherwise, like in resume driven development, your data infrastructure might disappear. This begs the question. How do you ensure the data infrastructure you’re building doesn’t get replaced as soon as you leave in the future? In this article I wanted to dive into the problems I often come into that require me to replace the current data infrastructure and how you can avoid it. So let’s dive in. https://lnkd.in/geP3sQ2W