Intricity的动态

1,978 位关注者

8 个月

Marketecture vs. Innovation. If we're making big investments we need to be careful that we're not doing so because of marketecture. Sometimes what is "old" is actually just best practice... #medallion #dataarchitecture #data #datacloud

Transcript

Hi Jared Hillen, If you followed my videos over the last 13 years, you'll see a consistent trend of showing a graduated data curation architecture. Data gets loaded from source systems, it lands in some kind of raw landing zone, then it gets somewhat cleaned up so that it has a source pristine representation in a single place for querying. Then it graduates to aggregated analytics. That architecture has been around since the late. 90s yeah. The only thing that has changed is that non relational data has become a greater part of our world. But this is where we get into a kind of annoying thing about the data world. Every three to five years, somebody has to put a marketing spin on things. The staging area or operational data store became the data lake, and it was going to change the world. And after a few years, nothing really changed. Some would point to the lake house as groundbreaking because you can get real time results, but honestly we've seen that with old data federation and operational data stores. And this brings us to the medallion architecture. I actually like the way of thinking that a medallion architecture brings. It's sort of simplifies to laypeople where they may want to get their data. But if you think the medallion architecture is some kind of special new data feature that you didn't have in your last company. You're likely in for a bit of a disappointment. This is what I call marchitecture. It's architecture that has been spun by marketing departments as something new. There's nothing nefarious going on in most of these cases, it's just people learning about something for the first time and then thinking it must be new and then spending millions of dollars promoting it. And I think with the Medallion architecture, it's a net positive. Where I think things get a little sideways is when business decisions are made solely on the hype of marchitecture. So I want to call out the why of each layer so in five years from now when the next type occurs, you'll be able to spot it. First you have application data. Data and application databases isn't physically set up for rapid and massive read events, so it has to be moved into something that is. This is what the medallion architecture calls the bronze layer. Now, this layer is very raw. You probably shouldn't let human hands access this layer, because the data itself isn't even reliable yet. You can have all sorts of issues that need to be cleaned U as part of data onboarding. The next layer U is where some of the messiness in the data has been resolved. This produces application pristine representations of source systems that produce that data, along with some other structures that make that data easier to query. And the medallion parlance, this would be called the silver layer. Consumers here are operationally oriented reports as well as data scientists trying to run experiments against the data. And this is also where your data sources come from. Machine learning and AI events, So much of what the business runs on is fed through this layer, including the next layer. So to make truly data strategic and consumable for large audiences, you start aggregating the data into structures that make data manipulable. Like conformed dimensions and measures that can be assembled into insights and Dr. action, Data scientists and AI can live here as well. If you need aggregated results, there's no sense in rebuilding something that already exists. The medallion architecture would call this the gold layer. Now, within each of these transitions, there is logic to generate the next medallion layer. That logic has to live somewhere. And the interim data that progresses that logic has to live somewhere as well, whether it's in memory or physical storage. Now, whatever you want to call what you're looking at really doesn't matter. This is essentially the same architecture that has existed for nearly 30 years, which is kind of crazy, but it's true. That doesn't mean that the demands on that architecture haven't changed. The volumes of data and the types of data have definitely changed. But the stickiness of this architecture has me questioning the properties of data more than anything else. There's something about this need for graduated curation that is built into data itself. Stay tuned for my next video. I'm going to try and pry open the mystery on what forces have influenced this kind of architecture and why we keep landing on it. So I've written a white paper on the medallion architecture, which you'll find a link for in the video description. Also, if you're planning that architecture out now, this is one of intricity specialties, so I recommend you reach out to talk with one of our specialist.

要查看或添加评论，请登录

最相关的动态

Jared Hillam

GTM VP
8 个月
举报此动态
Marketecture vs. Innovation. If we're making big investments we need to be careful that we're not doing so because of marketecture. Sometimes what is "old" is actually just best practice... #medallion #dataarchitecture #data #datacloud

2 条评论
赞评论
要查看或添加评论，请登录
GetOnData Solutions

2,924 位关注者
9 个月
举报此动态
In the fast-paced world of business, staying ahead means understanding the power of your data. Data virtualization brings your data together, breaking down barriers and unlocking endless possibilities. It’s time to embrace innovation, streamline your operations, and fuel your growth. Dare to transform with data virtualization! ???? https://lnkd.in/dPagCnJK #DataVirtualization #datamanagement #businessintelligence #dataanalytics #DataStrategy #EnterpriseData #CloudData #digitaltransformation #MasterDataManagement #datagovernance #realtimedata #Datalake #getondata
赞评论
要查看或添加评论，请登录
iTech Governance Consulting

143 位关注者
2 个月
举报此动态
AI is changing the game—and it’s putting new pressures on data centers! Our latest article dives into the challenges legacy data centers face as AI adoption accelerates. From power demands to cooling inefficiencies, the landscape is evolving rapidly. How will this impact businesses relying on in-house or colocation facilities? Read our full article to explore the implications and strategies. Have questions about how these trends might affect your business? Reach out to us—we’re here to help. #AI #DataCenters #Innovation #ITGovernance #Infrastructure https://lnkd.in/gh6Q3gYx

AI’s Impact on Data Center Power & Backup Power Needs

itechgovernance.com
赞评论
要查看或添加评论，请登录
Telmai

3,073 位关注者
5 个月
举报此动态
Modern data platforms are evolving, but the shift from traditional data warehouses to decentralized data lakes creates opportunities and challenges for organizations. With data storage becoming more flexible, organizations face a common issue: How do you balance scalability and flexibility while keeping a check on performance and cost??? Vikram Gundeti, CTO of Foursquare, dives into the core problem—explaining why the unbundling of traditional data warehouses has caused many organizations to rethink their data strategy. He highlights the rise of modern data solutions like data lakes, distributed storage, and modern compute engines and how the right tools can help solve performance and scalability issues while staying cost-efficient. Click the link in the comments to read Vikram’s full breakdown and explore how to optimize your data platform for the future! #datamanagement ?#DataPlatforms #DataLakes #DataWarehousing?
1 条评论
赞评论
要查看或添加评论，请登录
Rui Neves

Senior Partner at McKinsey&Company
5 个月
举报此动态
Is data center real estate overhyped? Digital and cloud revolutions—and the scale-up of AI, a $4 trillion global opportunity—are fueling enormous demand for data centers. In turn, there are a variety of ways that real estate companies can participate in the growth of the industry. My colleagues Katy McLaughlin and Pankaj Sachdeva discuss the sector, its real estate growth opportunities, and the long-term opportunity for investors in the new interview below. #AI #genAI #datacenters #realestate

What the real estate industry needs to know about data centers

mckinsey.dsmn8.com
赞评论
要查看或添加评论，请登录
Satish Hegde

Sr.Director | AI & Analytics : AI Engineering : Enterprise AI : Agentic AI : Generative AI | Data : Cloud | Thought leader : Advisory & Consulting |
7 个月已编辑
举报此动态
As data professionals, we've witnessed numerous architecture patterns, tools, and technologies emerge. However, it's crucial to acknowledge that the most vital feature for a data platform is "end-to-end lineage". This means having the capability to trace changes from consumption to source proactively or on-demand through a single control plane, even in a ""multi-technology"" stack environment. But how many organizations have achieved this essential capability? Does your enterprise have it? #Data #AI #Lineage #ModernDataPlatform
赞评论
要查看或添加评论，请登录
Britta Lietke

Senior Knowledge Expert at McKinsey & Company
4 个月
举报此动态
Is data center real estate overhyped? Digital and cloud revolutions—and the scale-up of AI, a $4 trillion global opportunity—are fueling enormous demand for data centers. In turn, there are a variety of ways that real estate companies can participate in the growth of the industry. My colleagues Katy McLaughlin and Pankaj Sachdeva discuss the sector, its real estate growth opportunities, and the long-term opportunity for investors in the new interview below. #AI #genAI #datacenters #realestate

What the real estate industry needs to know about data centers

mckinsey.dsmn8.com

1 条评论
赞评论
要查看或添加评论，请登录
Nimblechapps Pvt. Ltd.

11,138 位关注者
3 个月
举报此动态
Exploring the Next Frontier in Data Management! ?? Explore here: https://buff.ly/41y74tY The evolution of data lakes is shifting from centralized models to federated approaches, promising enhanced flexibility, scalability, and collaboration. As businesses embrace this transformation, the future of data storage and analytics is becoming more decentralized and dynamic than ever. Dive deeper into the journey towards federated data models and their impact on the digital ecosystem. #DataLakes #FederatedModels #DataManagement #Nimblechapps #MobileAppDevelopment #WebsiteDevelopment
赞评论
要查看或添加评论，请登录
Joseph Carlton - Director of Data Analytics, Speaker & Data Evangelist

6 位关注者
8 个月
举报此动态
The Importance of the Modern Data Stack In today’s fast-paced digital world, businesses need to leverage data like never before. The modern data stack, which includes tools for data ingestion, storage, transformation, and analysis, is revolutionizing how we handle big data. With cloud-native platforms, real-time analytics, and scalable infrastructure, companies can make data-driven decisions faster and more efficiently. Are you leveraging the power of the modern data stack to drive your business forward? #DataScience #BigData #Analytics #ModernDataStack #DataDriven
赞评论
要查看或添加评论，请登录
Steffen Fuchs

McKinsey Senior Partner and global co-leader of Industrials & Electronics client service
4 个月
举报此动态
Is data center real estate overhyped? Digital and cloud revolutions—and the scale-up of AI, a $4 trillion global opportunity—are fueling enormous demand for data centers. In turn, there are a variety of ways that real estate companies can participate in the growth of the industry. My colleagues Katy McLaughlin and Pankaj Sachdeva discuss the sector, its real estate growth opportunities, and the long-term opportunity for investors in the new interview below. #AI #genAI #datacenters #realestate

What the real estate industry needs to know about data centers

mckinsey.dsmn8.com
赞评论
要查看或添加评论，请登录

1,978 位关注者

查看档案关注

Intricity的动态

更多文章

BIG Changes to Intricity are Coming Next Month!

Hybrid Tables, Partitioning, Medallion Architecture, & more!