Data-related works never ends

Data-related works never ends

During the early stages of my career, as I transitioned from working with ERP systems to delving into the realm of Big Data, I found myself feeling confused and uncertain. A very good friend and mentor of mine who encouraged me with a statement : "Data related cheesien kabhi khatam nahi hota (in hindi)"

"Data-related works never ends"

After a decade in 2023, While offering guidance to someone eager to transition into Data wave from Mainframe tech, it evoked vivid memories from my own past, and took me back to those years.

Back in 2012, the primary focus within the industry revolved around migrating enterprise data warehouses (EDWs) to Big Data platforms such as Cloudera and Howtornwork, Apache Hadoop Open source, Aster Data etc. However, within a span of couple of years, in 2020 the trends shifted towards the migration of Data into cloud platforms— commonly referred as "Cloud Migration". Simultaneously, the volume of data being generated experienced a significant surge. Previously, most organizations used to capture only transactional information. However, with the widespread use of electronic devices and the growth of digital businesses; every event, nowadays, is being captured within stream data analytics platforms.

Now, let's take a moment to reflect on the present state of Data (Big Data). Big Data encompasses three fundamental aspects(3 V's of Big Data): Volume, Velocity, and Variety. By utilizing a distributed processing framework, we aim to achieve faster and more cost-effective data processing.

No alt text provided for this image

It is worth noting that in the past, handling "high-velocity" data and managing variety data was challenging. However, in recenttimes, the landscape has evolved, with the support of tools like Kafka, MQ etc enabling efficient processing of high-velocity data. Additionally, the growth of Cognitive Services, encompassing AI, ML, NLP, and more, has made it easier to process unstructured data(Variety) such as text, images, audio, and video. So Big Data at its peak in 2023.

In some ways, we can perceive these advancements as "old wine in new bottles", as the ultimate objectives remain unchanged:

  1. Collecting data
  2. Storing data
  3. Processing data
  4. Automating the aforementioned tasks
  5. Utilizing data effectively

We are witnessing the emerging concept like Data Lakehouse. Microsoft Fabric Data Architecture Platform built on the Data LakeHouse concept.

Best of (Data Lake + Data Warehouse) = Data LakeHouse        

The saga of data continues, with constant innovations and advancements.

"Data-related works never ends"

"Data related cheese kabhi khatam nahi hota"


#data #microsoftfabric #microsoft #bigdata #fabric #datalake #datalakehouse

要查看或添加评论,请登录

Balaram Panda的更多文章

  • Ensuring Value from Generative AI: Five Crucial Rules

    Ensuring Value from Generative AI: Five Crucial Rules

    Stop Treating Gen AI Like Just another software: Software Leaders need to stop treating Gen AI / AI like just another…

    1 条评论
  • Optimizing Cost in Generative AI Transformation

    Optimizing Cost in Generative AI Transformation

    In 2024, organizations are shifting their focus from Digital Transformation to AI Transformation. While organization…

    4 条评论
  • AI and Data Privacy Paradox

    AI and Data Privacy Paradox

    "Privacy is a fundamental human right that underpins freedom of association, thought and expression, as well as freedom…

  • Big NOse of?AI

    Big NOse of?AI

    As an engineer, It's not fun to be regulated in cutting-edge technology. During one of my past project on an AI-driven…

    1 条评论
  • AI in Marketing

    AI in Marketing

    Successful Marketing happens when a marketer converts a product or service pitch into sell. As a consumer, every day we…

  • Data Science during and after COVID-19

    Data Science during and after COVID-19

    Things are changing and changing rapidly. Customer behavior has been impacted due to COVID-19, so the organization…

  • AI Technical Landscape

    AI Technical Landscape

    According to a Forbes and DigitalOcean report, while only 26% of developers are currently using AI or machine learning…

  • Five senses(????????) of AI

    Five senses(????????) of AI

    Intelligence has three major parts. Data Collection Information Extraction Logic Processing In this article, I have…

    1 条评论

社区洞察

其他会员也浏览了