Data-related works never ends

During the early stages of my career, as I transitioned from working with ERP systems to delving into the realm of Big Data, I found myself feeling confused and uncertain. A very good friend and mentor of mine who encouraged me with a statement : "Data related cheesien kabhi khatam nahi hota (in hindi)"

"Data-related works never ends"

After a decade in 2023, While offering guidance to someone eager to transition into Data wave from Mainframe tech, it evoked vivid memories from my own past, and took me back to those years.

Back in 2012, the primary focus within the industry revolved around migrating enterprise data warehouses (EDWs) to Big Data platforms such as Cloudera and Howtornwork, Apache Hadoop Open source, Aster Data etc. However, within a span of couple of years, in 2020 the trends shifted towards the migration of Data into cloud platforms— commonly referred as "Cloud Migration". Simultaneously, the volume of data being generated experienced a significant surge. Previously, most organizations used to capture only transactional information. However, with the widespread use of electronic devices and the growth of digital businesses; every event, nowadays, is being captured within stream data analytics platforms.

Now, let's take a moment to reflect on the present state of Data (Big Data). Big Data encompasses three fundamental aspects(3 V's of Big Data): Volume, Velocity, and Variety. By utilizing a distributed processing framework, we aim to achieve faster and more cost-effective data processing.

It is worth noting that in the past, handling "high-velocity" data and managing variety data was challenging. However, in recenttimes, the landscape has evolved, with the support of tools like Kafka, MQ etc enabling efficient processing of high-velocity data. Additionally, the growth of Cognitive Services, encompassing AI, ML, NLP, and more, has made it easier to process unstructured data(Variety) such as text, images, audio, and video. So Big Data at its peak in 2023.

In some ways, we can perceive these advancements as "old wine in new bottles", as the ultimate objectives remain unchanged:

1. Collecting data
2. Storing data
3. Processing data
4. Automating the aforementioned tasks
5. Utilizing data effectively

We are witnessing the emerging concept like Data Lakehouse. Microsoft Fabric Data Architecture Platform built on the Data LakeHouse concept.

Best of (Data Lake + Data Warehouse) = Data LakeHouse        

The saga of data continues, with constant innovations and advancements.

"Data-related works never ends"

"Data related cheese kabhi khatam nahi hota"

#data #microsoftfabric #microsoft #bigdata #fabric #datalake #datalakehouse