Decoding Big Data: From Concept to Revolution

Big data is everywhere. Whether you realize it or not, we all are contributing to it. Here’s how:

At any given instance, there are countless incidents and activities happening in the world. Now imagine all of these events, every breath you take, every gush of the wind and every ripple in the waters are recorded in a giant container. Every single one of these seemingly insignificant events are stored and recorded as they are happening.?

One might wonder how this container is collecting all this data. The answer is not very far from us. Look around, and you’ll likely spot several electronic devices. From the heart rate monitor in your smartwatch to the TikTok that you watched on your phone, the digital chip in your passport, the GPS in your car, all these devices are a part of? continuous data exchange.?

In simple words, big data is enormous volumes of data collected in real time. It can be in any form, from structured data like in a database to unstructured data such as a video or a social media post, or semi-structured like an email. The concept of Big Data has evolved significantly over the decades. This evolution is driven by technological advancements and increasing data volumes. Here’s a detailed overview of this evolution:

Early Beginnings (1970s-1980s):

• In the 1970s, data processing included managing small datasets (in MBs). Data was stored in Relational Database Management Systems (RDBMS). The decade marked the beginning of SQL and ETL.
• The 1980s brought advances in database technologies and initiated data warehousing, providing the foundation of modern data analysis.

1990s: Birth of Big Data

• With the growth of the internet, the 1990s marked the widespread and easy access to data.?
• The launch of Google in 1998, marked the beginning of a new era as the search engine later initiated the development of numerous other technological innovations, including in the areas of machine learning, big data and analytics.?
• NoSQL, an open-source relational database was developed by Carlo Strozzi.?

2000s: Conceptualization and Technological Advancements

• The rise of the internet and digital devices significantly increased data generation. The 3Vs of big data (volume, variety and velocity) was coined encapsulating the true definition of big data.
• ?In 2004,Google introduced MapReduce, a programming model and processing framework for efficiently processing large datasets across distributed systems.
• In 2006, Hadoop, an open-source platform for scalable and distributed data storage and processing was launched. Hadoop uses the MapReduce processing and includes various analysis tools making it a versatile solution for handling big data challenges.?
• This decade also marks the beginning of web-based computing infrastructure services, now known as cloud computing by AWS.

2010s: Evolution and Expansion

• 2010s saw a huge inflow of data leading to development of IBM’s supercomputer Watson.??
• The Web 2.0 era brought semi-structured and unstructured data. Companies such as Yahoo, Amazon and eBay started to analyze customer behavior by analyzing their internet behavior. The arrival and growth of social media data greatly pushed businesses to adapt to new data types.? New technologies, such as networks analysis, web-mining and spatial-temporal analysis were developed.
• ?In the 2010s, the biggest challenges facing big data was the advent of mobile devices and the IoT (Internet of Things). Explosion of data sources from smartphones, IoT, and smart devices, led to diverse data types and volumes.?

Present Day (2020s and Beyond):

• Advancements in AI and machine learning continue to enhance data insight extraction and innovation. Tools like TensorFlow and IBM Watson are making predictive analytics more accessible.?
• There is an increased focus on data governance and security. There's growing emphasis on ethical data use, including privacy, fairness in AI, transparency, and societal impact.?
• Tailored analytics solutions are becoming prevalent, addressing the unique needs of different industries like healthcare, finance, or sustainability. IoT devices are generating vast amounts of data, which is being integrated with big data analytics for improved operational efficiency and insights. As data complexity grows, effective visualization tools are becoming essential for interpreting and communicating insights.

Today, Big Data is reshaping the way we do business. It is not only altering the data dynamics- it is completely revolutionizing it. Businesses are able to gain insights into consumers' needs so precisely, it may sometimes feel like they know what people want before the consumers themselves. Let's take an example of e-commerce. I’m sure I’m not alone in opening the app to buy one item and ending up with ten things I didn’t realize I needed! Have you ever wondered how they know exactly what products you might be interested in? These suggestions are made by analyzing your purchase history and browsing habits. It’s all about making customer interactions more relevant and personalized, which, in turn, fosters greater customer satisfaction and loyalty.?

Big Data unlocks immense potential for innovative research and valuable business insights. It enables organizations to make well-informed decisions. For instance, companies can predict sales volumes based on weather forecast, seasonal trends, festivities and strategically shift their promotions and inventory accordingly. Moreover, big data is a game changer for operational efficiency. It identifies inefficiencies, highlights areas for cost reduction, and supports effective risk management.?

“With great power comes great responsibility.” This saying fits perfectly to this scenario. On one hand, big data analytics offers incredible opportunities for businesses. On the other hand, there are serious concerns about privacy and data protection.? It’s a bit unsettling to think about how much of our personal information is out there and who can access them. While it is no surprise that we’re constantly contributing to a massive pool of data. It makes me wonder, how much of this information is truly necessary?

It is disheartening to think about how easily our personal data can be misused. Take for example, the revelation of Period tracking apps. In May 2022, Periods tracking apps such as Flo came to light when it was found that they have been sharing sensitive and intimate information from their database to third parties without users' explicit knowledge. When data is collected without our consent, it feels like a breach of trust that undermines our fundamental right to privacy. I wonder if convenience is really worth the increased risk of breaches and cyberattacks.?

From humble beginnings to today’s data driven world, data analytics has come a long way.? As we navigate the complexities of big data, it’s essential for us to stay informed and engaged. With the power of big data at our fingertips, it is important to be mindful about the data accesses we provide. I often question myself, are we sacrificing our privacy for the sake of convenience?