Big Data Analytics: Evolution Captured

Analytics is not a new discovery although the term has got a huge emphasis in recent time. It got enriched over a period of time due to various disruptions. Analytics is being practiced ever since a brain is applied to take a decision. The brain by itself does thousands of calculation on the data stored in memory from the past experiences. If we go with this than an experienced one would take a better decision and this actually is true. In internet world there has been several disruptions from information technologies where the dependency on the memory of an experienced person is passed on the expert systems. And we never realized as we were becoming more and more dependent on technologies for taking business decisions.

Two parallel streams of IT disruptions. This evolution to its current form has happened in two parallel streams of technology disruptions. One stream is of enterprise business applications and other stream of database management systems (RDBMS) for storing and retrieving the data. The two streams kept improving as the processing speed and data retrieving speed kept increasing while cost kept decreasing.

Both these disruption streams kept supporting each other. In the first stream of disruptions the enterprise resources planning (ERP) systems were developed which were more for managing the functional areas of businesses. And the enriched database management systems (RDBMS) from the second stream has been storing the data known as operational data store (ODS). These systems are to keep managers updated on their key performance indicators (KPI) reflected on their dashboards to take decisions.  

More developments in both the streams. Over a period of time the amount of data created kept increasing in the ERP database (ODS). However, maintaining a huge ODS beyond the certain period would not add much value for ERP systems. On the contrary it was counterproductive by making the systems slow. This challenge was solved up to large extent by new IT developments such as faster extraction, transformation and loading processes (ETL) and enterprise data warehouse (EDW). ETL applications help to extract the data from one source, transform it as per the destination database and upload there. Whereas EDW is historical database layer. Thereby the practice is keeping the snapshot of transactions of recent period in ODS and rest is exported to the EDW.

Business Intelligence tools were developed for digging information from EDW. In the period of 2006 – 2013, the popular buzzword had been Business Intelligence (BI). These BI tools were playing a major role while taking strategic decisions like acquisitions and expansion by enterprises. BI tools helped in data mining mainly from the historical data warehouses, and representing the information in various form of data visualizations and reports. Statistics has been the core of BI. Also information technology now equipped the data scientists to create not only the two dimensional reports but multi-dimensional cube reports (OLAP). Till the data mining was around RDBMS the enterprises were immensely benefitted by BI.

However the biggest disruption thus far came in the form of Internet of Things (IoT) and cloud computing. These two technologies IoT and Cloud became inseparable which makes smart devices communicate to each other. Which also make possible the various functional areas like marketing, finance, HR and operations be realized in e-Business. Search Engines and Social Networks also are the components of IoT.

Data from IoT is mostly unstructured/ semi-structured and massive in size, well beyond the scope of RDBMS which was mainly structured data. Now most business transactions and critical data has been through IoT as well. Actually the bulk of data created by business enterprises is more in the form of unstructured and the structured data is now a days a very small percentage of entire enterprise data.

Unstructured and massive amount of data now require new technologies leading to Big Data Analytics. GFS-Google File System used in Google search engine also is a big disruption in itself and is the basis of the buzzword Analytics in contemporary business world. Business Analytics is all about equipping the decision makers with certain architecture models and associated tools which are capable of data mining from not only structured data but also the unstructured and semi-structured data. The models that are scalable theoretically to dig any data size like Zettabytes and above and are based on distributed storage and distributed computing. The Big Data Analytics is being given a huge emphasis as this is capable of giving very accurate data analytics as to give accurate trend patterns and forecasts. Huge knowledge regarding the customers, market, products and service quality and various measures from analytics for e-Business are now possible through Big Data Analytics tools and that’s why this is a new buzzword in contemporary world.