The evolution of analysis: OLTP, OLAP ... RTAP

Big Data and IoT brings new challenges but also new opportunities to get augmented results into our companies.

Over the time there were multiple ways to store, process and analyse data. We've started to build reports on top of OLTP, them we've evaluated to more intelligent analysis using OLAP concepts, but now given the volume the variety and the velocity of the data now a day, we might have new needs on how to analyse data. That's what Real-time Analysis Processing (RTAP) can bring us. New ways to analyse a huge volume and variety of different data as also we need to analyse it faster.

Once up on a time we could build reports on OLTP ...

Once up on a time, companies had the need to summarise data to take decision, but the queries and reports that were built at the time, did take some time to developed and usually were built on demand. It was impossible to determine if you had the right version of a spreadsheet, with the most current data and logic in it. I am not too old (at least I like to think that way), but I can remember companies were building reports on top of a OLTP database, that could be nice because business users would get fresh results, but sometimes causing performance issues on their day to day work.

Reports were not much interactive and most of the times delivered to people in paper (even if I had some fights to stop doing it and bring them online). But OLTP was characterised by a large number of short on-line transactions, maintaining data integrity in multi-access environments and an effectiveness measured by number of transactions per second.

Business users were able to create simple logic on data structured in rows and columns and create their own analyses in a spreadsheet to business problems, but, it was almost impossible to determine if you had the right version of a spreadsheet, with the most current data and logic in it. So that was not really the best solution.

Getting clever by using OLAP for analysis ...

As is referred in [Wikipedia], databases configured for Online Analytical Processing (OLAP) use a multidimensional data model, allowing for complex analytical and ad hoc queries with a rapid execution time, and, able to provide three basic analytical operations: consolidation (roll-up), drill-down, and slicing and dicing. That brought a great advantage when building analysis, where any user is able to build them in a easy way, share them with another users, anyhow there was still the need to look for opportunities, risks, ...

Whereas a relational database and OLTP can be thought of as two-dimensional, a multidimensional database considers each data attributes a separate "dimension" that brings the ability to create multidimensional analysis.

Being smart enough we were able to bring Machine Learning and Data Mining into the picture, doing more clever stuff like predicting what would a user buy, or just improve usability of software, build better products, and even do sentiment analysis, among many others.

This was the era known as the era of Business Intelligence.

But as data is becoming faster to acquire and as companies now understand that data is a big advantage on business, the volume of data started to increase very fast, so the process to analyse it, also are improving.

The new generation of analysis ...

Now we should imagine how would it be if we can bring analysis or do predictions in real-time, easier and faster to build. The variety of data, the big volume of data we are storing and the velocity we are doing it as also the velocity we need to analyse it, brings us new challenges. If Big Data could be too big, with IoT the challenge is becomes much bigger, and also interesting.

That's were Real-time Analysis Processing (RTAP) comes into the picture and can give a great help. It enables us to get smart decisions without the need of the user interaction to get the best solution, decision and preventing mistakes. We are now in a time we have better insights, without the need to request for it and in useful time. We shouldn’t have the need to request for insights, the insights should get to use when there is a need to take decision, even decisions should be delivered to us, we just need to decide if taken or not. We can even get the change to know when the best time to take a decision, but we need to be able to do it before it's too late, so it's something who need to take action now, we need to know it previously.

We are able to use data in motion to predict business opportunities, while at the same time we can improve business operations, identify risks and comply with laws or regulatory requirements, among many others. This is not new, but there are now new ways to bring them to companies.

There are still some customers, that mix the concepts trying to build analysis on top of OLTP, or just trying to apply Real-time on top of OLAP, and many other cases. There are customs that don't even know we have all these capabilities now a day. It's up on us, consultants and expertise on the area, to help them to understand, to get them to know what are the options, what they need to build and use based on their real requirements, and not based what they think they need.

Check the blog post: https://mfgaspar.github.io/2016/OLTP-vs-OLAP-vs-RTAP/