#1 - Still loading data into memory

Recently I posted a number of signs that your BI & analytics strategy is falling behind. Several people asked to elaborate a bit more on the various topics so in this blog series I'll tackle each sign one by one. Starting with number one:

Your end users still cannot perform deep and wide analytics on large data sets as they're limited by the amount of data they can load into memory

What is causing this?

Most legacy BI tools originating from the previous century and the somewhat more recent self-service (prettier) analytics tools weren't developed with the latest database and web development technologies in mind. For them to be able to deliver a rich, user-friendly & fast performing analytics experience they designed solutions that have to load a portion of the data into memory of either the server or the desktop client application. And obviously that amount of memory is limited, and in the era of Big Data far not enough.

How are users and your business impacted?

As a result, users will be limited by the kind of analysis they can do on their data by themselves. They need to make a choice: lots of data on 1 topic (deep & narrow), or a little bit of data on multiple topics (shallow & wide). Larger wide & deep analysis will then be performed (usually in batch) by some data scientists using other tools which massively delays the process and increases the cost. It also may cause your users having to spend additional time (re)modelling their data as they pull individual data sets into memory instead of using the models defined in the database.

As many organizations are moving to solutions like Snowflake, or already have high performing analytical databases like SAP Hana, Exasol or Redshift this means the processing power of these is not really taken advantage of. They become dumb query engines to offload data from.

The inability to perform deep and wide analytics by end users means as a business you could miss critical business opportunities before your competitors do. 

What do you need?

You need to look for an analytics platform that can leverage the analytical horsepower of the underlying database infrastructure. It's architected to get the database to perform the queries instead of pulling data out into memory and then doing calculations. But be careful, some solutions will claim they do this, but only for a small set of simple queries, for advanced queries they default again to pulling the data into memory. This is because it's not always easy to translate a user query into a query for the underlying database, especially one that will perform. So make sure your BI & Analytics platform was architected with pushing queries to the database as a key strategy and not as some afterthought so it could tick a feature box.

With the right solution architecture, your business users will be able to perform deep and wide analytics on large data sets at the speed of thought, enabling them to spot business opportunities and issues instantly. Empowering your end users and reducing the need for additional tools and processes.

Please share what other symptoms and challenges you see in the comments!