3 Steps for Developing a Successful Analytics Architecture

Many businesses have undertaken Big Data and/or Analytics projects, but for every successful project, there are dozens that have failed or have stagnated. Customers are realising they need to understand both the business outcome and technology in order to target their goals more accurately.

If you realise it or not all organisations are on a journey that results in the need to process thousands of times more data than they have done previously, process that data and operate in real-time or near real-time and then use the insights from that data to deliver personalised and the enhanced services.

To do that successfully requires you to think about the Journey from two points of view. One focused on the business aspect and one focused on the technical aspect. What is the business problem we are trying to solve, what is the data you have available to you and how can you simplify the technology to accelerate those business outcomes. 

The reality of course is that most IT infrastructures have not been designed with Big Data and Analytics in mind. The journey to digital breaks traditional IT infrastructure and there is need for a new architecture and new infrastructure to support data driven business.

1. You need to be able to ingest from a growing number of internal and external data sources 
2. You need to store more data, for longer, cost effectively. The more data you can collect, the better analysis and insights you can gain
3. You need to be able to surface that data quickly and easily across the business, or to consumers through applications

Customers with traditional OLTP databases are struggling to leverage these technologies for analytics and for good reason. In the same way that I would not recommend an OLTP database as the engine for advanced analytics I would not recommend an OLAP database for transaction processing. Yet many organisations continue to try and support/leverage their existing investments by buying additional bolt-ons from their incumbent supplier to try and resolve their challenges. The net result is significantly more cost, vendor lock-in and increasing complexity.

Some organisations chose to invest in Hadoop and the associated ecosystem of open-source solutions. Fast forward 3-5 years and many have reached the point where traditional Hadoop Architectures are causing their own challenges. They have a cost effective (maybe) platform for storing large amounts of data, but they are quickly realising that getting insight from that data can be extremely expensive in terms of hardware and again complexity.

Some organisations are just getting started, learning to leverage visual analytics to simply replace their historical Excel reporting. Even here many of the companies I talk to are experiencing challenges. Like mobile applications, visual analytics solutions need to be engaging and they need to be fast. In todays world, if you have to wait more than a few seconds for the answer the solution you have deployed will probably be considered a failure by its users.

So how can you resolve these challenges whilst leveraging the investments you have already made? Firstly despite what you might have been told, there is no single solution. Big Data and/or Analytics is an eco-system play. Certain products are better suited to certain use cases and in many cases certain technologies are complimentary. The overarching question is how can you leverage multiple back-end data stores with multiple front-end tools whilst reducing complexity and increasing performance?

Step 1 - Create a high speed, scalable, cost effective Enterprise Analytics Hub. 

In todays world, speed is everything. Whether it be surfacing the latest information to the business in a near-realtime dashboard or running the latest risk prediction algorithm, performance is key. Even if your current performance is acceptable I have not yet met a customer who would not like to do more or add more data points to their calculation to get even finer insights. The solution is an analytic database designed from scratch for speed, that aggregates your various data sources and front end tools whilst accelerating your analytic workloads.

Step 2 - Simplify your environment

Inevitably most organisations have multiple tools to ingest, store, wrangle, surface and act upon data. Multiple data sources means creating a mesh of interconnectivity between tools and data sets which adds complexity and can impact performance. A better idea is to connect them all through the concept of an intelligent, centralised hub. This enterprise analytics hub should be able to accommodate the needs of all the data consumers and their tools. For the BI analyst it should support SQL. For the line of business it should integrate with visual analytic and dashboard tools. For the Data Scientist is should support advanced analytic functionality and integrate languages such as R, Python and Lua so that code can be executed natively within the database.

Step 3 - Connect your data silos and use them for what they were designed for

Technically, consolidation of data silos, is pretty simple to do, politically it can be a nightmare. Data, like knowledge, is power and consolidating data across different business units can be incredibly difficult to do. Couple this with the issue that many organisations have data in the cloud as well as on-premise and the challenges become exasperated. The solution is to connect, rather than consolidate, using a data virtualisation framework: The ability to aggregate sources of data whether they be on-premise or off-premise, structured or un-structured. Whilst at the same time being able to push relevant analytic queries to those underlying technologies that are best suited to store and process the data that they have, e.g. Hadoop for unstructured data and Oracle for structured data OLTP data.

The result of these steps is an architecture that empowers analytics within any size business, whilst leveraging previously purchased technology to maximise your return on investment. 

What are your thoughts?