The most overlooked factor in Data Analytics – People

I’ve posted a number of articles and opinion pieces, on LinkedIn looking at various aspects of analytics, from Master Data Management, how to run an effective PoC and through to Analytics in the Cloud. All have included an element of ‘how to’ or ‘what to avoid’ and the feedback has been largely positive. But a theme has been nagging at me in the background, People. Who does all these analytics, who’s it aimed at and who is asking for it? I am going to address this omission here.

I work for Teradata, a software company that makes the best analytics database engines available. But when describing ourselves we tend to talk about solving business problems with better data management and analytics before we ever get down to talking queries, I/O, Petabytes or other ‘techy’ terms. And in the market I focus on, pharmaceuticals, my work is about helping companies get new and better drugs to market faster. With these approaches I like to think I’m reasonably well positioned to take a metaphorical step backwards and look at where people fit in to a wide range of analytics projects.

Let’s start by getting more specific on ‘People’ in analytics. I break them down into three main groups; the customer (in retail this would be the shopper and in pharmaceuticals it could be the patient or prescribing physician); the analytics producers (data scientists, business consultants, data engineers, architects etc.); and business managers (brand managers, VP of marketing and anyone else trying to get answers to their business questions).

Of all the groups, customers are probably the best catered for. On-line shopping, digital marketing and an ever-competitive market place means almost every business has to be more customer focussed and more are turning to analytics for answers. We’ve been asked by a national train company to help improve customer perception of their service and analytics led us to look at why trains failed and were cancelled. In turn, this led to improving train reliability through far more accurate predictive component failure and better maintenance programs. Today, they fix the trains before they break down, which means less cancellations and much improved customer perception and loyalty.

Where a lot of these businesses fail is when then simply put all of their customers into one big amorphous blob… ‘our customers’.

A key trend in marketing driven analytics is to be as personal and targeted as possible. This is becoming even more the case as we see the uptake of AI and machine learning where customer interactions and behaviours trigger specific actions on behalf of the company. For example, a bank runs the analytics on people over 25 who are making significant monthly payments to building societies, financial institutions and other banks. These are likely to be mortgage payments and so the bank can offer competitive re-mortgage offers when appropriate.

In this way clever analytics leads to a better understanding of the customer, identifies a particular subset with specific needs and leads to increased revenue opportunities.

The business managers asking the analytics questions pretty well sit in the middle. As such they need to understand what they should be looking for (their customers) and what they can expect from analytics (the producers, data scientists etc.). What business managers must not become is the bottleneck. All too often they are the ones specifying and defining the systems and naturally, they think of one thing cost. Cost per server, per core (unit of processing power), per user license etc. These are the wrong metrics to be concentrating on.

With this mindset what you end up with is a point solution that is good at solving a small number of queries on limited data. Your data will probably still be stuck in silos, so you will not be able to leverage valuable insights from combining the data and you won’t have the capacity or flexibility to scale up your analytics services. Business managers need to take a different focus. What are the business problems you are trying to solve – what do your customers need and how can you get them to interact with you? And what do your data scientists and analysts need to get the most detailed and specific answers you need?

So if you are a business manager with the responsibility of choosing your new analytics platform, think in terms of cost per query, over the long term and across the whole business. Look for solutions that can integrate the widest range of data sets, SQL, SAS, SAP and more, as well as a platform that has the power and flexibility to upscale and downscale as and when your business needs. How Cloud-friendly is it? Finally, make sure it works with the applications and toolsets your producers are using. The chances are most of them will not be too many years out of university and well used to open source tools such as Spark, Anaconda, R, GraphLab and more.

Which brings me neatly to the producers of analytics. In the pharmaceutical world this can be clinical data scientists looking at new drug trial results and looking for the compound that delivers the best outcomes, lowest levels of toxicity (side effects) and even down to specific patient populations. In this way a new drug may have failed in the general population but may very well be effective for womens’ early stage breast cancer in a particular ethnic group.

These producers have specific roles and needs. Data Scientists use analytics algorithms and build predictive models to solve business problems. Sometimes their task is to find patterns in data, and then oftentimes to use those patterns to predict behaviour. E.g. Using patient level data from electronic medical records to predict that a particular patient will have an asthma exacerbation in the next 6 months. They have domain knowledge and expertise, are used to working with data, but want to answer questions not spend time wrangling data or trying to download database drivers so they can access the data in the first place.

If you are managing data scientists, or again looking for a platform that they are going to use, bear in mind that they can spend up to 80% of their time doing non-core tasks – finding and accessing data, cleaning up and data integration, labelling and feature engineering or even just waiting for long running queries to finish. And remember that their favourite tools are R, Python, Deep learning libraries e.g. TensorFlow, Jupiter notebooks and SQL.

Business analysts (consultants) – Use analytics to answer questions but cannot code to the degree that a data scientist can. They have domain expertise and are I.T. literate with a number of power user tools. In my sector they may be tasked with ‘Build a dashboard to show all the patient pathways for newly diagnosed lung cancer patients’, or ‘Show the top 3 treatments and how long patients are on these drugs. To build this dashboard they often rely on BI tools to get access to the data, and ‘point-and-click’ tools to answer the user’s question. They also do not like data wrangling – cleaning or programming – to get what they need. They would rather use self-service tools to get the answers they need without talking to people in I.T. Their favourite tools tend to be Spotfire, Qlikview, Microstrategy, Tableau, PowerBI and Excel.

The people who are tasked with ‘find me the data’, or ‘get me this data in this format’ are Data Engineers. They use programming tools and others to get data into usable state for downstream analysis. They may be working with raw data coming from various different sources into a data lake. Their task is to clean up and validate the data and then create structures which are easy for data analysts and scientists to consume. Often, they write extract, transform and load (ETL) procedures to get data from one source to a more manageable view. Many times they work behind the scenes and their hard work is not often seen by the business managers. Often they are frustrated by changing business requirements, and are trying to always keep up with the data requirements of the business. Their favourite tools are Informatica, Talend, SQL, Scala, Hadoop tools (Hive, Pig etc.), Presto and S3.

Finally, Architects have the huge task of engineering the data and analytics IT ecosystem so that it is easy to track the sources of data and how they are transformed and consumed by various users and applications. They decide on the big picture architecture such as bringing in raw data into a data lake, transforming cleaning and deriving data to be used for decision making, storing that data in a data warehouse, and then creating various consumable data marts to feed end user applications.

Their job is complicated these days by the fact that they often have to manage on premise systems as well as systems in the cloud. This makes managing the entire ecosystem of data, analytics and applications more complex; and sometimes leads to compromises (e.g. if a certain software is only available on premise, or if a certain software is only available in a specific cloud and that is not the cloud the company has chosen for the majority of its applications). The architect often has a grand vision of how things should work and is often frustrated when people ignore the architecture and create ‘shadow IT’ organisations using tools which ‘do not fit in’. Annoy them at your peril! Their favourite tools: Visio, AWS console and DevOps tools (Chef, Puppet etc.).

At Teradata people are front of centre of everything we do. So although we are essentially a software company – brilliantly fast and scalable analytics in the Cloud, also available on premises – what really sets us apart are all the really smart business consultants, architects, engineers and data scientists who pull it all together to deliver the right solution and services set.

As you’ve seen above, if you better understand what these people do, either within your business or as external providers, you will get better insights into the people that are going to impact the bottom line of your business…your customers.

I would argue that when it comes to factoring in ‘people’ in your analytics program or project, do what we at Teradata do – be business led, data driven and make sure your managers, producers and customers are front and centre.