Will the last BI vendor please turn out the lights
As someone who believes strongly in the potential for business intelligence (BI) to empower people and transform organizations, I have an important concern to share with every like-minded BI pro: We aren't being disrupted, it's already happened. Despite increasingly powerful dashboards and data exploration tools, traditional approaches to business intelligence are struggling to meet the expectations of the modern data-driven workforce.
Fortunately the core tenets of BI — that combining data makes it more valuable and that people are more powerful with data — are alive and well. What has changed is both the volume and diversity of data, as well as the expectations of end-users who depend on it. The result is that data demands are increasingly being satisfied with tools that simply aren't BI. This should excite us all as it presents a huge opportunity to rethink our data strategies and heighten the impact of our teams. Examining three mega-trends influencing the BI market tells us that the future looks very different and helps point the way forward.
Trend 1. The data driven workforce has arrived.
It should be abundantly clear that everyone is comfortable with data nowadays. It's embedded in our personal lives in all sorts of ways: the reviews when we shop online, fitness tracking when we jog down the street, movie recommendations when we chill on the couch, social apps feeding us the latest personalized news, and so on. This pervasiveness of data translates to the workplace as well because everyone needs data to get their job done. Not just traditional analysts. Not just quantitative marketers. Not just growth hackers. Everyone including factory floor workers, pizza delivery drivers, and even school teachers get more powerful with data.
And, although it pains my heart to say it, these modern data consumers don't expect classic BI reports, or even fancy natural-language enabled dashboards. They expect the data to come to them tailored in an interface designed specifically for the task at hand, ideally integrated into a tool they are already familiar with.
Of course we still need great BI tools for our analysts and data jockeys. But we must also be aware that there are a rapidly increasing number of data-enabled workers who view the idea of using dashboards the same way that you or I might view using a rotary phone. And that’s okay. I'll claim that the future of BI doesn't look much like a BI application at all, and that you don’t need everyone to become familiar with analyst tools to have an insight-driven business. People aren't going to go to BI, BI has to go to the people. This is already happening in a big way.
Trend 2. The proliferation of SaaS applications.
SaaS adoption has absolutely exploded. Think for a moment about the number of SaaS applications you've used today. I'm writing this post at 10am and I've already used Namely, Paylocity, DataDog, JIRA, and about 10 other tools — all of which I love, all of which are essentially polished user interfaces on top of rows of data.
There is an awesome purpose-built SaaS app for most every problem you could encounter at a company. Looker uses about 140 SaaS apps to run our business and we’re not remotely unique. Mary Meeker’s recent Internet Trends report found that the average enterprise is using about 1,000 SaaS applications, and that’s trending upward.
While this is incredibly powerful from the perspective of the average employee, it’s a nightmare challenge for IT departments and data analysts. We BI pros fundamentally believe in the power of connected data. 1,000 SaaS apps means 1,000 little data silos that should be stitched together. But let's be honest, is that really what's happening in our organizations today?
According to a McKinsey Digital study, only 1 percent of all the data created in the past two years has been analyzed. This should come as no surprise. You aren’t alone if you feel like your data team spends 90% of their time trying to wrangle data and keep the chaos at bay. But no amount of workbook automation, no amount of SQL enablement training, no amount of ETL code is capable of wrangling the explosion of data volume and complexity. And there is no going back, this is the world we live in (someone in your company has probably signed up for a new SaaS app while you’ve been reading this blog post). Fortunately, there is hope for data teams in the form of more powerful infrastructure.
Trend 3. Modern data infrastructure.
Modern massively parallel processing (MPP) data warehouses (e.g. Google Cloud BigQuery, Snowflake, Amazon Redshift Spectrum) have made step-function improvements in just the past few years. They can hold immense amounts of data, query it all in seconds, and even do advanced analytics directly in the database — all this at a cost which is bafflingly cheap compared to last generation technologies.
Looker is not traditional BI and we took a huge bet on MPP databases early on. We predicted that they would lead to fundamental changes in data infrastructure. In a world where we can dump as much data as we want into one place, query it fast, and pay pennies for the privilege, whole steps in the traditional data engineering workflow can be simplified.
Rather than doing the heavy lifting of creating aggregate tables, massaging data, and data prep outside of the database, we do much of it in-database by transforming data when it's queried. And since queries can be directly executed against the data warehouse, that data is more fresh, more detailed, and more trustworthy. It’s better data for less effort. This significantly improves the lives of data engineers because they can spend less of their time building and maintaining data pipelines and more time doing what they really want: empowering end-users with data-driven experiences.
So what does this all mean for the future of BI?
One thing remains constant: the analyst is still the hero. It is their knowledge and passion for data that will provide the deepest insights to businesses.
But in a world where everyone — not just analysts — depends on tailored data apps to get their work done, where data complexity and volume is increasing at an incredible rate, and where data infrastructure is exponentially more powerful, I believe the analyst has a new set of responsibilities. In addition to being the hero who wrangles and interprets data with their own powerful set of tools, they must also learn to empower others with data in new and unique ways. This means taking a fundamentally different approach to BI:
- New infrastructure that connects SaaS apps and departmental data with a BI fabric that flexes and scales with the immense demands of the modern data ecosystem.
- New governance that provides a single source of truth for business data. One which is unified, trusted, fresh, and comprehensive.
- New purpose-built data experiences that go beyond dashboards, reports, and embedded visualizations to provide actionable insights delivered when and where they’re needed.
In doing so, analysts can unlock a new frontier in which people from all backgrounds, in any type of company or department, including those who traditionally have not worked in data, will be empowered to work smarter. In this world everyone, each in their own way, becomes a data hero.
rock on
Curious how these three BI trends are impacting actual companies? Check back here in a few weeks as Nick will detail some real-world scenarios.
Follow me if your users don't use your dashboards ☆ Down to earth data visualization & storytelling ☆ Make your analytics count ☆ Chart Doktor Data Product School
5 年Hi Nick, this article is really inspiring. I would love to read more about custom-built data products use cases! I my experience as a data visualization designer specialized on data products, the power of purpose-built data UIs is their ability to show not only data points (that are measured at different places in a complex system), but also the underlying structure, process or system. This context gives the user the chance to form a mental map. Without a mental map of the system, it is not possible to make sense of the data. For example, how could you interpret the relevance of a user review on Amazon without having a mental map of the timelines of product lines with their quality issues, usage duration of the users, identical product offers that are not grouped here, different products (like audiobook version) that is grouped here but not relevant, etc.? We have to make sure that users of our product have all the information they need to form their correct mental model. But this is not going to happen if we have available only the very limited toolbox of a BI tool like bar charts and pie charts. This becomes especially true if we add systems using AI and data science to the equation.
Healthcare - Website -Connections 1976 Employment 2012- Healthcare 2020 Ongoing at Healthcare News
5 年Thank you I believe that you are in the right BI update. One does read review plan revise extends even tried shorten important connected data. On change, business calls, duties change but for the better.
Ed Tech Strategy
5 年This is a prescient statement: “...modern data consumers don't expect classic BI reports, or even fancy natural-language enabled dashboards. They expect the data to come to them *tailored in an interface designed specifically for the task at hand*, ideally integrated into *a tool they are already familiar with*.” (emphasis added). If we want to extend the reach and impact of data across our workforce, we can’t just rely on increasing our people’s “data literacy”, we must conform the results of the data to their existing culture of work, simplifying it’s output and embedding it in our workflow.
Global Data & Analytics Director - Retired (Open to Short-term Consulting)
5 年Bill:? In my view, you are?spot on correct.? Few companies want?to invest the time and effort to find the causal measures/metrics or the underlying evidence drivers that connect people's action to the outcomes that matter most.??I call this?research and collectively we have invested too little time in outcome based research that would have engaged senior leadership....and?without engaged senior leadership any?effort it is likely to struggle. ???
Senior Economist and Methodologist. Statistics, Applied Econometrics, General Analytics, and the Data Sciences. Incisive Thinker, Writer, Researcher, Teacher. Entrepreneur. Author, Writer, Editor, Blogger, Poet.
5 年With all due respect, these are not the reasons that BI is struggling. Paul Weiskopf, in his comment, below, is much more to the point. I would add that what should have been the main focus was overlooked a long time ago, and that was to test, identify, and present metrics that are statistically validated indicators of e.g., inflection points in trend data, or are statistically significant correlates with other internal or external changes that an organization believes are crucial to its continued success. This would have been a BI dashboard that managers would have embraced. But the difficult front end work of identifying those kinds of metrics has rarely if ever been done. And BI practitioners, enamored as they have been, for so long, with the Big Data craze, often denigrated statistical work. In fact, statistical data science, as I call it in my upcoming book, is the indispensable precursor to getting real business intelligence into the hands of those who need it most. It takes raw data--which cannot speak for itself, a myth that the Big Data naifs apparently refuse to abandon--separates the signal from the noise, and allows analysts to process that signal into useful information. There's your BI that never was, but still could be.? ?