New year's resolution – Real-time insights for all!
When the New Year comes, we often set resolutions for ourselves: How do I continue to evolve and improve this year? While we often set resolutions for our personal lives, we also set them for our work roles and our organizations.?I would offer up that real-time insights could be one of the best resolutions you can make for your business in 2023.
Whether it's real-time e-commerce, streaming, content engagement, API consumption, or financial processing - there are an increasing number of technical and business factors that impact customer experience and revenue in real-time. Enterprises seeking visibility into these experiences have invested heavily in instrumenting the product and collecting all data in a central data platform. While the product teams and other stakeholders have access to much more data in the central data platform, they find a significant gap in their ability to identify and react to these factors in real-time.
The central data platform of modern enterprise
Before we look at the gap, here is an overview of a typical central data platform:
There are two primary sources of data in the modern enterprise:
The first primary source is the Transactional event data from System of Engagement: As users interact with the application, tools like Segment, RudderStack, Snowplow, etc., are used to collect events that capture who did what and when.
The second is the Business metadata from Systems of Record: This is the data about the Customers, Products, Contracts, etc., which is typically in enterprise applications like CRM/ERP or database tables powering the application. Tools like Fivetran, Hevo Data, Debesium, etc., are used to synchronize this metadata with the central data platform.
Such a data stack has multiple benefits:
Real-time insights on the modern enterprise data platform
While there is a lot more data available in the modern enterprise data platform, teams face death by dashboards due to the complexity of the business. Product teams are looking for actionable insights from this vast data source to be delivered to them in real-time.
Some of the key characteristics of such a real-time insights system are:
A Semantic Business Model
The first step to democratizing context-rich insights is to have a rich semantic data model that describes your unique business accurately. The model should natively describe the different Events, Journeys, Business Entities, and Relationships between them specifically for your business. Such a model hides the complexity of the underlying data and allows users to ask questions in a natural business-friendly language.
Data collection in the modern stack is inherently fragmented, whether it is event data (user experience events vs. backend events) or your business entity data (such as customer or product data). Some of the tools allow you to describe the schema/model for different fragments of data. However, a good model abstracts the fragmented nature of the data allowing users to query without having to worry about the source of various fields and what kind of joins or stream merges are required.
Query-time contextualization
Typically, contextualization of event data with business context happens while loading into the data-warehouse (in an ETL pipeline), leading to severe limitations. ETL jobs tend to run a day or more behind. In a modern business where revenue can be impacted by the minute, teams need to act on insights in real-time, not in retrospect. Also, as business changes continuously, data needs to be retransformed through ETL on an ongoing basis to contextualize with the latest business metadata, which can be expensive.
Query-time contextualization, on the other hand, allows users to query the events in real-time without the lag caused by ETL pipelines. Also, since enrichment picks up the latest business context at query time, users do not end up chasing red herrings due to outdated data.
领英推荐
Anomaly/opportunity detection
Visualization tools allow users to create dashboards with various slices of the data available in the central data platform. However, modern complex enterprises have many factors that impact the business; this results in numerous dashboards where it is humanly impossible to track all the dashboards and identify issues and opportunities manually.
Instead, a good anomaly and opportunity detection system learns normal patterns of the data for each factor and highlights the ones that need attention without anyone having to stare at thousands of dashboards. The anomaly detection system needs to be simple, so business users can subscribe to insights of interest without knowing machine learning concepts like “Hyperparameter Optimization.”
Notifications, Reverse ETL, and Automation
Users want the insights delivered to them at their chosen time and on a channel of their choice (Slack, Email, Text, etc.). In a complex business setting, the insights need to be delivered to the right team responsible for that part of the business.
Some of the teams might want these insights to be available in business applications used by the team, for example, CRM or Marketing Automation tools. Therefore, users should be able to Reverse ETL these insights back into the business applications.
For the most common remediations or actions, users should be able to trigger automation via webhook, Kafka message, etc.
Drilldown/Triage platform
Teams want more than just insights about what is happening, but why it is happening so they can take appropriate action and remediation. For example, if there is a surge in orders, is it limited to a specific product or campaign? Or if there is a sudden drop in revenue isolated to a particular mobile platform, and did we release a new version of the application recently?
Drilldown is usually one of the areas in which users are most underserved as it involves downloading data-dump or having someone write custom scripts or queries each time.
Users should be able to seamlessly drill down into "the why" without waiting on others. Sometimes the root-cause of the issue could be an external factor like a software/business change. If there are software and business change events in the data platform, users should be able to view the correlated external change events that could be a factor.
Are you looking to democratize real-time insights on your enterprise data platform?
We built Bicycle from the ground up to help modern product teams identify various factors affecting the business and react to them in real-time. With a backdrop of an uncertain economy, and slowing of global ad spending, there is no better ROI than to focus on customers who have already expressed interest in purchasing your products. Acquiring a new customer can cost five times more than retaining an existing one.
With Bicycle’s real-time analytics, anomaly detection, and alerting, companies can quickly stop revenue leakage, optimize the ecommerce storefront in the moment, identify self-service issues driving unnecessary calls to customer support team, and identify quickly emerging trends to increase revenue and profitability.
Bicycle sits on top of your existing event streaming platform and data-warehouse without requiring new pipelines or infrastructure. Unlike a typical data project, Bicycle can be up and running in a few days instead of months.
If you are considering adding real-time insights into your resolutions this year, contact us at [email protected] to give Bicycle a spin.