Enabling Real-time Predictive Analytics for Connected Transportation: Critical Lessons from the Telecommunications Industry

Technology advances in data science such as statistical techniques, predictive modeling, and pattern-mining machine-learning algorithms now enable predictive analytics to be applied to large real-time streaming data sets. In the tolling industry, as the IBTTA (International Bridge, Tunnel and Turnpike Association) mentions in their white paper How Toll Agencies can make the Best Use of Big Data, such data sets are now available: "With increased connectivity and communications among vehicles, organizations, systems, and people, unprecedented amounts of data are being generated".

This data has brought forward exciting opportunities to use real-time predictive analytics, as one component of an overall Big Data Analytics strategy for tolling agencies, to dramatically improve transportation systems. Envisioned applications include nowcasting to travelers, predictive road usage and incident management, and traffic flow optimization.

But exploiting this potential has two major problem areas: the first is the design, implementation, and tuning of algorithms appropriate to the problem domain. Most companies are focused on this part of the challenge and will engage trained data scientists for purpose-built solutions.

The second problem, equally critical, is finding a robust solution for real-time data capture and data pre-processing required to create the right inputs into a predictive algorithm. Streaming data from connected cars, sensors, and other sources must be collected and processed in a number of ways in order to be useful. Processing often includes format recognition and normalization, filtering for partial readings or other corrupted data cases, aggregating to the right granularity to reduce huge raw volumes, correlating records from separate streams into the same time window, and finally passing the ‘golden record’ in real-time into predictive analytics algorithms. 

This time, I had the chance to meet with Bert Dempsey - - who explained to me how the technology used in the telecommunication industry can help the transportation industry in the path towards robust, high-volume, real-time data management critical to reaching the potential of predictive analytics solutions.

[MS] Bert, What can the transportation industry learn from telecom when it comes to real-time data collection?

[BD] Much.

The telecom industry builds large graphically-distributed networks, involving many thousands of network equipment machines and millions of end-user devices. In recent times, major carriers have extended support for sensors and devices associated with Internet-of-Things (IoT) solutions. This is the same model the new generation of the transportation industry is moving to.

In order to monitor and control its distributed network equipment, telecom operators were required long ago to develop high-volume real-time data management software. Real-time services like prepaid calls and prepaid users checking their balances must take place within the delay tolerances of human interactions, which is on the order of tens of milliseconds. Network monitoring solutions must continually collect large amounts of status information on all parts of the network fabric, building dashboards for rapid response to problems. These requirements are strict since any service outages or service degradations are damaging to an operator’s brand and business.

To address these hard real-time requirements, the telecom industry developed a software layer called mediation for real-time data collection and data management. The telecom industry uses mediation platforms, such as SAP Mediation, to collect, validate, monitor, correlate, aggregate, and otherwise process raw network events streaming in real-time from network equipment, platforms, and management elements.

Key features from telecom mediation that will be needed for the sensor-heavy future of the transportation industry are flexible collection, complex data format, Monitor-Alert-Validate, Correlation, multiple sources –

Correlation, in a single stream – Telecom mediation also must be capable of building a “session view” of data coming from one network element. For example, a phone call will be recorded at a network switch as a START event, then multiple IN-PROGRESS events, and finally a STOP event. Mediation captures all these events and collapses them into a single session event, the CALL event, for billing purposes.

Similarly, with IoT frameworks in the transportation industry, individual transmissions from devices will require session-control functionality for multiple different use cases. Two examples are (1) filtering and aggregation of data from a sensor within a specified time window and (2) the process of smoothing sensor readings by averaging values over multiple samples and/or taking out extreme values.

 [MS] What SAP Mediation will do differently from other ETLs or Enterprise Buses for this predictive analytics use case in the tolling industry?

[BD] As we have noted, telecom mediation developed out of a need to support high-volume networks (millions of users) with multi-format data (highly complex, including nested binary formats) and multi-transport technologies (standards and vendor APIs). This mediation platform must comply with the highest grade of performance requirements where it will collect many billions of events per day, process millions of transactions per second, and be measured on latency where some use cases wouldn’t allow for more than 15ms of latency.

Traditional enterprise data management tools are not capable of telecom mediation. Traditional extract-transform-load (ETL) tools were born out of the data warehouse needs in the finance industry. The main objective for ETL products is to perform a point-point integration, typically migrating a table from one vendor’s database to a target table in another vendor database or loading a text-based file (or excel) into a database table.

The ETL model can involve high volumes, but, unlike mediation, there is no focus on

·     dealing with diverse data sources,

·     processing data in real-time with delay constraints,

·     extending easily to new data transports, or

·     bi-directional data exchange with a target system for real-time distribution of actions.

Enterprise Service Bus (ESB) solutions are another traditional enterprise integration solution. The ESB design comes from an application-application integration background with an emphasis on translating all application messages into a common format for full interoperability in message passing. Enterprise Bus software does deal with multi-transport technologies and format diversity. Mediation differs from Enterprise Buses is by offering much higher performance in real-time, strong features around configurable and extensible session controls, and extremely agile correlation-aggregation logic.

These differences cited above are significant: no mainstream ETL or ESB products are candidates for the data management capability needed in real-time streaming-data analytics for tolling.

Finally, in IoT solutions using cloud-based data storage, it is important to know that high-volume data capture technologies such as Kafka and AWS Kinesis are widely available. These technologies are complementary, not competitive, with mediation because mediation adds the broad functionality above required for robust real-time predictive analytics.

 [MS] The transportation industry is talking about millions of transactions per day. Is SAP Mediation prepared to support this increasing volume?

[BD] The transportation industry should not be concerned about it. As the leading next-generation mediation platform, SAP Mediation already processes more than 500 billion events or transactions per day in a single customer deployment (Tier-1 wireless operator). SAP Mediation handles all the data in real-time, averaging 4 million transactions per second over the entire 24-hour cycle of the day. In this type of deployments, SAP Mediation maintains these high data volumes driven by millions of network users while responding on average in under 15 ms to signaling from network equipment, such as prepaid balances.

In addition, SAP Mediation and SAP Billing and Revenue Innovation Management have been already successfully implemented for collecting and monetizing IoT data in the automotive and transportation industry.

SAP Mediation is part of the SAP Billing and Revenue Innovation Management product suite.

About Bert Dempsey

Bert Dempsey is a Senior Solutions Consultant for the SAP Mediation software. He has worked 20 years in technical roles within software product companies focused on telecom mediation and usage data management software. To start his career, after obtaining a PhD in Computer Science, he was an Associate Professor at the University of North Carolina, Chapel Hill (1995 – 2003) where he co-authored 35 research papers on topics in real-time networking and distributed systems.