Requirements of Fast Data Within the Enterprise Data Architecture
Enterprises are discovering that data is becoming increasingly more valuable. Along with this has come the realization that applications need to interact with fast streams of data and analytics to fully take advantage of data as soon as it enters the system in real time. Fast data may be a new frontier, but it is an inevitable step organizations will need to take when they begin to deeply integrate analytics into their data management architectures.
Systems designed to handle fast data need to be responsible for merging the capabilities of operational databases, real time analytics, and stream processing. Committing to a path that does not address all these capabilities means that an organization could potentially be rewriting systems far sooner than expected.
Understanding this shift, application developers and architects are incorporating the necessary interaction of ingesting fast streams of data along with the desire for real time analytics on that data. The next step is clearly knowing how to build an architecture for fast data.
What are the top three requirements for building a fast data architecture? Some requirements of fast data applications are negotiable, but any decision to waive a requirement should be driven by the application’s needs, not by a limitation of the data management technology chosen. The typical requirements for a fast data architecture are:
1. Ingest/interact with the data feed
Typically at the core of any data pipeline being built are data sources ingesting high speed data entering organizations from more sources, and at a greater frequency. However, simply ingesting this data isn’t enough. It is important to remember that there is an application facing the stream of data, and the “thing” at the other end is usually looking for some form of interaction.
2. Make decisions on each event in the feed
The way in which the aforementioned interactions are enhanced is by using other pieces of data - previous events, events from other end-points, static data - to make decisions, as well as incorporating much-needed context. If an event is taken only at face value, the architecture is missing the context in which the event occurred.
Take, for example, online shopping. Typically, businesses will offer recommendations for similar products other shoppers may have also purchased. However, often times these recommendations are timely, but not necessarily relevant or personalized. To improve these recommendations and the consumer’s experience, businesses need to act on each event - each ‘cue’ from the shopper. The ability to interact with the ingest/data feed means businesses can know what the customer wants, at the exact moment of his or her need.
3. Provide visibility into fast-moving data with real-time analytics
One thing that distinguishes fast data applications from old-school online transaction processing (OLTP) is that real-time analytics are used in the decision-making process, thus making operational decisions informed by the analytics themselves. The ability to take more than just a single event into context when making a decision makes that decision much more informed.
Beyond the three general requirements, it is also critical that fast data systems seamlessly integrate into systems designed to store big data, and to have the ability to serve analytic results and knowledge from big data systems quickly to users and applications, closing the data loop. Not every customer is looking to solve all requirements of fast data at once, but it is risky to gloss over these requirements if a data architect believes ingestion is their only current concern.
At VoltDB, fast data is our thing, and we want to share our knowledge on the topic with businesses and systems ready to incorporate fast data into their architecture. If you want to know how to complete this integration, or learn more about other requirements of fast data, download VoltDB’s eBook “Fast Data and The New Enterprise Data Architecture,” written by co-founder Scott Jarr. (https://voltdb.com/fast-data-and-new-enterprise-data-architecture)