REAL-TIME ANALYTICS AND DATA PROCESSING FOR SCIENCE AND RESEARCH

We know how important the storage, processing, analysing and visualising of large data sets can be, especially in the fast-paced world of Science and Research where speed, accuracy and flexibility are all significant factors.

At Spicule we have the knowledge and experience to design, implement and support robust data processing platforms tailored to the specific needs of our clients across a wide range of Science and Research sectors. Here’s how we used Apache Druid to solve one particularly unique challenge.

THE CHALLENGE

When a satellite is launched into space, getting it into orbit is only the beginning of the journey. Once it’s up there the satellite will receive, generate and transmit a monumental amount of data, all of which needs to be securely downloaded and stored so that the scientists back on Earth can analyse what it’s telling them in real time, instantly compare it to historical trends, and respond to any changes or anomalies as they happen. The data can take many forms depending upon the satellite’s purpose, but it will also include crucial details like spacecraft positioning and orientation and the functional status of onboard hardware and software. After all, if something’s close to failure or your satellite’s orbit is decaying, you’ll want to put another satellite up there as quickly as possible to fill the gap.

When you consider the petabytes of data a single satellite could produce over the course of its lifetime, think how much more complicated and data-heavy the problem becomes when a constellation of satellites is launched into space and each of those satellites has to be tracked and analysed individually, historically, and in relation to all the other satellites within that constellation.

Our challenge was to design and build a platform that could a) store the massive quantities of data each satellite was producing on a second-by-second basis, b) analyse the data in real-time while simultaneously comparing it to historical trends, and c) generate the results in a format that was quickly and clearly understandable via an intuitive custom-user interface.

OUR SOLUTION

Find out how we solved this challenge here and also the benefits of Apache Druid for Science and Research