The Amazing Ways NASA Uses Big Data

“Billions and billions…”

It’s a phrase made famous by physicist Carl Sagan on his popular Cosmos TV show, and he was referring to the number of stars in our universe.

But it could also easily be applied to the bits of data NASA is collecting about that same universe.

It’s a daunting task — and with its dozens of missions and hundreds of scientists taken together, it may constitute the biggest big data project ever undertaken.

NASA’s big data by the numbers

So how big is big?  Just a few facts about the data NASA collects, stores, processes, and analyses today:

• NASA’s deep space spacecraft send data to Earth on the order of MB per second.
• NASA’s near Earth spacecraft send data back on the order of GB per second.
• Data is currently sent by radio frequency, which is relatively slow, but plans are in place to use optical (laser) communication in the future, which will increase the speed and quantity of data 1000-fold.
• NASA tested the laser system in 2013 and set a new data-speed record, sending data from the moon to Earth at 622 megabits per second (Mbps).
• Missions being planned today will generate and send as much as 24TB of data each day — that’s roughly 2.4 Libraries of Congress every single day for a single mission.
• Climate change data alone are projected to grow to nearly 350 Petabytes by 2030.
• The NASA Center for Climate Simulation (NCCS) Discover supercomputing cluster ranks among the top 100 supercomputers in the world and supports research for more than 500 NASA scientists and others around the world.
• Discover uses more than 35,000 processing cores to calculate more than 400 trillion operations per second. “By comparison, it would take every person on Earth adding pairs of seven-digit numbers at the rate of one per second more than 17 hours to do what Discover can do in one second.”
• The Square Kilometre Array telescope, set to open in 2016, will generate 700TB of data per second when it is operational.

So how does it handle all that data? Here are some examples of what NASA is doing today.

Managing and Processing

NASA uses a system called the Mission Data Processing and Control System (MPCS) to manage and process data. It provides custom data visualizations that are used by the flight operations team, and does it all  in real-time — a process that used to take hours, if not days to accomplish.

Storage

NASA has several storage centers for all this data, including the NASA Center for Climate Simulation (NCCS), which focuses on climate and weather data. It currently houses 32 petabytes of data, with a total capacity of 37 petabytes. It also has a one-of-a-kind 17-by-6-foot visualization wall, providing a single high-resolution surface for scientists to display data visualisations.

Archiving and Distribution

One example of how NASA processes and archives all this data is the Planetary Data System (PDS), focused solely on planetary science. It archives and distributes all data from NASA planetary missions, astronomical observations, and laboratory measurements into a single website and offers access to more than 100 TB of space images, telemetry, models, and anything else associated with planetary missions from the past 30 years.

Commercial cloud computing services

For its cloud computing, NASA is using commercial services just like any other company. For the recent Mars Science Laboratory mission, NASA migrated its legacy content management system and websites to Amazon Web Services, which needed to be able to deliver more than 150 Gigabits per second of traffic to a global team of operators, scientists, and the general public. As the data came in, every image from Mars was uploaded, processed, stored, and delivered from the cloud.

And all this is really the tip of the iceberg. Luckily, NASA understands the size and complexity of its probable future data needs and is already planning for a data-heavy future.

Thank you for reading my post. Here at LinkedIn and at Forbes I regularly write about management, technology and Big Data. If you would like to read my future posts then please click 'Follow' and feel free to also connect via Twitter, Facebook, Slideshare, and The Advanced Performance Institute.

You might also be interested in my brand new books:

• Key Business Analytics: The 60+ Business Analysis Tools Every Manager Needs To Know A book outlining the most important analytics tools and approaches and how to apply them across all business functions.
• Big Data in Practice - How 45 Successful Companies Used Big Data Analytics To Deliver Extraordinary Results A book full of real-life case studies outlining how big data and analytics are changing our world - from finance to manufacturing, from retail to science.
• Big Data For Small Business For Dummies - A book that helps smaller businesses and business units get a very practical grip on big data and analytics.

Photo: Shutterstock