Can There Be Too Much Big Data?

Big data hype is only matched by its actual growth. Experts are predicting a 4300% increase in annual data production by 2020. In 2013, there were 3.9 billion email accounts generating oodles of data which will grow to 4.9 billion by the end of 2017. Big data is here to stay and companies that can analyze large pools of data are in an advantageous position to make better business decisions by discerning patterns. This tremendously competitive advantage allows companies to stay focused, enhance productivity, and create even more value. But is there such a thing as too much big data?

Data Overload

We have all heard of information overload, well the same can be said of too much data. Data has a certain life span of usefulness as over time it becomes less valuable, outdated or even irrelevant as needs and computer networks change. Most companies can only analyze a limited amount of data they currently collect, so as big data gets bigger the task of collecting and analyzing will bury many companies. Understanding which data sets are important is key or companies may end up over their head with too much data. Just look at law enforcement for one example of this.

After months of delays, the NYPD has finally released official proposals to outfit their entire police force with body cameras. That’s 23,000 officers by the end of 2019 and a lot of data! This is just one example of potential data overload coupled with all the social, political and legal ramifications that accompany it. And we didn’t even get to the issue of security yet.

How Secure is Big Data?

If data networks are pumping out so much data that they can hardly keep up with the collection and filtering of it, how can they even begin to address the security of all this data? Corporations need to not only secure their own organization’s data but also all customer data. As most technologies blossom, security seems to be an afterthought and with big data, this same corollary holds true. Security needs to be carefully considered at the onset and not after a compromise.

Most organizations are not likely to build their own machine learning environments in-house so cloud and big data have become inextricably linked together.Naturally, cloud security is a great concern as more than ever, workloads are moving to these cloud-based platforms. And the fact that a company’s data is stored in the cloud and not on the premises does not remove the company’s responsibility for protecting that data.

Big data analysis is not only effective in finding patterns for client needs but also in detecting and preventing advanced persistent security threats to those very clients and their data. Detecting threats at an early stage by using advanced pattern analysis and taking a close look at multiple data sources can be instrumental. When breaches occur, logs are sometimes ignored unless a specific incident occurs. With big data, there is an opportunity to analyze many logs automatically from several sources instead of in isolation. Patterns with the best chances of developing into useful data points are discovered before and during cyberattacks. This leaves more time and resources available for the more intuitive detective work by cybersecurity experts. This AI/human tag team effort enhances (IDS) Intrusion Detection Systems and (IPS) Intrusion Prevention Systems by continually adjusting and learning good behavior vs. bad behavior along the way.

Where Innovation Meets Data

Big data and machine learning technologies will aid the next generation of cybersecurity solutions since they can immediately adapt to the rapidly changing threat landscape. When machine learning is dynamic and intelligent, it can analyze large quantities of data and spot unusual activity by correlating it with other suspect events. Security personnel can then be notified to provide a solution that can be rapidly applied.

Due to its defensive nature, cybersecurity innovation is always the result of disruptive technology. But this innovation can be hampered by government agencies required to make political decisions over financial or technology related ones. Hackers do not suffer from such road blocks and often rely on the latest technology to accomplish their crime. They also have the ability to react on the fly. In order to keep up, cyber defense technologies must be proactive rather than reactive by using big data to feed models to anticipate threats and thwart attacks before it is too late. This near instant response time reminds me of the state of connected device security in the (IoT) Internet of Things world of devices. AT&T tracked a 400% increase in scans of IoT ports and protocols across their networks in 2016. This clearly shows that device recruitment is on the rise. Standards lacking any security foresight run rampant and user features and defaults are even worse. These security gaps coupled with the speed of a typical botnet attack poses a true network security challenge that demands a rapid response from innovative cyber defenses.

Big data can only serve cybersecurity’s best interests by simultaneously alerting experts to emerging threat patterns but also staying out of its own way. By combining big data with cyber innovation in proper proportions, companies can stay one step ahead of the cyberthreat landscape.

This post is brought to you by AT&T. The views and opinions expressed herein are those of the author(s) and do not necessarily represent the views and opinions of AT&T.

Scott Schober

CEO | Author | Speaker | Cyber Security & Wireless Expert at Scott Schober LLC

Scott has presented extensively on cybersecurity and corporate espionage at conferences around the globe. He has recently overseen the development of several cell phone detection tools used to enforce a “no cell phone policy” in correctional, law enforcement, and secured government facilities. He is regularly interviewed for leading national publications and major network television stations including Fox, Bloomberg, Good Morning America, CNN, CCTV, CNBC, MSNBC and more. He is the author of "Hacked Again", his latest book as well as a contributor for Huffington Post and guest blogs regularly for Tripwire’s State of Security series. Scott also writes for Business Value Exchange, Fortune Magazine and IBM Big Data & Analytics Hub.

Follow me on Twitter: @ScottBVS www.ScottSchober.com

@LinkedInEditors