Information Overload

It’s no secret that most companies today are at the breaking point when it comes to the amount of data they have to deal with. And this situation is bound to worsen. Experts predict a 4,000+ percent increase in data flow by 2020. As Forbes contributor Brendan Marr has stressed in a number of recent articles, the current digital overload is happening for good reason. Network speeds and capacity continue to increase at a feverish rate. The 5G revolution and the emergence of the so-called “Internet of Things” (IoT) are expected to push data traffic into the stratosphere. At the same time, in the developing world, economic growth and the rapid expansion of data infrastructure are bringing millions of new Internet users online year upon year. Data is increasingly cheap, ubiquitous and available. Yet data is a very different thing than wisdom, regardless of how revealing and valuable it promises to be. Ultimately, businesses that prioritize unstructured data collection over patient, rigorous analysis are doing themselves a major disservice. Data collection only serves its larger purpose if it is used constructively to advance understanding and improve performance. Absent this, businesses are truly better off if they avoid data collection altogether.

Why pursue data analysis at all?

Most businesses understand that effective data analysis is the key to making changes that will allow them to push past competitors. Yet mining data effectively requires far more than simply storing as much of it as possible and running it through a few business intelligence tools now and then. Effective data analysis demands clear objectives. One must know—generally if not specifically—what one is looking for. What is the goal of the search? What is one hoping to find?

Truly effective data mining by any business requires careful strategizing. Developing an effective strategy requires enumerating current strengths and weaknesses, short-, medium-, and long-term goals, and the current state of knowledge regarding performance. It is especially important to identify knowledge gaps—i.e. areas where knowledge is incomplete or non-existent. Guided by a clear governing strategy, stored data can be broken down so that the results are much more narrow, relevant and actionable. This is a difficult task, but one far likelier to produce useful results than the more scattershot approach favored by many businesses.

 Additional challenges (and opportunities) derive from the fact that almost all useful data is time-sensitive—i.e. its “window” of potential value is quite narrow. Success in the marketplace is all about using information quickly and effectively to gain an advantage over competitors. Yet even when businesses realize that the data they're storing is obsolete, they're frequently reluctant to dump it—despite the fact that they typically process only a fraction of what they hold. This is deeply unfortunate because a company's ability to increase its market share and revenues correlates strongly with its ability to not only collect data but to analyze and understand that data. At the same time, terabytes of obsolete data take up valuable space and increase power and security costs significantly. Raw data that stays raw not only fails to benefit those paying to store and protect it—it actively siphons away resources that could be spent more productively.

Data Overload and AI: Helping Front-Line Employees Cope

AI offers potential solutions. Relying on powerful algorithms, AI and related technologies are able to impose order on stored data by using a cross-section of variables to establish connections and relationships. Categorization and contextualization of data becomes far easier, especially through the use of metadata and tags. Some data emerges as an extremely high priority, other data less so. This also allows the technology to establish who can access the data and under what conditions. As a result, the likelihood that the data will be analyzed properly and used productively increases exponentially.

 Providing structure to largely unstructured pools of data inspires a level of confidence in front-line employees whose performance suffers when they don't know what their organization is storing, the best way to access that data, and whether the data itself is still accurate and relevant. According to a recent survey conducted by Dimensional Research and M-Files, nearly half of all workers surveyed expressed immense frustration at their inability to access data stored at more than one location within their organization. Four in 10 reported having to search more than three locations, often blindly, to find what they needed. Just under 50 percent remained uncertain, after seeming to find what they needed, that the data itself was still accurate and useful.

 AI is able to provide structure and consistency not only by organizing the data itself but by working to understand the human beings eager to analyze and exploit that data. AI-based systems, driven by “learning” algorithms, quickly absorb the tendencies, expectations, outlook, and goals of both individual employees and the larger organization. They are then able to act in accordance with this knowledge, satisfying data needs as they arise or even before they arise. The more exposure the system has to an organization and its personnel, the more reliable the system becomes at organizing and parsing data and delivering valuable information. The potential benefits are incalculable.

 Beating Back the Data Onslaught: The Example of Email

AI's ability to impose order on stored electronic communication testifies to its potential to tame data overload. An AI-based system could, potentially, use the information it identifies in emails to help adjust workflows involving different departments. John Brandon, writing for CIO, envisions data management apps with the ability to “read” email and social media messages and draw conclusions after considering a near-endless number of variables—everything from the nature of marketplace competition in a particular area to scheduling patterns and past performance. The efficiency gains and competitive advantage represented by actionable insights of this type are mind-blowing.

 Such tools, seemingly revolutionary, represent the natural extension of much existing AI technology. This includes current technology employed by Google and others that proactively attaches files to emails and automatically introduces bits of information into messages as they’re being written. IBM’s Verse messaging application makes subjective decisions about which stored messages to highlight and which to delete forever. Salesforce IQ’s AI-based platform goes further, using the data contained in emails, text messages, social media exchanges, and phone calls to help separate promising clients from time-wasters. The performance of these and similar applications, like virtually all AI-based systems, tends to improve markedly over time. John Brandon stresses how AI, with its ability to delve deeply into innumerable pools of data, is facilitating the offloading of boring tasks (like answering email) to machines, giving employees greater opportunity to “...focus on mission-critical topics.”

Countering Data Overload at the Top: Helping the Decision-Makers Decide

For senior managers concerned with everything from economic conditions and patent issues to trade regulations and industrial espionage, AI offers the growing ability to extract valuable insights from modes of communication that have long been resistant to a reliable analysis by most human experts. This explains a 2017 Forrester research report predicting that investment in AI will triple in the coming years. Owners and managers increasingly understand the level of competitive advantage they can gain through increased spending on these cutting-edge technologies.

For example, AI is getting better and better at identifying and highlighting hidden meaning (i.e. unconscious or disguised meaning and intent) contained in vast pools of quantitative and qualitative data, including financial reports, government documents, political speeches, policy statements, social media posts, emails, text messages, and audio-visual recordings. AI-based systems are proving adept at identifying extremely subtle shifts in language and tone—effectively identifying disguised intent in much the same way that they identify the objectives of average employees who interact with the system over time. As with employees, the more exposure the systems have to such data, the more reliable their responses become.

 Recruitment is another area where AI's data management skills are making technology indispensable to owners and managers. The AI-based program Cornerstone, for example—which employs machine learning to recommend specific job candidates to specific employers—has been found to be more reliable than other forms of screening and hiring. Personnel recommended by Cornerstone, whose algorithms organize vast pools of personal and employment data to draw conclusions about employability, consistently turn out to be better employees. Decision-makers in HR departments around the world will soon have no choice but to embrace AI's head-hunting skills. As AI continues to demonstrate, the very best decisions, whether in recruitment or other areas, generally come down to assessing available data effectively

The Path to Wisdom

Ultimately, AI can take advantage of the coming data onslaught to help both front-line employees and key decision-makers. Organizational efficiency and productivity will increase across the board—even as data flows rocket upward, as most observers suggest they will. Organizations need to strategize effectively to make the best possible use of AI as a tool that can help them understand and benefit from their own data. AI has the power to help employees find the data they need and to interpret that data. It also has the power to take over a number of mundane, repetitive workplace duties. This improved ability to navigate and exploit data ultimately benefits the entire organization, including key decision-makers. This makes perfect sense since the more AI engages with an organization’s data and the more familiar it becomes with that data, the better-placed it is to offer reliable, data-backed analysis. Ultimately, AI-based analysis can promote greater wisdom within an organization, but only if that organization makes a full commitment to AI and develops a clear strategy. In the end, an organization’s data is only as valuable as the organization's level of commitment to understanding that data. Absent this, data collection becomes little more than a waste of time, resources and storage space.