Intelligence, Data Processing and Government Decision-Making

Intelligence has always been viewed as essential for Government, as well as for private sector, requiring cutting-edge data. The need of "true information" has become increasingly decisive because of the information density explosion. Today too much information is available, but also too much is inconsistent. Processing this information has become crucial to extract "real" knowledge.

Intelligence, as it emerged from the NATO standards, defined, in its confidential business, the data collection as the dominant mission. For this reason, ICTs are fundamental today both in the government and in the corporate intelligence world, helping to manage information complexity and overload.

The basic question is: how, and how much, can we transform large amounts of text information into knowledge useful for public decision-makers?

The fine-tuning of Big Data to the Government Intelligence needs is much more complex than the corporate intelligence, given the bureaucratic structure of the Government structure and the difficulty in creating significant quantitative indicators for the public action.

The application of Big Data management to public and private Intelligence is even more complex because of the nature of information, more textual than numerical. When available, numerical data are -almost always- fragmented, and making sense of them is not easy. Statistical inference is often influenced by biases and data scarcity. This scenario gets worse when human mistakes are added, due to superficiality, inaccuracy, ignorance or negligence. In this case, the result is a definite failure.

Managing textual information, an interesting subset of Big Data is “text mining”. Basically, text mining is the process of combing through countless pages of plain-language digitized text to find "hidden" information in plain sight. Different than keyword searches and other algo forms of web data analysis, text mining is about finding "unseen" connections and patterns in plain-language narratives like documents and reports, but also newspaper, website articles, research papers, blog entries, patent applications.

Following the Introductory Overview of statsoft.com, the purpose of text mining is "to process unstructured (textual) information, extract meaningful numeric indices from the text, and, thus, make the information contained in the text accessible to the various data mining (statistical and machine learning) algorithms."

You can analyze words, clusters of words used in documents, determine similarities between them or how they are related to other variables of interest in the data mining project. In the most general terms, text mining will "turn text into numbers", which can then be incorporated in other analysis methods. These methods are described and discussed in great detail in the comprehensive overview work by Manning and Schütze (2002).

I recently posted on my blog a review of the Professor Potter's book on "Economic Intelligence and National Security", the only valid text I know on the subject, giving space to a reflection on the Economic Intelligence definition and concept.

You can find the posts here and here.

The conclusion drawn is that, given the slowness of the government bureaucracy and the technological speed of the private sector in data processing, it is likely that, in the near future, certain functions linked to the Government Economic Intelligence could be centralized in supranational entities, taking advantage of increased budgets, but loosing the national characteristic of the Intelligence associated with the legitimate defense of the national economic interest.