Connecting the dots

When you need to understand a problem or a concept fast, it helps a lot to know its relations with other objects / entities, together with the nature of these relations, i.e. its extended ontology. This includes the knowledge of a timeline of related events.

We often struggle with two extremes. Either we are presented with an over-complicated picture (not always accurate though) or provided with just a handful of context-deprived information. And it is usually not possible to find relevant information on the internet. Hence, we desperately need a new kind of analytical frameworks.

For a little more than a year, I've been working (in a free time I do not have) on an application allowing identification of relations between concepts and ad-hoc construction of the knowledge network around a given topic.

The picture above presents such a network around "big data" query. Colors were modified based on the object tags - dark blue for company, green for concept, gray for product, so far. A special type of nodes - yellow rectangles - present events connected with the nodes. Shadows behind the nodes indicate text notes. URLs are also assigned to many nodes, but they are not currently exposed.

In addition to visualization, there is a growing set of exploration tools returning text output.

The picture represents a tiny and the most generic part of the underlying facts base. Still, I had to remove a couple of nodes. Fortunately, the graph is both dynamical and editable.

The new nodes can currently be added manually only. The automatic data retrieval module is still under testing, and is used as a support tool. Similarly, the inference mechanism requires more work. Even the core of the application cries for re-development. That's the price of adding new functionalities without re-thinking all the key consequences. But, as I wrote earlier, it is a deep moonlight project ;)

My application is far from perfection. The progress is not breathtaking. Still, I'm satisfied with the results so far, and hope I will be able to push the project much more further.

Unless somebody provides a decent tool with some similar functionality. One may spot on the graph a system called Palantir Gotham. Although its nature is about discovering relations between certain type of objects - like looking for information during criminal investigations or employees surveillance, there may be some similarities in the approach to connecting data. But there is nothing like that in the commercial space.