Crossing the Knowledge Chasm Trap

Humans are “meaning-making” machines. Our brains are hard-wired to spot and interpret patterns. We consciously and unconsciously crunch data to establish mental connections between ideas, events, emotions, moods, etc. We empathize and sympathize with things. We relate and associate. We inquire. 

Citizen analysts are as what William Kent describes in his book Data and Reality, “intent on describing a portion of "reality" (some human enterprise). In comparison, Database Administrators (DBAs) focus on the data processing activities needed to support the query. Sometimes a “knowledge chasm trap” exists between these two roles. 

A play on ideas: The Chasm Trap

The “knowledge chasm trap” relates to chasm trap problems in data modeling. The chasm trap occurs when two or more “many to one” tables are joined to a single table which has not been resolved by any context. The result can lead to over counting and faulty analysis. Sometimes the problem originates when tables are designed and populated without an analysis use case in mind. 

The “knowledge chasm trap” manifests itself like this. Say the analyst wants to know all customers who have purchased a product within a certain region. Pretty simple, first order level thinking. The analyst could go to the DBA (or if she knows SQL could script and fire the query herself) and request a result set for the question: “what customers bought what product within what regions”. The query would form a relationship between the product, customer and store tables and turn out the resulting data. For the most part, analytical databases are built and optimized around this exact type of work. If the data engineers and DBAs can “pre-plan” as many of these questions as possible, they can optimize the tables and queries, resulting in a fast-moving query and response service. Remember, the DBA roles are focused primarily on the processing activities and often paint themselves in a corner by planning data design around a slurry of first order level inquiries (an Inventors Paradox). The chasm widens as the analysis dives deeper.  

In developing Data Literacy among our citizen analysts, we must teach them how to spot potential knowledge chasm traps. Citizen analyst are not typically versed in data modeling and are often given data resources constrained by DBA bias. Qlik’s Associative Indexing Engine enables citizen analysts to challenge and reinforce incumbent data structures. Associative Indexing implies relationships, it does not explicitly define them. As such, data structures can be better analyzed, improved, challenged and reinforced. Associative indexing is a powerful data profiling tool that can improve how analysts describe a portion of business operations.