Google for numbers

The following is a rough transcription of a pitch meeting between an entrepreneur and an investor that happened in 1883.

Investor: “Did you say you are going to reduce daily commute time with your new invention?”

Entrepreneur: “YEP! There are these carbon-based life forms that are dead and buried miles below the ocean floor. Hundreds of millions of years of heat and pressure have turned them into a form of crude oil.”

I: “eh..ok”

E: “I’m proposing that we create some underwater explosions and study the sonic waves and figure out where these oil chambers are. Then we go down there and drill a mile or so and use steam to exert pressure and extract the oil and ship it back to the shore.”

I: “But, how does that help reduce a commute?”

E: “Let me finish. We will take that crude oil and heat it the hell up. At extreme heat, it will separate into a highly flammable liquid and a few other toxic substances.”

 <Investor shifts in her chair restlessly and looks at her Apple. ...and she’s not even hungry.>

 E: “Now comes the fun part. I want to take that highly flammable liquid and send it all across the country through pipelines and trucks and store it in small underground tanks.”

I: “When you say ‘all around the country’ surely you don’t mean literally, just in some big cities, right?”

E: “Oh no, I do mean all around the country. Cities, villages, mountains, islands, everywhere”

I: “Look, I really like your vision, but I don’t think this is the right investment for our firm. I would like to keep in touch with you though.” (This is usually how investors let you down easy)

E: “But I didn’t come to the best part yet. We are now going to take this highly flammable liquid and put it inside a hollowed out cast-iron block. We will pump air into this liquid and light it on fire”

I: “Whoa.. what the heck?”

E: “Don’t worry, these explosions are going to be well-timed and controlled and will happen thousands of times a minute. We will use this fire to expand air and use that air pressure to move a piston and use some very complicated gear systems to spin the wheels. We will build a metal cottage on top of those wheels and shorten our commutes”

 <Entrepreneur sits down and pushes her hair behind the ears and looks up to the investor with a proud smile>

 Investor gulps down a bottle of Vitamin Water (Those days it was just called water) and says: “I see, so *that’s* how you are planning to eliminate horse buggies?”

 ---

Historians could debate if this really did happen or not, but when you next stop your car at a gas station for a routine fill-up, you may wish to reflect on what a series of amazing human accomplishments you are filling your car tank with.

Simplicity is the best way to consume sophistication

 Just as driving that car is now simple, well-understood, and taken-for-granted, so is Googling for words. When I typed, “who made titanic”, Google came back with the right answer at the top: “Harland and Wolff”, the maker of the ship Titanic. Google also had 62,000,000 other answers that it dug up within 0.64 seconds, but its PageRank algorithm knew that I wasn’t looking for “James Cameron” (Maker of Titanic, the movie) for an answer.

Google has crawled the deepest corners of the Internet and created an incredibly massive and blazingly fast index of all web pages, articles & books. It matched my query with a Wikipedia page that had those words in the same relative order. With its cookies, location and other affinity algorithms, Google was able to assume the best fit out of those 62 million answers and show me the one that made me happy.

 Googling words is a really hard problem, but Google solved it elegantly, with a very simple user experience. We live in magical digital times, but like the gasoline-powered automobile, we also take it all for granted.

 Why Google for numbers is even harder

When ThoughtSpot went about creating a Google-like search platform for numbers, it became clear that this was an even harder problem to solve than searching for words due to the following reasons:

 ●     Undefined Search Intent: Human languages are well defined, but in the case of Googling for numbers in a Data warehouse, nothing is well defined. It’s just the opposite - everything is open to interpretation. There are no universal way to define any column or header in a database. We had to figure out the user’s intent and interpret what she wanted us to find.

●     Lack of Content: In the case of Google, the content already existed; Google just needed to point us to it. When combing through numbers, the ThoughtSpot answer is often computed. Every single question every single time requires a perfect visualization drawn up from thin air. And the response time must equate to the speed of thought (or typing)

●     A Googol of Combinations: Computing answers can get very complex very fast due to the PowerBall problem of analytics. Imagine if you need to find a perfect answer for a problem within a matrix where every value could potentially combine any other value within the matrix. We are talking about finding the one answer within billions or trillions of combinations (though not actually a googol)

●      Determining Context: Context is important while Googling for words, but context is everything when it comes to numbers. For example, comparison of product margins can take on a different view when it's put in the context of the Purchasing Power Parity of the country. Having the power to model that with a simple ad-hoc query can be quiet enlightening and powerful for a merchant.

●      Security: For the most part, Google can operate out of a single large index -- security isn’t really a big problem. In the world of corporate data, security needs to be built all the way down to the cell level and mapped to both roles and context. Answering a question such as, “Find my highest margin product,” may differ depending upon who is asking the question. A merchant manager for the US will find the top grossing US product, while a person with worldwide scope may get a totally different answer.

●     Complex Schemas: Googling for data in a corporate environment requires dealing with very complex schemas. Customer data may be in your SaaS CRM cloud, while finance data may reside on-premises database. We need to know which tables to pull what rows, headers, how to join or aggregate and in what order, and how to ensure fan traps and chasm traps are solved for. Then we need to translate it all into the package the user can understand.

●     Only one Correct Answer: In the world of corporate data, we don’t have the luxury of finding 62 million answers for you in 0.64 seconds. We need to compute that one correct answer and generate the perfect visualization in 0.64 seconds!

Simplicity and agility are explicit business needs

When you next login to a ThoughtSpot search engine and type a complex question in natural language with the help of autocomplete, you will see that your one precise answer was computed and beautifully visualized for you on the fly. We sincerely hope that you will take it all for granted because that’s exactly why we hid it all behind a simple, elegant search bar.

We don’t think about the complex process of extracting crude oil from the depth of the seas while we are enjoying our drive. You shouldn’t worry about complex data analysis or data science when you are looking to make a business decision based on the insight hiding in the mountain of data in your business.

Hope you will join us at our inaugural user conference, ThoughtSpot Beyond2018, in DC in November where our customers will share the analytics industry is heading next.

Follow me on Twitter --@sudheenair

<A quick note of Thanks to Ajeet, Amit & Steve, for their help>