Why are there so many BI tools?

This question comes up a lot lately. It's usually followed by questions like, "which one should we choose?", "which one is the best and is there an open source one?". So before I answer the question, or maybe even plural questions, let's first prove that there are many BI tools out there. Below is a partial list. If I forgot any major player, let me know and I'll update the post:

• Business Objects (BO) by SAP.
• Cognos by IBM
• OBIEE by Oracle
• Hyperion by Oracle
• Power BI by Microsoft
• Microstrategy
• Sisense
• Pyramid
• QuickSight by Amazon (AWS)
• Yellowfin
• Pentaho
• Qlikview
• Spotfire by TIBCO
• SAS BI
• Pivotal by EMC
• Tablue
• Looker
• WebFocus
• Bime
• Style Intelligence by InetSoft
• Spago BI
• Bizzscore by VCD
• Birst
• Redash
• Board
• JasperSoft
• Clear Abalytics
• Gooddata
• Ducen

Well, if the list above is not long enough, here are a few pointers. This is not the complete list. My favorite tool isn't even listed above. I am trying to stay impartial here. Most people working in the BI field have never even heard of more than half the tools on the list. If you want to look at this from a different perspective, a BI tool is considered infrastructure. If you look at different areas of the infrastructure world, you will find a much shorter list of options. OS: Win, Linux, Unix, OSX. Databases: Oracle, MySQL, SQLServer, DB2, Postgre, Sybase, Maria, Hive, Impala. Of course there are many specialized derivatives like VoltDB, Vector-H, CitusDB and PostgreXL, RedShift, BigQuery and others but these are specialized derivatives, most of them derive from Postgre. So we do have a huge influx of BI tools. The question is obviously why. Well the answer is, as expected, composed of a few  factors.
They are not difficult to write. The vast majority BI tools are not a technological break through. They are simply a set of tools and GUIs who's goal is to provide and intuitive and eye catching access to data. It's about making sense of data using visualization, exploration and automation. No rocket science.
BI tools are infrastructure that is used by business people and is often the customer facing facade of the application. This means everyone has an opinion and a say. In other words, unlike other infrastructure tools BI tools can not be measured on a set of price performance characteristics. Customers purchasing BI tools can not simply choose the best price performance tool for their scenario like in a DB or Application server or OS. This is an emotional decision. The look an feel, the ease of use, the ability to display a certain perspective or explore a specific data-set have an enormous impact. Price is still a factor, but it does not play a huge roll. This generates a myriad of solutions. Fortunately for all, the size of the market is able to accommodate everyone.
Which brings me to the next reason, market size. It's a huge market and it's constantly growing. This allows for a lot of companies to flourish. The market size also constantly attracts new comers. Since this is an emotional looks based decision, it is not uncommon to find more than one tool in the organization as sales like one tool, marketing prefer another and finance are still heavily relying on the old one.
The last, but not least factor is the fact that the earth beneath them is moving. All BI tools work vis-a-vis on data sources. They either query the data in it's original storage, or extract the data and run on their local copy. In recent years with the influx in data a.k.a BigData and the creation of the "Data Lake" or even lakes (or maybe just ponds) the data storage has diversified. From NoSql storage like Cassandra and HBase through columnar and in-memory databases all the way to massive file storage in different formats in HDFS, it's all new. The API is different. SQL does not always work, and even when it does, you need to optimize the queries for each DB. New data storage means new opportunities for the up and coming BI tools. The emergence of the cloud and it's data storage, plus it's BI tools is another major change in the landscape.

I hope by now the reasons are clear. Just to quickly summarize. No technology breakthroughs, huge market, emotional looks based decision, New data storage technologies and the cloud.

Now comes the more personal question: which one is right for me?
So first, if you are the technical person, a word of advice. Be careful which options you put on the table, as you might end up with a product that looks very sexy but can not scale, or even worst a tool that is "free" open source but requires a lot of work to setup and maintain.
So how do you choose the right one? Here are a few questions that you want to address:

1. As always, first ask yourself, what is my use case? Internal? External (customer facing)? Heavy lifting (explore & investigate data) or mostly visualization?
2. Who are my users? Are they comfortable with SQL? Is Excel their tool of choice? Can they program? Do they have the time?
   Please note, it is usually the case that you have multiple groups of users and you might end up with more than one tool. This however is not recommended. A tool that fits everyone, is usually a much more valuable choice.
3. Where is my data? Not all BI tools connect to all data sources. Don't assume that just because "everyone" is using this database the BI tool can connect. Also don't assume that the fact that it does connect means these two work well together. There is an issues of optimizing queries to the underlying database.
4. How much data do you have and what is your projected growth? While most tools are great with relatively small (TBs) of data that resides in traditional databases, only a select few can deal with huge sets of data that resides in locations like RedShift, BigQuery & Hive and only a few of these are efficient at it.
   Please note, some of the tools require/prefer to ingest the data and run on their internal storage. This is a scaling limitation.
5. How long does it take to create a report, dashboard etc...? As data grows and schema changes are more frequent than ever, the life span of your reports, dashboards and other artifacts is constantly shrinking. The ability to quickly create what you need now is a huge benefit in the long run.

Obviously, there are more questions, like the no-brainer "Does it look good?", which you are probably going to get three opinions on, from each individual of course, and the learning curve of the product. However, I do believe these are secondary to the questions above.

So whether you are shopping for your first one or you looking to switch to a better one, I hope clarified and helped.

If you have any questions or comments, please ping me I'll update the post.

Sam.