Random Thoughts on Analyst Firm Research and why you must thread carefully

Many analyst firms produce so-called research that has the illusion of objectively when comparing different vendors. What goes unnoticed is the slight of hand practice of changing the criteria ever so slightly every year. These changes are often confined to small print, but these changes create winners and losers that have real business impact.

Analyst firms will tell their clients that this so-called research is designed to help narrow down a vendor search, not tell a customer which vendor they should choose, but by this argument, if a vendor is not on the list of vendors in the report, it will not form a part of the decision process.

Imagine a scenario where you are an analyst having to go through a rigorous process to review say the same 19 vendors every single year. Ask yourself, what if you decided to tweak inclusion criteria to reduce out two vendors you had to review? Who would get upset and would a vendor be able to make a viable complaint? Taking this one step further, you may have an interest in a new startup vendor and want to get them included but need to reduce out another, so you tweak the criteria and find a way to make this happen.

What do you do if the best vendor in the marketplace happens to be an open source product offering produced by a really large well known wallstreet entity? This challenge goes well beyond inclusion criteria and requires an understanding of biases. If the goal of any analyst firm is to sell research, Consultancy and events, and open source products tend to not participate at this level, do they ever make it to anyone's shortlist?

One product offering that I believe is world-class in the Enterprise Architecture Management space is a tool produced by 德意志银行 known as Waltz that will run circles around other highly rated tools, but otherwise gets zero ink dedicated to it.

The real issue is that a listing is often a prerequisite to being invited to the table to pitch or run a proof of concept for a prospect. After all, with so much choice in the market it’s difficult to see the wood for the trees – so taking an analyst research report seems a logical step when wanting to whittle down a selection list. The problem is that the same vendors dominate a majority of these reports – multinationals such as Microsoft, Oracle, Google, IBM, and so on. This is stifling growth as a whole; it creates a situation where you need to be big to get the business.

Much of the published research is written by big enterprises about big enterprises and is sold to big enterprises. With the additional hurdles for vendors to be listed, it is no longer a complete assessment of the market. You need to ask yourself it that is what you need or do you need options that include open source.

When conducting your research, ask yourself, where are the comparison benchmarks? Who are the customers and are they satisfied? How long has the vendor been around? Let’s look a little deeper – try the smaller analyst firms who know their stuff – analysts from Constellation who are informed, independent and passionate about technology.