Consultants: What I've Seen, and How It SHOULD BE.
Dr. Eric Woodell
World's #1 expert in data center resilience. I audit and certify colocation facilities, ensuring secure, continuous operations—insured by Lloyd's of London.
My Introduction to Consultants
Working in the mission-critical facilities arena, we all end up eventually working with consulting firms.
Not surprisingly, I’ve worked with quite a few of the big-name firms. I’ve briefly written about it at our website, but I wanted to expand upon those experiences here.
We’ll change the names, to protect the (ahem) guilty…
When I started working for one large company, responsible for a portfolio of ten sites, I quickly discovered a “problem child” data center that was no less than 37 years old. It had started out as an office building, and some areas were turned into server closets, and it sort of grew and grew over the years.
The original electric plant- designed to carry only 1 megawatt of load, was actually carrying 1.7 mW; the system had been somewhat bolstered in the past with extra patches and feeds coming in, a generator was added with an automation system literally designed and built in the shop upstairs. The automation system- such as it was- had no schematics, the engineers who’d built it had retired long ago, and the automated functions often didn’t. The breakers were so old that replacements were literally found throughout the country in barns and warehouses, brought back and “refurbished” as best as possible, despite being so old replacement parts were no longer available. The Motor Control Centers (MCCs) were literally so old and unsafe that none of my electricians wanted to service them, as they could not be safely upgraded. Most of the safety features of the breakers didn’t work anymore; the plant was unsafe to operate.
The cooling system relied upon five chillers, two of which were decommissioned in place because the internals had rotted away over decades of service, resulting in refrigerant leaks and no functionality. The others were slowly giving up the ghost, and the associated chill-water piping was so frail with simple erosion- and rust- that it was only a matter of time before we had a catastrophic failure, causing our primary data center to permanently shut down.
Our backup site, the “new” data center had been in service 24 years without major upgrades or repairs, so it was only a few years behind the old one, in materiel condition.
Being new to the company, and having to deliver such bad news to directors and VPs, I felt like Captain Edward Smith on the RMS Titanic, looking at disaster about to unfold in front of him. Naturally, when I submitted all of my findings to the executive leaders of my company, they were unhappy with my reports, and were hoping I was prone to exaggeration. [Since I was working on my Doctoral degree at the time, they started calling me “Dr. Doom.”]
However, given the dire nature of my warnings and the potentially lethal effects for business continuity if I was indeed correct, they called in one of THE most respected consulting firms in the world, to check out the facility with a fine-toothed comb.
A 2-man consulting team arrived, worked ~6 hours each day, and were there for three work days. They examined all of my reports, talked with the building engineers and chief, examined all of the same schematics and diagrams, looked at all the same pipes, switchgear and cabling… And then they went home.
A month later, they sent us a bill for $75,000, along with a 72-page report which merely confirmed my findings. They didn’t offer a single shred of new information.
Admittedly, I should have been happy that such a prestigious firm was agreeing with my findings, but I was appalled by the fee. Imagine my reaction when the travel-bill of $5,000 arrived a week later! Really? How does anyone rack up a $5,000 travel bill for two nights of lodging and airline tickets?
Of course, they kindly offered to help us develop a plan to mitigate the risks, solve all the technical problems and make the facility viable for business for another 15-20 years… all for the paltry sum of $250,000! I was incensed at such audacity.
How I’ve Seen Consulting Firms Work
That was my first experience dealing with consultants in this industry, but the truth is that every consulting company I’ve had to work with has behaved this way. They come in and bid on a project with a narrow scope of work, and charge exorbitant prices. The idea is that they need to maximize their profits, because they’ll have only one shot at your business; after that, they’ll likely not see you again for a long time, if ever. There’s no prospect of having a long-term relationship with the client, so they’re after every dollar they can get, right now. Of course, serious projects can take months or years to develop, and the consulting firm- to be truly effective in solving the clients’ problems- needs to foster a long-term relationship, with meaningful communication and collaboration.
Having spent quite a bit of time in the Critical Facilities Manager role, I knew that a consulting company coming in the door, would be thinking about how to get in and then get out quickly, and move on to the next… victim. In other words, the ‘hired gun’ wants to be pointed at the target, and after it’s hit, he’s moving on. Gee, thanks, what happens if something else pops up, like procedures, training, enhanced maintenance or improved efficiency? The narrowly-defined scopes of work make getting assistance on peripheral matters problematic.
If the results aren’t quite what were expected, is the hired-gun going to stay and help improve the situation, or do we have to develop a new contract, new scope of work… and come up with more funds, to get the consultant to keep working with us? During one project the consulting firm- of Professional Engineers, had literally specified the wrong equipment, which the customer had bought and installed. I came in during the commissioning, and realized it was wouldn't work. After the spectacular failure of the equipment to perform as intended, the consulting firm came back with a new scope of work- and the expectation of more money- to fix the mess they had created!
I finally came to the conclusion, there has to be a better way!
How Consulting Firms SHOULD Work
Why can’t consultants use a model that is specifically tailored to meet the clients’ needs?
- Why not charge a low monthly rate per data center in the portfolio that they’ll be helping, and the scope of work includes anything that they can possibly assist with; engineering and operations, training, procedures, energy efficiency studies, the works? Life does not fit into a neatly-defined scope of work, nor does the operations of a mission-critical facility.
- Keep the monthly rate significantly lower than the loaded labor-rate to hire an internal expert, and you have the flexibility to continue or discontinue the relationship as you see fit.
- And finally, provide whatever technical assistance, training or other recommendations to your field staff that they desire.
This gives the client the flexibility to get as much expertise as they need without a huge upfront cost- OR an outrageous travel bill!
The goal should be to foster long-term relationships with clients who own and operate mission-critical facilities, because effective management and operation of those facilities is not a quick or simple solution.
Of course this requires time to build relationships with the players, teamwork and collaboration. It requires constant monitoring, feedback, adjustments to plans, and then checking the results. This takes effort and time to have lasting, meaningful outcomes.
It requires a commitment to the client, for the long-haul.
I have never seen this approach used by the consultants. But it IS the approach that I, as a former Critical Facilities Manager, would want.
And that’s what we do!
Director Of Engineering And Operations at Critical Energy Storage Solutions
8 年Eric, my firm, Somerset power Systems, has been in business for 4 years, your model has been our MO since day one. Not all consulting firms operate in such an unethical manner as you describe. If we make a mistake, we pay for it, and we do make mistakes every now and then, it's how you respond to the mistake that makes the difference. Our three partners all agreed on how we would treat the customer before we ever hung our sign on the door. Just know that there are a few of us that still live and work ethically and treat our customer as the boss, the one we work for, Thank you
Independent Information Technology and Services Professional
8 年Excellent article. When I joined an IT group in my last 9 years before retirement, I could find nobody in the company that could give me answers to how data centres, telecom rooms, and radio sites should be set up or run. I had to learn on my own although I did come across a couple of people in the industry who did help me out in that respect and those are the persons who got my business.
People and process integration in Engineering Solutions with foundations LSS | Championing Continuous Improvement and respect.
8 年My exposure to consultants has found many use arrogance as a method to indicate how competent they are. Instead, what your describing, requires some humility and being a good listener. The arrogance may come of being so successful but those of us that want grow need to be more thoughtful. Imo