“You can’t measure the success of an IT project.” Oh, really?

Over the last few months I’ve had some encounters that have spurred me to action. This article is the product of said action. In each case I was in a virtual meeting of business folks, project managers, analysts, and their tech counterparts from the IT department and external consultants. The representatives of the tech community were young-ish, Agile enthusiasts who subscribed to the motto “We’re Agile we don’t have to plan.1” A position even the authors of the Agile Manifesto don’t agree with, but that’s a discussion for another day.

Somewhere mid-meeting one of the tech team turned the conversation towards the impossibility of evaluating the success of IT projects. Once the premise had been introduced a chorus of affirmation arose both agreeing to and extending the argument. Indeed, according to those present evaluating the success of an IT project is a fool’s errand. You can’t do it and you shouldn’t even try. You’re just wasting your time.

Having listened to this discussion for what seemed like an eternity (probably five minutes), I was compelled to speak.

“You know,” I said, “I must be coming from a different perspective than you guys. In my experience you can always tell whether an IT project is successful.”?

They’re never IT projects. They’re business projects that happen to be enabled by technology.

First, you need to call it what it is. You’re not talking about IT projects. They’re never IT projects. They’re business projects that happen to be enabled by technology. That means you can’t evaluate them from a technology perspective. You have to evaluate them from a business perspective. And that’s a piece of cake—if you know what you’re doing and are willing to engage in a little advance planning.

Second, there’s almost always nine completely objective measures of project performance, and at least one quasi-objective measure. Whether your development model is Agile, Waterfall or anything in between, these measures apply. The first three evaluate project execution. The remaining evaluate project efficacy. Start with the three standard measures of project performance, scope, schedule and budget.

• Did you deliver the functionality you promised?
• Did you deliver it on time?
• Did you deliver it within budget?

If so, congratulations. If off by a little (less than 20%), you ran over. If off by a lot, you’ve got some explaining to do.

Now, if we’re being totally honest, those three really only measure how well you planned whatever it was you thought you were going to do. A project can be executed within its scope, schedule and budget and still not do what it was supposed to do. That’s why we need the other seven measures. They focus on the business process being automated by the project. They ask the question, was the intent of the project successful—did the project achieve its objectives?

A little illustration—albeit a highly simplistic one—might be helpful. Business is pretty simple. You move a “ball” whatever that ball is from Point A to Point B to hopefully create value. You can visualize this process as a pipe with a crank in the middle. You load stuff into the beginning of the pipe. You apply some work to the middle, and something comes out the end.

Assuming you’re working with an existing process and not on something that’s never been done before, there are seven criteria that you can measure to baseline how well the “As Is” process is working:

·?????? Inputs required: How many resources go into the process?

·?????? Transit time: How long does it take to go from Point A to Point B?

·?????? Output volume: How much volume comes out the end?

·?????? Output quality: What’s the quality of that output?

·?????? Cost: How much did it cost to transit the process?

·?????? Profit: How much profit did you generate?

·?????? Customer satisfaction: How satisfied is the customer? 2

If, after the project is complete and the new system goes live:

·?????? Fewer inputs are required;

·?????? Transit time went down;

·?????? Output volume was higher;

·?????? Output quality was better;

·?????? Costs went down;

·?????? Profit went up; and

·?????? Customer satisfaction rose,

Then congratulations, the project was a success. If not, it wasn’t.

You’ll rarely get all of these measures, but you’ll usually get enough to determine whether or not the tech enabled business project you undertook was worthwhile. And, yes, while there may be other mitigating factors, if these criteria are not met, it’s going to be hard to argue the project achieved its goals and was a success.

As I reflect on the conversations that prompted this piece what I find most concerning is how pervasive this kind of thinking seems to be. Otherwise smart people appear to have no idea that tech enabled business projects can be so easily evaluated. This isn’t rocket science. The solution I’ve described hides in plain sight, but it’s obviously not being widely discussed in either business schools or computer science programs. Sadly, this isn’t the only issue like this I encounter. Maybe there’s something to be said for experience and pattern recognition after all.

?

1.???? Don’t get me wrong, contrary to what you might assume, I’m not Agile bashing. Agile is great when applied with some discipline. I’m truly a fan, but more development sins have been committed in the name of Agile than you can shake a stick at. Too many times it’s been used as an excuse to throw “stuff” against the wall to see if anything sticks, and then to excuse such behavior when nothing does.

2.???? Again, in fairness, customer satisfaction is a little squishy as it requires the subjective evaluations of a customer both pre and post new system implementation.