Implement Anything: Measuring ROI - Making it Rain (Part 4!)

So, you have a fledgling implementation plan (Part 1 and Part 2), a general understanding of how you can move the needle for your business unit's top or bottom line (Part 3), and you start your practice technology implementation journey... now what? Well, break out your pocket protector and TI-82 graphing calculator Bueller - it's measuring time...

Measuring ROI

Measuring sounds so simple, so easy, but with ROI it rarely is! Unfortunately, in our industry we do a pretty terrible job of measuring our companies' financial performance and thus have little ground to stand on when trying to prove out the value of an investment in technology.

Perhaps that is why our industry lags most others in IT investment, R&D spending, etc... Hmmmm. We might be on to something here.

As an industry we just don't typically measure the business at a detailed enough level to prove out the value of those investments in this industry. To be fair, some of this isn't laziness, some of it is structural as well. But, regardless, we lack the information to determine a clear cause-and-effect relationship to revenue or profit to prove the worth of almost anything we do.

Unfortunately, the same cannot be said for say, opening a new office in Raleigh; or starting up a new line of business – those can be viewed in a traditional way since the service itself is the main outcome of the investment. Those are easy to measure, and thus easy to evaluate and justify – which is why these types of expansions (or acquisitions) are so common in services industries. It’s the “easy” improvement.

But, that doesn't mean we're at a dead end (otherwise, why the heck did you read this?). We do have a couple of tools we can leverage to give us reasonably good information presuming others buy into how you're measuring things...

The Basics of a Baseline

So, if we don’t have a clear way to measure the innovations impact on net profits, how in the heck do we measure ROI??? This is what I struggled with for years (being anal retentive). If I couldn’t get hard numbers, I didn’t want to do it. So, I get it if you’re in that boat. But, get over it. Soft numbers are what we’ve got to work with people. So, how do you take soft numbers and stiffen them up: Use historical data to derive baselines.

That’s right, let’s get statistical! Don’t get scared by the word. All you need to do to get a baseline is use historical data to get an average performance for any metric you think your implementation can improve. This isn't a "perfect" approach because you can't definitively say that your implementation is the ONLY thing that lead to any measured improvement, but it sure beats the heck out of nothing at all.

Let’s take using BIM for documentation as an example. It is a service, so profitability is driven by productivity. So let’s think about establishing a baseline for productivity...

In a perfect world, we’d have amazing metrics on productivity inside our companies. If you do, you’re a lucky bastard. In my old gig, all our architectural staff was on salary. They recorded 40 hours a week on their time sheets, no more, no less, no matter what. So, we had no idea how many real hours it took to produce a job. We did have our fee budget worksheets that used some black magic math to project man hours based on job type, size, etc… but those were based on some “lick your finger and hold it up to see which way the wind is blowing” style logic. The margin of error was huge on any given job, and it was buffered by the staff’s willingness to work bonkers (not bankers!) hours and the project manager’s willingness to sacrifice on quality. So, pretty much like every other Architecture firm I’ve worked with before, during, and since.

So, what’s an aspiring BIM Manager to do? Productivity is measured in man-hours right? But that’s burned as a metric on both ends in an Architecture practice… so, you find a different metric for productivity.

In discussion with our Principals it became evident that we didn’t manage productivity or capacity (the other side of the productivity coin) in hours. We managed at the level of people. As in, how many people (and thus how much salary) do we need for that job, hours be damned. Of course, I was worried that the people metric had the same back end risk as hours (people using them for other stuff). But, since it was the thing that was managed, projects didn’t just get people because it was on the budget. Projects got people when they demanded it IF they had the budget for it. And yes, some people were split between projects complicating things a bit. But, when they split time the quantity of time might not be correct, but you better believe that the proportions were carefully watched by the project managers because that drove job costs.

So… that became our metric for productivity. We had good historical data on how many people were on a job any given month (by percentages of their capacity which was represented by the number 40. 40 no longer represented hours for me, it represented a person-week. We were able to look at the last 10 years of projects and build up a rough X many people per square foot per month metric for each market sector. Healthcare needed more people per square foot, Retail needed less, etc… But, we got a baseline. And, we were able to watch that number drop as we reached a mature BIM adoption in each market sector. And that can be translated to a measurable ROI in a couple of different ways!

We could do a similar thing for BIM when it comes to risk avoidance. While Risk Avoidance is hard to measure granularly, a baseline approach also works to document the number of RFIs on BIM projects vs. CAD projects. But, since Architects have almost no direct liability from risk factors outside of what is covered by E&O insurance, the main value from better coordinated drawings was also productivity at the project scale in the form of fewer CA hours. So the man-week approach neatly wrapped up the main risk avoidance and productivity increases in turn.

The last thing we might want to capture is the impact implementing BIM had on growth. Early on in the rise of BIM in the US, it was a real differentiator at bid time. We won competitions where BIM played a major role in the win. We were the only competing firm that had demonstratable BIM experience on some projects that required it. And, as we started documenting real productivity increases, our firm realized that it’s capacity was growing and it could both offer slightly more competitive fees on projects and pursue more projects with the same staff. All of these things contributed to growth, and again, since our productivity metric was capturing at least some of this we had some capability between that and anecdotal information on specific wins to document some impact on growth and revenue for the company. But, this is by far the most tenuous of the impacts, so documenting it is a tricky thing…

The psychology of ROI measurements

Something I ran into pretty early in all this was the danger of being over-optimistic about the impacts an implementation might have on any particular value back to the company. Well, of being perceived to be over-optimistic is a more precise way of describing it.

So, let’s say we get a win on a project with a new client and the owner says: “We picked you because we really want BIM on our campus and you guys blew the other companies out of the water with your BIM expertise.” (Yes, this happened to us on a job). Now, let’s say you walk in for a quarterly report and say – “The BIM implementation just achieved a net positive ROI because we won work with a new client due to our BIM expertise and that project is going to bring in 200,000 in projected profit.”

Now how do you think the principal in charge of that relationship is going to feel about that statement? Her team built the relationship. Her team put the proposal materials together. She missed her daughter’s dance recital to make the pitch to the client… Would your BIM have won that job without her team? No. Same goes for the marketing team, etc…

So, what percentage of that revenue is fair for you to count? There is no math formula for that. Period. Good luck finding it. And if you try and come up with a percentage on your own – you’re stuck with one big stumbling block… other people’s internal percentages of their own impact. We all like to think that we’re the one that won the job for the company. I promise you the marketing team is thinking they should get 25% of the credit. The Principal is thinking they should get 75% of the credit. The BIM team thinks they should get 50% of the credit. The CEO things they should get 25% of the credit for referring the client initially. The BD guy wants his cut of the credit, and on and on it goes… they sum to much greater than 100%.

Fortunately, there is a fantastic strategy you can use to get broad buy in on how to count financial impact on these super-squishy issues. It’s called… a meeting.

No seriously. Get the right people in the room (you probably know who that is already) and ask them to tell you a percentage. For my example, it was in the weekly management meeting with the principals and project managers. Rather than lay claim to 100% of the profit for my ROI, I just asked the room what they thought was reasonable. They argued about it for 45 minutes, but I got a 25% cut of that $200,000 projected profit in my ROI spreadsheet. I guarantee you if I’d staked out a 50% or 100% claim I’d have ended up with 10% at best, or more likely 0%. Let the people affected by the outcome propose something that they think is reasonable, and go with it. If you make a claim that seems too high to one of those people, they will distrust every other thing you ever tell them about ROI. It is a sure-fire way to shoot yourself in the foot.

We did something similar with ROI on 3D coordination with Navisworks. Just like with BIM, I went digging into project data to try and establish a baseline for the cost or frequency of coordination errors. But, the data seemed so much lower than I'd expected from all the belly-aching I was hearing from people about the poor quality of construction documents. I was talking it over with my boss and he drew this little gem:

Thanks Rick! (diagram by Rick del Monte, FAIA) ((Drawing not to scale)) (((Yes, the ink is brown, if you knew Rick you wouldn't have to ask)))

His advice was that the numbers we could see were just the tip of the iceberg. That only mistakes that made it to a change order were documented anywhere, but there were a lot more and those did have a cost to the job overall. He suggested I ask the superintends (the people at the front lines of coordination issues) thought the cost was per field issue.

So, we polled the superintendents at the company on the average cost of a field issue at their annual meeting. After the meeting I got back $4000 from the very people who were known to be gun-shy about 3D coordination on projects. So, I used $2000. I didn’t get a single peep from any of the field teams when I started presenting the value of 3D coordination in reducing the number of field issues because they’d all bought into 4K and I was just showing 2K. After that presentation I had a bunch of people come up to me telling me I underestimated the impact. But, the ROI case worked at 2K – so the fact that they all thought the number was low was pure gold.

So, anytime you feel like you’ve got a soft number, find the people most qualified to have an opinion on the impact and get them to give you a baseline or a cost or whatever you need to make your case. It might be soft, but if anyone questions it you can just say:

“Well, the superintendents told me to put in $4000, but that seemed optimistic so I put in $2000.”

That gets you instant buy-in, and that buy-in is priceless. With it, people focus their time asking you about how it can deliver that value, and how quickly or slowly can you ramp up, etc… Without it, people clam up and discount everything else you say as pie-in-the-sky technobabble. So, even though it might be scary, engage the company in helping you set your baselines and targets, your metrics and costs, etc… They might be a little low, or not, but if you get that buy in your case for ROI is so much easier down the road – and you’re not the only one who made a mistake if you were overly optimistic in the end. ??

Projecting ROI

Of course, a lot of those examples have been focused on ROI in the past tense. To get an implementation off the ground you need to provide ROI in the future tense. So with that, let’s use everyone’s favorite BIM application to do some ROI projections!!!

That’s right, Microsoft Excel!

Now, since this was a class at BILT I have a dataset to share, so you can open this on your own and not squint at the pictures if you'd like. This way you can play with my numbers, or make your own. The math isn’t actually complicated. If you can Dynamo, this should be child’s play. But, for now, we’ll take a look at how this spreadsheet is set up and how you can use it to project out ROI for an AE or C company (though some modifications would be needed for a contractor as the fee relationship isn’t quite the same. ??)

Now, this spreadsheet may look intimidating… but it is actually pretty simple. Basically, it lets you note cost and revenue resulting from your implementation. So, let’s talk through the numbers you’ve got to put in…

- For AE firms, put the projected annual net fee

 - Costs specifically for the implementation including hardware, software, implementation staff, loss of implementation staff, and costs incurred paying legal fees for the implementation

 - Direct measurable revenue (if any) from getting staff costs covered at cost, at a fee, or otherwise

 - Productivity gains and cost avoidance due to the implementation

 - AE centric way of calculating project win contributions as a result of implementation

So, basically, you plug in these numbers and, Viola!, you’ve got the start of an ROI projection. Let’s look into each at a little more detail...

Costs

Implementations aren’t free. One can rack up a lot of costs rolling out new practice technologies. Try and be both honest and consistent about the costs you’ll need to incur. In the BIMROI example file, you’ll see the first cost started being incurred back in 2005 with the first Revit license we purchased. That license, and a second starting in 2006, were used for experimentation during down time by one of our lead designers and a few other geeky staffers who were happy to play with a new toy to better understand it….

Early 2007 was the first real Revit project, and that brought a ramp up in license costs. It was a short project that was a disaster… but mainly due to some poor choices managing that job. So, we decided to double down in July on additional projects and see if we could get better results. We did thankfully, but I started full time as a BIM Manager at that point too. We also engaged our in-house counsel to oversee any BIM related contract language, NDAs, and other legal stuff starting in 2008, billing an hour a month to our implementation for simplicity...

Now, fast forward to late 2013… We had two full time well paid BIM staff, software was running over $100,000 a year for our Architecture firm, and we had some hardware costs for a Revit Server, etc…

Think that’s bad, wait till you see 2015… for reasons we shall not discuss, we lost our BIM manager and had to find a replacement. Don’t underestimate the costs of losing great personnel. You spend a ton of time recruiting. While you have an empty seat problems blow up in your face you don’t have the staff to handle. You end up hiring consultants to come in and keep the ship together. You fly people in to interview. You pay signing bonuses and moving costs to get them there. You have training and other “get up to speed” costs. You might even pay a recruiter a hefty percent of someone’s salary. You get the picture. Losing people is expensive.

Now, at this point, I left the company about 6 months later and stopped updating these things with actual data and we just have the projected data used for the implementation plan. Projected out to 2020, with the company’s growth and additional offices, we end up with a projected budget of $867,000 a year on the cost side. 5 full time staff, $175K in software costs, Revit Servers in every office (on VMs), etc…

Of course, that isn’t how things actually turned out. These projections were initially made back in 2013 more or less. Implementation plans change and update, other implementations provided some relief in those offices, and you make some changes and see how it affects our ROI overall.

But, over the 15-year period this is tracked/projected – the total costs dedicated to the implementation is $6,185,000 plus or minus. For Revit. At a firm that had less than 100 people in 2005 and will probably have 220 to 250 in 2020.

So, that’s a lot of money, right?

Well, yeah. But what was the value???

The Value

So, per our earlier discussion, value can be measured a lot of ways… at the end of the day, things can drive profit, things can drive savings, and things can drive wins. So, how does that show up in this spreadsheet?

Well, right when we first started, while I was a dedicated BIM Manager, I didn’t spend all my time on overhead. I billed ? of my time to projects. Now, departmentally, that time was billed at my fully burdened cost to the project, and while the project made money on that time at our billing rate, I was doing “Architecture” work so the fee was not for BIM. So, that ? of my cost came back into the implementation as revenue at cost that covered part of my cost to the implementation. We started marketing our use of BIM pretty heavily after the first job, and had a project brought in that people thought we should get some credit for in 2008, and we also started seeing some productivity benefit on one of the two projects, and recorded it.

This spreadsheet has a grossly over-simplified way of accounting for the revenue that hides the complexity of the real process that came up with a number. But, projecting ROI (the original purpose of the spreadsheet) is a game of approximations and averages. The actual data that was used to confirm and fill in actuals is a lot more complex of course…

Lets see the formulas though…

• Calculating the at cost reimbursement at 50% is simple enough: =(AM6*0.5)*-1
• Calculating the time savings is a little more complex: =(2080*5*0.01*85)/12 If you don’t already know, 2080 is the number of work hours in a year assuming 40 hour weeks. Here we’re taking the 5 people on the project, claiming a 1 percent increase in productivity, multiplying it by their average billing rate, and dividing by 12 to get to a month’s value. Now, wait, didn’t I just say we started tracking productivity at the man-week level using # of people on a project? Well, yes. We did. How does that relate to this? Highly indirectly. This was a psychology thing. While we were seeing much better than 1% improvements in the project in question, the impact on the firm overall was tiny. And, people were in training, etc… So, it was basically agreed that given the improvements we could measure that the conservative thing was to claim initially that this had a 1% net improvement on productivity on the project from the standpoint of returned value back to the firm. I thought this was stupidly low given what we were seeing on the job, but I used it because it had buy-in. I was able to milk that percentage up to 5% over the course of a few years, include the whole firm in the hour calculation once we’d fully implemented, and even at that tiny percentage it does have a significant impact on the value proposition.

For the record, at the project level we were seeing a 20% reduction in staff compared to the baselines; which was WAY more than a 5% net productivity improvement.

• Calculating the contributions of winning jobs was even more “fuzzy”: =Insert Formula I had our annual revenue projections, and based on the markets we were targeting and their interest in BIM, the principals conceded a contribution of net revenue based on our efforts supporting bids and checking the “BIM Box” on RFPs. I don’t know if it is scary or reassuring, but most years even when considering specific project wins where we had a major hand in getting a few jobs (like in 2008 and 2009), we were surprisingly close to the projections we had made in terms of dollar value even though the historical calcs were done totally differently.

Now, these formulas aren’t “THE FORMULAS”. They’re the equivalent of a back-of-the-napkin sketch way of projecting an impact that people were comfortable with. I could just as quickly crank out a formula based on the number of people freed up and the additional revenue they brought in for the “savings” entry. All of these will inherently be soft numbers and highly debatable. So, the only thing that matters for projections is to get that buy in on the black magic you are going to use so people don’t crap all over it.

Tracking actual metrics and backing up (or hopefully exceeding!) the projections is where you need to be more scientific. You can always go in and plug actuals in to update the projections. That’s just how it’s done.

Now, in 2013 we started charging construction jobs a fraction of a percentage point for BIM support for time spent helping them that they didn’t want billed to the job directly… and you’ll see by then the contribution we were making to wins was minimal as the market had mostly caught up and it was a rare project where BIM was a differentiator.

And looking into the future, we thought that win contributions would pick up as a result of some new initiatives that were part of our broader definition of the implementation from the updated plan in 2016 that included use of tools like Enscape for immersive VR design review processes, etc… Basically, we wanted some additional investment for new personnel so we could leverage new technology within the scope of the existing implementation of BIM for design and documentation. So, again, these things aren’t one and done with a plan and projection and track it for 15 years. They’re a continual work in progress as things change and your plans change with them.

In the end, including rough alignment with actuals (intentionally conservative on purpose) this implementation over 15 years of tracking/projections was set up for a return of $12,704,000 dollars give or take.

Yes, that’s right, double the money in 15 years. #NotToShabby. And this is why being conservative at projections is usually OK. You’d be shocked at how small of a productivity improvement you have to deliver to provide a LOT of value at a firm with 100 people. That’s a lot of people to get value back from one way or another.

The calcs…

The rest of the spreadsheet is basically a series of calculations that take those numbers and figure a couple of different values you can use to fine tune what you’re wanting to do…

 - This section calculates monthly profit and loss by taking the revenue and subtracting the costs for that month. Since the numbers are squishy, there are multiple rows so someone who things wins are BS can see the return without wins included for instance.

 - The cumulative calcs are similar, but rather than just being a snapshot of each month’s P&L, the cumulative total is calculated with that month’s results and all prior months. You know, cumulatively. Cumulative. Cumulativsolutely…

 - Running numbers are calculating the actual ROI ratio for each of these sections. Again, ROI is calculated by taking the Net Profit of the investment (the cumulative P&L) and dividing it by the cost (the running or cumulative costs).

Now, rather than look through more spreadsheets, let’s play with charts!

That’s going to be hard to read without a magnifying glass, but you’ve got the Excel file so you can look it up there. This shows a couple of fun things…

The additive area graph is showing the contribution of each category of costs and each category of revenue / savings each month. Labor of course makes up the bulk of the cost, but savings (even at a couple of percentage points) makes up the bulk of the returned value.

The lines are showing the P&L for the implementation each month. You can see the big downward spike in 2016 when we lost our BIM Manager. Ouch! More importantly, you can see that between the revenue coming in from billing time to jobs and the savings being delivered even at just a couple of percentage points of increased efficiency, the implementation is net positive as early as 2010 and remains net positive month to month from there on out.

Also, you can see the huge impact a couple of early project wins had to the overall value back to the firm.

This chart is taking the same monthly P&L data as the prior chart, and putting it as a background to the cumulative data for each category at an order of magnitude higher scale.

The cumulative data smooths out the monthly spikes and shows you much clearer trending information over time, and gets us where we can see the real break-even points for the investment being made.

What’s interesting to me in this data is that the “big bang” of the early project wins meant the implementation pretty quickly broke even, but on their own wins were not enough to keep it net positive (yellow line). Costs eventually swallowed that return back up. And even if we were able to implement cool new differentiating add-ons to BIM (Dynamo, Enscape, Etc...) by then the costs of the larger department and support staff make it hard for wins to pay the bills.

So here's the headliner from this data: The majority of early-opportunity for ROI is only available to those who adopt the technology and attain proficiency while it is still a competative advantage in the marketplace. In other words, the BEST time to invest in new technology (for achieving ROI) is as an "early adopter". This doesn't mean leading edge when the technology is unproven - as that carries risk of failure and usually limited initial benefits as the technology goes through growing pains. And it disproves the late-majority adopter position that you should wait till a technology gains broader market acceptance; because by then everyone will be able to say "we do BIM too" and your ability to win work and get that early ROI bump is gone. So, USE this information to convince your firm to invest early in practice technology!!!

If we only consider direct revenue and savings (orange line), it takes a lot longer to break even, but it’s a steadily rising tide as the implementation gets better, delivers more efficiency, and affects more people in the firm positively. Sure it takes until 2013 to cross the break-even point, but after another 7 years it’s delivered over $2,500,000 in value. So, not bad.

Of course, the green line is “all in”, and if we believe all the positive metrics not only does the implementation break even quickly thanks to those early wins, it also projects over 6.5 million in returned value by 2020. Go BIM!!!

Of course, we’re here to talk ROI. And, here’s the actual ROI graph for those same break-outs. Now remember, ROI is a ratio. So, anything less than zero is a loss, anything more than zero is a gain.

I’ve also added in a blue line that represents the ROI that would have been achieved by the company by taking the same amount of money month to month and investing it in the stock market assuming a 5% ARR. So, just like your 401K. ?? This is a good benchmark to use as a minimum because you should deliver a better ROI than this blue line. Otherwise, your company is better off not giving you that money. #SorryNotSorry

Our red “direct revenue only” line is of course a negative ROI. But, we can see the slow and steady march towards positive ROI of the orange line, as well as the erratic impact of the yellow wins line. All in with the green line we’re sitting pretty well above the blue make or break line.

So, looks like it was a good call to do this BIM thing y’all! Phew!

I can (of course) show similar graphs for things like 3D coordination...

And right away you'll notice a big "Ouchie!". Yes, you're reading that turnover correctly. Three people. Punch in the gut. On the up side, we had much more success using our troubled 3D Coordination implementation for winning work with owners. So... not all bad! To fully air the dirty laundry, we had the original "Navis Guy" depart Beck for a BD role at another company after the completion of the first real job. After that, we more or less half-assed it for a while (no real leader for the implementation) until eventually the guy who did the actual work on the first project was talked into taking it over for the whole company. Then, he left to be a missionary in eastern Europe with his family a few years later leaving the guy already running the Synchro (scheduling) implementation to carry the torch. Not surprisingly, running two full time plus implementations by himself burned him out and he left a year later for a software company.

I told you we didn't always have our shit together...

Shortly after this we incorporated all our practice technology implementations (design and construction) under one department, started resourcing them properly, and not surprisingly saw substantial improvements from the more concerted "whole-assed" effort. I'm happy to say that since my departure Mason and his team (yes, there are now multiple people supporting 3D coordination instead of 1/2 of one) are kicking butt and taking names (whatever that actually means).

Looking at the cumulative numbers, despite our screw ups we managed to scrape by and get some reasonable ROI pretty fast. A lot of this was because we were able to keep a vast majority of the actual man-hours billable to jobs over the first 5 years, so actual costs to the company were minimal. Thus, even the small value enhancements we were able to achieve made up for the costs of the software and other bits we couldn't reasonably pass on to the jobs (read owners). And, because of the BIG spike from early wins with new customers for the firm we could call this one a success despite our idiocy with the staffing of the implementation early on.

Really really real?

If you're wondering... are these numbers really real? Well, the answer is - it doesn't really matter. I mean that in the sense that it is pretty hard to "prove" any of these numbers outside of the costs. But, they aren't "my" numbers as a practice technologist. The metrics I used to make these numbers were all "provided" to me (with a lot of urging and meetings and e-mails as long as this series of posts) by the people running the business. These are as much their numbers as mine. So, in that context, whether they are accurate or not is somewhat irrelevant. They are the numbers we all believed in and thus acted upon when it came to decision making. I could churn up similar numbers for our reality capture service department, or our scheduling implementation.

And, while this specific format is something I pulled together for these presentations and blog post, these numbers are the same core numbers we showed in our internal implementation planes (one of which is included in the download link) though that has been redacted to protect the innocent (and the guilty). So, while these charts will be unfamiliar to my old peers and bosses at Beck, the numbers behind them are what we used to justify most of the budget increases we got the last 5 years I was there

And, assuming the numbers are even within the ballpark of reality, I think they speak for themselves.

So, "real" or not - if you get buy in on the metrics and track progress faithfully - you can absolutely measure ROI in meaningful ways and have good data to present back to your company's leadership on the financial impact these implementations are having for your firm. Of course, HOW you present them matters too... which will be the 5th and final topic of discussion on this never-ending thread! See you in the comment section you brave and persistent reader. And, thanks as always for making it to the end! See you for Part 5!

And, if you want to play with the materials discussed above and in future parts you can download them all:

Implement Anything Materials

ROI Materials

Have fun!