Adventures at Work, Episode 7: Cost Reductions I'm most proud of

How many times have you been asked the interview question, "What accomplishment are you most proud of?"

My own estimate to date is around ten times, and I've asked it myself as an interviewer more than that. It's the kind of open-ended question that keeps the person talking and gives insight to their skills, personality, and what excites them. This and "Tell me about yourself" were two questions I always tried to ask candidates, to see what areas they'd focus on without direction.

I often choose cost reductions as my answer, for several reasons. First, because they tend to be interesting stories with a lot of challenges. Second, they have objective, quantifiable benefits with hard numbers. Third, they tend to highlight my unique value. Product development is a team effort, with a lot of players. It's sometimes difficult to feel true ownership among so many contributors. I'm prouder of cost reductions because I was more directly responsible, as an individual, for their success.

Cost Reduction 1: Radio Battery

Batteries were the bane of my existence, across different product types and companies. Many technical PMs will share the same sentiments. The Samsung Note 7 team certainly will, after their exploding batteries burned customers, cost the company billons of dollars in market value, and damaged the entire brand.

Batteries are large, heavy, and messy to manufacture. Even the best modern batteries are ruled by the constraints of chemistry, and are just incremental improvements to the rechargeable concept from the mid-1800s. They lose capacity over time. They lose capacity the more often you charge them up. If they're left dead for too long, they won't charge anymore, permanently. If they get too cold, they stop working. If they get too hot, they can swell, vent, or explode. They're the direct or indirect source of most customer complaints.

None of this is a slight on battery engineers, who do a lot with what they have. It's just a technology that hasn't seen a large evolution yet. Until someone invents cold fusion, safe miniaturized reactors, or some other amazing innovation that provides a huge leap forward, anyone in the tech field has to deal with these finicky components that enable the entire product to function.

Our new radio product at Harris had launched and was selling well, but the margins were lower than preferred. Management tasked my boss with a cost reduction effort. At the kickoff meeting, he began going through the bill of materials line by line, with group discussion about shaving pennies off here and there on simple passive components. I interrupted the meeting after about fifteen minutes of this.

"Hang on, I don't think we're going about this the right way. Sure, we can look for low-hanging fruit, but that's only going to save us a dollar or two, and that's not going to move the needle. I have a BOM sorted by component cost, let's look at the top ten and apply our big efforts there."

Our largest cost component, by far, was the battery. It was single sourced, and I knew from speaking with Purchasing that they were having several difficulties with the supplier. I also knew that battery developments were difficult, and other teams had failed in the past.

My boss then issued both a challenge and an opportunity: "Who's going to do a battery development? Are YOU going to lead it?"

"Yes, let me take it, I'll do it."

I had really stuck my neck out on this one. I knew it was going to be high-risk and a lot of work. If it had major issues or failed, it'd be almost exclusively my fault. If it succeeded, I'd get a pat on the back and a hot cup of coffee (oh, the joys of corporate life). I also knew the potential payoff to the company was huge.

Over the next months we prepared and issued an RFP to multiple suppliers, vetted them out, awarded, and worked closely with the chosen supplier on development and testing. I wrote in another article about the specification problem we uncovered and corrected. Aside from just reducing cost, this was also a chance to make minor improvements, carefully balanced against finishing on schedule. Our products were frequently operated in very cold conditions, beyond what the cell manufacturers specified, so we met with them to review their custom discharge curves and did our own cell evaluations as well to select the best option. We retained the same outer dimensions and interface of the existing battery, but it was a full electrical and mechanical redesign. Achieving the MIL-STD-810F environmental standard (drop, immersion, etc.) was challenging, as always. We were able to increase capacity in the same envelope and also improve performance at cold temperatures compared to the existing battery.

After plenty of challenges and a lot of engineering testing, we were able to release a functionally superior replacement on time for less than half the cost of the original. We had a good relationship and good contractual terms with the supplier, making Purchasing even more happy. And the increased capacity plus improved cold weather cell performance reduced one of our top customer complaints, cold temperature runtime. This cost reduction saved the company millions of dollars across the lifetime of the product, it was a massive win.

Pats on the back all around. Man, that hot cup of coffee tasted good!

Cost Reduction 2: Mac ARM prototype build

One of the challenging aspects of working at Apple is the level of secrecy. For context, I worked for a defense company that made encrypted radios for over a decade. Every time I visited the cryptography area, someone had to punch a red button on the wall, which lit up a siren that told everyone to stop and hide work while the visitor was present. We had white noise generators outside of conference room doors. I had a government security clearance. I was even registered as a courier, and transported secret materials through airports while exempted from TSA search.

With all that, the Apple culture is even more secretive.

I personally never minded not knowing what other groups were working on. I liked being surprised and delighted at a launch, just like our customers. Product silos did make it harder to share lessons, and all the lockdown areas made it much harder to casually interact with colleagues, but those are challenges you learn to overcome. The real bummer was conversation with friends outside Apple became pretty boring. Whereas before at lunch or happy hour we'd swap stories about work, it was now one sided. I couldn't say much without giving information away, and "I'm working hard on some stuff" isn't that thrilling.

Now that it's public, I'd like to talk about something I worked on from the very beginning, the Mac silicon transition over to ARM. Hopefully you understand if I give a bit less detail than usual.

As a brief history recap, way back in their classic Macintoshes of the 1980s, Apple used Motorola 68000 processors, then changed to PowerPC in 1994. At WWDC in 2005, Steve Jobs announced the switch to Intel. The first Intel-based Macs were released the next year in 2006, twelve years after the first PowerPC Macintosh shipped. Intel reigned for the next 14 years until WWDC 2020, when Tim Cook and Johny Srouji announced all Macs would transition to use in-house Apple Silicon, based on the ARM architecture. The first systems based on the Apple M1 processor shipped that November to rave reviews.

In 2018, our Senior Director mysteriously pulled me into a conference room one day and asked me to join a small tiger team working on a Mac silicon transition. It was very early in the process. Coming from iPad, I'd be the only EPM on the team that had released products with Apple silicon. I was meant both to help out directly as well as act as a consultant, educating the Mac personnel on the differences of engineering development with our own silicon. The whole project was kept secret at an even higher level than usual, with codenames for codenames and a main team of only a dozen or so at that time.

It's very cliché to say so, but the whole project was run like a startup within Apple. The team was tiny and physically sectioned off from the rest of the company. Without access to any of our normal support teams, we had to find workarounds to do everything ourselves, which was extremely challenging. I have a lot of fond memories from this period. It was very difficult but great fun, even when we had to constantly re-invent everything.

I was directly responsible for all of the prototype builds, so one of my big challenges was getting enough development units built. Ordinarily, like many companies, we'd build just a few boards locally to prove out the design, then transition to our production facility (in China) to build units through the engineering assembly lines. In this case, the requirement was that everything stay local. We had no existing system for this, and we had to make substantial numbers of units for the development teams to use.

To minimize costs, I kept things as simple as possible and did them in stages:

• Early non-form factor units were built in the lab by just a couple of engineers
• The next small batch of form-factor units were also built in a lab by engineers
• For a larger batch of form-factor units, we set up a miniature production line in a dedicated building at six lab benches, with printed operator instructions per build station and a few test stations cobbled together for final unit verification. We actually used some Directors as operators, because they were among the few who were disclosed on the project at this time! Everyone saw the humor in what we were doing, it was pretty funny.
• For the largest build, we needed to churn out hundreds of units. This wasn't possible without real test fixtures and more space. On top of all the other support tasks we had to locate a new facility, ship sets of test fixtures from China, set up a replica line, and find assembly operators to work full time for weeks building units.

We had to be unusually scrappy all along, but never more so than on the big build. When we first spoke to Facilities about getting a location, they asked for a plan of what we'd be doing, and then quoted us a price in excess of a million dollars. I just about choked. What followed was a series of "let's get real" meetings, where I got that down to essentially zero.

• "Electrical upgrades are needed to support the power draw." We'll space out the fixtures to balance the load among breakers.
• "HVAC upgrades should be done to handle the heat load." This was due to an error in the planning sheet we gave them, there was no way our minimal equipment would generate enough heat in a large open space to matter.
• "More furniture and seating is needed." We'll just borrow some from another unused portion of the building and move it ourselves.
• "Earthquake requirements have to be met for racks." Ok, we'll use short racks to keep them below 4 feet.
• "Locked storage areas cost $20k and have a six week lead time." No problem, I'll order a couple of locking cabinets from Uline for $800 and ask Facilities to assemble them.

In the end, we moved in for nothing and spent just a little bit to temporarily outfit the space for our needs. The build event was completed successfully on time and our team got to participate in a really cool part of Apple history.

Cost Reduction in general is part of being a PM

I do want to highlight that keeping costs down in general is a regular activity for any PM. Monitoring budget, tracking schedule, reviewing capital expenditures for equipment, reducing factory cycle time, improving repairability, and negotiating the number of units per engineering build are all normal management tasks focused on saving money, with some differences depending on the company or product.

• At Harris, military product volumes were low, so build count and units per build was already strictly controlled, plus or minus just a few. Engineering hours were the main focus for cost control. We individually reviewed project hours spent per staffed employee weekly, overall budget and schedule with Finance monthly, and budget and schedule again with Senior Management quarterly.
• At Synapse, hours spent, and scope creep affecting that, were the main cost areas. We were the design partners for very small startups that built only single-digit numbers of early units, as well as large consumer electronics companies that intended to ship millions. For both customer types we put a lot of focus on an early non-form factor unit, a collection of boards with plenty of test points, to prove out functionality before shrinking into the final design. We gave guidance on factory management, since most of our larger customers were using ODMs in Asia. Like Harris, our work hours were tracked against budget from the weekly timecards each employee was filing. Unlike Harris, we were consultants billing our hourly work to the client, so keeping our hours within budget was a joint management task between us.
• Apple was the first company I worked at that didn't require weekly timecards. Also unlike the others, we didn't get a say in which engineers were staffed to our programs, or visibility to how much time they spent working on them, it's a total unknown. Production volumes are so large, there are entire teams dedicated to reducing costs after release (components, test, factory line operations, etc.). Our primary levers for controlling cost during the engineering phases were the number of builds and the number of units per build, which were orders of magnitude higher than other companies I'd worked at.

Side note: Stop laser targeting travel, it doesn't work

Speaking of overall program costs, I'm going to take a moment to rant about business travel as a cost reduction. It's a huge pet peeve of mine.

At every company I worked for (less so at Synapse), business travel got an overwhelming amount of scrutiny and constant second-guessing from outside parties. Why? Not because it was a top expense in and of itself, but because it's highly visible and segregated. It's one of the few areas Corporate Travel/Finance is empowered to attack without directly meddling in a program and angering a bunch of VPs and Directors. It's political, usually undertaken as a mandate from the C-suite, and the results it achieves are based on perception versus reality.

Every time cost cutting came up, reducing business travel spend was the first rallying cry. PM's calendars filled up with meetings to justify every person and every trip. Supervisors started pressuring cutbacks and remote design reviews. Travel policies got written that literally didn't make sense, by employees who didn't travel.

Am I talking about crazy extravagance and $5,000 bar tabs? No, not at all. There will always be a few bad actors, go after them directly instead of writing restrictive policies that punish everyone. I'm talking about being forced to take redeyes and work exhausted because the company won't pay $100 for an extra hotel day. Having to shell out cash at the dinner table, because the company per diem doesn't even cover a normal meal. Getting called by Finance because they scrutinized your $14 receipt and are disputing the Dramamine you bought for the flight.

Here's a radical statement: Business travel, as a percentage of program expenditure, is negligible.

On over a decade of my programs, across multiple companies, business travel was in the low single digits as a percentage of the overall program budget. That's it.

So let's say it was at five percent, aggregated across the company for all programs. The new policies come along, and it gets squeezed down to 3.5%. That 1.5% company wide gets translated to a number and presented to senior management as a catch-free savings, just a reduction in inefficiency. What a great job!

It wasn't catch-free, though. It's inevitably harder to solve problems remotely, particularly for hardware products, increasing program hours spent (which directly relates to cost, but is now obscured in a different number). It impacts the team relationship by reducing face to face interactions. And clamping down on spend during travel pisses off every single employee who is already sacrificing time away from friends and family for business, now to be harassed on top of it. It causes tons of harm to morale.

Moreover, it's a drop in the bucket compared to a meaningful cost savings. Engineering units are very expensive to build during development. On larger programs, reducing the build count by 10% can essentially translate into a 10% program cost reduction. Ten percent. Without alienating every single employee who is now resentful that their sacrifices aren't appreciated.

So if 10% is possible, versus 1.5% (which isn't real, since it's just shifted elsewhere) and making people so mad they often quit the company or leave a program, why isn't it as common?

It's hard, and it's political in a different way.

Finance can't do it themselves; they can't mandate how Engineering runs their program. It requires a broad commitment from the development teams to work together. The primary PM has to meet with each team and negotiate how to achieve their goals with fewer units. Teams that request large numbers, such as Reliability, will have a harder time sequencing their tests. It adds work and complexity, but the savings are well worth it. I did this throughout my tenure at Apple, and it was worth spending the time getting to know each team and how they run their testing so I could provide a fair number of units per build without overproducing.

Let employees enjoy a business class flight across an ocean, or a good hotel, or a nice meal after a long day. It's a small compensation for the sacrifices they're making on the road, and has almost no impact on the bottom line. Make the hard choices and collaborate on those larger savings elsewhere.

Conclusion

Managing cost is just an everyday part of life for any PM, but these situations are some of my favorites that bring back fond memories. I hope these stories are interesting if new to you, and spark happy memories for others that lived through them.