What I learned packing boxes at Metaswitch, and how it could save your VoIP network

In 2012 I was plucked from my role as Director of Metaswitch Support working from home in Ohio, and sent to our Operations warehouse in California. What I learned there totally changed my thinking about business, and these same principles could be crucial to surviving in the current telecommunications market.

After graduating from college I wanted to be a computer programmer, and so I got a job at a software company named Data Connection (which later became Metaswitch), but after a few years writing software it became apparent that I wasn't particularly good at it - at least not compared to those around me. I mean, I was decent, solid, competent, but I wasn't exceptional. So when I got the opportunity to work as a Customer Support Engineer (a CSE) and move from England to the USA, I figured I should give it a try.

This proved to be an excellent choice. Firstly, and most importantly, my British accent was a key asset in capturing the heart of my wife, who would have been out of my league otherwise. And from a career point of view, it turned out that I was well organized, good at communicating, and enjoyed helping my assigned customers with their problems. Suddenly I'd found my niche, and over the next few years I was promoted a couple of times until I was running a team of about 12 people responsible for installations, commissioning and PSTN migrations for all new customers. Life was good. I was working from my home office in Cleveland, OH, and our first daughter was learning to walk.

But I'm always keen to learn new things, and so when I was asked if I fancied moving my family to San Francisco to take up a role in Metaswitch's Operations team, with a particular focus on improving shipping quality, I was willing to give it a shot. It would be a big change to our family life and a big change in my role at work, but change means learning new things, and that's good, right?

I didn't really know what to expect when I showed up for that first day in San Francisco. I knew my new boss Jim, and maybe two other people, but otherwise the team (and more so, the role) were brand new to me. I knew all about class 5 switching and the associated technology, but the Ops team had a whole other set of terminology and way of working that didn't match anything I'd seen before.

For some reason the team was really proud of their work benches - everyone's work station was set up in exactly the same way, with the same set of equipment positioned in the same places, all neatly labeled - and I later heard that the layout the whole work area had been carefully designed to take into account who needed access to what equipment and when, to limit the number of steps each worker needed to take.

And then Jim told me I needed to learn how to pack boxes in the warehouse. I reluctantly agreed, but on the inside I was pretty skeptical. I'm not a very "handy" guy. Give me a text book or a computer and I can dazzle with my smarts; but ask me to fix a leaky faucet, and you can bet there's going to be water all over the floor. And anyway, I was a manager, wasn't I? I thought my job was to ask other people to do all the actual work?

Thus was I introduced to the idea of Lean Manufacturing. Over the next few weeks I packed quite a few boxes, read several books, admired some pretty graphs, and began to view the world in a different way.

But before we get into all of that, I'd like to step back and take a look at the current state of the telecommunications landscape.

As I see it, we live at a critical time in the world of telecommunications, and the next few years will see some big winners and many losers as the landscape changes. 

There are two big trends that are colliding to create this environment.

Trend #1: It's getting harder to run a high quality network

• Many moons ago, each call was manually connected by an operator at a switchboard, by physically moving a cable to connect the call. This was pretty labor intensive, but at least it was simple. Since then we've seen many advances in our technology, and while the overall impact has been very positive, each advance has made the network more complicated and harder to run smoothly.Line concentration (e.g. GR-303) allowed multiple lines to share a smaller set of trunks, thus reducing the capacity needed between the switch and the service area. But it also introduced the need for separate signaling and media channels on the subscriber side of the network.
• On the network side, when SS7 replaced CAS (MF) as the primary PSTN trunking technology it increased the services on the network (e.g. call forwarding, voicemail, caller ID services), but it also introduced the need for separate SS7 signaling links to STPs and a whole bunch of new SS7 routing logic - again adding complexity.
• Adding Voice-over-IP into the mix on the PSTN side of the network promised great cost savings over traditional long distance carrier trunks, and as VoIP has become the dominant technology for connections to CPE and ultimately handsets (or soft-clients) service providers can retire their old copper-based network entirely, and run everything over IP. And again, that's great - but there's a lot of complexity involved in running a high quality voice service over an IP network, particularly if you don't own it, and particularly when you're dependent on "smart" handsets from third-party providers.

All of which is to say: new technology is great, but it makes it harder for the service provider to run a high quality network.

Trend #2: The market is becoming increasingly competitive

Thirty years ago an IOC or ILEC was basically defined by geographic service boundaries. These boundaries limited the opportunity for the business to grow but also made it hard for any competitors to significantly harm their business. 

This all began to fall apart with the introduction of competitive local exchange carriers (CLECs) in the 1990s, but it is VoIP that has truly opened up the floodgates.

Today, I could use my broadband internet connection to access phone service from one of hundreds of service providers nationwide, not to mention all the free over-the-top software tools that include voice calling in their product. (All assuming of course, that the new FCC chairman doesn't put an end to net-neutrality.)

The geographic boundaries no longer exist, and in fact, with so many options, and with relatively cost in moving to a new service provider, service providers really struggle to retain their customers. "Customer churn remains a costly and pervasive issue" (according to this Broadsoft survey) and it's easy to understand why.

In fact, given the difficulty in running a high quality service, the pervasive competition, and the ease of switching providers, it's no wonder that Vonage are reporting annual churn of 29% on consumers and 18% for business customers. On the consumer side, that means that each year they need to recruit enough new customers to replace 29% of their entire customer base, just to get back to the starting line. No wonder that their revenue has basically been flat for the past 8 years, and they currently have a profit margin of about 2%.

But this isn't about Vonage - I've picked on them just because they're a public company so the data is easily available. My point is that all service providers are struggling with the same issues. If you have 1000 customers and you need to find 290 new ones every year just to maintain your current size, then that would be a problem for anyone. Even if you have a fantastic sales and marketing team that can reliably recruit 400 new customers every year, then guess what? Your maximum customer base is 1379 - once you reach that point, growth will stop.

But there's some good news here too. In this marketplace it's easy for users to switch, which means that if you genuinely provide better value (a combination of service quality, features and price) to your customers then ultimately you should triumph over the competition. So imagine for a moment that your product is so good that only 10% of your customer base churns each year - and you can recruit 400 new customers annually - how far can you grow? Answer: 4000 customers. And if you can reduce churn to 5% you can grow to 8000 customers. 

So if you can significantly reduce churn then that will dramatically increase the value of your marketing and sales efforts allowing you to build a much bigger business.

Could "low-churn" be your competitive advantage?

Of course, that's not very sexy, so how about we flip it around? What prevents customers from churning? A high quality service, great support, market-leading features, and a good price. In other words, your product delivers great value to your customers. If you can find the secret to delivering fantastic value to your customers then they are much less likely to churn, and that's how you could become one of the winners in today's competitive market.

Back to the warehouse

?Funnily enough, it turns out that this idea of Lean Manufacturing (what I was learning about when I joined Operations) is ALL about how to deliver value to your customers - as much value as possible, while eliminating waste.

This could be the key ingredient in succeeding in today's marketplace, but it's not easy. It requires a new way of thinking.

Before I started this journey, I prided myself on rapid decision making - I was busy, and so I thought one of my key skills was to make a quick decision and move on to the next thing. But what I learned about Lean is that sometimes you need to take the time to invest in quality for the long haul. Sometimes a significant investment of time and attention now can solve a problem that reaps benefits for years to come.

But what is Lean again?

The concepts of Lean all started at Toyota, where they had an obsession with delivering quality products and reducing waste at all steps of the process, while also empowering the individual workers in their plants to take responsibility for their areas of expertise - by seeking to continuously improve.

OBSESSION #1: QUALITY

To give just one example of the quality obsession, each worker in a Toyota production line has a button they push whenever they encounter some kind of quality problem with the parts they're working on, and if the problem is not quickly resolved they can even use this button to stop the entire production line while the problem is investigated. Toyota are so obsessed by quality that they'll stop production entirely rather than continue with an unresolved quality issue.

?OBSESSION #2: WASTE

The second plank in the Lean methodology is that we must not be wasteful in our pursuit of quality. If you go to great efforts to produce a high quality product, and as a result the price sky-rockets, then that doesn't deliver value to your customer. The customer wants a high quality product at a certain price point, and any wasted effort or materials or time or money all contribute to higher prices for the customer and lower margins for the business.

Taiichi Ohno (the father of the Toyota Production System) originally identified seven types of waste, and Lean experts seek to identify and eliminate waste from business processes. As a very simple example, unnecessary movement (motion) is a form of waste, so workers should always have all the tools and materials they need within easy reach (hence the carefully designed and labeled workstations in my Ops warehouse). Another form of waste commonly seen in an office environment is producing, processing and storing unneeded information (a form of inventory waste).

One interesting corollary of this obsession with waste, is that in order to eliminate waste, it must be easy to see - so a lot of the ideas in Lean are partly related to exposing waste, whether that's through keeping really tight inventories, or visual management techniques. (For more on this, read the parable of the Lake and the Rocks).

OBSESSION #3: PEOPLE (AND CONTINUOUS IMPROVEMENT)

Henry Ford is famous for introducing mass production into the US auto industry, but there's a major philosophical difference between how Ford and Toyota viewed their workers. To Ford, the beauty of the production line was that workers were basically interchangeable, whereas at Toyota the workers were each viewed as experts in the piece of work that they did, and the workers were given responsibility for identify waste in the process and for testing and implementing improvements in both the process and their work environment. 

As a result, Toyota created a culture where everyone was engaged in their jobs, and everyone was focused on continuously improving their part of the process. And ultimately it's this culture of engagement and continuous improvement that creates a sustainable competitive advantage over the long haul.

But isn't this for manufacturing only?

?Since the Lean concepts began at Toyota and have most famously been applied within manufacturing environments, many people are skeptical that these concepts work well in other types of businesses - and certainly some of the tools and methods are focused specifically on a factory environment where people are performing manual work in a production line.

However, many of the principles of Lean can be applied to other types of businesses, and if you look back at the three core obsessions listed above (quality, waste, and people) these ideas are not specifically about manufacturing.

There are many examples of Lean principles being applied in other types of environments, but I'll share just three here.

Grand Rapids, MI

The City of Grand Rapids, MI was facing significant budget pressures and implemented a Lean program in an effort to continue to provide great services to the public despite budget cuts. They improved a variety of services across their departments, but one of the simplest to understand was the way the public library handled book (and media) returns.

Before the changes, each returned item would typically be handled by 9-12 different people and would take days or even weeks to get back on the shelves after being returned. After a team analysis session walking through the end-to-end process, the public library was able to reduce the average lead-time on returns from 126 hours to 3.5 hours. As a result, the most popular items returned to circulation much more quickly which allowed the library to serve the public much better and to save money by holding a smaller stock of popular books and media.

Read more here.

The Dentist's Office

It's hard to imagine anything further removed from a factory than a visit to the dentist, but Dr. Sami Bahri of Jacksonville, FL nevertheless applied many of the seemingly manufacturing focused aspects of Lean (such as removing batch processing, analyzing takt-time) along with an analysis of all the steps a patient goes through during their treatment and the value they receive at each step to implement significant improvements in his dental practice.

The end result was an 82% improvement in capacity/efficiency for the dentists (with the same staffing levels) AND an 81% reduction in overall treatment time for patients. You can read more here but this is a great example of significant improvements in both quality (value to the customer) while simultaneously removing waste from the process.

Silicon Valley Startups

Over the past 5 years there's been a significant movement among tech startups, where new companies follow the principles described in Eric Ries' book The Lean Startup. Many successful businesses have used aspects of this approach, including DropBox, Groupon, Intuit and Wealthfront.

Interestingly, the Lean Startup methodology actually focuses on just one part of Lean (the Build, Measure, Learn cycle, which is very similar to the Plan, Do, Check, Adjust cycle in Lean), and applies this form of iterative learning to the key question in a startup: What do users want?

In more mature companies the product is usually fairly well understood, and so Lean tends to focus more on delivering that product with higher quality and lower cost, but even in a startup environment you can use continuous improvement to figure out what customers want through an ongoing series of short experiments.

Telecommunications?

Which brings us to our key question: Can we apply Lean principles to a voice service provider and see the same kind of results that have been demonstrated in other industries?

That's the question I'm seeking to answer in my business, and I'm looking for a handful of service providers to join me in this journey - as we seek to build The Lean Network - please contact me if you'd like to be considered.

I believe there's every reason to be confident. When I was running the Metaswitch Operations team we certainly weren't perfect at implementing the Lean methodology, but even in our imperfection we got some great results. If you recall, the primary reason I was asked to move into Ops in the first place was to improve shipping quality. We hit this problem hard through improvements to our shipping process, our information flows, our tools, and the way we investigated any problems.

So how did we do?

There was no magic bullet, but over 3 years, we reduced the number of errors in our shipments by 80%.

Not to mention that we were simultaneously making big improvements in efficiency and inventory management. But let's ignore that for now. Let's just focus on that one stat.

What would happen if you could reduce the number of trouble tickets in your business by 80%? What would that do to your customer satisfaction? What new features could you roll out thanks to the extra time available to your team? What impact could those new features have on sales?

What's Next?

In the next article I'll take a look at one key tool we used to improve our shipping processes, and how you can do the same for your business.

This article originally appeared on AWardConsulting.net. If you found this interesting please comment or share this article on LinkedIn, or contact me directly.