Why Telecom Billing Support Systems (BSS) are not more plug and play?

The dream

The dream probably started in mid-90s and it goes like this: “BSS systems could be built just as easily as Lego sets with bits and pieces provided by best-of-the-breed providers. The switching from legacy systems would be easier due to the common consistent framework to be developed under TM Forum Information Framework (SID) and to be widely adapted by all the players.”

That dream was so tempting to so many:

• Service providers who wanted to move away from large and cumbersome systems with possible vendor lock in.
• New players who would develop not the entire OSS/BSS (Operational / Billing Support System) solutions, but smaller, more digestible pieces.
• Army of consulting companies who were eager to help with integration of those Lego pieces into a shiny flexible, reliable yet inexpensive OSS/BSS marvel.

By 2000 huge amount of money was spent, many new companies started, a lot of conferences and good discussions occurred but it was miles away from making any serious impact on anything when 2000 bubble exploded and most of the companies working on it went out of business.

The strongest survived, the largest one picked bits and pieces for cheap and got even bigger. The movement continued but at much lower pace.

Now, almost 25 years later, Information Framework (SID) model become much more comprehensive; many companies claim some level of compliance with it (to look like good citizens and to reap benefits of easier integration in some areas), but, IMHO, it becomes similar to an Esperanto language – comprehensive and universal, yet not used too much.

Why?

The rest of the story would be my attempt to explain WHY:

• Why today’s large Service Providers are still using their legacy systems (internal or from vendors)?
• Why post-merger system integrations take way longer than originally expected and sometime just abandoned? 
• Why vendors who acquire one competitor after another are not too fast to consolidate, but instead often retain acquired solutions as separate product lines?
• Why vendors who provide full OSS and/or BSS support for a service provider (for some or all of the product lines, like wireless, long distance, data, regulated, etc.) often deploy large (usually offshore) support teams dedicated to that service provider for what (in theory) supposed to be out-of-the box solution?

The short answer is very simple: there is a lot of complexity there and that complexity often gets underestimated.

Does BSS have to be so complex?

In the ideal world telecommunication billing should be easy and fairly universal because all major building blocks are largely the same and look simple at the first glance:

• Data is coming from standard voice and data switches.
• There are customers who are associated with phone numbers, IP addresses, etc.
• There are products that are sold to customers. Those products have features and charges associated with those features.
• There are some discounts / promotions / special pricings (depending on the terminology).
• All of that finally appears on the invoice along with the right tax information and there is some reporting / BI (Business Intelligence) layer to support financials, marketing, sales and security / fraud mitigations.

But there is a lot of under-the-hood complexity with each of those steps. The levels of complexity accumulate over the time. It is often hard to retire the old complexity. In addition, the new technologies, products and customer expectations add more complexity along the way.

Here are few examples of that complexity:

• Customer structure and customer identification complexity: There are multiple ways to define and parse the customer hierarchy and to tie the large customer who has multiple subdivisions that purchase multiple products under multiple contracts to all instances of the phone numbers, dedicated lines, IP addresses and other means that customer uses to consume the services. Customer identification also gets complicated by external events such as line disconnects, NPA (area codes) splits, etc.
• Contract structure complexity: Contract pricing and terms could override regular product pricing in multiple ways. For commercial contracts it is a result of tough negotiations. For government contracts it could be a CLIN (Contract Line Item Number) based product and pricing structure that have to be adopted. Contract expiration and potential penalties for breaking the contract commitments need to be addressed. Many contracts last for ten or more years which complicates the retiring of what otherwise could be an obsolete functionality. Integration of contract terms with sale support systems is required to meet customers’ expectations.
• Rating and Discounting complexity: This could be as broad as combined imagination of the marketing team over the last 20 years. That also includes marketing teams of the competitors, as many companies try to provide better value based on the comparable to competition rating structure. The number of dimensions that are used to effect rates or discounts are almost unlimited. Some are harder to compute than others. Here are few examples of interesting challenges: diverse rounding of both billable quantities and resulting price, including “three for dollar” kind of rounding across specific sets of products and services for a given customer; multiple layers of volume (total spend or total quantity or total number of events) discounts that are calculated against specific set of customers, products and services and then applied to potentially different set of customer, products and services; rate guarantee algorithms where you could be switched to the newer rating plans yet with your price never exceeding the one you had when you sign up for the first time; dealing with promotional pricing that you can get for limited duration but where the start date is triggered by different events such as when you asked for it, or when you made your first call or on your birthday.  
• Cross-system integration complexity: This includes integration between mediation, fraud, rating, discounting, taxing, invoicing, ordering, quoting, CRM, reporting systems. Sometimes, if there are multiple billing systems are used, there is also integration across those systems to deal with situations, for example, where services handled by billing system A contribute to discounts on services handled by billing system B.
• Implementation / Technological / SLA (service level agreement) complexity: Some systems are designed to be simpler but rely on a large staff of billing analysts to enter the products and customer specific pricing data. That is prone to data errors and can result in very large data sets. Some systems are more generic and allow just few billing analysts to define all the product pricing. But the data model for those systems is much more complex. SLAs drive additional complexity in terms of code implementation. Technical debt could result in less reuse and business logic implementations that might be hard to comprehend, test and extend. Complying with regulatory requirements could add another layer of complexity. Resources who understand the system might no longer be with the company and the additional complexity could be added due to the lack of full understanding of the system architecture. The list could go on.

Why is it so hard to consolidate two Billing Support Systems into one?

If we agree on the complexity of the BSS, it is easier to see why integration and consolidation is hard. There are largely three things in play here:

1. Business requirements addressed by different BSS systems (even when well documented) are often different, so there are gaps to fill. That might require significant amount of new development or customization. Often those requirements are not easily available but instead are spread among partially obsolete documentation, database set up and millions of lines of software code where some of that code was not accessed for years and the last person who touched it retired and moved to Florida  five years ago.
2. In addition, for each complex software problem, 25 great software architects could come up with 25 different ways to solve it. Therefore, even though at high level systems might be doing similar things, the mapping from system A to system B could be a complete nightmare.
3. Finally, the interfaces with systems that might not be a subject to a consolidation would have to be revised that might be fairly complex.

And even if you are up to all the challenges above, you have to assure that you can test it well so there are no surprises to the customers.

That is why after multiple mergers, the number of BSS systems does not go down nearly as fast as everybody would hope.

Even vendors, who would have a strong motivation to consolidate the systems they acquire to improve their profits, often continue to maintain acquired product lines and have an army of support engineers (often offshore) for each large service provider.

It is just too messy and too complex to both integrate the functionality and convert legacy customers to a new system. For example, Kenan Systems started in 1982 and developed one of the competitive BSS solutions among the several that were available on the market in the 90s. It was acquired by Lucent in 1999. The timing was not ideal, so it was sold at loss to CSG Systems International in 2001. Comverse (another prominent BSS player who had their own BSS solutions) bought division that had Kenan Systems in 2005/2006. Comverse was acquired by AMDOCS in 2015. And Kenan Systems are still there, more than 20 years later, along with many other competitive but complex BSS solutions (both commercial and in-house).

Could it at least live on the Cloud?

Why not? This is a different challenge driven by practicality. Is it more efficient (from cost, performance, reliability, security and scale perspective) to replace your data centers with the cloud? If answer is “Yes” then cloud infrastructure these days is mature enough to handle the load.

It is still sizable effort to wrap everything up in the right way, but the results are more predictable there if system functionality and architecture largely stay the same.

Are there elements of BSS solutions that could be consolidated?

If process, or group of processes, have well defined input and necessary reference data is readily available, the common building blocks could potentially be used across multiple BSS systems.

It would still require a due diligence to assure a return on the investment where you compare maintenance cost of the legacy system with the conversion cost and the maintenance cost of the incremental complexity. And you want to factor in your strategic roadmap for the applications in terms of new product support, technology, etc.

For example, single taxing engine could serve multiple BSS implementations. Below are the high-level steps to inject “new and shiny” taxing engine into your legacy BSS system:

1. Discover both functional (all covered taxes and mandatory fees, level of taxes – if some of them are aggregated across multiple events, additional summarization data might be needed) and non-functional (performance) requirements.
2. Convert / enrich your input data to match taxing engine requirement (Extract-Transform-Load - ETL layer). That could be an additional service that prepares the data or a process that converts the file.
3. Address functional and performance challenges (if any).
4. Bring output tax data back to BSS system (another ETL layer).
5. Test it well and deploy.

However, many BSS components such as mediation and rating are often very tightly coupled with other systems and internal data models (such as customer and product structures). That would make system consolidation significantly more complex. 

Why consolidation efforts are often underestimated?

First let’s say that there are well executed consolidations. And some consolidations are just easier than others. Often enough however it takes several years more than expected to consolidate large systems and sometimes it just never happens.

The key reason for that is not enough of up-front due diligence and therefore simplified assumptions about processes, data structures, data volume, availability of the business rules, availability of right resources, abilities to adjust the products, fundamental compatibilities of the sub-systems.

The root causes for that luck of due diligence could vary and include any combination of the following:

• Not enough time is allocated for the analysis
• Challenges of assigning and keeping right people on the project
• Pressure to hit the imposed date that is not tied to the realities of the project
• Comparing this effort to prior successful projects that had a different level of complexity

Those root causes are usually unintentional and sometimes even subconscious. For example, people making the decisions often have a lot of great accomplishments behind them. Thought their careers they were not necessarily involved in all the intricacies of the systems they manage at the moment and it is natural for humans to project the past experiences to the current problems. The past accomplishments could lead to over confidence and therefore more optimistic estimates.

The more collaborative culture at the company could help to overcome it by engaging all levels of the contributors into the decision-making process and allocating enough time up front to understand potential problem in more details.

Also, system consolidation efforts are often triggered by mergers. And after the merger there is a tremendous pressure from the owners / shareholders to execute on synergies. That might trigger a little bit of wishful thinking. Again, allocating enough time and involving the right people to do the due diligence would help to find your optimum solution given all the constraints.

Where to go from there?

Well, the reality is that telecom BSS systems are not plug and play. Some could be consolidated and some just could not (with any reasonable effort). It might be possible to move from system A to system B, but not the other way around. It might be possible to move both systems A and B to system C. And it might be more practical just to keep the multiple systems (as annoying as it is). The goal, typically, is to maximize shareholders value both short and long term and that is what should, at least in theory, drive the decision-making process.

It is almost always warranted to research the ways to minimize the number of systems that do the same thing.

The first step / phase / project there is to understand the complexity because understanding the complexity is the first step in solving it.

To be successful there is not different from any other successful execution:

• You assemble the right team that has both technology and subject matter experts, that have executive support, that embraces both collaboration and independent thinking.
• You methodically execute on the project identifying and drilling into all complexity aspects:  functional (for example, business rules), non-functional (for example, service level agreements) and organizational (for example, resource availability).

Once it is reasonably clear what would it take to consolidate the systems the strategic vision for the company could drive the next steps.

And that is the story.