How to measure real CX -Customer Experience- across and in wireless technologies? Part II

Disclaimer: The opinions and ideas expressed in this article are my own and do not necessarily represent those of my employer.

In Part II I am trying to answer some of the questions that I posed at the end of Part I of this article.

What is the best way to measure customer experience across multiple wireless technologies such as Wi-Fi, CBRS, LTE, and 5G whether to compare between them or to score an overall user experience?

Google's Site Reliability Engineering book released in 2016, the authors mentioned the factors of the site reliability, laying the foundations for a clear definition of Cx. I will be using that concept to my approach by prescribing the four golden parameters that are used by Google as the cornerstones of our Cx, which also made me think that we can use the same concept to measure any network performance wired or wireless from the customer point of view, these cornerstones are latency, traffic, errors, and saturation.

I will be starting by implementing this concept to measure Cx for the Wi-Fi network then it can be expanded to other networks (LTE and CBRS). so I will be focusing on the customer touchpoints along any wireless network lifecycle

1. Network discovery and availability: Assuming the service is present and available, network discovery depends heavily on the protocols defined in the UE (User Equipment). Service providers use design tools standards calibrated to work with various range of user equipment, such as the iBwave. The robust network design will include best practices of UE hardware and protocols standards as used by Apple iPhones and Samsung Notes at the network end.

2. Connection: Wi-Fi alliance or WFA designed open and secure network mechanisms to connect to Wi-Fi networks. At the same time, most commercial networks use an open network with a captive portal authentication. A transformation is taking place in wireless network implementations with a shift towards Passpoint 2.0 (or sometimes called Hotspot 2.0) as a secure and reliable method. The WFA introduced the WPA3-Enterprise security protocol in June 2018, which offers the equivalent of a 192-bit cryptographic strength, utilizing Simultaneous Authentication of Equals (SAE) handshake, as defined in the IEEE802.11s standard.

3. Onboarding: Onboarding users begin with a probe request to establish L2 connectivity with the network and end with the first data packet received from the internet. This passes through simple to complicated nodes, and of course the business captive portal if exists.

4. Saturation: Several factors affect Wi-Fi service saturation. The conditions of the RF, different HW components specs and capabilities, and the servers that provide the service. A high interference from an RF source (like SNR: Signal – to – Noise ratio and CCI: CO-Channel interference) can negatively impact the modulation schemes and consequently affect the quality of the physical layer. The Access point / Switch /Gateway condition affects the transport layer and consequently the service, for example, firmware bugs, CPU and memory utilization of the Access point. And the precise AAA / DNS / DHCP resource planning can positively or negatively impact the service delays. network backbone delays are caused by the bandwidth problems. Also, at the service provider side, wiring and cabling issues can have a tremendous impact, and finally the status of the destination application hosts that provides the application such as a website or email client. Any of these components can impact the saturation and cause a poor user experience.

The challenge we are addressing in this article is to define the best indicator that can describe all of the challenges at different components and stages. For that what we need is an agent that can time, count, measure, collect and send us this information, taking into consideration various User Equipment (UE) environments used by the user such as computers with different Operating systems, mobile phones, tablets, and even IoT devices.

A software agent is built and installed on the User Equipment or UEs (such as Windows OS, Linux OS, macOS, Android, and iOS) or a code injected in a specific free or commercial APP to collect a set of measurements considered as our “Probe”, a probe may also include specially built IoT devices running a light version of Linux OS.

The probe will be collecting measurements from our wireless networks based on the four golden rules of the site reliability, these values can be anything we determine it is important to our user for, the next table will be a data model for what a probe could collect from every session:

What is defined as the measurement in the table above is a mere example! Additional or different counters, timers or metrics can be defined by the customer experience teams for specific applications or protocols such as Adaptive video bitrate, RTP and Concealment metrics for Voice. The flexibility of the formula allows the addition of any required measurement. Please notice that these measurements should be geographically and time-stamped.

The next step is to define the appropriate satisfactory values for every measurement collected by the probe, this process is a very flexible process, for example, a latency of 50ms might be acceptable for a user watching a movie but not to a user using a VOIP application such as Skype, WhatsApp or IMO.

Once that process is done, we define an “artificial weight” for each of the measurements we are collecting from the probe, this weight will rank the values based on importance to our calculation. To explain this further I will use the following example.

In Wi-Fi networks, We prefer a 5GHz band over the 2.4GHz band. The 5GHz band has more non-overlapping channels compared to the 2.4GHz band, and it is known for a fact that due to power, SNR and Spectrum the 5GHz band has fewer interference problems than the 2.4GHz.

Now that we have our measurements collected and ready, we need to process these values to get our metric point for every session, this is a relatively easy process by implementing common statistical principles we can identify a score for every session the UE is generating.

Data collected from the probes can also generate a heatmap representing Cx, such heat maps can represent either a service we are measuring a detailed or the overall user experience:

This is the first step to build an idea about a user experiencing the Wi-Fi network. Assuming we were able to run this data model for Wi-Fi, CBRS, LTE and 5GNR network we can get an overall ratio for each technology and get an idea of how our customers think of our network.

The method of weighting and rating measured values is a proven way to measure Cx, quantify and justify network resources allocated to each user and build a practical customer happiness index based on the real-life network deployment in a cost-effective method that serves the business goals. Data accuracy and the calculation is significant in understanding Cx, a service that is invaluable in determining if the offered service is desirable or not!