The Disappearing Network and Work From Anywhere Support

By Ariel Pisetzky & Nuriel Alzam

You are alone, who will hear your call for support?

Companies quickly adapted and moved to a work from home model.? For most, this was a new way of thinking and for others this was an expansion of an existing arrangement. So, what’s new? Why share this blog post now? What makes working from home harder and more challenging than working from the office? For your corporate IT team, the list is long and includes the shutdown of IT logistics, the main location of support, internal client connectivity and protection to name a few. For many companies, the office is also a mini-hub for IT logistics. Something doesn’t work?? No problem, come in, meet an IT expert and we will help you out. Need an upgrade? A cable? Anything really, come in, we will set you up.

Not any more. The center of IT service isn’t there anymore.

Location, Location, Location (and your ability to work from home)

The users are home, the help they need (in many cases) is also at home. While some geographies are returning to the office, at least on some level, the office life as we knew it has changed dramatically. We now have to bring the logistics to the employees. Delivery services provide an alternative for the need to make it to the office, albeit at a cost. Yet the most significant change is the barrier of reaching personal support. Many larger support organizations have remote desktop support or call centers. For many smaller IT shops, this isn't the case. You can't just drop by when you have an issue, you can't just ask someone at an adjacent desk to open a ticket for you.

On top of all that, you are home, which could be in the middle of the ocean. Your home IT infrastructure might not meet the minimum requirements needed to operate effectively and securely. Anything from home wifi coverage, ISP speeds, ISP performance issues and VPN connectivity problems (for the few resources that still require VPN access) can be deficient. Anything and everything has just become a bit more difficult. There is no corporate level support for home networks and now we have exponentially more users on the home network that were not there just a few months ago (kids remote schooling, significant other Zoom sessions or roommates remote working to name a few).

From Corporate IT to Home IT

From the perspective of the Corporate IT department in just a matter of days, the single corporate network became hundreds of home networks. Think about it for a moment, from a single network that is under close IT control and support, there are now hundreds (and in some cases thousands) of home networks that are now a de facto part of the Corporate IT realm (whether you like it or not.) Every home network issue becomes a productivity issue. Multiple users with the same problem might become a shared ISP problem or an extremely chatty application consuming too much bandwidth or causing a DNS issue and the list goes on. Because of this shift to home offices, there are so many new problems to consider and address.

Improving the user experience with data

So we decided to put on our thinking caps and take a crack at this. How can we improve our user experience? How can we bring some order to this newly founded chaotic environment? Well, we are the IT team so we first looked for data, statistics, metrics and observability to help solve our problems. Being data driven is what helps us learn and improve daily in Taboola.

We looked for a way to bring in data from every laptop and device connected to our applications and systems. With data we can try to find patterns and help support the users in a meaningful way.? Support that would bring value to the users and create a better user experience. Something that is part of this new “work from anywhere” age and beyond. Collecting metrics could help us find patterns to proactively identify and resolve both individual user issues along with regional and/or home network related problems.?

Where to dig for data?

In reviewing months of “work from home”support tickets, we found out which statistics would be the most useful:

• OS statistics (CPU, memory, disk capacity, running apps, versions, uptime)
• Network data (connection type, signal strength, DNS and ISP)
• Network statistics (packet loss and RTT to multiple targets - over VPN, fixed points and 2nd hop)

This data could help us provide users with quick answers to many of the questions regarding the state of their connection and potential problems that they may be facing.? From a weak wifi signal to a saturated home Internet uplink.

We found our “Why”

The ability to help direct our employees and improve their work experience is part of what we do in IT. Our goal isn’t to manage corporate networks, our goal is to provide business enablement, empowering our employees to do their work. Simply put, this is our “why”. This is why we come to work and this is what the business needs from us. Enabling the team to focus on what can truly improve day-to-day work and productivity of the organization was a project everyone enjoyed working on.?

We decided to build our own “Home-IT” proactive monitoring system. This needed to be a very light and agile solution that would provide our IT team and employees with clear visibility of their working environment.?

Project Spec

The solution we set out to build had to solve our unique working environment. On top of that, we were on a mission to use what we already have so we wouldn't have to deploy new tools for this new work environment. To that end, we defined the following:

• The solution should support both Windows and Mac workstations.
• No new agents should be added to the OS.
• No installations of any tools into the working environment.
• VPN should not be a factor when collecting the data.
• Minimal impact to user devices.

To our benefit, we did have the advantage of a managed OS environment. All our workstations are centrally managed and have some agents on them already, so anything we did had the ability to use our central management system.

Solving the data collection challenge

Following a brainstorming session of all the limitations and restrictions we self imposed; we found that creating a local script using only the existing OS tools could actually work. The data we are looking for is readily available with command line tools, which is the “easy part”. Checking for connectivity, CPU levels, OS version, or IP information, no problem! Checking the packet loss statistics for the second router up the chain, again, no problem; there is a tool for that as well. The main issue was the data collection. How do we bring in all the data in a secure fashion, yet avoid the use of VPN?

Our choice was to upload the files to a cloud repository. Using the cloud as a go between and providing the end user with a destination unique to that user machine was done for convenience. Creating a write-only key that doesn't have read permissions was done as part of the security design. If the user can only write and never read, no data will be exposed even if the key is lost before the next rotation.? On the local drive we keep only a limited size log and manage the log rotation (as we didn’t use any log tool but a script, that logic needed to be taken into account as well).

How did this help us? (and how this can help you!)

We did this project with the belief that it would make a difference. Little did we know how much of a difference that would be. Now, when a user calls in for support, we already have lots of baseline and historical data about the user's environment. This data is well beyond the local OS and crosses into the working environment of the user. Packet loss on the local network? Weak wifi signal? VPN connected with no DNS update and so much more - all of this is now available in the data that we can easily access.?

Support calls have reached an exponentially faster resolution due to the data being already available for the support engineer to read. For the more technical users that want to check on their own, there is now a local log of multiple potential issues they can see.?

The impact on the user's support experience is profound. There is no longer a need to open a ticket and provide a call back with “check this / do that”.? The calls now start from an insightful point and we are already seeing the time to resolution of support cases shorten. We are also seeing the comprehension of the support reaching deeper into the users work environment. The ability to help beyond the single laptop, understanding the environment it operates in - one more win for this project.

Last Note - Privacy

We take user privacy very seriously and make sure to collect only operational data that can actually help solve problems. The data collected is limited to system monitoring data, the same data normally collected from servers (on prem or in the cloud). The data collected from the laptops is hardware and operational related with no user actions being collected in any way.?