Testing at Sea with AI

Software powers the world and that means your favorite cruise line too. Not surprising, but most software testing happens on land. It’s a function of our environment, we live on land and it’s the most convenient place for us to test. When testing happens at sea — things get a little more interesting. For those of you that haven’t been on a cruise recently, today’s cruise ships are high tech and heavily dependent on software. Either through software embedded on large devices such as screen kiosks that show the next stops, onboard events, interactive maps of the ship, or a fleet of mobile apps to entertain the kids, order drinks, gamble or even open your stateroom door, all this onboard software needs to be tested and monitored at sea.

Note: Thanks to Janna Loeffler for her insights, experience and sharing this info with the world!

The software on these ships not only needs to work for the passengers and crew but also needs to work for the business. More so every year, cruise lines depend on these apps for cost reductions and increasing their top-line revenue. Those mobile apps that let passengers order a drink mean faster orders and fewer servers. Each spin of a mobile app’s roulette wheel or blackjack hand means more money straight into the coffers. Those apps that entertain the kids mean the parents can spend even more time in other more profitable apps. Modern cruise ships and experiences are transforming into software experiences and that software needs to run reliably to support these new business models.

Testing Challenge: Realworld Conditions

All this software can be tested on land, but if it doesn’t run on the waves, none of that testing matters. It would be a horrible experience for passengers and crew and the business if this software failed at sea. The testing isn’t complete until it is verified running on specific ships, with specific internet connections, specific routes/GPS locations, and it all needs to be monitored 24/7. Most software deployments happen while in port or while guests are off the ship so they don’t notice, and to reduce risk, but these onshore testers need to be working odd hours with never enough time. Software testers must sail on the ships to test and verify new versions of all these apps work in the real world.

It sounds like a fun testing job to be at sea, and sure the off-hours are great, but much of your time can be spent walking 50–90 flights of stairs a day, to and from their stateroom, and running all over a ship that averages the length of three football fields, testing the same test sequences day after day. Sometimes the internet can be flaky, so any test automation needs to be able to run offline, with no connection to the cloud. For the business, all this means more expensive manual and automation testing, paying for all those testers on the ships, and losing the money they could have had if those cabins were filled with guests instead.

AI-based testing is beginning to help these sea-fairing test pirates. Firstly, AI-based test automation means test cases can be created more quickly than traditional code. The AI-based tests also need less maintenance as the app changes versus traditional hand-coded test automation solutions. Some AI-based solutions easily run offline, so there isn’t any downtime, which is especially helpful with monitoring these on-ship systems. The AI-based test machines also take up less room and eat less food than human-powered traditional test automation, so the cruise lines can open up those cabins to paying customers. AI is starting to make real-world testing on cruise ships, better, faster and cheaper.

Testing Challenge: Platform Matrix

The software on these ships executes on everything from giant HDTVs running web apps bolted to the inner walls, to the matrix of the passengers on personal phones and tablets, all with custom-built user interfaces and technology stacks. Most cruise ships have a mix of web, Android, iPhone, and Unity-based user interfaces that need to be tested. Traditional software test automation requires different testing frameworks and often different programming languages to test such a matrix. Test automation for this matrix is almost too daunting for most teams to attempt, and when it is built, the teams can quickly succumb to the costs of maintenance. This leaves them needing ever more people to manually test and check all day.

AI again comes to the rescue. AI-based test automation doesn’t need to be re-written for each platform, the core test logic can be re-used across this matrix of interfaces. All the AI needs is access to the screen, since it sees the app just like a passenger sees it. The AI interacts with the applications like a human would, so it doesn’t need access to custom testing hooks, APIs or custom testability interfaces so AI-based tests can be written by non-programming testers and even re-used across different platforms. Faster and cheaper means it is plausible to have tests running on all platforms with the fixed testing budget.

Testing Challenge: Deployment

These ships make money when they are at sea, so when is all that software powering the ship and the guest’s experiences updated? The software can really only deployed safely when the ship is docked, when the engineering team has a great internet connection, access to all the hardware and software and can fix anything that goes wrong during updates. This creates a rush of effort to get the software updated and tested with all the latest features and bug fixes in as little as a day.

AI-based testing is helping by making it quick to build tests for all the new features and spend less time repairing older test scripts that were broken by the new changes. The AI-based tests can also ensure that the performance of these apps and systems from the viewpoint of the guest by analyzing the user interface and making sure the user experience hasn’t accidentally degraded — even though the API calls and underlying operating systems thing everything is fast, it doesn’t matter if the user experience is still slow or broken. Soon, AI might well analyze all the system logs after the update and compare it with the previous voyage to look for any anomalies so the team knows what to work on before the ship arrives back at port. AI is helping turnaround time and confidence with deployment

Summary / Promo

Testing on the high seas sounds glamorous but it is full of danger for software quality and testing. AI is starting to help solve these ship-specific testing problems. I’ve been working with the folks at PinkLion.AI to help solve some of these interesting testing problems and wanted to share. The PinkLion.ai folks are also happy to share what they have learned deploying AI-based testing solutions, including some of the technology we’ve developed at test.ai, with some of the largest cruise lines in the world. If you find yourself testing at sea, or another difficult space, the PinkLion.AI team can help.

It is fun to think that our little testing AI bots are having more fun than the engineers that created them — living a life at sea :)

— Jason Arbon, CEO at test.ai

Image credit: https://awesomeocean.com/top-stories/summer-cruise-bad-news-for-ocean/