CHAPTER 5: ROBOT WARS

We are shortly to release the 6th edition of the mobile banking app review. In this article, we share an extract of Chapter 5. To find out more and get exclusive material visit our website.

The ‘social’ experiment

Live chat is a feature now supported by over half of current account apps, with a third offering a chatbot prior to being transferred to agent support.

In August and November 2020, we tested the chat functionality of all the current account banking apps offering this feature to customers. We tested two queries; what we considered a standard query ‘how do I pay in a cheque?’ and a more nuanced query, ‘can I add my other accounts to the app?’. This was to help test the sophistication of the chatbots, but also to see how comprehensive the FAQ and support functions are within the apps.

The first step was to see whether we could find the answer to the questions by searching the FAQs and help sections (where available, surprisingly supported by less than half the market). Next, we tried the chatbot, followed by a live agent. Where these options were not available, as a last resort we tried searching the browser of the provider in question.

In the chart right, we can see which apps support which functionality, prior to testing. Halifax, Lloyds, Revolut and Monese support all elements. Halifax and Lloyds ‘search FAQs’ feature works in a unique way, searching the entire app as opposed to a set of pre-populated answers to frequently asked questions[1].

Barclays and Virgin Money are the only apps offering chat that is not asynchronous, meaning the user must stay in session to prevent the chat expiring before receiving a response from the agent. It is perhaps surprising that one-third of apps do not offer their customers any recourse for resolving queries via the mobile app.

Final answer

NatWest reported that interactions with Cora, their AI chatbot, doubled compared to 2019 (6 million compared to 3.9m)[2]. Considering a user has to engage with Cora prior to a live agent, this gives an indication of the level of engagement with ‘live chat’ functionality but does not tell us how many queries Cora is helping solve without the need of an agent.

With the more standard query of finding out how to pay in a cheque, just 19% of the apps offered an answer to this query through a simple search of the in-app help section.

For the standard query, in all cases the chatbot (where offered) was able to solve the query without the need of a live agent. Barclays, Virgin Money and Kroo required a live agent which took under five minutes for Barclays and Virgin Money, and 20 minutes for Kroo. It is worth reiterating however, that we had to wait for the full five minutes with the screen engaged for the responses from Barclays and Virgin Money, whereas Kroo allows you to leave the app and notifies you when a response is received. For those offering cheque imaging, it is arguably a significant oversight to not (at a minimum) include this in the FAQs.

For the second, more nuanced query the chatbots struggled to find us an appropriate response. The only chatbot that did was Revolut’s Rita, which gave a detailed response directing the user how to add the accounts to the app.

In contrast, Santander ended up in an endless loop of query / response, which repeatedly told the user to download the Santander mobile app – despite the fact we were already in it – suggesting the chat functionality has not been optimised by channel, reflecting the response you might receive had we been logged in on the web. The customer also must specifically type ‘Chat with agent’ to get transferred (which is not made clear). Had we not been seasoned app users, we may never have reached the agent support!

In fact, in our testing experience generally, the chatbots were generally unhelpful unless the customer is asking a standard query they could find the answer to in the FAQs and help section of the app. The chatbots can create a point of friction between the user speaking to an agent and getting their query resolved quickly.

What if you need real people?

In both tests, Virgin Money and Barclays were the quickest to respond (within 5 minutes) – which is fortunate considering they are the only apps where a user is unable to leave the app without losing the chat altogether! Both apps direct the user straight to an advisor, skipping the chatbot stage delivered by peers and saving the customer precious minutes in the meantime. Although arguably those minutes are then lost waiting for a response with the screen open.

In both tests, Virgin Money and Barclays were the quickest to respond

During the second test NatWest, Santander, Dozens and Lloyds all took under fifteen minutes, most likely considered a reasonable time for a resolution, considering the customer can leave the app and receive a notification when the agent responds. That said, Santander did call us the wrong name (although, quickly noticed their mistake) oops! We asked HSBC if there were any plans for adding third party account functionality to which the response was “we did have an updated app on the way, but I have no idea on the given timescale for this’.

Monese took the longest to resolve. Although they responded within half an hour to the initial query, the response from the agent did not provide a satisfactory answer to the question and we had to go back and further clarify. The final response providing the answer wasn’t received until over 24 hours later (by a different agent).

We had a genuine reason to retest Monese recently when a standing order failed to go out of the account. It appears they have had some major issues recently from reading the spate of 1 star reviews on Trust Pilot. The query was raised on the 9th October, and we eventually received an email on 20th October (not related to the chat query) explaining a mistake had caused a payment not to be made and a similar response to the chat query shortly followed.

The new normal

We expect live chat and servicing help through mobile to become critical features of the mobile app if they are not already. It is an area of innovation and testing we will be following in detail over the coming editions. If it’s not already front and centre of their app development roadmap, we suggest it needs to be!

We'd love to here what you think of the study - reach out on social media to any of the team with your thoughts or feedback.

[1] Since completing this test, the Halifax and Lloyds apps have expanded the search to include some browser-based FAQs

[2] NatWest Q3 2020 Management Presentation