5 Things We Learned Running a Chatbot for Social Good

Remember when the ICT for Development sector was buzzing with excitement (and anxiety) about brand new, game-changing tech that could revolutionize the way we can reach, engage, and impact people around the world? No, we’re not talking about November 2022, when ChatGPT was released to the public, but about April 2016, when the Facebook Messenger API was launched, allowing developers to build chatbots that could interact with users who messaged their page. A couple of years later in 2018, WhatsApp followed suit, opening its API to a select group of Business Service Providers such as Turn (then ‘Engage’) and infobip , via whom organizations could set up WhatsApp based services.

At the time, the hype around chatbots centered around their potential, rather than their actual functionality: development practitioners were promised a near future where expensive, time-consuming tasks could be fully automated, allowing us to deliver life-changing services remotely, cheaply, and at scale (sound familiar?). However skeptical you may or may not be about this vision, it is undoubtable that chatbots had a profound impact on how we do our work - but not (yet) because of AI.? Rather, they allowed us to build more direct, personal relationships with users where they are most likely to be already spending their precious data:? on chat apps.

*

ONE is a campaigning and advocacy organization focused on fighting extreme poverty and preventable diseases, particularly in Africa. ONE had been using mobile technologies to reach new audiences since 2012, leveraging SMS, USSD, IVR and mobile-optimized websites to provide a simple and easy for citizens across the African continent to take action. In 2018, an audience survey revealed that for the first time, as many ONE supporters preferred to be contacted via Messenger as via SMS - a tipping point catalyzed by the release of Messenger Lite in 2016.

ONE, who had by this point accrued a mobile supporter base of 2 million users, has realized that regularly re-engaging with these users via SMS or IVR was prohibitively expensive, and saw chatbots as a potential channel for more effective long term relationship building. ONE also hoped that chatbots would be a way to bypass Facebook’s increasingly obscure newsfeed algorithm, sending meaningful content directly to users’ chat inboxes using a personable, conversational format.

It was against this backdrop that the organization decided to experiment with chatbots and assess their merits against multiple objectives: reaching new and existing audiences, raising awareness of campaigns and activities, getting users to take action (for example, by signing a petition), and keeping users updated on campaign outcomes. ONE’s big questions were:

1. Would users be interested in engaging with messaging based services?
2. Would users understand and enjoy an automated experience, including an ‘on-rails’ one not supported by a sophisticated AI engine?
3. Would chatbots perform better than other mobile channels in getting users to take repeat actions?
4. Could ONE save time and money engaging users via a chatbot rather than continuing with our other mobile efforts?

To answer these questions, ONE set out to design and test a chatbot in the Spring of 2019. 5 years later, ONE has produced 5 iterations on its first MVP (Minimum Viable Product), and wants to share key learnings with the sector, so others can see what worked and what we’d do differently.

ONE’s chatbots today

Today, ONE runs 2 chatbots. The first is a Messenger-based chatbot connected to its global Facebook page , which allows users to learn about ONE, become a supporter, and see answers to FAQs which would usually be addressed by a Facebook page manager.

The second, more significant chatbot offering , is a pan-African bot which can be accessed via Messenger (including via a widget on the ONE Africa website), or WhatsApp. It currently reaches over 38,000 unique users in Kenya, Nigeria, Ethiopia, South Africa, Senegal, and DRC. It is available in both English and French (French speakers represent 28% of the audience), and enables users to learn about ONE, become a supporter, learn about campaigning issues, sign petitions, take quizzes, see answers to FAQs, and contact the ONE team. Nigerian users (who represent the biggest audience group, followed by the DRC) can also join their local WhatsApp activist group to get involved in real-world campaigning activities. The chatbot includes re-engagement mechanisms to automatically or manually reach out to existing supporters to get them to take repeat actions. Over 17,000 actions have been taken via the chatbot, with most of those engaging taking multiple actions.

In the backend, the chatbots are integrated with a WhatsApp Business Provider (Turn), a chat-flow builder (RapidPro), and a CRM platform for managing supporter numbers (ActionKit). Data produced by chatbot usage is visualized using Tableau and built on ONE’s Google BigQuery data warehouse. Neither chatbot is supported by AI (yet!), but uses a simple decision-tree format, with some keyword recognition.

Learning # 1 : Start small

It has taken ONE several years to land on a version of a chatbot that we feel confident delivers on what we had initially hoped to achieve. A first recommendation is therefore to feel comfortable with starting small and proceeding slowly. Because of the way funding as well as hype-cycles work, there is often an emphasis on delivering exciting results within months. There is also the temptation to ‘move fast and break things’, which often translates as ‘move fast and break things, at the expense of users’ safety and your budget’. ONE’s chatbot work was approached pragmatically and experimentally, and was not initially tied to targets other than learning.

Using a cycle of user research, designing, launching and testing prototypes built around one specific campaign and market at a time, ONE determined that a chatbot for Nigerian youth, accessible on Messenger and WhatsApp, had the highest chance of success. Its launch coincided with the Covid-19 pandemic, and ONE was rapidly able to update the bot with content promoting trusted Nigerian Covid information and support services , as well as case-tracking data.

Building on the learnings from this version, ONE eventually landed on the current iteration, the ONE Africa chatbot, which offers similar functionality, but with an increased number of actions for supporters, as well as localized content for different markets in English and French.

Starting with a focused offering with minimal functionality and content still involved a steep learning curve related to the design of chatbot flows, writing, localizing and uploading content, integrating successfully with 3rd party platforms, building and testing flows, integrating data-tracking, making sense of data, marketing, and figuring out costs and scalability. All of this took time so make sure you have the space to properly learn and iterate.

Learning # 2: Prioritize user testing and feedback

At the time that ONE started experimenting with chatbots, very few organizations were using them to reach their target audience. ONE’s design research yielded only a handful of examples of ‘chatbots for good’, including UNICEF’s U-Report , and an innovative story-telling chatbot by WaterAid . At the same time, a chatbot's conversational interface was in many ways more familiar and accessible to citizens in the global south, more accustomed to using USSD and IVR services than mobile websites and apps. Beyond getting to grips with the technical work of designing and building a bot, ONE was therefore firstly concerned with understanding how users would feel about them.

To understand this, and start assessing our chatbots’ usability (covering effectiveness, efficiency, satisfaction and error tolerance), we regularly consulted with members of our target audience leveraging ONE’s network of youth activists in Nigeria and other markets. Using WhatsApp groups to gather qualitative insights, we also monitored feedback via polls and open-questions in the bot itself to continually learn what was working and what wasn’t.

After testing our first chatbot, users employed words like “interactive”, “funny”, and “easy”, particularly enjoying the conversational tone, and localized language. They were also excited to be using an innovative technology, exclaiming “Naija, no dey carry last, in fact we be the first man!” Even the use of buttons to progress the conversation at each ‘fork’ in the conversational branch, was seen as an efficiency rather than a hindrance.

Overall, by testing early and often, we learned that a specific segment of ONE’s audience saw the chatbot as a great way to engage young activists with their use of innovative tech, and to enable them to quickly and easily introduce what ONE had to offer. In comparison, they reported that the ONE website (which has since been redesigned itself), tailored broadly to desktop or smartphone users, felt overwhelming.

User testing also enabled us to identify pain-points and ideas for improvement, including message length, the need to be responsive to a minimum amount of small-talk (e.g “hello”, “thank you”, “OK”), and adding increased personalisation (such as being greeted by name). We were also able to fine-tune our use of language - users indicating that they would prefer multiple language options such as English, Pidgin or Hausa, to the pidgin-English mix we had started with. Whilst analyzing backend data would give us the ‘what’, regularly getting feedback from users gave us the ‘why’, and ultimately, the confidence to continue investing in chatbots.

Learning # 3: Anticipate marketing spend

Sadly the famous “if you build it, they will come” line from the film Field of Dreams does not apply when it comes to digital tools. Although chatbots are touted as a potential cost-saving channel, they still require marketing budget to drive users towards them. Unless your use-case already involves a built-in onboarding mechanism (for example, in-person referrals via Community Health Workers), you will need to rely on 3rd party ads, predominantly those running on Meta platforms, using call to action buttons that click through to your chatbot.

ONE experimented with a variety of internal and external marketing approaches to understand more about the costs and implications of growing its chatbot userbase. For example, in an experiment re-engaging existing Nigerian email subscribers, ONE achieved a CTR of 8.5%, and 3.4% of those emailed went on to sign a petition via the chatbot. Because there is little to no publicly available, comparable data, it’s hard to benchmark this performance, however this chatbot-focused campaign actually performed better than previous, comparable email campaigns ONE had run.

This exercise also highlighted an important challenge related to attrition: ONE noticed a significant drop off between link clicks and conversation starts, likely because of cross-app compatibility issues (users clicking on links who may not have Messenger or WhatsApp on the device used to open the email).

When experimenting with Meta ads driving male and female Nigerian users aged 18-36 to both Messenger and WhatsApp chatbots in 2020, ONE quickly learned that it was easier and cheaper to drive users to Messenger than to WhatsApp: one of the best performing campaigns cost $0.17 per new Messenger user, compared to $1.77 for new WhatsApp users. As a result, only 14% of our userbase is on WhatsApp, despite the additional cost required to run a WhatsApp-based service. It was also cheaper and easier to convert male users, with female users 2.5x more expensive to acquire, in line with the gendered nature of internet usage. As digital marketers reading this will know too well, you also need to account for regular changes to Meta’s advertising platform and major differences in the user experience from an ad promoting a Messenger versus WhatsApp chatbot, which can massively impact conversion and cost.

The total spend on user acquisition compared favorably to other lead generation campaigns driving users towards the ONE website, which was a positive indicator that a chatbot could be even more effective at growing engaged supporters. However, all these experiments took time and money, and were often stymied by the stranglehold Meta has on the digital advertising ecosystem.

Learning # 4 : Plan for re-engagement from the start

Most products pin their hopes on providing information and functionality that will meet a users’ needs or interests so strongly that they will be compelled to come back again and again. But as anyone who has developed a digital product (or indeed any product) will know, this does not reflect real human behavior: users’ attention is extremely divided, and organizations are competing with any number of digital and real-world distractions in our ambitions to have that user interact with our service again. Like many, ONE did not design with re-engagement baked in from the start, and as a result have only recently started to grapple with the limits of user retention.

This task of re-engagement is even more challenging because of the rules set by Meta in re-engaging users outside of a ‘free’ 24 hour window opened by a user messaging your chatbot. Outside of this window, organizations and brands can only send unsolicited messages, at cost, that match a very specific set of use-cases, or pay for sponsored messages that in turn have major limitations. In the case of WhatsApp, each message has to be signed off by WhatsApp themselves, and broadcast messages are restricted by a system which throttles the number of (unsolicited) messages sent to users in tiers of 1-100k messages.

ONE tackled this challenge by running a series of internal tests to see what could be learned, designing a fairly complex automated re-engagement system that nudged users to chat with the bot again in the weeks and months after their first engagement. As a result of what we learned in these tests, namely how challenging it was to work around the broadcast message rules, but also to make sense of the re-engagement data generated, ONE significantly scaled back the re-engagement approach. It is for now confined to one re-engagement nudge less than 24 hours after the first chat, when users are most likely to want to re-engage if they do at all, and a system to re-engage Messenger users with broadcast messages outside of the 24 hour window (having deciding that for now, the WhatsApp restrictions made it prohibitive as a broadcast channel). At the time of writing, data shows that 25% of users nudged using this mechanism take at least 1 additional action within the bot, a figure that ONE hopes to improve on in the coming months.

Finally, the challenge with re-engaging users isn’t just about? cost or technical restrictions, but has to do with wider audience engagement strategy. In ONE’s case, we realized that we had not mapped out a chatbot supporter journey beyond the first engagements, meaning we didn’t always know what to drive existing users to next. This too, needs to be factored in when designing your chatbot.

Learning # 5: Ensure chatbots add value to staff, not just users

ONE’s chatbot work being, for a long time, experimental, meant that its development was largely conducted separately from day to day activities, and relied on a small group of staff and external experts who had the technical expertise required. Whilst the chatbot team worked with local and global colleagues to understand campaign objectives, develop content, coordinate user testing, and share out learnings, for the most part, the chatbot work operated in a silo. Making sense of performance data on a regular basis, and generating motivation and accountability for the continuous improvement that all digital products require, was a particular challenge.

In retrospect, we should have made a greater effort from the very start to better integrate the work across teams and ensure that staff across the organization were sufficiently invested. We also would advise identifying an internal chatbot ‘champion’ - someone whose role it is to promote the integration of the chatbot into day to day thinking. This state of affairs is not exclusive to chatbots, however, but reflects the way digital platforms are often (wrongly) perceived: as a discrete output or simple comms channel, rather than as a tool with a holistic, cross-cutting role to play across multiple workstreams. For example, the chatbot work enabled ONE to better understand its audience in Africa, which in turn added value to campaign and policy teams alike. Similarly, the chatbot raised questions about supporter journeys, which fed into wider discussions on this topic at an organizational level.

Finally, there is the question of capacity - when teams are already stretched, adding a new platform, with its long tail of upkeep tasks, from performance monitoring, to content updates, can result in disengagement. Unless the chatbot can demonstrate value to team members as well as to users, i.e show how it can help people do their job easier or better, it may be sidelined in favor of other priorities.

The future for ONE’s chatbots

Looking back at the initial questions ONE set out to answer is both gratifying and challenging. On the one hand, chatbots have proven to be an effective channel for ONE to reach and engage citizens across Africa - driving awareness of campaigns and the issues they tackle, and giving young Africans a platform to share their concerns, support causes they feel passionate about, and connect with other activists. Our ongoing feedback tracking and user testing has shown that a significant majority of (connected) users can easily access and use Messenger and WhatsApp chatbots, and furthermore, enjoy them.

On the other hand, the technical challenges with re-engaging users after a first chat mean we have not yet definitively answered the question of comparative value in relation to other digital channels such as Facebook, X, or e-mail, especially relating to user retention. Similarly, we feel that we have only begun to scratch the surface in terms of optimizing our chatbots’ UX and improving our bounce and drop off rates. Finally, the question of cost-effectiveness remains unclear. As organizations in the non-profit sector grapple with a challenging fundraising environment, it remains difficult to make smart choices on where we should invest our limited digital and marketing budgets. A key lesson for ONE from our chatbot (and other digital) work has been the need to demonstrate a clear Return on Investment for initiatives with a focus on meaningful data rather than vanity metrics. Ensuring you have a robust reporting infrastructure in place that focuses on small amounts of meaningful data enables you to better analyze performance and make changes as needed.

The advent of more powerful forms of AI since ONE undertook this journey also begs the question of whether this type of on-rails chatbot, and the learnings detailed above, are still relevant. We believe that they are: marketing costs, retention strategies, and staff expertise and capacity are all tech-agnostic hurdles. However, what may change is users’ tolerance for services which are not backed by sophisticated AI. As Generative AI becomes mainstreamed into other social media platforms used in the global south, as it has in the case of Snapchat for example, more connected users will, rightly, start to expect the same standard from us. Ultimately, the rapid growth of AI opens up a huge opportunity to rethink the chatbot approach and how organizations can utilize the power of generative AI whilst maintaining a strong (and safe) experience for users. For example, rather than having to design individual user flows for each new campaign we could use AI to automatically pull the relevant information from our website. And instead of manually adding and updating information about our issues to help answer users’ questions, we could build a custom version of a tool like ChatGPT that combines instructions, extra knowledge, and any combination of skills to provide a richer user experience.

At the same time, economic, gender, and structural barriers to connectivity still remain. Generative AI powered products are reported to drain phone batteries at a faster rate because of the amount of data continuously being exchanged. AI powered tools’ reliance on communicating regularly with vast databases also makes them challenging in environments where phone credit is a precious resource. Women and girls continue to have their internet and phone usage monitored by gatekeepers. And in a world where citizens are rightly asking questions around the environmental sustainability of technology, how do we address the fact that making a single image with generative AI uses as much energy as charging your phone .

In other words, more basic chatbots, SMS, and IVR still have a role to play for the foreseeable future. So too do real-world interactions: one of the most popular conversations on ONE’s chatbot was how to join a local activist group. And that’s the biggest lesson our chatbot work has taught us: what users really want, is connection - with us, and with each other. Let’s make sure we give them that.