AI realism (part two)

Emotions are running high about AI technologies. In this 2-parter, I do my best to make a rational case for the state of AI, and how we can respond to it. This is the second part; catch up with part one here .

Today, we’ll talk about developing a company culture that thrives on experimentation and unpredictability. I’ll describe the conditions that can keep a product company nimble and healthy during a period of rapid change, enabling it to take advantage of emerging technologies.

I’m also going to recap the weeklong AI exploration project we ran at Raygun, the tactics and processes we used to make this initiative a success, and a couple of examples of what we built.

I won’t claim to have all the answers, but I am proud of Raygun’s response to the shifts we’re seeing, and I’d like to bring you along.

The fart stage

In these early stages, like any period of mercurial change, there’s a short-term chance to exploit AI. Understanding is low, hype is high, and there’s a quick buck in simply being first. I recently heard someone relate it to the launch of the iPhone and App Store, where there was a huge proliferation of simple apps that did almost nothing. One of the most ridiculous examples was a fart app , which was essentially a digital whoopee cushion.

Right now, we’re in the fart app stage of the AI evolution. Some interesting products are launching, but they’re more of a novelty, and not delivering huge rewards yet. During these early stages, we should be constantly inventing, rather than assuming that the limitations we have now are what the “real” version will look like.

But before we talk about adopting a paradigm-changing technology, we have to achieve a culture that can support successful adoption and adaptation.

Lightning in a bottle

I’m a builder at heart. As a kid, I loved Lego and Technics (in fact, I still do). So I always wonder; why just live in a pre-built reality? Why not create the world we want? Most people see things they want to change about the world, and almost nobody actually wants to do anything about it. Why accept the status quo when you can build things?

I like being a founder. I like the creativity and the short list of rules you have to follow. I like the challenge of taking on almost everything as an opportunity, feeling like it’s our team versus the world. For most of us, when we encounter a problem, it’s easy to feel a loss of control and become stressed and frustrated. As a leader, though, I find it can be unexpectedly helpful when a problem constrains the available options to just a few.

This optimism and drive to build are the biggest prerequisites for leading an innovative culture. The mentality that if something’s not right, that’s our chance to build a solution. When we discuss culture in tech, there’s a lot of emphasis on responding to issues with blameless postmortems and empathy. That’s all well and good, but we shouldn’t overlook the cultural value of mucking in, backing ourselves that we can fix things, and carrying on.

This is the idea of “failing forward”. If you deploy code and have an issue, rather than pulling it back, applying the fix, and putting it back in, you just deploy the fix on top. You’re always rolling forward, looking for those opportunities.

At Raygun, we look for people who want to solve hard problems and scale through technology. We try to avoid a “business as usual” approach and embrace critical thinking. When I hear a leader enforcing the status quo or best practices, I know that’s a person who doesn’t want to think about the problem. They’re just accepting an established view. That’s not to say we can’t learn from precedent or build on proven successes, but it does mean we don’t go with a method solely because it’s worked before.

Fail forward, scale small

Raygun’s products help software teams identify and diagnose what’s crashing, impeding, or slowing down their software, and then make it better for their customers. Basically, we help other tech businesses build better experiences. So when we talk about how Raygun should scale, we remember that we’re effectively a data business. Scaling people-first, which makes sense in a services company, doesn’t make sense. I often refer to examples like WhatsApp, which had 55 employees when it sold for $19B, or Instagram with a team of 13 when the Facebook acquisition happened. Product companies don’t need lots of people. If I look back on the last several years, I’d say one of the biggest issues is that we’ve been scaling product companies as though they were services companies, adding headcount thinking that bigger is better, when we should focus on the impact of an individual and how that scales.

So what does a product company need? Well, again, it’s culture. If you’ve got the right internal culture in an organization, risks are often mitigated by default because you’re curious, you’re innovating, and exploring.

So at Raygun, we try to foster a culture of experimentation. Some of our most loved features and designs have come from tangents and curveballs, peering over each other’s shoulders and discovering that our “what if” ideas might actually be viable. Engineers love to solve problems, but I don’t want to focus so much on fixing and optimizing that we forget to create. I love hearing the team say to each other, “Hey, come have a look at this!”. We encourage these small, casual demos of things they’ve found or made, teaching each other, and often incorporating these experiments into their projects. It’s embracing that idea of “failing forward”; accept that things will go wrong, and start trying stuff. Raygun processes billions of software exceptions for people every month, so we’re an integral part of everybody’s mistakes and repairs. That helps us get comfortable with the idea that things break, and that’s okay.

If you’re a product company, your goal shouldn’t be to scale to thousands and thousands of employees. Instead, it’s the aptitude of those people that is absolutely paramount, and their willingness and ability to participate in this culture of experimentation and innovation.

AI week: F##king around to find out

Leading up to our 2023 All Hands meeting, I could see that AI was going to bring big changes, but I didn’t feel confident that I — or anyone — understood it just yet. This was a problem because I wanted to provide a clear direction. I even considered delaying that meeting until I felt I could come in and give the team a definitive plan.

That’s not what I did. I remembered that I’d hired a bunch of smart, curious, creative people. So, at that meeting, I told the team that I didn’t have all the answers, but I had some ideas about how we could start asking better questions. And that led to AI week.

We agreed to allocate a full 5-day work week, Monday to Friday, putting all our regular tasks and projects on hold. The only exception was support — if a customer contacts us, we still need to engage with them professionally and responsibly.

The goal of the week was to have informed discussions, across every team, about the application of AI in our business. I had already noticed gaps opening between those who were fully immersed in AI exploration and those who had seen a few headlines about ChatGPT. This gap in understanding creates challenges – how can we make great decisions if we can’t have great discussions? So, instead of producing a prototype, a product, or a campaign, the goal was just to investigate what’s out there and how it works. Nobody even needed to write any code; the focus was on exploring.

We also wanted to get everyone collaborating beyond the team they ordinarily worked with. Raygun is roughly a 30-person organization. I didn’t want to see a cluster of engineers working on one problem and the customer success team working on another. I wanted cross-functional teams. So we set up small blended teams across the business, each with their own project.

These projects were only a guide, not a mandate. We presented them as “if you’re stuck for an idea, here’s something to get started on.” In the end, every team actually chose to run with their suggested project, improving on the brief as they went.

Remember, the real goal of this week was to get everyone up to speed, with a shared frame of reference. That meant every single role, not just the nerds. One remarkable thing about GenAI is that it’s self-referencing; the tools themselves contain the knowledge to make everyone an expert. If you don’t know how to do something, ChatGPT can teach you.

So come Monday morning, for the first four hours, each person in the company tackled our coding challenge. This is the technical test that we give to any software engineer who wants to join Raygun. Everybody in the business, whether a designer, customer success executive, accounts, attempted the coding test using a range of AI tools. This demonstrated to the whole team that they could complete (or at least take a damn good shot at) a software engineering task when they weren’t even a developer.

Now that each of us had a sense of what we could achieve, the teams formed. They were told they’d be presenting whatever they had built on Friday at midday. Effectively, they had four days to see how much they could achieve.

A small showcase

Now, half the projects I can’t share here because we’re bringing them to market! These are just a couple of examples that are particularly interesting because they’re applicable to most organizations.

Robbie

‘Robbie’ is a chat indexing agent. The team had indexed some of our internal data into this, then took an AI model for doing voice-to-text, (OpenAI has a really good model called Whisper), and an LLM (in this case, an alarm-based one that we could host ourselves). In their demo, they asked aloud, “Hey Robbie, did we have any issues or outages in February?” Robbie reported that we’d seen an issue in production, how long it lasted, and how we’d resolved it.

Within four days, a team who started out knowing almost nothing about AI has built an internal agent that can ingest data that I can talk aloud to, freely, and will give me insightful, accurate responses. For context, Siri started development in 2003 and was released in 2011. The game has changed.

Jarvis

Jarvis was somewhat similar, but chat-based, trained on a wider dataset, and running inside our internal Slack, as though chatting with a teammate. For their demo, this team asked Jarvis about audit logging. The full Raygun platform has audit logging , meaning whatever action anybody takes within the application goes into the log. When one of our developers is building something, say adding a button on a form, we want to log how often that button gets clicked.

So the team asked Jarvis, how do I add something to the audit log? Jarvis, with access to our Slack history, internal documentation, and full context of the Raygun product and technology stack, provided the precise steps one of our developers would take to make this happen.

What about our customers, though? The next part of the demo asked Jarvis how to set up Real User Monitoring , one of Raygun’s core products. This time, Jarvis drew on our documentation, on resolved past queries in our CRM, and our onboarding resources, to give detailed, interactive instructions.

Both of these examples solve problems that exist in virtually every organization using data that we all have. If you look at the archives in your communication channels (say in Slack or Teams), there’s a lot of solved problems and knowledge, or in your CRM (Zendesk, Intercom etc). I personally estimate that I spend about an hour each day answering questions from the team about procedures, architecture, past projects, etc. Of course we have documentation and guides, but the reality is the founder has often been around the longest and knows where the bodies are buried. Plus, people like to ask questions in a conversational format , so many of these questions come to me.

In less than one week, a small team was able to build a replacement for much of that knowledge. That’s hours back in my week. That’s a solution the team might have hesitated to ask for when they don’t want to bother the boss. That’s the removal of delays when I’m in a meeting or preoccupied. Not to mention the potential time-saving and experience improvements for customer-facing queries.

So what?

We’re so happy with these outcomes in terms of product concepts, team cohesion, and learning, that we now want to do this once a quarter. Not focused on AI every time, but always on a technology or challenge that can bring the organization together. We’ve had hack weeks in the past, but without a shared focus, people would either go and fix something that only related to their own work, or chase a feature they’d always wanted to build with limited applications for our customers. The AI theme provided the constraints to keep us focused without limiting ambition or imagination.

I believe that much of the success of this initiative comes back to that culture of innovation and curiosity. An established sense that it’s okay to go off-map, and if you make a mistake, just work out how to fix it. All those things in our existing culture meant that it worked when we said “no process. Do whatever you want, and work however you want.”

Of course, pausing everything to conduct exploratory projects with your entire workforce is a luxury. For many businesses, when people stop working, they stop billing. Raygun is a recurring revenue business, so our only major “cost” is the regular work that we already had planned. I acknowledge that an intensive “boot camp”-style initiative isn’t viable for every organization. However, I’d still encourage you to borrow the essential elements that made this a success. Set up small, cross-functional teams and challenge them to solve a problem using these new tools, whether it’s across days, weeks, a few Friday afternoons – whatever makes sense for you. Give them permission to play around, to go on tangents, to try things that will never work, and to fail. Worst case, they’ll realize it’s okay to try stuff. Best case, they’ll blow you away.