Schr?dinger's bath bombs

Episode 26: 01/10/2023

The bath bombs fiasco was when it really hit home how much we’d learned about building websites.

?

My mother wanted to buy a large volume of bath bombs for a family member’s birthday.

?

Bath bombs in Lush cost about a tenner each. As an economist, I can practically guarantee that this is in no way related to how much they cost to make. Lush can just about get away with them at this price point because, normally, you only buy a bath bomb as a one off.

?

This was not a one off. Judging by the quantities involved, the family member was planning to use bath bombs as a normal part of their daily bathing routine. For at least six months. My mother cast around on the internet for cheaper suppliers, and eventually found an online-only outfit selling bath bombs for much less.

?

So it came to pass that one day I woke up to find a massive purchase order for bath bombs, lying in ambush in my work email inbox. It was my mother, asking me to buy them on her behalf.

?

This is a familiar technical support scenario to most of us, but actually my parents are pretty internet savvy. That made the request suspicious. But investigating the political machinations behind the request was highly likely to take longer than just doing it. I took what appeared to be the easy route out, and bought the bath bombs.

?

It was not the easy route out.

It was actually the start of a multi-week tale, of order errors and miscommunications between left and right hands. It made me miss the good old days. You know, when you went into a physical Lush store and shelled out sixty quid to make the bathwater go all sparkly for a few days.

?

The blow-by-blow of what happened is unimportant. It is the underlying reason is relevant here, in a blog about founding a tech company. Because the bath bombs supplier was a startup, and all the root of the order problems was its website.

?

Over the last few months, we’ve had to create the advertising section of the El Toco search engine’s website. Whereas many startups would use a template for this, we’ve had to do it from scratch. This meant I've had to learn how to program the e-commerce parts of this from scratch, because my development experience in finance and AI has proven to be of zero relevance. And it turns out that there are a great many pitfalls when you program the shopping bit of a website.

?

Pitfalls like the simple matter of generating IDs for orders and customers.

You’d have thought IT people cracked this decades ago. They have not. We’re in 2023, but topics like the usefulness of anti-aging creams, or the amount of Kim Kardashian that has been surgically altered, are glimmering temples of certainty compared to the incredibly common problem of how to generate unique order numbers.

?

There is simply no consensus on the best way to do it. The solution can be wildly different, depending on your database engine, how quick your application needs to be, and how many orders you will be receiving. Among many other things.

?

As a developer coming to it cold, you’ll investigate all this, choose the best option, and spend hours implementing it.

Then, a week later, you’ll notice something odd has happened, and on further investigation discover that your solution is doing something hopeless, like giving two different orders the same ID.

?

One of the things that is an absolute no-no when generating these numbers is for them to be sequential. This reveals information about your business that you might not want to give away. For example, if you’re pretending to be a busy dating website but a new customer notices that their ID is #cus_00003, it kinda clues them into the fact that this is one sea where there aren't many fish.

?

So it was with some sympathy that I spotted our bath bombs order was order ref 0000012.

Bless them, I thought. Not only are we practically their first customer, they’ve also had to cut corners on development.

But that was only the beginning.

?

Another thing, when you set up an e-commerce website, that you have to learn about is inventory management. Your website needs to know the instant you run out of stock. Otherwise, you can end up selling things that don’t exist.

?

Even though El Toco sells advertising, we’ve had to solve this problem too. Because if two conflicting ads are placed for the same website, chaos will ensue.

?

At first glance, this seems ok? An item is either in stock, or it isn’t. So long as the website is in sync with the inventory in your warehouse, it’s a black and white issue. Right?

?

Wrong. The reason is checkouts.

When there’s one bath bomb left in stock, and somebody puts it in their virtual basket, your website needs to block anybody else from also putting it in their basket.

?

But that doesn’t mean the bath bomb is actually out of stock yet. A whole host of things might happen between this point and the order completing. The customer’s card might be declined, or their spouse might come in and force them to minimise the window for forty-five minutes.

?

We can see that after going into the basket, but before payment is taken, the bath bombs are in a sort of Schr?dinger's cat situation. They’re definitely not out of stock, because they haven’t been sold yet. But they’re also not in stock either.

?

The website needs to be super on the ball here. It must make sure that, whatever happens next, the inventory is eventually reduced to zero, or released so that other customers can buy it instead. This includes handling the situation where the customer just closes the window, which does not trigger any sort of signal for the website to act on.

?

Handling all of this requires reams and reams of code. It becomes almost dizzying in its complexity, when you realise that many of the scenarios can happen in combination. Like a payment being declined while the spouse is in the room and the window is minimised.

All of those scenarios have to be tested. It takes a huge amount of time.

?

So we can sympathise with the bath bomb website, whose developer didn’t bother with any of this. Many of the bath bombs I ordered were actually out of stock, so we had to effectively re-do the order manually, by email.

?

Manually ordering the bath bombs by email took weeks. At El Toco right now, there simply isn’t the resource for that level of customer service after we go live. There's me. That's our resource. If I spend one day manually correcting orders because of system glitches, we’re going to be in water that’s far too hot for a relaxing bath.

?

What we’re doing with the bath bombs story is slipping in a worked example, to show how even the simplest e-commerce website is its own world of complexity under the hood. There are no books explaining all this, you have to teach it to yourself. It is only when spotting errors in other people’s websites that you realise just how much you’ve had to learn.

?

Doing this much coding properly is particularly hairy when you’re on a deadline. A tight deadline, like the one we have at El Toco right now.

?

?

The advertising section revamp

?

In May 2023, we pulled the plug a week after launching El Toco. Too many things were going wrong all at once. Chief among those things was that it couldn’t make any money.

?

It couldn’t make any money because our website was missing the advertising section, which is the bit that makes the sales. We’d already built this part of the website, back in 2022, but the whole thing needed a facelift and the order processing still had some holes, like the ones in the bathbombs story above.

?

As with the search section, this meant we’ve ended up rebuilding the advertising section from scratch, even though it was supposedly already finished. Piba Studio , our designers, done a great job with the visuals. The result is modern and much more appropriate for our professional user personas:

?

The design for the advertisers' section was passed over to development in July. With the expectation of finishing by October. Which is what I'd told our investors.

??

Our investors, who came on board in summer 2022, had already been through the trauma of an El Toco Missed Launch Deadline.

Actually they'd been through it twice. We were supposed to launch the search service in winter 2022, but had to put it off because the crawler wasn't ready. The wounds from that first delay hadn't really healed when we had to cancel the launch in 2023, because the advertisers' section didn't exist. There was no overt aggression on the conference calls, but believe me when I say there was zero appetite for this to happen a third time.

We had to make this deadline work for once.

?

It was in early July and we’d only been a-developing for a week when I started to twig that we were in trouble. Coming back to the original code we wrote in 2022, we were noticing a great many things that needed a rework from the ground up. For example if a customer's card was declined on a rebilling date, there was nothing in our system which would deactivate their ads. They'd continue getting their ads for free, until somebody manually switched them off.

?

Startups normally solve this sort of looming train crash by dividing the work between more people. We don’t have that option. There’s two of us. Alex Cachia [[Ka-Key-A]] does the front end, I do the back end.

?

With a that sense of impending doom which has become all too familiar over the years working on this project, Alex and I went through the design, slashing everything that wasn’t strictly necessary. This moved the goalposts, but only by a metre or so. It was already a pretty sleek website. There wasn't really that much we could cull.

?

?

We had no other choice but to work faster.

To increase efficiency, we started programming in tandem. Alex worked on the front end during the day and committed her changes at night. I got up early and did a round of corrections first thing.

?

For the front end, this worked. Alex wrapped up her side in a matter of weeks, right on schedule.

For the back end, where I was focused, there was no such luck. This was partly because, while doing it, I had spotted a seemingly innocuous bug in the database.

?

The seemingly innocuous bug in the database

?

Yup, if you're a regular reader, sorry. It's time for another anecdote involving databases!

Let’s just describe the atmosphere first.

?

Generally in films, when there is a programming scene, they show the person sitting in a moodily-lit room, hammering relentlessly away at the keyboard, in time to a grimy techno soundtrack playing in the background.

?

During my many years of development I can tell you that this is about as far from reality as it’s possible to get. Most of the time, you gaze out of the window thinking, generally with your beverage of choice. You type smallish amounts of code into the computer, and then go back to window watching again. This all happens in silence, because it’s easier to concentrate.

?

However, the reason you work like this is because under normal conditions you have to pace yourself. This is especially true at a startup, because if you burn out there's nobody else to pick up the pieces.

Right now, we are not under normal conditions. After all these years, we've reached the part of the race where you stop pacing yourself. The part of the race where you just go hell for leather until it's done. The more I focus, the more outside distractions become increasingly unwelcome. Like noises. And also light. I find myself sitting in the dark, typing as fast as I possibly can, listening to techno music to drown out distracting sounds. Maybe they're onto something in the films after all.

?

One day, this intensive work comes to an abrupt halt. I've just spotted that, when you place an order, the position your ad appears in our search results isn't always correct. The ranking of the ads doesn't always reflect how much the customers are paying.

Techno music off. Hmm. What’s going on here?

?

What I discover, to my complete and utter horror, is that a feature of our database isn't working properly. It is stopping the ads from being shown correctly when you run a search.

I own a book about this database. It doesn’t help. I post messages on the database’s support forum. They are ignored. It’s an open source database, but the source code is way to complex for me to understand at such short notice.

?

At any other moment in the El Toco project, this kind of road block wouldn't even be worth mentioning. Indeed, we’ve solved hundreds of such issues over the years. But, right now, the investors are going to hang, draw, and decompile me if we end up on another conference call to discuss a new showstopper. Especially on an area of the product that I said was done and dusted eighteen months ago.

Also, this error must be solved, otherwise our advertising service simply doesn't work. All those prior years of effort, everything we've discussed in the previous 25 episodes of this blog, now hangs by a thread.

?

What to do?

I have no idea. In desperation, I discuss it with a friend over a painstaking conversation by text message. She is a scientist, not a developer, but thinking coolly about the problem, which I almost can’t do, comes up with some ideas.

?

All of her ideas might work. The problem is that, knowing what they probably involve, none of them looks doable before our October deadline.

?

One of the database contributors has set up a consultancy in London to help businesses in exactly this sort of situation. It's a cute business model: you work on an open source database, and set up a sideline charging people to help set it up. Unfortunately, their daily rate is approaching what we pay for entire sections of the website. That sort of expense will trigger the investor meeting that I’m so desperately seeking to avoid.

?

This is one of those times when you’re so worked up about a problem it actually helps to stop thinking about it. I spend a long afternoon walking around the parched, dusty cobbles of Lille in midsummer.

?

The plan that emerges from the walk is to go back to basics.

I create a test database with an example advert, to isolate precisely what is going wrong. The behaviour we're seeing is exactly the opposite of what the database documentation says should happen. Maybe it's a bug. If I can find out what triggers it, I can find some way to compensate.

?

After a few days' work, to my immense relief, this turns out to be the escape route from the pickle. I discover, almost by chance, that the error is triggered by a setting which controls how capital letters are handled. The setting determines if cat, CAT and CaT are considered to be different words. Nowhere in the docs does it ever mention this, but if you flick that feature off, suddenly the database behaves properly, and our adverts actually work.

?

The larger the horror, the larger the relief when you escape it. It was a Tuesday but we literally opened a half bottle of champagne that night, to celebrate.

?

Which was nice for a couple of hours. But the following morning, it was straight back to programming in the dark, just like the movies.

?

Light at the end of the seven year black hole

Despite everybody's best efforts, October 1st has arrived and we are not online.

?

Another deadline has sailed merrily past. Unlike with previous delays, the responsibility for this one falls entirely on my shoulders. It is not for lack of trying. It's just too much work.

?

Although nobody is happy about this final hour hold up, we have been able to douse the pain with two things.

?

Firstly, my to-do list for the advertisers’ section is definitely getting shorter now, instead of longer. At the end of every day, there are less items on it than there were in the morning. Based on current progress, I’ll be done in weeks. Done done. The real launch, that we've been waiting for since early 2017, depending on which business plan you're going by.

Secondly, in late August, we switched the search section back on to the public. It is not accompanied by any marketing, so the only traffic is bots, who have presumably picked up the URL from our LinkedIn profile. But, thus far, it has not crashed. Let’s not jinx it by saying any more.

?