Becoming agile

Episode 23: 19/11/2022

?

If you spend long enough trying to gain mastery over a subject, there eventually comes the moment where it clicks.

?

It can take a very long time. Something really worth striving for will probably need years. But, regardless of whether it's sports, cooking, or charcoal sketching, you always get there in the end.

?

When it does click, the realisation that it has seems to be almost subconscious, at least at first.

The subconscious brain says, hold on a second. If my opponent hits the ball here, then I hit the ball there, then he is forced to do that, which I can counter with this, and I’ve won. And, I can do this time and time again. Because, somehow all of a sudden, I always know what to do.

?

?

It would be nice to say that, in 2022, the El Toco team mastered website creation. Being strictly objective, I don’t think we’re there yet. There are still some things on the website which a more experienced team would have got right on the first pass.

?

What we have done, however, is mastered the process. The increase in productivity is like going from night to day. It is the payoff after years of trial, error, and numberless things going wrong.

?

In November 2022, El Toco's search website rolled off the production line. What took years to do previously, we completely rebuilt in a mere handful of months. Today we will review how we did it.

?

?

The W team

Wait. What!? We're delaying our launch again, to revamp the website again!?

Yes, we are, for two reasons. Firstly, with our new focus on medical and scientific equipment, the bright, playful graphic design feels out of place.

Secondly, I looked at a lot of other startups during our fundraising round in summer 2022. The one thing that really stood out is that their websites were better.

?

People subconsciously spot errors in a product when judging its quality. As explained in the enjoyable book Car Guys vs. Bean Counters, cars feel low quality when the bodywork has gaps or doesn't align properly.

?

Websites feel low quality if the spacing or size of the elements is wrong. On many of our pages, the scale was just somehow... off. Despite being a vast improvement on the earlier iterations, it still wasn't good enough.

It was a sad day when this sunk in, but that's life so let's get on with sorting it out.

?

My one friend who actually works in tech suggested, politely but firmly, that now would be the time to get some seasoned user experience designers. These are not graphic designers, and focus more on the user's interaction with the product. After casting around for a few months, the search bore fruit in the form of Piba Studio , our new design team.

By pure chance, I found them just as they were striking out on their own. They'd created a dummy portfolio of imaginary work on the artists' website Behance and it was clear just from that they knew what they were doing. El Toco is their first real client. The mishaps during our previous rounds of web development were myriad but, had any of them not taken place, I would have never found Piba Studio, because they didn’t exist yet.

?

A particularly serious run of those mishaps took place earlier in 2022. That culminated in the hiring of our new web developer, Alex Cachia [[Ka-Key-A]] .

We glossed over Alex's first project for El Toco because that blog entry was getting too long. The summary is that she picked up the mess made by the previous developers and straightened out the website in short order.

?

As a recruitment test for a software developer, you couldn’t have come up with a better one than the mess our website was in at the time. If you’ve ever had the unfortunate job of taking ownership of a spreadsheet made by somebody else, firstly I feel for you. Secondly, taking over somebody else’s programming code is exactly like that. Only worse. In our case, it was a pie that four separate people had their fingers in at various points in time, so the code was a real mixture of styles, which made it especially tough.

So it was impressive that Alex cut through it like a knife through spaghetti code, wrapping up the whole thing in a fortnight. I made a mental note that there was somebody to keep hold of.

With Alex and las Pibas on board we have a good team but the meat of today's episode concerns how we actually do the work.

How agile development helped us

The philosophy of agile development grew out of project managers trying to correct the age old problem of software being delivered late.

In some other sectors, people can get it right. For example the arts. Ballerinas do not cancel shows at the last minute because they need a bit more practice. Meanwhile, in IT-land, it seemed during the 1990s to be almost impossible for any software project to be delivered on time.

The proposed solution to this is a project management philosophy called agile development. It came of age in about the year 2000. Since then, it has become increasingly dogmatic, and it is worth noting in passing that it is not totally clear whether it has definitively solved the problem of slippage on IT projects.

If you aren't interested in video games like Cyberpunk, which are still delayed all the time, consider Birmingham City Council. They went bankrupt in 2023. Among the contributing factors was the epically bad decision to move some part of their admin system onto a cloud platform hosted by Oracle. Not the whole council, mind you, just some of their admin stuff. As IT projects go, it's a solid case study in how not to do them, the cost of the program jumping from ￡20m to ￡80-100m, depending on which sources you believe.

What really went on in Birmingham council's doomed IT project would make an interesting book, even more so than El Toco's storied past up to now.

The point being, agile development is not necessarily a panacea. In our case though it has been very useful.

Earlier in 2022, I started experimenting with the web-based project management tool Jira. This, already was a sea change. By having our to do list in Jira, the problem of not knowing what on earth everybody was doing was immediately and permanently put to bed. No more email chasers. You just check whatever is "In Progress" on Jira.

With this round of work we've gone further, by taking as many leaves out of the agile development bible as we can, given our tiny team.

The first one is the concept of sprints. We break the work down into self-contained mini projects. This sounds obvious, but in IT projects it is often not obvious. People try to produce the whole shebang in one go. I know they do, because this is how some of our previous contractors tried to work. The result is chaos. You get bogged down in the treacly details, unable to escape, and you are then easy picking for the bugs, which swarm all over and finish you off.

In the parlance of agile development, these mini projects are called sprints. By breaking the work down into sprints, it is immediately compartmentalised. So instead of thinking "Argh there's too many details! How are we ever going to get it done!?", you concentrate on what's immediately in front of you. Everything else is forgotten, because it's not part of the sprint.

You can take that logic and apply it again, by nesting the sprints inside each other. For the search section of the website, we had a design sprint and a development sprint. Within those sprints were smaller child sprints, focused on different features that we wanted to finish.

At the end of each sprint, I swoop in to provide feedback. In the past, all the ways that we handled feedback were clunky. With one contractor, we had some dedicated bug tracking software, which sounds great but was so badly designed that took longer to file bugs then just sending them by email. Emails, which we've also tried, probably sounded awesome in 1985. But they take ages to write politely, and then have to be translated into a todo list somewhere else, which creates more work. Previously in this project, things often degenerated to the point where I found myself doing the bug tracking manually in Excel. And that's when you're really in trouble.

We've binned all those feedback methods in favour of two. For long, woolly stuff, we record videos. Videos mean everybody can watch the feedback whenever they like. They can replay it, pause it, slow it down, and a host of other things that you can't do with a physical meeting. For concrete bugs, we file a report in Jira. This lets us prioritise bugs, but also turn them into todo items, for the next sprint.

By making sure that the testing happens during each sprint, we also avoid annoying the developer. Another bad thing that used to happen is we'd wait until the developer did a pass through the whole project before doing any testing. I'd then do a monster round of testing and dump a huge number of bugs on the developer, some of which went right back to the design phase. It was a real morale killer for the developer, because it felt like snatching defeat from the jaws of victory. By testing continuously and giving the feedback in bite-sized bits, each sprint feels like it progresses more slowly but when development wraps up, so has testing, so you're ready for release.

?

Those are the main things which have helped us. There are more details, but you get the idea. Things are way more efficient now. The improvement in productivity speaks for itself:

?

?

The actual output also feels like a big step in the right direction:

If this all sounds too self-congratulatory, don’t worry. This is still an El Toco blog. Something will always be going wrong, somewhere.

?

Actually, we've got two things going wrong. One is the snap decision to accept the drastically new look which the designers have given the search section. The implications of this are major. Now, we have to revamp the advertising section before launch. That was not in the business plan approved by the investors. Uh-oh! We'll see how that pans out in future episodes.

?

The other thing going wrong is the crawler.

?

?

Crawling in my skin

?

While the website team does an excellent job, I’ve been stuck in a nightmare wrestling match, with El Toco's crawler.

?

El Toco’s crawler is the program which flits from website to website, hurriedly downloading all the text. It is one of the biggest building bits we've actually had to make ourselves, taking several thousand hours between 2016 and 2020.

?

Now, with the move to medical and scientific equipment, the crawler can't just go around collecting any old websites. They've got to be relevant websites. This is more subtle than it sounds. It’s not going to be much cop if you’re looking for a stethoscope and it brings back a plastic one from the Medical School Barbie Accessory Pack.

?

Thus a whole new can of worms is cracked open, leaving me on my hands and knees scrabbling round to pick them up off the kitchen floor.

It's mainly tough because whatever solution has to be totally automated. It will be too slow to tell the crawler which pages to visit manually. I’m having to write some new logic to govern this, which takes time.

?

It takes so much time, in fact, that it’s blown our latest launch plan. That sentence is so commonplace in this blog it’s a bit like saying “and then I had breakfast and brushed my teeth”. But here we are again.

It wasn't supposed to be like this. With the impetus from our friends and family funding round, we had a new launch date of early November. The website team executed perfectly and revamped the whole search section in time for this. But the crawling has gone and blown it, so much so that the launch has had to be postponed indefinitely, until I can sort it out.

?

Morale has, however, been boosted by one small nugget of news. Later analysis will reveal that it is actually the harbinger of a much bigger tech shock headed our way, but initially the news sounded like a point in El Toco's favour.

?

Reports have started surfacing that owners of independent websites are getting really cheesed off with Google's crawling. Ironically, while I’ve been setting up a fully automated system for El Toco’s crawlers, Google, it turns out, had a whole team of people handling this very job. It's unclear whether they were just auditing the results, or literally telling the crawlers which websites to visit. But that's a moot point, because apparently they made a lot of them redundant. The website owners are getting cheesed off because now, when they update things, Google is still showing the old version of the website.

?

It is welcome news. Our hilariously tiny startup, managed from the corner of a living room in France, is once again bumping into and solving problems which our much larger competitors are also wrestling with.

But time will show that actually there is probably more going on here than meets the eye. Little do we know it, but ChatGPT and its contemporaries are just over the horizon.

?

Finding purpose

?

While fortunately not a big tourist destination, Lille is a big city by European standards. Including all the suburbs, it has a population of about 1.5 million. In UK terms, this is bigger than anywhere but London. Lille is roughly what you get if you add Birmingham to Bristol and subtract one bankrupt city council.

?

As a big city, Lille hosts big events. A highlight with our family is the independent wineries exhibition. It takes place every November and includes unlimited free samples. I can confirm to all those students on the perpetual search for the cheapest way to get absolutely trolleyed, this is that way.

?

Roughly five hundred independent wineries descend on Lille, setting up small tables in the vast exhibition hall. For the entrance fee of six euros, you get a glass and permission to wander around the room, tasting all the wines.

?

The room is packed with people who have a similar goal. You don’t buy wine by the bottle here. You buy it by the crate. People bring sack trucks, patiently wheeling them in and out of the crowds.

?

All the tipsy activity is particularly fascinating if you are an economist. There are very few such events in the world, where you can see the supply side of a large market gathered together under one roof. What’s really impressive is that these isn't really the supply side. These are just the independent guys. Big wineries don’t come to this exhibition, it's too small. It really makes you realise how large the wine industry is in France.

?

A question I am often asked about El Toco is why we are bothering, when everybody just uses Google. Surely Google is so ubiquitous that it's pointless to compete. Surely their market position is so cemented they're impossible to dislodge.

?

The answer to that question is here, in the wine hall. These independent wineries are not all striving to drive each other out of business and become the world's only dominant winery. Some of them rise, some of them fall, but in general, they’re happy making their wines, they do it well, and people enjoy drinking them.

?

There is a presumption about startups, particularly in tech, that they are all trying to reshape some industry from the ground up in order to turn themselves into a monopolist. Sometimes, they are.

But often, it’s a good enough goal just to create something, with care and attention, that other people will use. Watching the new website team as they work, we can finally see this taking shape.

?

?

?

This episode is dedicated to El Toco’s website team, for creating a product with care and attention. Just don’t get too comfy. We’re not done yet!