Startup thoughts

May was a big month for our little startup — we went from 2 paying customers a month ago to 6 today. Obviously, by the end of the year we'll have 4,374 customers, right? Simple extrapolation!

That has me thinking philosophically about this weird roller coaster. Some days you feel doomed, some days divinely inspired. It's nice to have had more of the latter, of late ... but I'm sure there will be months that feel the opposite.

Not least because we're trying to do something new, not a better version of something that exists today. Granted, high-performing teams already realize that consensus understanding & alignment is a chronic problem for all software projects, and often already generate regular reports not unlike ours. But doing that automatically, with LLMs? That's new, and therefore somewhat suspicious.

So I'm extremely pleased we've now established, at least to my satisfaction, that there's a real market for it. This was always inescapably logical from first principles. Every output of a software project is either 1) code or 2) communication about code: a Jira ticket, a Notion document, a PR comment, a Slack message, a Figma design, an email, a meeting. Those communications are enormously important — if they weren't, people wouldn't spend so much time and effort on them! — and therefore sifting, analyzing, summarizing, and assessing them is no less important.

But logical and amenable to real people in the real world can be orthogonal, he understated. For decades many software teams have handled all these communications in a haphazard, ad-hoc way, and thought nothing wrong of it ...

... and they have suffered. Oh, how they have suffered. I saw all the failure modes in my consulting years: disconnects, or even different consensus realities, between product and engineering, between execs and managers, between teams, within teams. "Why are we moving so slowly?" "Wait, there's a whole different API?" "What do you mean, this feature isn't a strategic fit?" "You sold what?" "How is that even supposed to work?" "Why can't we stop and refactor before it breaks?" "Why can't I trust my dev team?"

I won't claim a regular cadence of objective (or consistently biased) reports will fix all your problems. But imagine equity investing with no analyst reports; VCs with no deal memos; international affairs with no intelligence dossiers; hiring with no CVs. Imagine making all decisions based on vibes, recollections, and a few data points devoid of context, without any consistent attempts to step back and assess. That's the reality for many software teams.

Our thesis is that just as, really not so long ago, software engineers used to (amazingly) eschew automated tests — table stakes today, of course — automated AI analysis and reporting will become essential for managers and executives.

The "reporting" part isn't so eyebrow-raising. Weekly reports have been a universally popular form factor for centuries, from broadsheets to magazines to equity research to Substacks. (In a way we're simply offering to write a Substack for every project at every company.) This is a natural winner; "projects you work on" or "projects you manage" are, one would hope, subjects of considerable interest to everyone!

It's the automated AI analysis part which is tricky / interesting. A defining feature of LLMs is that it's easy to craft mindblowing demos ... and hard to get them to just work, consistently, week in and week out, against all the vagaries and variations of real-world data. I am very pleased and proud that we've been able to do that. They're still stochastic machines, so we do still get minor peccadillos, or occasional wacky behavior ... but only occasionally, and then we just run an eval, notice the anomaly, re-run the report, and voilà, 99.9% of the time, this time it gets it right.

I don't want to anthropomorphize. That said, (very) metaphorically, LLMs don't want to hallucinate. If you give them good data, they'll use it. And their analysis is surprisingly good! Risk analysis, code analysis, issue estimation — they're much better at all of these things than you might expect, especially as we make them more agentic.

All that said, it's still a roller coaster. The early days of a small startup are a three-sided stool — product, sales, fundraising — and also a constant game of Whack-A-Mole, as you never feel like you have quite enough time for any of those things, much less all three. We're gearing up for real fundraising soon, and I'm already wincing at sacrificing the product & sales time. But it's all part of the journey, they say.

Anyway: if you're interested in what we're doing, want to hear me talk your ears off about LLMs' real-world capabilities and limitations, or are on a similar roller coaster and want to commiserate, please do reach out, here or at [email protected]. We also have a Twitter and a Substack if you want to follow along more distantly. (And I have a separate Substack for bigger-picture AI musings.) Cheers to May, and here's to June—

Jon