GPT-3 reviewed ProdPad. Here’s what it said ??

If you haven’t heard, there’s a new kid in town. GPT-3 is the AI that’s blowing the minds of many, and seems to be able to take anything you throw at it. 

I first saw GPT-3 as this ‘natural language to code’ demo, and at first, assumed it was a well curated demo of something that definitely couldn’t do half of what it was implying. I then started seeing some other fascinating examples coming through, and I’ll admit, I was intrigued. I mean, check out this GPT-3 Figma plugin example that turns words into editable website designs, or what it can do with legalese. 

I don’t have access to GPT-3 myself, but one of our customers happened to be playing with it. I posed a question of what applications could GPT-3 (or its evolutions) have in ProdPad? After all, a lot of the work product people do is highly creative and human-centric… though some days, when you’re slogging through user stories or writing acceptance criteria, it can get a bit dull. Maybe AI can help ?? (after all, we already have DotBot, who’s an absolute star player when it comes to helping to deduplicate your backlog!)

Adam Wintle went ahead and posed a challenge to GPT-3 (in bold italics below), giving it just a taster of ProdPad, and it came back with the following:

Apparently the conversant tone, including the follow up questions, are GPT-3’s own, choosing to ask and then answer itself in an interview style.

I grinned ear to ear as I read this, loving how ingenious it was, while conjuring up a million questions about how it pulled it off. What magic! 

Now, GPT-3 missed its first assertion by a bit, which made me laugh out loud on the spot - first of all, it didn’t have access to see our user interface, so what does it know?! And second of all, our user interface is ?? - we get tons of love from our customers on the care and attention we put into it. But I’ll come back to this point, as it’s important.

The second part about the free trial, was actually really insightful! GPT-3 wasn’t wrong, we probably would benefit from giving a more in-depth free trial period to let people get the hang of ProdPad. What GPT-3 doesn’t know is that we just launched the Sandbox environment to solve exactly this problem, a free explore-on-your-own version of ProdPad that’s preloaded with example data to really give people a feel for how ProdPad fits together and with other tools. Here’s the Sandbox now: https://sandbox.prodpad.com/ 

GPT-3 probably also doesn’t know that we also have a unique magically extending trial period, meaning you can try ProdPad as long as you like, and that you actually earn trial time by using the product. Here’s a review we came across the other day of Peter Sharkey exploring our onboarding and discovering this aspect for themselves: https://www.dhirubhai.net/feed/update/urn:li:activity:6694532194976702464

I also love that GPT-3 spotted a couple other tools in the same space as alternatives - Github and Trello were both called out as good tools for organizing your products, though I’d like to point out to GPT-3 that product management is more than just managing lists of features and dev tickets ??  

Good thing we have integrations with Github and Trello (and many more like them!) so you get the best of your product strategy/discovery in ProdPad, and delivery management in the tool of your choice.

Is GPT-3 intelligent?

While the above looks and reads like something a human might write, I wasn’t fooled so easily. 

As Francis Jervis pointed out, it wasn’t so much that it was nailing the content, but that it was generating the most probable content based on the instructions. He said:

If you prompt it to create a list of the most common complaints about products in this category, I'd bet these would be top 3. 

He’s not wrong. Ask any SaaS founder or product person what they’re working on or where a sore spot in their app is, and it almost always has to do with the interface or the trial period. There’s entire ecosystems built around optimizing both of these things! 

My bet is that GPT-3’s output is about as good as a horoscope. Here’s mine today:

Catch the wave of activity that's spinning in your direction today, Taurus. You'll find prosperity and good fortune within your reach. Expand your horizons and connect with others intellectually. Hurdles will shrink to nothing as long as you maintain a positive attitude. It will be tremendously easy for you to make great strides in any project you're working on.

While this was meant to speak to the Taurus’s of the world, I bet that this jumble of words resonates with a good chunk of the population, regardless of where the stars were when they popped into this world.

Likewise, GPT-3 output is sufficiently generic that it speaks to whoever reads it, and yet under scrutiny, it doesn’t really say anything insightful at all.

I shared my thoughts with our Tech Lead, Will Newmarch, who’s also got his Masters in AI, and is known to roll his eyes when people get into debates about whether AI is ‘intelligent’ or not. He’s got good reason. As he pointed out to me this morning:

There are something like 70 recognised and heavily referenced definitions of intelligence, it’s not something we’ve ever agreed upon.

If we don’t know or agree on what makes a human intelligent, how are we supposed to judge if some code passes the bar? 

I’m not going to dive deep into this topic, but I will share an insightful thread from Simon DeDeo on GPT-3 and intelligence.  I love this take (in depth article from MIT Technology review) too, which posits that we’re not really getting anywhere with AI if we aren’t agreed on the goals or which problems we’re aiming to solve. As a bonus, that article links off to another about giving GPT-3 a Turing Test, with hilarious results:

Essentially, it seems that GPT-3 is amazing at synthesizing concepts and putting them together in unique ways. In many ways, that’s what a lot of us knowledge workers have spent a lot of our time doing over our careers. I can see this having an interesting impact on content writers, legal and technical writers, and even some elements of product management, in time. That is pretty clever. 

But it’s not human and we should be careful allowing ourselves to be tricked into thinking it is. It might be able to generate something that looks and feels like feedback on a product, but it’s not representative of what a human actually thought about the product; just a good guess at what a person might say if asked to give feedback about a product. It lacks the context of the emotion and behaviour, and until those are corrected for, will produce misleading results.

Will Newmarch summed it up:

GPT-3 is impressive, but it’s like saying that a magician is clever for ‘teleporting’ themselves across the stage. They can’t really teleport, they’re just tricking you into thinking they did.

In the meantime, I’m a sucker for a good party trick, so show me your favourite applications of GPT-3 or other clever-looking AI’s, or leave ideas about how AI will change product management, in the comments ??