Things that drive me nuts about ChatGPT and coding #2343

TLDR

AI cIodegen really does offer a boost to productivity - on certain tasks

But there is very nuanced point of complexity beyond which it will slow you down

AI codegen has the potential to deskill less experienced devs

I suspect, but can't evidence, a significant difference in the quality of AI codegen according to the programming language

There are 3 things you need to do to get the most out of AI codegen

AI CodeGen offers a great boost to productivity - on certain tasks

I am all in - I love me some ChatGPT and Amazon Q is my daily companion in VS Code. I would hazard a guess that I get a good 20% productivity boost as a result of using AI codegen for back-end dev (which in my case is mostly Python, Node, or Go) and it gives me a goodly 50% boost on front-end dev (React). The reason for the difference, is that I'm way stronger where it comes to b/e dev than the front-end and also that writing good back-end code is as much about thinking time as it is coding time.

Where it comes to back-end dev, I use it extensively for housekeeping/mundane code -

Say I have a struct -

type Organisation struct {
	ID uuid.UUID
	Name string
	Lat float64
	Lon float64
	SectorCode uuid.UUID
	Parent uuid.UUID
	PrimaryContact uuid.UUID
}        

And I plan to marshal it into JSON... Well we should play nicely with our macchiato-slurping-avocado-on-toast-scoffing millenial F/E brothers and sisters and format our JSON nicely so, we ask ChatGPT to add JSON tags to it, and boom...

type Organisation struct {
	ID                      uuid.UUID `json:"id"`
	Name                string    `json:"name"`
	Lat                     float64   `json:"latitude"`
	Lon                    float64   `json:"longitude"`
	SectorCode        uuid.UUID `json:"sector_code"`
	Parent                uuid.UUID `json:"parent"`
	PrimaryContact  uuid.UUID `json:"primary_contact"`
}         

Is this something that would have tested me?

No (although it usually takes me a moment to remmber where the "`" key is) - but this is way quicker than typing all those tags out myself.

When it comes to reformatting data, stuffing things into files, and even basic Postgres Queries - AI codegen chucks out pretty decent, reliable code.

It even produced a pretty nifty Haversine function (for calculating the distance between two lat-lng co-ordinates); but then let itself down by producing pretty poor test data when I thoguht I'd be lazy and get it to write the tests for me.

But there's a very nuanced point of complexity beyond which it will slow you down

Here's an example of something that in my earlier days might have cost me many hours. I'm chatting with ChatGPT over a rate limiting middleware function, and it chucks out the following code...

var visitors = make(map[string]*rateLimiter) 
var mu sync.Mutex 

func (s myServer) addVisitor(ip string) rate.Limiter { 
limiter := rate.NewLimiter(s.rateLimit, s.burstLimit) 
mu.Lock() 
defer mu.Unlock() 
visitors[ip] = &rateLimiter{limiter, time.Now()} 
return limiter 
} 

func (s myServer) getVisitor(ip string) rate.Limiter {
 mu.Lock() 
defer mu.Unlock() 
v, exists := visitors[ip] 
if !exists { return s.addVisitor(ip) } 
v.lastSeen = time.Now() 
return v.limiter 
}        

When a visitor arrives, we call getVisitor, with their IP address. We look for that IP address in our visitor map. If it's there, we return the limiter that relates to it, if it's not there we add it to our visitor map and then return the limiter that we've created.

ChatGPT is cunning enough to know that we need to add some syncronisation - so we are delighted to see a Mutex rock up.

I suspect that the really eagle eyed reader will have already clocked the problem - and to be fair to little old me, it was the first thing I spotted when my server just froze.

But... that's because I learned a long time ago that the presence of mutexes invites calamity (the joys of writing firmware for IOT devices... good times).

What actually happens is that the getVisitor function grabs a lock - and won't let that lock go till it's done. If it calls the addVisitor function that then tries to grab the same lock - but it's locked, so it waits. Deadlock.

So I edited the code...

func (s *myServer) getVisitor(ip string) *rate.Limiter {
	fmt.Println("get visitor")
	mu.Lock()
	defer mu.Unlock()
	v, exists := visitors[ip]
	fmt.Println(exists)
	if !exists {
		newVisitor := &rateLimiter{rate.NewLimiter(s.rateLimit, s.burstLimit),                     time.Now()}
		visitors[ip] = newVisitor
		return newVisitor.limiter
	}
	v.lastSeen = time.Now()

	return v.limiter
}
        

I looked at the addVisitor function and concluded that, since it's basically only doing one little thing I could just fold that into the getVisitor function and avoid the deadlock drama altogether.The other alternative would have been to do the unlocking by hand rather than just defering it - And if I could see a good reason for adding a new visitor outside the getVisitor function I might have done that - but in this case we're dandy like this.

It didn't occur to me to take this up with ChatGPT till I'd settled on this solution - but then, out of curiosity I posed the question... and here's where it turns into an eyeball roller.

I asked it to review its own code, and it praised itself for using "defer" to unlock the mutex saying:

You lock the mutex and defer unlocking, which is correct.

And then hits me with...

Deadlock Risk: There is a risk of deadlock if addVisitor is called within the lock held by getVisitor. If addVisitor needs to lock again, it will cause a deadlock because getVisitor already holds the lock. This is the primary issue in your code.

The proposed solution was to just remove the locking from the addVisitor function... which is TERRIBLE advice. But at least it sort of found the issue.

AI codegen has the potential to deskill less experienced devs

So I'm coming to the conclusion that for newcomers, AI codegen might be a bit of a mixed blessing; it creates an appearance of speed and productivity that can easily turn out to be an illusion. It's not that it's useless - but it does create pitfalls which can often be properly gnarly to unpick.

That said, I've seen really new developers do some pretty cool stuff with it - and I use it a lot for the inital scaffolding of react or node projects (and I'm a comparativ noob at both) but I'm coming to both react and node with quite a few years (and plenty of screw-ups) under my belt.

While I don't want to sound like an "old timer" - there's a lot to be said for experience gained fighting with freeRTOS, or having to follow a dozen tutorials. So, much like the cranky woodworking teacher who forced me learn how to make a mortice joint with a chisel before letting me loose on the fabulous morticing machine, I think that there's a core of expertise that needs to be build up before AI codegen really delivers the goods.

I suspect, but can't evidence, a significant difference in the quality of AI codegen according to the programming language

I can't prove this - but I do have a feeling; I have a growing impression that chatGPT is better at some programming languages than others. I think this is a function of the volume of code that's out there (and much of it comes from stack overflow, I am sure) and the aggregate quality of that code. If you have a very popular programming language - javascript, there will be lots and lots of code out there, but a goodly proportion of it is a bit... well.... rubbish. Whereas the GoLang code that I'm served up feels as if it's a bit more reliable - And I wonder if thats a function of the quality of the code examples that exist?

Three things to get the most out of AI codegen...

Develop your prompt-fu

Writing very clear prompts, and offering up samples of your code really helps. If the output isn't right, try again with a different prompt.

Bully the AI!

I am always polite to ChatGPT, in part because yanno... when it becomes our overlord, it'll have recorded the names of people who were unkind to it (This is also why Iwould never ever be caught on camera knicking an AI powered robot dog over). But you can be insistent. Bizarrely, sometimes just asking "Is there not a better way to implement that?" will produce a response along the lines of "Oh, of course! Here is a much better way to implement this function!"

Focus on using AI to do the easy but tedious stuff so you can deal with the gnarly problems

I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes.

I believe this quote is from Joanna Maciejewska (@AuthorJMac on the twitter) and it nails it.

Let's face it, nearly 90% of all code is doing trivially simple things that are obvious and easy to implement - It's fetching something from a database, adding sales tax to an order line... this is definitively not what gets me out of bed as a developer.

That 10%... about optimising complex algorithms, thinking about how to partition data for scale, worrying about the impact of future changes to the business logic and how you might protect your future self from a load of refactoring... that's the "art" in sofrware engineering, and as long as AI codegen is grabbing it's stuff from stack overflow, I promise that these problems will be better solved by humans for the foreseeable future.