Using AI to write top of the line software

"If?the only tool?you have?is a?hammer, it is tempting to treat everything as?if?it were a?nail." ~ Maslow's law of instruments.

Is a popular saying with regards to applying AI into software projects. Have customers complain? Institute an AI to categorize their emotions to get heaps of feedback. Have a big data silo? Apply AI to make sense of it. Don't want to read 3 books before making your final essay? Just have AI write you a summary.

Yes it's true, we have crossed the line. We now have AI as a new hammer that we can apply to a multitude of different day-to-day problems and let it solve things for us. However, next from the ethical debates which are endless in and of themselves. I am most curious about how we can solve problems together with AI. I'm not talking about letting a program run with rules to automaticaly solve issues FOR me, but rather use it's intelligence together with mine to solve issues WITH me. Not only will this continuous exercise broaden understanding of true intelligence in software machines, it will hopefully also speed up productivity and free up time and effort that it can take to solve (complex) challenges.

So what have I done?

We recently saw an opportunity to write a new piece of scheduling software that would connect and integrate between several API's. One API exposes schedules of individuals and another would be able to ingest 'shift' data that it would use to forward and re-route alerting and monitoring features of our expansive landscape to the responsible engineer or analyist on-call.

The first step in engineering new software is to make some decisions and don't worry I won't go too much in to them. There are always some considerations in using a main programming language, an environment, build tools, plugins, deployment environments, secret management and so forth and so forth. Most of these decisions an AI cannot take for you. You will have to learn about them and their sometimes subtle differences.

After I made some rough outlines and sketches about the program outlines I used ChatGPT to write me the large chunks of software. A question as simple as 'Give me an function in Java that calls an API with so and so headers and so and so body. Make it use of x library if you can and omit inner functions and callbacks'

As you can see it did quite well and this isn't suprising for most of you. ChatGPT is quite great at giving you exactly as specified and filling in gaps as it goes. But don't be fooled, the deeper the rabbithole you go the more challenging it becomes to work with it.

For example. I was writing test-cases with it to test if my API calls, headers and JSON body was set correctly. Something I've done before and always struggled with. It's always a pain since dependening on the version, the mock injection of Mockito works differently. Sometimes you don't actually use the right mock and sometimes you just need to find some obscure reason why interactions or agruments are null instead of the values you think you provided.

Writing these test usually costs me between 3 to 5 working days to cover code I wrote in a day, brining a new feature upwards of a week or 2 to deliver. Given the complex work environment and legacy code I didn't look too hars at this number but I was never satisfied with it either. The days felt wasted often just debugging a niche or obscurity. And with the AI tooling it funnlily enough felt no different.

So what did I find?

While the initial answers of chatGPT were astounishing the moment you encounter a bug it's like a dumb child either hyperfocussing on the new information you give it or getting stuck in a loop of self-idolization. Let me explain.

• "Hey chat, here is my class can you write me a test class?" :: `Sure here is one which doesn't actually test anything' - (but I believe in you! You can probably find where to add it! ;))
• "Hey chat, in your previous answer there is an error since I'm getting this nullpointer" :: `Ah yes sorry for the confusion, here is part of the code rewritten using a different framework with a complete new setup missing several importent variables`
• "Hey chat, in my previous answer I asked you to correct yourself and fix the issue with the argumentCaptor not accuratly capturing the StreamInput, you gave the exact same answer as before and I am still having the issue" :: `ah sorry it must have been a problem with the formatting of the response`

These are just some of the issues I had debugging one test. Often when you go on with it too long it devates too much from it's inital, quite amazing, first answer. Starting a new thread and combining certain key learnings from your previous thread might give you the desired result in just one or 2 questions again. I.e. asking it to write a test class for this and this class but with this and this libraries, don't use this, do use custom setters for this and this variable and omit writing your own byte[] method overrides since anonymous inner classes can't easily be mocked.

What I found most curious was that this turned out to be a real conversation and collaboration between man and machine. And even though it had trouble really understanding what it was doing wrong and unable to formulate a working response, it was able to adapt to my feedback and drastically reduce the time I could deploy my code to market. It all took just 1 day to write and just 1 other one to write all the tests.

Sometimes you want to experience writing a piece of sofware in a new(er) language, use a new library or experiment with unfamiliar technologies. And the only place to really toy with it is often just a new project, especially when other team members have experience in one of those. Using AI could be a great help into getting yourself up and running in hours rather than days.

So recommend?

Yeah, I could say every developer that is at least a tad lazy should try it out. However, don't use it to write software that you do not understand or do not want to understand! In my last example, I asked it the following question: `Hey chat, I am using certain API keys that are stored on the remote repository however for my local testing I want to inject these in my code during initialization can you make sure that if the apikeys couldn't get parsed they would get injected from my local .env file?`

And well.. it gave me a nice response, or did it? A bit of a programmer might look at this and wonder, why the OR?

Note that this example is not nessecairy wrong, the code would work and switch using Dotenv when one of the API Keys couldn't be retrieved by the earlier code but it isn't neat either. It assumes that if one key cannot be found the other cannot be found either and then both initializes them in one go.

This code on the right is a fuller, more streamlined answer that is as complete as I would make it. It neatly initializes Dotenv only if it needs to switch to .env and depending on which key is missing it will fill that variable.

Now this example won't blow most of your socks off. It's a syntatical novelty which might just be a prefference from one engineer to another. But when building large pieces of complex software these logical fallacies and strutures could really start to matter. Which begs the question, can we 'trust' the AI to handle our logic for us? To which the answer is probably not.

Even github-co pilot was at times so eager to give me a respond it auto-completed my test case in such a way I'd be integration-testing the third party API with real calls instead of spoofed or mocked ones.

So yes, do use AI but no, don't be too lazy with it. Check it as you would check a junior engineer's merge request. Don't just run it without reading and when you run into exceptions or error's, prone deeper and don't be afraid to start over.

With the help of the AI I was able to write an production-grade application, package and bundle it -using Github actions, write documentation about decisions and diagrams using mermaid and get it fully tested within a week all by myself.