When AWS Charges Turn Into LLM Challenges (and Solutions)

It all started with an AWS bill and a simple wish: to connect to any application or tool, get the data I needed, and not have to fully understand how it worked.

AWS was charging me $0.02 per day for a service I wasn’t actively using, and I wanted a way to monitor what was going on without having to log in and navigate through the AWS billing console every time.

I was imagining the possibility of simply telling a Large Language Model (LLM) what I wanted and having it just do it for me. Sounds enticing, right? I could think of a few horror stories about handing over the keys like that, but the idea was just too tempting to ignore.

What came next was the creation of an app that does exactly that — dynamically executes Python code generated by an LLM based on user input. Turns out, this was a much, much bigger bite to chew than I’d expected. ??

Creating an application that runs code generated outside of it opens up a Pandora’s box of challenges. And the biggest one? Security! Unless you own your AI and know exactly what you’re doing, this is an area with big flashing warning signs. Even if you do know what you’re doing, it’s still risky.

What felt like a breeze in Jupyter Notebook turned into an almost insurmountable task in building my own executor app. I faced challenges like:

• Dynamic requirements: The generated code needed libraries and connection variables that my app didn’t know about in advance. These had to be installed and populated dynamically at runtime.
• Unpredictable LLM output: While OpenAI gpt-4o and o1-preview consistently generated Python code that was clearly delimited from the rest of the output, o1-mini suddenly decided to add something completely unrelated. Unfortunately LLM output consistency can’t be trusted.
• Redirecting execution outputs: The biggest challenge of all was managing the execution outputs between my app’s own code and the dynamically executed code.

Honestly, if not for OpenAI o1-preview, I wouldn’t have made it. It analysed and debugged code in a way that made gpt-4o feel like talking to a child. But beware - it comes with a hefty price tag!

Here’s the kicker: while I finally resolved my SageMaker issue and stopped the $0.02 charges, I discovered that my quest to avoid logging into AWS came with a cost of its own. Turns out that the Cost Explorer API requests I used to track my charges are billable. Looks like I’m still donating to AWS ??

If you’re curious, you can check out my app on GitHub. It’s functional, and you can even use it—provided you know what you’re doing! Better yet, you can help improve it because there’s a lot of potential there.

https://github.com/jenyss/OneConnection

My app isn't quite granting special wishes just yet, as you can see below - so for now, please stick to connecting with tools here on Earth ??

As for me, I’m moving on to a topic much closer to my heart: Product Discovery and how to leverage LLMs to help with it. Stay tuned!

And finally, the reason I’ll keep writing and sharing my stories is that I believe only through experimenting and understanding can we truly uncover the potential of AI and find our own paths forward.

And it’s only through those explorations that we can take on the biggest challenges—and maybe, just maybe, figure out how to start solving them.

Imagine what could be achieved if you have a bigger motivation than $0.02 ??