A.I. is not trustworthy—yet.

Tech. enthusiasts continue to going viral on social media sites like LinkedIn and Youtube after GPT 3.0 was released in November of 2022. Nearly two years later, a common proclamation persists:

"A.I. is going to replace you."

While the claim has caused some concerns amongst professionals and creators, is the replacement imminent?

If you're entering or already in mid-life crisis mode, you have likely had to do many things from the ground up. Figuring things out with little to no resources, though sometimes inefficient, has helped me grow and become more self-sufficient. So, when I started experimenting with tools like OpenAI , Perplexity , and MindStudio, I was looking for technology that could quickly perform and/or automate these repeatable/boring tasks.

After testing most of the publicly available platforms out there, I have concluded that while adoption of these tools has increased exponentially, the accuracy and reliability of the A.I. is still in the "Intern" phase.

What do I mean by that?

Have you ever been asked to allow a new hire to shadow you at a job you've been at for a while? Someone fresh out of college or interning? There is no doubt that the recent hire is intelligent but they know absolutely nothing about the intricacies of your specific business.

Simply put, though full of knowledge (like ChatGPT), they lack context. This is where most of the A.I. tools fall short. Tons of information, very little wisdom.

Let me give you a real life example.

Recently, I was researching an organization American Independent Medical Practice Association to gain a better understanding of their interests and how they fit into the healthcare market. I learned that they submitted a compelling response to a Request for Information from the DOJ on the topic of consolidation in the health care market. If you're in healthcare, I'd recommend reading it here. As I learned more about the function of MSO's, I wanted to know what other people said in response to the RFI. To my shock, over 2,000 comments were received and available to review on regulations.gov. I began to wonder, what was the overall consensus amongst commentors? Are people happy about the consolidation in healthcare or unhappy? Who is responding to this RFI? Companies, patients, or healthcare workers? The weirdo in me just had to know, but that guy also has his actual job to do (hint: I don't get paid to analyze data).

I was left with a dilemma. I needed to know more about these comments, but I didn't have the time to analyze the data. Until, I remembered one of those A.I. hype men here.

In his video Skill Leap AI demonstrates how great Chat GPT4o is at data analysis. Previously, I used several of the GPT's in combination with 3rd party apps to review PDF's and .csv's but found that they were inaccurate and became easily confused. I took my list of comments from regulations.gov and after confirming with GPT4o that this was possible, I got to work

Here was the first result after asking for it to analyze for common comment themes (not word pictures) :

Though I didn't ask for an image, I figured it would be best to specifically state what I was looking for and so I asked, "Can you categorize these themes?"

While it made a solid attempt, it simply supplied me with a list of 5 buzzword titles and random SEO-like words under each "theme". It was not grasping the concept of reducing data into themes. I decided to press it a little further to see if maybe it could tell me how many comments existed under each supposed theme...but...another word salad:

At this point, other than the fact that this .csv contained words of a relevant nature to the subject, I still had no idea of what the overall public sentiment was for this RFI.

But I wasn't ready to quit yet.

I forgot that A.I. knows what you're feeling by evaluating your tone of voice and/or writing. I remember some influencer/A.I. peddler saying that once (or a thousand times). In fact, entire companies have been built around this. Aviso AI is one that I've used in the past and their sentiment analysis is ...well... sad.

At first glance, the tool obeys after every prompt and provides visually appealing charts/answers that seem relevant.

However, once I dug into the chart details, I realized many positive comments were marked as negative and visa versa. GPT4o is a liar.

So it makes a few mistakes.

Here's the point:

If a plain language model cannot understand what I'm asking for in plain language, why are so many people putting an emphasis on its ability to replace the full functions of a working human being? It's been nearly 2 years since the hype began, with several enhancements to existing models. If I can't trust it to count the columns on my Excel sheet, how can I trust it to ingest and understand thousands of lines of data?

With that said, I still love these tools.

Are there use-cases for them? Absolutely. Will they replace jobs? Probably. Will they do these kinds of jobs better than an experienced human and with less error? Probably not.

Prediction: We're going to need some human data checkers and fast.

Bonus example of a simple prompt failing where an individual excels:

If you enjoyed this article, let me know and I'll start creating more.