Most Chatbot Suck (And It’s Not Because of the LLM)

It’s easy to plug in an API and get a chatbot to respond. In fact with the level of abstraction available today – bots/RAG systems can be created at a very fast pace.

?Example being:

?response = openai.ChatCompletion.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)        

?Boom – here is your working chatbot!

But this is where the troubles start - this naive implementation won’t attract any user adoption.??

Imagine talking to someone who tend to forget everything you say the moment they respond. That’s exactly what happens with most chatbots today? - they give you and answer and go to la-la-land!????

A few examples...

??Dumbest lot: Chatbots without any memory:

Have you ever encountered this?

• User: "I need help tracking my order #12345."
• Bot: "Sure, what’s your order number?"
• User: "I already mentioned it—12345!"
• Bot: "I'm sorry, can you repeat that?"

Result: Instant rage quit. ??

?? What went wrong?

No concept of memory/history at all.

??Slightly "intelligent" ones: Naive memory management:

Some developers try to fix above issue by simply storing conversation history in an array or even a basic database.

Here is what they do:

?chat_history = []
def add_message(role, message):
    chat_history.append(f"{role}: {message}")

# When a new message comes in:
add_message("user", "What's the weather like?")
add_message("assistant", "It's sunny!")        

Here, we’re just appending messages to a list - seems fine, right? Wrong. This breaks as soon as you introduce sessions.

Naively storing messages in a list for a single session.

At a glance, this seems sufficient. However, Here’s what happens next:

?

??Too Much Memory without session Management:

With a simple list - every time you switch the session, the same conversation will be appended again. Confusing and unprofessional.

?...an example

? User: "My WiFi keeps disconnecting."

? Bot: "Have you tried restarting your router?"

? User: "Yes, I already did."

? Bot: "Try restarting your router. That usually fixes most issues!"

? User: "I JUST TOLD YOU I DID!"

? (Frustrated, the user closes the chat, comes back an hour later...)

? User: "My WiFi is still unstable."

? Bot: "Previously, you mentioned that you restarted your router. Let me suggest something else..."

? (So far, so good... but then:)

? Bot: "However, earlier, you also asked about slow speeds. Would you like help with that too?"

? User: "I NEVER ASKED ABOUT SLOW SPEEDS!"

? Bot: "Here’s a summary of our past interactions:

1. Restart router ?

2. Slow speeds ?

3. Connection dropping ?

4. Unplug modem ?

5. Factory reset ?

6. Contact ISP ?

Would you like to continue troubleshooting?"

? User: "What is this, a test? Just help me fix my WiFi!"

Result: Keyboard slam detected. ????

?

?? What went wrong?

• The chatbot over-remembers past issues instead of focusing on the current conversation.
• It retrieves unnecessary past context (the "slow speeds" issue was never part of this session).
• The structured memory retrieval confuses the user instead of helping.
• It doesn’t distinguish between resolved and unresolved issues, making the conversation frustrating.

??? Failing to Maintain Sessions’ longterm history: The Half-Memory Chatbot

A chatbot that remembers part of the context but fails to maintain continuity across sessions.

Scenario: Travel Booking Chatbot (Session Inconsistency)

A user books a flight but comes back later to modify it. The chatbot doesn’t properly link past and current interactions.

? User: "I need to book a flight from NYC to LA for March 10."

? Bot: "Got it! Your flight is confirmed for March 10 from NYC to LA. Do you need a hotel as well?"

? User: "Not now, thanks!"

? (Later, the user returns to modify their flight.)

? User: "I need to change my flight to March 12."

? Bot: "I see you have an upcoming trip from NYC to LA. What would you like to do?"

? User: "Change it to March 12."

? Bot: "Would you like to book a hotel for this trip?"

? User: "NO, I just want to change the date!"

? Bot: "I can assist with that! What date would you like to fly?"

? User: "March 12. I literally just said that."

? Bot: "Okay, let me confirm your details… What’s your current flight date?"

? User: "You already have that info! It's March 10!"

? Bot: "I’m sorry, I need you to provide your current flight details before making changes."

What went wrong?

• The chatbot recognizes the booking but doesn’t link previous user intents (changing the date).
• Instead of acting proactively, it forces the user to repeat details it already knows.
• The chatbot remembers the trip but not the right part of the conversation—failing to link the change request to the original booking.
• The conversation feels robotic, forcing unnecessary verifications that a human wouldn’t need.

?

Example 3: The Phantom Session Problem

Issue: Wrong session retrieval leading to mixed-up conversations. Scenario: A healthcare chatbot that allows patients to check their prescriptions and request refills, but session mismanagement leads to cross-session confusion.

Interaction:

• User: "Can you show me my active prescriptions?"
• Bot: "Sure! You have an active prescription for Amoxicillin."
• (User leaves and returns later.)
• User: "I'd like to refill my medication."
• Bot: "Got it! Refilling your prescription for Ibuprofen."
• User: "Wait… I never asked for Ibuprofen. I need Amoxicillin."
• Bot: "Your last recorded prescription was for Ibuprofen."
• User: "NO! That must be someone else's!"
• (The user is now panicking, thinking their medical data has been mixed up!)

Result: Immediate uninstall. ??

?? What went wrong?

• The chatbot retrieved the wrong past session or mixed past conversations together.
• It falsely associated the user's intent with an unrelated prescription.
• Instead of instilling confidence, it destroyed trust—critical in sensitive areas like healthcare or finance.

?

?

??The above examples are enough to show case ?why even Naive memory Fails

Developers might think that simply storing the chat history is enough to give the bot memory. But here's why even a basic approach falls short:

• Context Overload: A chatbot that attempts to consider the entire conversation history will mix up current context with irrelevant past details, leading to responses that are either vague or off-target.
• Duplicate Data: When session history is naively stored without proper state management, switching sessions can inadvertently create duplicate entries, cluttering the UI and confusing users.
• Scalability Issues: Simple in-memory lists don’t scale well. As conversation history grows, the system slows down, and fetching the right context becomes challenging.
• User Experience: Ultimately, a user doesn’t need to see every single message; they need a bot that understands what’s relevant now. Naive memory doesn’t filter out noise - it simply repeats everything.

?

??What’s Next?

In my next series of posts, I’ll dive into why memory is the missing puzzle piece that transforms a chatbot from a frustrating Q&A machine into an assistant that feels truly intelligent. Stay tuned!