Example: Using LLMs to Classify Email Bounces

This weekend I've saved myself significant work by using a local LLM to classify rejected email for a non-profit I work with.

The Background

In addition to my work with OptiML AI and Sportslytx , I also volunteer time for non-profits. One non-profit I have worked with for many years (let's say it is domain @example-non-profit.org) provides members with email addresses like [email protected] or [email protected]. These can be FORWARDED to their main email, such as [email protected] or [email protected] if they don't want to check it on our servers.

The Problem

We've provided this forwarding service for more than 25 years, yet we haven't culled this set for bad addresses in a long time. There is a fairly significant number of members who have forwarding addresses that no longer work. These addresses that no longer work are called BOUNCES.

• We recently sent a message to many thousands of members and got over 2,500 bounces. That's a lot of email to read and process! No one was looking forward to that.
• A bounce message typically says something like 'Your message could not be delivered to one or more recipients.' and gives a technical reason. These reasons - and their wording - vary for each message.
• Even though these messages are not uniform or consistent across domains, a human can read them and tell whether or not it is a PERMANENT or TEMPORARY rejection.
• Each bounce needs to be processed by issuing an UPDATE command to our SQL database using information found in the rejection message.

LLMs can process text like humans - let's take advantage of that.

The Solution

We will take advantage of LLMs ability to read like a human and let it do much of the processing for us! But we need to do this programmatically and also not expose any personally identifiable information to public LLMs.

I wanted to keep all the data private, so I tested out local LLMs using LM Studio v0.3.6. You can download it for free at https://lmstudio.ai/. Note that as of January 4th, 2025, v0.3.6 is very new so my antivirus blocked it because it didn't have enough downloads!

I downloaded and tried several models, choosing qwen2.5-14b-instruct for this project. This model is an 8.99GB download that was last modified on November 27, 2024. It supports 128k tokens with yarn rope settings and 8k token generation and is "trained for tool use" - something that I will try soon.

Using the chat feature of LM Studio , I loaded the qwen2.5-14b-instruct model and tested various prompts. I settled on the following system prompt and inserted various rejection messages in the {MESSAGE_BODY} part.

A typical bounce message will have technical information like what is shown below - sort of cryptic to a layman, but good information for those that have experience with email systems:

The result from the LLM for the above bounce would be JSON like this:

The processing took about 5 seconds locally on my PC (Windows 11 Pro, i9-14900KF, 64GB RAM, NVIDIA GeForce RTX 4090).

That means the LLM can process all the messages in 3.5 hours rather than a human reading them individually, which would take at least 4x as long to read and extract the information. I'll take a computer doing the work over me spending 14 hours on a repetitive task any day!

Note that it has everything that we need to make a decision later:

• SUBJECT and DATE of the message so that we can confirm it was a rejection (some other messages that are not bounces come to this mailbox too).
• USERID, TO, and ORIGINAL_TO so that we can confirm the email addresses and user ID in our system.
• DETERMINATION, REJECTION_REASON, and EXPLANATION so that we can better understand what the LLM thought about this message and why it made the decision.
• SQL_COMMAND so that we can run this command later if everything looks good.

Doing it Programmatically with Python

Once I had this satisfactory prompt, I needed to automate the processing of 2,500+ messages via Python.

First, I started the local API endpoint in LM Studio for qwen2.5-14b-instruct - it is literally as easy as pushing a button.

I got some assistance from ChatGPT 4o and created the following loop in Python:

1. Make a SQL query to the database that holds all the bounced email messages.
2. For each message, use functions from python's email library to parse out the MESSAGE BODY.
3. Call the API for each message (system prompt + MESSAGE BODY).
4. Append the resulting JSON to a file that will be used later.

Still some errors every once in a while, so I'm not going fully Agentic yet...

There are still a few cases of mistaken TO or USERID, so I'm gathering this as a JSON and will process the SQL updates programmatically with Python later.

Next step will be to make it agentic and let it modify the database itself!