When Cats, Dogs, and Birds Become One: Understanding Soft Labels and Monte Carlo Conformal Predictions

Have you ever looked at a picture and thought, “Is that a cat, or maybe a dog… or wait, it has feathers?” Probably not, because that is an oversimplification to the idea I am writing about today.

I am using this example, because it is an abstract way of explaining the complex identification of industry problems like https://arxiv.org/pdf/2312.07460 classifying skin lesions. This is a tough problem, even for AI algorithms. Instead, we will break down the new Monte Carlo Conformal Predictions as it relates to simple classification. But in the back of your mind, think medical imaging.?

If you want to jump to the code, bippity-boppity, Google's code is now your property: uncertain_ground_truth/monte_carlo.py at main · google-deepmind/uncertain_ground_truth · GitHub

Label Confusion

This confusion in labeling objects is exactly what soft labels capture—except we use numbers (probabilities) to describe the uncertainty instead of just saying “I’m not sure.”

In a typical classification problem, you might tell the computer:

• Class 0 = “cat”
• Class 1 = “dog”
• Class 2 = “bird”

and each training example comes with a single hard label like 0, 1, or 2. But there are times when you’re not 100% sure, or maybe multiple people offered different opinions. That’s where soft labels come in. Multiple opinions happen quite frequently in medical imaging, that is why people seek a second opinion!

What Exactly Are Soft Labels?

Soft labels (sometimes called “smooth labels” or “probabilistic labels”) represent the fact that an image might be:

• 20% cat, 50% dog, 30% bird,
• or maybe 10% cat, 10% dog, 80% bird,
• or even 0% cat, 100% dog, 0% bird.

Each line above is a distribution over the three classes. Instead of shouting “it’s a dog,” we say, “the data suggests it’s probably a dog, but it could be something else.”

Here’s where Monte Carlo Conformal Prediction enters the stage. Conformal prediction is a mathematical tool that helps us wrap a “guarantee” around our guesses. It says something like: “We’ll output a set of labels for this new image, and we can guarantee that the correct label is inside this set at least 90% of the time” (or however confident we want to be).

But how do you do that confidently if each example has a range of possible labels rather than just one?

Monte Carlo Conformal Prediction—The Big Picture

Start with Uncertain Labels We have a bunch of examples—some are easy (100% dog) and others are uncertain (20% cat, 50% dog, 30% bird). These “soft labels” tell us how likely each category is.

Randomly Pick a Label, Repeatedly For each uncertain example, imagine rolling dice that land on dog, cat, or bird according to those probabilities. If you do this multiple times, you’ll end up with different “possible realities” for that example:

• First roll: “dog”
• Second roll: “bird”
• Third roll: “dog” again …and so on. This repeated sampling is known as Monte Carlo because we’re randomly drawing from probabilities, like pulling the lever at a slot machine over and over.

Expand the Calibration Data In order to figure out the “confidence threshold” or the “p-values” for our conformal predictions, we need a set of labeled examples (a calibration set). But if each example might have multiple true labels, we handle that by simply duplicating each example for every label we sampled.

• Example #1 could appear 3 times, each with a different label guess that came from its soft distribution.
• Example #2 might always appear as “dog” if it’s 100% dog, so there’s no guessing.

Compute Coverage Guarantees Once we have this expanded dataset—where each example is repeated for each sampled label—we can apply standard conformal prediction methods. The math behind conformal predictions ensures that if we produce a set of possible labels for a new, unseen image, there’s a high probability we’re not excluding the correct label.

• Without further adjustments, we might get a slightly weaker coverage (like 1–2α), because we “broke” some statistical assumptions by duplicating data.
• We can do additional corrections (like ECDF corrections) to climb back up to the ideal coverage (1–α). This deserves its own article, so don't think about it too much. ?It is essentially redistributing the probabilities via some resampling and stuff.

That’s it. If an image looks part-cat, part-dog, part-bird, Monte Carlo Conformal Predictions will (in a sense) treat each scenario as possible and calibrate accordingly, giving us a final “prediction set” that honestly reflects our uncertainty.

Sneak Peek of a Chimera

Imagine you’re labeling this bizarre (but adorable) mash-up creature. How confident are you that it’s a “cat”? Maybe 20%. A “dog”? Possibly 50%. A “bird”? 30%. The exact percentages come from your intuition, other models, or multiple human annotators. Regardless, if you pass these numbers into a Monte Carlo Conformal method, you’ll get robust set predictions—like “this image is either dog or bird” with 90% certainty.

Why This Matters

• Capturing Uncertainty: In real-world applications (medical images, complex images in self-driving cars, etc.), sometimes there isn’t a single “hard” label. Soft labels let the system reflect real ambiguity.
• Robust Guarantees: Conformal methods provide coverage guarantees without making strong assumptions about the data distribution.
• Flexible: You can use any underlying model (could be a neural network, random forest, etc.) to get the “conformity scores.” Conformal wraps around those scores to give a final confidence measure.

Key Takeaways

1. Soft Labels are probability distributions over classes, acknowledging that an example might not be purely cat, dog, or bird.
2. Monte Carlo Conformal duplications + random draws from these soft labels let us expand the calibration data and ultimately give us confidence sets for new predictions.
3. Practical Uses: Everything from medical imaging (where multiple diagnoses might be plausible) to label smoothing in machine learning (where we intentionally avoid 100% confidence).
4. Avoid Determinism: Deterministic labelers can lead to hallucinations and lower accuracy.

Conformal prediction doesn’t magically turn uncertainty into certainty—it just helps us formalize “how uncertain we are” and translates that uncertainty into predictive sets. Whether you’re labeling quirky cat-dog-bird hybrids or diagnosing diseases, it’s a powerful framework for honest, reliable predictions.

Thanks for reading!