Computation and Language - How much do LLMs learn from negative examples?

Alright learning crew, Ernis here, ready to dive into some fascinating research about how we teach AI to be, well, less wrong! We're talking about Large Language Models – think of them as super-smart parrots that can string together sentences in amazing ways, like ChatGPT or Bard.

These models learn in stages, kind of like going to school. First, they're just exposed to tons of text – that's the unsupervised pre-training. It's like letting them wander around a library and soak everything up.

Then comes supervised fine-tuning, where they get direct instruction: "Here's a question, here's the right answer." But what about learning from mistakes?

---

That's where this paper comes in. It looks at the final phase of training, where these models are shown negative examples - incorrect answers, rejected responses, the AI equivalent of a big, red "X". Think of it like teaching a dog to sit. You don't just reward the "sit," you also correct the "stand" or "lie down" at the wrong time.

The researchers used a clever technique called "Likra" to carefully control how much influence these negative examples had. Imagine Likra as a volume knob for "wrongness." They wanted to see what happens when you turn it up or down.

They focused on multiple-choice questions, which provides a clear way to define "right" and "wrong." What they found was really interesting:

• Negative examples can be super-effective. At a certain point in training, showing the AI what not to do led to a much bigger jump in performance than just showing it more correct answers. It's like suddenly the AI "gets it" in a way it didn't before.
• Not all wrong answers are created equal. The most helpful negative examples were the ones that were plausible but incorrect – the "near misses." These are the tricky ones, the answers that sound good but are subtly wrong. Correcting these really helps the AI sharpen its understanding. Think of it like learning to play chess: it's not enough to know the basic moves, you need to learn how to avoid common traps and blunders.
• Negative examples help squash those hallucinations. Showing the model wrong answers helps it learn to more accurately identify those tricky, plausible-sounding but ultimately incorrect responses. The researchers found that while positive examples alone didn't do much to reduce the likelihood of these "hallucinations" (when the AI confidently makes stuff up), negative examples were much more effective.

---

So, why does this matter? Well, for a few reasons:

• For developers: This research offers a powerful new tool to make our AI models more accurate and reliable.
• For users: This could lead to AI assistants that are less likely to give you wrong information, making them more trustworthy.
• For society: In areas like medicine or law, where accuracy is critical, this kind of improvement could be a game-changer.

This research suggests that showing AI what not to do is just as important as showing it what to do. It's about teaching these models to not just memorize, but to truly understand.

Here are a couple of things that popped into my head while prepping this:

• If negative examples are so powerful, how do we ensure they're not biased or misleading? What guardrails do we need to put in place?
• Could this approach of using "near miss" negative examples be applied to other machine learning tasks, beyond language models? Think self-driving cars - can we teach them to avoid accidents by showing them examples of near-collisions?

Alright learning crew, that’s the tea on negative examples in LLMs. Let me know what you think!

Credit to Paper authors: Shadi Hamdan, Deniz Yuret