Computation and Language - Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start

Hey PaperLedge learning crew, Ernis here, ready to dive into some fascinating research! Today, we're unraveling how to make AI models that can not only see and understand images but also think about them in a really smart way. We're talking about multimodal Large Language Models, or MLLMs.

Now, you might've heard about these AI models that seem to have "aha!" moments, like suddenly understanding a complex problem. Researchers often thought these moments were all thanks to a special type of learning called Reinforcement Learning or RL. Think of RL like training a dog: you reward good behavior, and the dog learns what to do. But this paper throws a bit of a curveball.

The researchers discovered something cool: these "aha!" moments can actually show up before the RL training even starts! It's like the model already has a hint of understanding. However, and this is crucial, these early "aha!" moments don't always mean the model is getting better at reasoning. So, what's going on?

---

That's where the real innovation comes in. The researchers developed a two-step process to really boost the reasoning skills of these MLLMs:

• Step 1: Supervised Fine-Tuning (SFT) - The Cold Start: They start by feeding the model lots of examples of structured, step-by-step reasoning. Imagine teaching a child to solve a math problem by showing them exactly how to break it down. This is like giving the model a solid foundation.
• Step 2: Reinforcement Learning (RL) - The Refinement: Then, they use reinforcement learning to fine-tune those reasoning skills. Think of it as the coach helping the athlete perfect their technique. They used a special type of RL called GRPO.

Why is this two-stage approach important? Well, it turns out that this combination is much more powerful than using either SFT or RL alone. It's like having both a strong foundation and expert coaching.

"This combined approach consistently outperforms both SFT-only and RL-only methods..."

---

The results are impressive. These researchers built MLLMs at both 3 billion and 7 billion parameters (think of parameters as the size or complexity of the model), and they achieved state-of-the-art performance among open-source MLLMs. That means these models are among the best that are publicly available! For example, their 7B model showed substantial improvements on tough visual reasoning tasks like MathVista (66.3 % -> 73.4 %) and We-Math (62.9 % -> 70.4 %). Their smaller 3B model even performed competitively with some 7B models!

So, why should you care?

• For AI Developers: This research offers a practical blueprint for building better MLLMs. The code is even available on GitHub (https://github.com/waltonfuture/RL-with-Cold-Start), so you can try it out yourself.
• For Educators: Understanding how AI learns to reason can help us develop better teaching methods, both for humans and machines.
• For Everyone: As AI becomes more integrated into our lives, understanding how it works and how to improve it is crucial. Imagine AI assistants that can truly understand and reason about the world around them!

This research isn't just about making AI smarter; it's about understanding how to make it smarter. And that understanding is something we can all benefit from.

Now, a few questions that popped into my head while reading this paper:

• Could this two-stage approach be applied to other areas of AI, beyond multimodal reasoning?
• What are the ethical implications of building AI models that are so good at reasoning? How can we ensure they're used responsibly?
• Since the "aha!" moments appear even before RL, what exactly triggers those moments, and can we leverage that even further?

That's all for this episode, learning crew! Keep exploring, keep questioning, and I'll catch you next time on PaperLedge!

Credit to Paper authors: Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, Weiran Huang