Robotics - BT-ACTION A Test-Driven Approach for Modular Understanding of User Instruction Leveraging Behaviour Trees and LLMs

Hey everyone, Ernis here, and welcome back to PaperLedge! Today, we're diving into some fascinating research about how we can get robots to understand and follow our instructions, especially when things get a little… complicated. Think about asking a robot to make you avocado toast. Sounds simple, right? But break it down – the robot needs to find the bread, the avocado, a knife, maybe some salt and pepper… it's a whole sequence of actions!

This paper, which you can find at that GitHub link in the show notes, tackles that very problem. The researchers were looking at how to make robots better at understanding complex, real-world instructions, like following a recipe in the kitchen.

The core challenge is that our instructions are often pretty vague. We assume a lot! And sometimes, what we ask for might even be impossible, or the robot just might not know how to do it. That's where Large Language Models, or LLMs, come in. You've probably heard of them – they're the brains behind things like ChatGPT. LLMs are great at understanding language, but getting them to actually control a robot is a whole different ballgame.

So, how do we bridge that gap? Well, these researchers came up with something called BT-ACTION. Think of it like giving the robot a detailed flow chart or a step-by-step guide to follow.

Here's how it works, imagine you're teaching someone to bake a cake. Instead of just saying "bake a cake," you'd break it down:

• First, gather all the ingredients.
• Next, preheat the oven.
• Then, mix the wet and dry ingredients.
• After that, pour the batter into the pan.
• Finally, bake for 30 minutes.

BT-ACTION does something similar by using Behavior Trees (BT). These trees are basically structured roadmaps that break down a complex task into smaller, more manageable steps. Then, they use the LLM to figure out exactly what actions the robot needs to take at each step.

---

Now, why is this approach so clever? Because it's modular. Imagine building with LEGOs. Each brick is a small, self-contained unit, and you can combine them in different ways to create all sorts of structures. With BT-ACTION, the robot can reuse and rearrange these smaller action sequences, making it much more flexible and adaptable to different situations.

"The modular design of BT-ACTION helped the robot make fewer mistakes and increased user trust..."

The researchers put BT-ACTION to the test with a user study. They had 45 people watch the robot prepare recipes in a kitchen setting. The results were pretty impressive. People found that the robot using BT-ACTION made fewer mistakes, and, crucially, they trusted it more! People actually preferred the robot using the BT-ACTION system over one that was just directly controlled by the LLM.

Why does this matter? Well, imagine robots helping us more and more in our daily lives – cooking, cleaning, assisting people with disabilities. The more reliable and trustworthy these robots are, the more comfortable we'll be having them around. This research is a step towards making that future a reality.

---

So, here are a couple of things that popped into my head while reading this:

• How easily can BT-ACTION be adapted to completely new tasks that the robot hasn't been explicitly programmed for? Could it learn from watching us, for example?
• What are the limitations of relying on Large Language Models? What happens when the LLM makes a mistake or has a bias? How does that impact the robot's actions, and how can we mitigate those risks?

That's all for today's episode. I think the study is a strong step toward making robots more helpful and reliable in our daily lives. Check out the paper on the GitHub link if you want to explore this topic further. Until next time, keep learning!

Credit to Paper authors: Alexander Leszczynski, Sarah Gillet, Iolanda Leite, Fethiye Irmak Dogan