Human-Computer Interaction - Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines

Hey PaperLedge learning crew, Ernis here, ready to dive into some fascinating AI research! Today, we're unpacking a study that looks at how well we humans are actually talking to these super-smart AI chatbots, like the ones powering your favorite writing assistant or customer service tool. Think of it like this: you've got this amazing, super-powered genie in a bottle (the LLM), but are we really making the best wishes?

The basic idea is that these Large Language Models (LLMs) are designed to understand us using everyday language. You just type what you want, and poof, the AI does its thing. Sounds simple, right? But the researchers found something interesting: even though these systems are supposed to be user-friendly, a lot of us are struggling to get the most out of them. We're not always asking the right questions, or phrasing them in a way that the AI can really understand.

---

Think of it like ordering coffee. You could just say "Coffee, please." You'll probably get something, but it might not be exactly what you wanted. Maybe you wanted a latte, or an iced coffee, or a decaf with oat milk! The more specific you are, the better the barista (or the AI) can deliver. This paper suggests that we often give AI systems "coffee, please" prompts when we could be asking for a perfectly customized beverage.

This study set up an educational experiment. They had people try to complete tasks using an AI, but gave some folks special instructions, or prompting guidelines, on how to ask better questions. It's like giving some coffee-orderers a cheat sheet with all the different drink options and how to ask for them. They looked at three different kinds of cheat sheets – one they designed themselves and two others as a comparison. Then, they tracked how people interacted with the AI, looking at the types of questions they asked and how well the AI responded.

"Our findings provide a deeper understanding of how users engage with LLMs and the role of structured prompting guidance in enhancing AI-assisted communication."

---

To analyze all this data, they used something called Von NeuMidas – a fancy name for a system that helps them categorize the common mistakes people make when prompting. It's like having a coffee expert watch everyone's orders and say, "Ah, this person forgot to specify the size," or "This person didn't mention they wanted it iced."

What they found is that when people got better guidance on how to ask questions, they not only asked better questions, but the AI also gave better answers! It shows that a little bit of instruction can go a long way in improving how we interact with AI.

---

Why does this matter? Well, for educators, it means we need to teach people how to effectively use these AI tools. For AI developers, it means we need to design systems that are more forgiving of vague prompts, or that actively guide users towards asking better questions. And for everyone else, it means we can all get better at using these amazing tools to boost our productivity, creativity, and problem-solving skills.

So, here are a couple of things that popped into my head while reading this:

• If we need to be "trained" to talk to AI, does that mean these systems aren't as intuitive as we thought?
• Could AI be designed to provide real-time feedback on our prompts, almost like a built-in tutor?

Let me know what you think in the comments! What are your experiences with prompting AI? Have you found any tricks that work well for you? Until next time, keep learning!

Credit to Paper authors: Cansu Koyuturk, Emily Theophilou, Sabrina Patania, Gregor Donabauer, Andrea Martinenghi, Chiara Antico, Alessia Telari, Alessia Testa, Sathya Bursic, Franca Garzotto, Davinia Hernandez-Leo, Udo Kruschwitz, Davide Taibi, Simona Amenta, Martin Ruskov, Dimitri Ognibene