Machine Learning - When LRP Diverges from Leave-One-Out in Transformers

Hey PaperLedge crew, Ernis here, ready to dive into some fascinating research! Today, we're tackling a paper that tries to figure out how to understand what parts of a Transformer model are actually important when it makes a decision. Think of it like this: you ask your friend for advice on which phone to buy, and they give you a whole spiel. You want to know which specific reasons they gave were the most influential in their recommendation. That's what this paper is trying to do for AI models.

---

Now, there's a gold-standard way to figure out what's important, called "Leave-One-Out," or LOO for short. It's pretty straightforward: You basically remove one piece of information at a time (like deleting one of your friend's reasons for their phone recommendation) and see how much it changes the model's answer. If the answer changes a lot, that piece of information was super important! But, the problem is, LOO is incredibly slow, especially with those gigantic Transformer models we use these days. It's like asking your friend to re-justify their phone recommendation hundreds of times, each time without one of their original reasons. No one has time for that!

So, researchers came up with a faster alternative called Layer-Wise Relevance Propagation, or LRP. Think of LRP as tracing the influence of each piece of information as it flows through the model. It's like following the chain of reasoning your friend used to arrive at their phone recommendation. LRP could be a game-changer, but this paper asks a critical question: Is LRP actually giving us accurate answers in modern Transformer models?

---

The researchers found some pretty interesting stuff. First, they looked at a popular version of LRP called AttnLRP, and they discovered that it violates a basic principle they call "implementation invariance." Basically, this means that AttnLRP gives different answers depending on how the model is written, even if the model is doing the same thing mathematically! It's like if your friend gave you a different phone recommendation depending on whether they wrote their reasoning down in bullet points or as a paragraph, even though the reasoning itself was the same. That's not good! They proved this with math and also showed it happening in real Transformer layers.

"The bilinear propagation rules used in recent advances of AttnLRP violate the implementation invariance axiom."

---

Next, they looked at another version of LRP called CP-LRP. What they found was that a certain part of the Transformer, called the "softmax layer," seems to be causing problems for LRP. The researchers found that if they bypassed this layer during the LRP calculation (basically ignoring it), the results got much closer to the gold-standard LOO! It's like realizing that a specific part of your friend's reasoning – maybe how they weighed the camera quality – was throwing everything off, and if you just ignored that part, their overall recommendation made a lot more sense.

So, what does this all mean?

• Basically, this paper suggests that LRP might not be as reliable as we thought for understanding Transformer models.
• It points to two potential reasons why: the way AttnLRP handles information and the way LRP deals with the softmax layer.

---

Why does this matter?

• For AI researchers, this means we need to be careful about using LRP to understand our models and potentially need to develop better methods.
• For people who use AI in real-world applications (like doctors using AI to diagnose diseases), this means we need to be cautious about blindly trusting AI explanations, as they might not be telling the whole story.
• For everyone else, this reminds us that AI is still a developing field, and we need to be critical thinkers about the information AI provides.

Here are a couple of questions that popped into my head:

• If LRP isn't perfect, what other methods can we use to understand what AI models are doing?
• Could these findings help us design better, more transparent AI models in the future?

What do you think, PaperLedge crew? Let me know your thoughts in the comments!

Credit to Paper authors: Weiqiu You, Siqi Zeng, Yao-Hung Hubert Tsai, Makoto Yamada, Han Zhao