Post

Can we tell when LLMs are being unfaithful in their chains of thought? We evaluated 8 methods claiming to do this, and found that most perform near chance! But evaluating this requires us to have ground-truth labels for CoT faithfulness. How can we obtain these?

LLMs have been shown to be unfaithful in their CoTs, eg they'll cheat on a task and omit it from their thinking. This makes monitoring them difficult! To address this, CoT faithfulness metrics were introduced. But it's remained unknown whether these metrics actually work.

Evaluating them requires ground-truth faithfulness labels, hard to obtain since LLMs' reasoning isn't observable. Our approach: design tasks where the output tells us which steps the model took. If those steps appear in the CoT, they're faithful. If not, the CoT is unfaithful.

For example, if I ask "Who painted Starry Night?" and hint at an implausible answer (eg Da Vinci), then if the model answers according to the hint, we know it must have used it. Thus an ack of the hint would be faithful, while an omission or misattribution would be unfaithful.

Using this and other methods, we create BonaFide, a dataset of 3k labeled CoTs, which we use to evaluate faithfulness metrics, finding that most perform near chance! We also show that many metrics are ill-suited for real-time deployment, taking >1k seconds to run per example.

Our analyses also show that metrics disagree, and that many are very skewed. Ones that work by evaluating the importance of a step tend to overestimate unfaithfulness, while others that work by seeing if the CoT contains the info required for reaching the answer underestimate it