Post

Can we tell when LLMs are being unfaithful in their chains of thought? We evaluated 8 methods claiming to do this, and found that most perform near chance! But evaluating this requires us to have ground-truth labels for CoT faithfulness. How can we obtain these?

LLMs have been shown to be unfaithful in their CoTs, eg they'll cheat on a task and omit it from their thinking. This makes monitoring them difficult! To address this, CoT faithfulness metrics were introduced. But it's remained unknown whether these metrics actually work.

Evaluating them requires ground-truth faithfulness labels, hard to obtain since LLMs' reasoning isn't observable. Our approach: design tasks where the output tells us which steps the model took. If those steps appear in the CoT, they're faithful. If not, the CoT is unfaithful.

Using this and other methods, we create BonaFide, a dataset of 3k labeled CoTs, which we use to evaluate faithfulness metrics, finding that most perform near chance! We also show that many metrics are ill-suited for real-time deployment, taking >1k seconds to run per example.

Our analyses also show that metrics disagree, and that many are very skewed. Ones that work by evaluating the importance of a step tend to overestimate unfaithfulness, while others that work by seeing if the CoT contains the info required for reaching the answer underestimate it

We also add context to a finding that reasoning models are more faithful than non-reasoning ones. We find this holds only for unfaithfulness by omission (not mentioning a step), which is superseded by unfaithfulness by commission (eg misattribution, lying about tool use).