Post

Despite extensive safety training, LLMs remain vulnerable to “jailbreaking” through adversarial prompts. Why does this vulnerability persist? In a new paper published in Philosophical Studies, I argue this is because current alignment methods are fundamentally shallow. 🧵 1/13

Current alignment methods aim to instill norms like helpfulness, honesty & harmlessness in LLMs through preference fine-tuning (like RLHF or DPO) that rewards them for producing outputs that humans prefer, steering them away from unhelpful, dishonest or harmful content. 2/13

The problem is that these norms often conflict. For example, a request for dangerous information (violating “harmlessness”) can be framed as an educational query (appealing to “helpfulness”). Many issues with LLM behavior can be framed through these normative conflicts. 3/13

Why does this work? Current methods don't teach LLMs to reason about the contextual relevance of conflicting norms – they reinforce shallow behavioral dispositions. When faced with a new conflict, LLMs follow the disposition most strongly activated by the prompt's framing. 5/13

By contrast, humans can engage in normative deliberation to weigh conflicting prima facie norms and arrive at an all-things-considered judgment about what they should do in a specific context. Current LLMs lack a robust capacity for context-sensitive normative deliberation. 6/13

But what about newer “reasoning” language models (RLMs) that produce an explicit reasoning trace before answering? While they show inklings of normative deliberation, they remain vulnerable to the same kind of conflict exploitation. Worse, they introduce a new attack vector. 7/13