Post

It's important to know how LLMs work: they predict how some text ends. That's it. That's why the early ones were literally "enter some text, and it'll finish it." The "magic" is that the text they see isn't just your input. It's surrounded by context, and filtered.

You enter "hi" and it sees sth like: "You are an advanced chatbot, [various rules and character description], the user says: hi; you reply: " That's why injection can work. It's still just using the structural patterns of language to determine the most likely followup, no understanding needed.

Input after stage 1: "The user is asking if 1000000 is divisible by 3. This is a math expression, so I should ask the Calculator tool. Its response is: <calc:1000000/3>" The system sees that tag and runs "calc 1000000/3", replacing the tag with its result.

Stage 2 input: "[...] Its response is: 333333.33333333" Feeding this back in again, it continues matching the pattern: "Its response is: 333333.33333333 - since this contains a decimal, the answer is No. I should reply: No, 1 million is not divisible by 3."

This is all well and good, but the final response still came from just statistical analysis of language. Earlier models were just as likely to finish it with "since this doesn't contain a decimal" or "since this contains a decimal, the answer is Yes". Those are all likely endings.