Post

It's important to know how LLMs work: they predict how some text ends. That's it. That's why the early ones were literally "enter some text, and it'll finish it." The "magic" is that the text they see isn't just your input. It's surrounded by context, and filtered.

You enter "hi" and it sees sth like: "You are an advanced chatbot, [various rules and character description], the user says: hi; you reply: " That's why injection can work. It's still just using the structural patterns of language to determine the most likely followup, no understanding needed.

In a more advanced case, one might be used to preprocess the user input to prompt another, for an exchange like: User: "is 1 million divisible by 3?"

Stage 2 input: "[...] Its response is: 333333.33333333" Feeding this back in again, it continues matching the pattern: "Its response is: 333333.33333333 - since this contains a decimal, the answer is No. I should reply: No, 1 million is not divisible by 3."

This is all well and good, but the final response still came from just statistical analysis of language. Earlier models were just as likely to finish it with "since this doesn't contain a decimal" or "since this contains a decimal, the answer is Yes". Those are all likely endings.

Since the exact expression is unlikely to be in its model, it reads like: "The user is asking if SOME_NUMBER [...] since this" - completing from there, any response is likely. Newer models try to "fix" this with more data, "peer review" by other models, etc, but the fundamentals are the same.