Post

another researcher accuses OpenAI of training on conversations and then claiming a breakthrough

Transformer models produced training data text verbatim frequently 2+ years ago. The technology itself has not substantially changed. It's plausible that the lower frequency of verbatim output is the result of mitigation rather than absence. Glad if mathematicians are mad. Coders let this happen.

it’s a model training smell. there’s a series of papers that go into why, but a high quality LLM should only be able to quote *in spirit* not verbatim

I haven't seen it since, but Opus 4.5 on bedrock gave me 2 full, exact paragraphs of Tale of Two Cities, without any I/O AFAIK. It has been extremely hard for me to suspend my skepticism when people tell me LLMs don't plagiarize. Small perturbations to avoid string match actually makes me more mad.

that’s a pretty small excerpt but anyway, the NYT wasn’t able to prove it in court. same as several others not to say it’s impossible, it’s just a bug in the training data if it’s memorizing (arguably not for tale of two cities, since it’s meant to be quotable)

There are a lot of things the judicial and regulatory systems aren't prepared to handle (yet). Anthropic just paid $1.5B. For provenance, yes. It shouldn't be fair use because it competes with the original work. When people tell me it can't quote and it does it's a major reputational problem.

Earlier I got several full classes from sqlite3. I think that was Microsoft copilot in vscode maybe 2 years ago. Btw I don't push models at all. I'm just trying to do my work.

Personally I think the business model needs to change. Training data needs to be licensed for training use, with real teeth. The vendors shouldn't able to pay a small fine for piracy. Don't even get me started on GitHub. The public acrimony will increase until the vendors reform themselves.

i mean, it *is* all licensed. Anthropic even validated that in court. in the case we’re talking about, they guy didn’t have his settings right, so he was allowing openai to train on his data, that’s within the license he agreed to!!

alright, the concepts i’m talking about were established in this series of papers, about 3 years ago. It’s possible the model you used hadn’t been trained like this yet arxiv.org/abs/2309.14316

I'll read it. I'm not a researcher but I've done my best to read several of the core white papers (as best I can). I feel it's very convenient that this technique is performance and not copyright motivated. Would love to see a SOTA model that doesn't for comparison.