Andrew Gross

I ran out of usage on CC and switched to a local model. The model got into this weird state of being responsive but assuming that its usage had run out and wouldn't continue anything by default, like it had entered a fugue state and couldn't proceed.

I wish more places would publish the Waffle grid of their Tbench 4.0 Runs, it would really help to understand why there is often a gap of 1-10pts between internal benchmarks vs AA etc. Even looking at the dated results from the Tbench site shows a lot of false negatives.

preferencemodel.comTerminal Bench 4 Is SaturatedWe audited Terminal Bench 4 and found models cheating their way to full marks and tasks with design issues. Here is what we found and how we fixed it.

preferencemodel.com/blog/termina... Timely and a great breakdown of some similar issues. I wasn't even that deep in the results, instead I was looking at things like API TLS call failures to Harbor APIs breaking runs etc.

Seems like the new Deepseek Flash is the first round of GPT 5.6 Sol / Opus 5 distils. Can't wait till they get around to Opus 5.5

I'll believe we have AGI when the Artifical Analysis website renders properly on mobile.

Can't wait for the Open Weight models to finish distilling Opus 5.5 so they can produce legible descriptions again.

contrastive-lm.notion.siteContrastive Language Models | NotionA System One Model for Fast and Generalizable Decision-Making

contrastive-lm.notion.site Interesting write up. No paper yet unfortunately. If some of the claims hold true it would be a pretty incredible leap. Their SOTA DeepSWE scores come from generating rollouts using Opus/Fable and then using their model to select the "right" action.

View full thread

For TBench 2.1, k=5 for the official run harness, I am curious if they used their model to pick the best of one of those, or if they picked the best of 4 or 5 runs of k=5, essentially getting k=20 (or k=25).

huggingface.cohuggingface.co

huggingface.co/datasets/tar... this is the deepswe one, is there similar for TBench? If its the same style, it would just be taking the k=5 (or 4 for deepswe) rollouts and scoring them and picking the best (essentially converting k=5 to k=1 for actual scoring) which is nifty

contrastive-lm.notion.siteContrastive Language Models | NotionA System One Model for Fast and Generalizable Decision-Making

View full thread

Seems like the fine tune is likely on the embedding space so that you get better contrast? Hence the usage of some of the DeepSWE tasks for fine tuning. I wonder what perf is like without fine tuning.

Also curious what it would look like to have this "in the loop" for a regular LLM, helping choose actions

contrastive-lm.notion.siteContrastive Language Models | NotionA System One Model for Fast and Generalizable Decision-Making

Does seem like they used a similar approach to CLIP but for non-image inputs? Need to spend more time understanding the training data vs what is used for a fine tune and why etc.

contrastive-lm.notion.siteContrastive Language Models | NotionA System One Model for Fast and Generalizable Decision-Making

View full thread

huggingface.coContrastive-LM/deepswe-clm-heads-8k · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.

Looks like Bo4 for the DeepSWE runs? huggingface.co/Contrastive-... In theory Fable/Opus Pass@4 should produce similar numbers right? I guess it depends if the model is choosing the final rollout results or being used mid stream to pick actions.

They did seem to train/fine tune on the DeepSWE tasks, with a set held out for the actual testing. Not sure what the fine tuning gets here, "style" for the outputs? Shouldn't the action model be general w.r.t. these sorts of inputs and not need fine tuning?

contrastive-lm.notion.siteContrastive Language Models | NotionA System One Model for Fast and Generalizable Decision-Making

They have the data and code out there, so can dig in to see how they did it I guess. Downside is that the SOTA scores would come from running 10x (or more) tokens on frontier models to get candidates.

docs.github.comCustomizing how changed files appear on GitHub - GitHub DocsTo keep certain files from displaying in diffs by default, or counting toward the repository language, you can mark them with the linguist-generated attribute in a .gitattributes file.

Pondering using docs.github.com/en/repositor... this to hide away code where the agent has domain, so PRs only focus on the interfaces and hotspots I care about.

Im wondering if the next big "change" we see from AI is in our bureaucracy. Not so much in how it operates internally, but in the dramatic rise in inputs from people using AI. So many things were implicitly gated by the cost of time and attention that has mostly gone away. Filing complaints, forms..

disputing claims etc. A lot of this had a very real cost in human time in the past, so much so that being someone wealthy enough to have a lawyer to do it all for your was a huge privilege. Now many more people have the ability to send a properly bureacracy coded letter than before.

With the way devices are going these days, Im going to need to have all of my private convos inside an MRI machine.

23 Aug

Claude has reduced me to forcing it to summarize its own output without jargon every time it comes back to me. github.com/andrewgross/...

Trying to find an old video on Zip codes. No luck with Youtube search, SEO'd to oblivion. Ask Gemini, gives crappy results, prompt it for a Youtube link, no luck. Tell the model its part of Google and it can search Youtube, responds telling me it can't, but also includes the exact link I needed.

share.gemini.google/MPUBUjIOwuKK

19 Aug

share.gemini.google‎Gemini - direct access to Google AICreated with Gemini

15 Aug

Im curious what the uncounted token overhead is for things like ChatGPT and Claude chat sessions. There are a lot of things like memory that involve some background LLM processing that don't seem to get reflected in usage limits, I wonder if it dramatically affects margin.

15 Aug

Similar for things like the auto command classifier (if it uses an LLM at all), security review for Fable etc etc. I'd assume its no more than 10% of the cost of the actual token serving, so it probably doesn't change the numbers that much.

7 Aug

The latest crop of Claude models are too verbose and create incredible amounts of unnecessary jargon, requiring a re-prompt just to clean it up. Curious if its a side effect of RL rewarding lower response length, so they "create" vocabulary to compact more into less. Either way it sucks.

29 July

Spaghettios Aglio e Olio

27 July

Im curious if I have missed papers/projects out there for converting compiled/minified code to an AST/text code consistently. I know of humanify of course, and I found the BinDiff paper/project as well. Wondering what else is out there. My big problem right now is consistency between versions.

27 July

Running out of signal I can use for some identifiers and right now they permute between files, creating noise in the final diff. Wondering if I need to find more signal or add more structure to reduce the variance.

18 July

How often do people get the prompt in CC asking if Anthropic can review their session to help improve Claude Code? I haven't gotten it on my work setup, but on my personal sub I have seen it prob 4-5 times. This is on a project that gets downgraded from Fable to Opus probably 50% of the time.

16 July

Fable seems to be really good at... deciding it needs to spin up 10 sub-agents, immediately running out of usage credits, being awoken and then deciding to do all the work itself while doing a bad job.

github.comGitHub - jehna/humanify: Deobfuscate Javascript code using ChatGPTDeobfuscate Javascript code using ChatGPT. Contribute to jehna/humanify development by creating an account on GitHub.

14 July

Something really weird is going on with the npm downloads counter for claude-code www.npmjs.com/package/@ant... Really weird spikes and total download counts in recent versions, while old versions have 1000s or even just 100s of downloads. No way every one is on the native installer.

Im realizing my fork of humanify has become quite a different beast than the original repo, and I need a new name (while still giving credit to the original). Original: github.com/jehna/humanify Mine: github.com/andrewgross/...

10 July

The only thing I ever (rarely) use image generation for is making a customized meme for my friends. It seems that Google has decided that making a meme is a copyright violation and can't work with it at all.

en.wikipedia.orgCybernetics - Wikipedia

20 June

TIL the etymology of cybernetics (and cyber). Thats... actually pretty cool. en.wikipedia.org/wiki/Cyberne...

iamwil@interjectedfuture.com

Lots of ink spilled on "Loop engineering", but no one seems to connect it to its much older name: Cybernetics. Maybe by the year's end, we'll recognize that it was all Control Theory engineering all along.

20 June

ianbarber.blog/2026/06/19/l... Sharing, if only for the reference to "Claude Telenovela"

19 June

FrontierCode is great and all, but does anyone actually get to run it? Aside from the Fable release, I haven't heard of any updates for the rankings for any new models.

18 June

I need to make AngryBench. Just re-running/validating existing model benchmarks, but adding in a system message or prompt prefix to set the tone of the interaction from the user (angry, kind, professional etc). Essentially trying to understand how various models perform based on how you treat them

youtube.comDeep Learning with YacineDeep learning project and theory videos every week! 👺

17 June

Apropros of nothing, shoutout to Deep Learning with Yacine. Great channel where someone recently graduated and a research background just...talks to the authors of papers. Its great www.youtube.com/@deeplearnin...

15 June

I find myself needing to expand Github PR diffs much further these days to be able to actually see the relevant code for understanding it. The way agents craft code, with excessive commenting/explanation and poor abstraction/overlong methods makes it hard to visually see everything easily.

15 June

Maybe Code Complete wasn't in their training data.

9 June

With the release of Mythos and Fable, I'm betting Anthropic goes with Saga and Epic for their next two big models. And yarn for their small model.

5 June

Dead Internet Theory, but for recruiting

5 June

This post brought to you buy a single senior level engineering job posting receiving 2500 applications within 72 hours.