Andrea de Varda

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)

A similar asymmetry emerges within the effort measures. Eye movements are driven mostly by length and frequency (context-independent). Brain responses are driven mostly by surprisal (context-dependent). (9/10)

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)

We test this on 13 datasets: 👀 4 eye-tracking, 3 self-paced reading, 1 Maze, 🧠 1 ERP (N400), and 4 fMRI. All responses are averaged across participants and brain responses are averaged across voxels/electrodes so brain and behavior are comparable (1-dimensional). (7/10)

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)

We use LMs to operationalize effort and meaning in one framework. From the same GPT/GPT-2 models we get (i) surprisal (+freq and len; SFL) for effort, and (ii) contextual embeddings (EMB), high-dimensional vectors that encode form and meaning. (5/10)

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)

One influential view is that responses to language are driven by processing effort, mostly captured by three word-level predictors: surprisal, frequency, and length (SFL). These are the "Big Three" in reading research and they also predict brain responses. (3/10)

New preprint! 🧠👀🤖 Behavioral and brain responses to language reflect different levels of linguistic representation w/ @whylikethis.bsky.social , @evfedorenko.bsky.social , and @rplevy.bsky.social (1/10)

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

Lastly, there is a fundamental tension in B&al’s pieces: the correlation cannot be both trivially guaranteed by training and fragile and easily dissociated.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

Imitation also predicts the wrong direction of the inter-model difference. RL moves models away from imitating the pre-training corpus, so alignment should weaken in LRMs. Instead, it strengthens. Optimizing for correctness with no human data in the reward increases the similarity to humans.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

The relevant manipulations exist in our data and all point the same way. Varying problem structure (digits, operands, carry, operation-type) affects tokens and RTs similarly. Varying training (standard LLMs vs. reasoning models) changes the correlations between tokens and RTs.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

B&al ask for causal evidence, but causal between what and what? There are three possibilities. (a) Tokens cause model accuracy (b) Tokens cause human RTs, which nobody would hold (c) A common cause: the computational demands of the problem drive both token counts and RTs.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

Above low effort, accuracy is indeed near flat. But e.g., a flat response above the effective dose is not evidence that a drug doesn’t work. It is what a causally effective resource looks like when the demand is small.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

Similarly, 60% of our participants obtained 100% accuracy in our dataset, but this does not make their RTs less indicative of effort. In fact, good RT studies try to keep accuracy at ceiling and analyze correct trials only, so that time measures are not contaminated by different processes.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

This is the core logic of reaction time (RT) research. Humans solve simple math problems at ceiling no matter how much time you give them but their RTs still reflect problem difficulty, and that variance is used to build theories of arithmetic reasoning.

Bowers and colleagues (B&al) have a new response to our paper on the cost of thinking in reasoning models and humans. Here we address the core disagreements.🧵

Some words sound like what they mean. In IconicITA we show that the (psycho)linguistic factors that modulate which words are most iconic are similar between English and Italian. Lots more details in the paper!