Post

Nick Huntington-Klein
@nickchk.com

I have a new working paper out today, looking at the impact of educational use of ChatGPT/AI on student test scores. Importantly, this is overall student-led use (incl for cheating!), not specialized AI tutoring tools. To my surprise, I found... no effect at all! arxiv.org/abs/2605.08812

arxiv.orgLittle Impact of ChatGPT Availability on High School Student Test Score PerformanceIn educational settings, AI can be used as a learning aid, but can also be used to avoid schoolwork, thereby passing classes while learning little. Many existing studies on the impact of AI on educati...
Sean Mackinnon
@seanpmackinnon.bsky.social

Great to see you doing rsearch something important. I didn't read closely, but it did look like you're using school-level data not individual student data. I suspect / wonder if the mismatch between experimental data and this has to do with this, a sort of ecological fallacy.

Sean Mackinnon
@seanpmackinnon.bsky.social

That is, thinking about it as a multilevel model with students nested in schools, you want level 1 (individuals) not level 2 (schools). Or at least, experiments are at level 1 typically, accounting for competing findings.

Nick Huntington-Klein
@nickchk.com

This is definitely a possibility (nonzero student effects, zero aggregate effects) and it's a part of the way I interpret the results in the paper. Notably though I also find no impacts on the lower or upper ends of the distribution either.

0 reposts5 likes

Sean Mackinnon
@seanpmackinnon.bsky.social

Sounds like I should actually read more closely! Naturally though, I think about the problem like a psychologist, concerned about individual level processes. I think your results suggest teachers are managing to pivot and adapt assessment to maintain the status quo though, which is heartening!