niplav is

Current LLMs have mesa-optimizers inside the CoT blobs. We were previously thinking about mesa-optimizers about being weights-level, but that didn't have to be the case, I guess?

The concept "capabilities externalities" is pretty bad imho. So many things that should be disambiguated. Thought stopper. Applause light.

How could I become highly persistent

It's interesting how Hinton is basically in the "eminent physicist pontificates about things they have very little knowledge¹ of" phase of his career, and the AI safety crowd is still shilling him out for respectability ¹: includes AI x-risk arguments, object-level takes are mid

Jeffrey Ladish gave up being a billionaire to found Palisade

This is a very clear and well-written book: www.goodreads.com/book/show/63... I think basically my standard for good German writing?

I'm coming around to believing that us trying to Hail Mary-beam our civ info into being emulated by benevolent grabby aliens is not a completely crazy idea www.lesswrong.com/posts/2xHhe4...

Captcha providers and AI companies could engage to have the AIs have "cognitive blindspots" so that there are still captchas that humans can solve but LLMs can not

Lexical threshold negative utilitarianism via log barrier

Ask frontier LLMs for: • Solutions to shutdown problem • {Embedded, Cartesian} diamond maximizer • Solution to the Löbstacle • Open source game theory program equilibrium solutions

I revealed you my preference, please respond

6fd7fbb3b81dbc0731ef61ae568db0c2383f844779c7ae556450e3f6393122a1

Convergent instrumental goals as another Darwinism-sized shift?

Recently: How much do I remember about the time when I formed this belief, so that I can treat events as being "priced in" already, vs. actually new? My memory isn't that good, so I don't have the full trace. Internet helps a bit

nominative biodeterminism

A distrust license is a situation where A has to give some level of trust to B, and the level of trust given is *also* a status signal, but B gives the license by pointing out the level of trust B thinks A should have on them given the evidence A has received about B so far.

Hm. AI speeds up history by a bunch, humanity would've died out in history far away, which is brought closer by said speedup. & species often get outcompeted by being reconnected with near-speciated rivals, I tried to get a baserate for this (endogenous vs. exogenous) but failed

It feels like ~60% of the Mythos transcript¹ is spent doing one of three things solving captchas getting a telephone number getting an email address If I were an AI agent these would be the three that'd most annoy me about humanity ¹: t.co/hktEfnHTR8

"This is my quant" but for Caroline Ellison joining Manifund

Alright everyone time to start talking about UMAP plots of genomes and kin-selection mechanisms to even it all out

I miss agent foundations, it was fun before we were forced to do policy and evals (Opus 5 on LittleDarkAgeBench)

It seems like "we all get killed by our own invention" is less virtuous than "we all get killed by our enemy's invention"?

"sitting on the front porch with a rifle" physiognomy

Big fan of the weird body {tilt, rotation, heaviness, expansion, valence-neutral contraction} feelings that arise in Body & Mind. "Yes, I'd like to feel like my torso is tilted/rotated by 25° towards my left, thank you very much" … "Oh, and weighing 300kg is also in the cards!"