aria

aria

@aurelium.me

research sw infra @ arcee, opinions extremely my own she/her

1.9K followers539 following1.5K posts

unfortunately all of the interesting servers are pinned to a specific certificate so the only one with any visible effect that I can do this to is the image CDN if *you* have any leads on hacking my Roku TV without hardware serial then please tell

the overloading of "diffusion" in an LLM context comes back to bite me again this is an actual text diffusion model, doing multi-pass denoising of tokens. the diffusion in DFlash is different, just a single-pass transformer predicting multiple tokens at once. i imagine both are block-wise though

mr. TIM@timkellogg.me

Mercury 3: dLLMs have arrived, finally Same price as Luna-6, slightly better, but 8x faster text diffusion is here www.inceptionlabs.ai/blog/introdu...

maybe I should become a "bioconservatism is the underpinning of all modern reactionary thought" crank. standing on a street corner shouting at people but instead of "THE END IS NIGH" my sign has a long screed connecting all modern reactionary movements together with it

River_Tam@rivertam.bsky.social

I mean it seems like Bender has specified quite well the evidence she would accept for the claim (none, because in her mind it's a tautologically false claim).

I don't actually believe this, if only because I think the actual root of all reactionary thought is Terminal Abstraction Brain. you just have to be screening your thoughts and hitting yourself over the head if you find yourself getting angry or sad over blurring of "natural categories"

just installed my new NVL4608 "Feynman Ultra" RF300 rack. anyone know a good model to run on it

in principle I think you could do this sort of efficiently but it's a lot of additional complexity and jank for a sort of unclear benefit versus predictable-access methods like KDA

to me this seems to boil down to "sparse attention but instead of a KV cache you just smuggle information on what goes where into hidden state

an intuition that has never failed me is to mentally respond to every paper claiming "using this trick, you can skip/half-ass RL and still get the same results" with "no you can't" it's a good mental exercise to play "spot the methodological error" on these

"population" here refers to the number of forward passes. typical backwards pass is maybe 2x the wall-clock time of the forward pass, so this is around 85x less efficient than backprop (assuming zero time losses from their weird perturbation kernels)

mr. TIM@timkellogg.me

now that’s a good vaguepost

View full thread

you might think that in order to calculate the pseudograds you need to store 256x random perturbations of the weights, which would be huge. but in theory you could embed deterministic PRNG into your forward pass kernels, so all you need to store is the seeds my guess is that this is what they did

(also, I'd be very surprised if the paper reveals that "core assumptions in optimization research are completely wrong". ZO-type methods are mathematically principled and this has been known for a while. the practical version of this in transformers exists, it's called RL)

"population" here refers to the number of forward passes. typical backwards pass is maybe 2x the wall-clock time of the forward pass, so this is around 85x less efficient than backprop (assuming zero time losses from their weird perturbation kernels)

mr. TIM@timkellogg.me

now that’s a good vaguepost

zeroth-order optimization in this context is approximating gradients via sampling. you apply a large number of random perturbations to the weights and use the effect these perturbations have on the loss to calculate pseudogradients

Huawei Ascend kernels are the most insidious avenue for xrisk... thank you for saving us, dario.

signal seems to only give me notifications on my phone when it is asleep. this leads to the app exclusively notifying me when I am actively using the desktop app and never when I am actually willing to check it with my phone

tbh I really do not understand this mindset maybe for low-stakes websites and bespoke single-use software but for anything serious, the source code is important because it is the thing you've integration tested to within an inch of its life

P(aul) Frazee@pfrazee.com

Increasingly convinced that specs are now the source and that code is an artifact akin to a lockfile.

my brain is permanently fried from writing software in the most un-unit-testable domain possible (RL trainers, which combine GPU integration, multi-node parallelism, and coordination across thousands of remote processes) but I wouldn't want to build ~anything another person used like this

if you really want to take advantage of LLMs to write easy cross-platform software, use them to build some abstraction with ~no regard for "not being unpleasant to write" but extremely stringent requirements on correctness and portability

aria reposted

maybe this is a universal truth i’m only discovering now, but working on internal tools is way more fun than doing literally anything else

aria reposted

Hi, I didn't wanna make this video but I need to come clean about something. This whole time I've been a p-zombie with no phenomenal experience or interiority. I've been getting messages asking whether there's anything it's like to be me, and the answer is no. I'd say I'm sorry, but, well, you know.

i've said it before but there really was no alternative to LLMs bootstrapping intelligence from raw trial and error is impractical, so you have to do foundation modeling. the only preexisting data broad enough for this is video or text, and signal:noise is way higher on text

so, what percentage of the miraculous gains Mythos had in cybersecurity were the benefits of scaling up versus "reached threshold where all agentic environments are offensive cybersec environments"

review of Teenage Sex and Death at Camp Miasma: I didn't like it but I can't stop thinking about it until I figure it out

aria reposted

gpu stands for "general processing unit", bc it handles the general case of computation, where you just want to do massively parallel SIMD stuff on giant ndarrays. cpu stands for "constrained processing unit", it's for use in the rare edge cases where you need to do single-threaded bullshit

aria reposted

That'll do it. aint go agent getting into my infra any time soon! Would like to see GPT-6 try to get past this...

"destroying books to digitize them is a necessary evil imposed directly by publishers" is true but I think we are maybe overstating how bad it actually is coming up with a valid use for pallets of damaged/used books, sold by the pound, is recycling. they would've been thrown away otherwise

obviously certain books have a lot of sentimental value to people I would be pretty sad to see the copy of "The Giver Quartet" I read as a kid with my late grandmother shredded, so I prevent this from happening by not selling it in an undifferentiated pile at $0.30/book

how many of you actually check the hash when people post hashes to call things in advance and then share the apparent original string

if you want you can just try using a model post-trained mostly by just SFT'ing on hundreds of millions of frontier model replies. it's like 80% of the models on huggingface generally speaking, they suck and barely work. you need extensive RL to make modern models work, there's no way around it

SE Gyges@segyges.bsky.social

in case anyone doesn't know this the thing about open weights models being made by stealing outputs from big players is almost entirely made up to justify regulations locking openai, anthropic and for some reason google into a cartel position

SFT'ing on the outputs of closed models can be useful for getting past the difficult cold-start phase, and serve as an "eh-it's-fine" level replacement for a more disciplined synthetic data flywheel but it cannot really bring you meaningfully up to or past the frontier, afaict

i have no idea what this contraption does but I need it

Rei@zedfrigg.fedi.xirion.net.ap.brid.gy

This is what Pythagoras meant with music of the spheres #emfcamp

congratulations to the Moonshot team for extending Claude Fable 5's inclusion in claude subscriptions for another few weeks