Post

interesting connection, but the transformer paper didn’t invent attention? arxiv.org/abs/1409.0473

Yeah I wondered about the conflation they’re making. I’d like to know who they’re talking about and hear their version of the story—which might make things clearer.

reading it again, maybe what the anonymous researcher meant is that the film inspired them to get rid of recurrence

It wasn't even the transformer paper that was first to show attention was all you need. Everyone forgets how aggressively folks were working on faster alternatives to RNNs ~2016, and another paper from Google did a pure attention model first: arxiv.org/abs/1606.01933

Why do you think AIAY got all the attention? Genuinely curious

They did the generation component + important pieces for making the model deeper, then a different set of authors did the BERT paper. The other bit of history is that ELMo did pretraining first using RNNs, but BERT was able to scale this up because of transformers.

Memory networks were earlier, attention only, and had position embeddings, but were not word/token level: arxiv.org/abs/1503.08895 They were later elaborated with the key-value distinction which is, AFAIK, where this terminology arises: arxiv.org/abs/1606.03126

Memory networks are amazing. Although they are not attention only. The first sentence of the abstract makes it clear they’re recurrent models. > We introduce a neural network with a recurrent attention model over a possibly large external memory.

I just bought the book. If it's not true, I think it should be. And I also understood the difficulty, as a Portuguese speaker, to understand English posts in social media: you use nouns as verbs. The sentence "I was about to history" makes no sense in Portuguese. It needs the verb "to do".

The reply with this sentence was deleted before I answer.

Also, here's the link to that part of the podcast: youtu.be/RCNwOQ_9Kg4?... I'm not aware of the 'primary source' of this inspiration but would love it if anyone has a direct source they're referring to

Regardless of whether this intuition was the original intuition for attention, this very idea of processing all tokens is it's Achilles heel in a world of data with the arrow of time, where compute, data is not infinite. Recurrence is a powerful inductive bias that reduces search space.

original person to catch this: didn't know he was also on bsky! bsky.app/profile/nwbr...

If I were a historian of science I would have to seriously evaluate how much causal weight this “inspiration” has. But since I’m just a literary historian I can enjoy the irony that Ted Chiang created the technology that produces his “blurry jpegs of the internet.” “Blurry JPEG of your life.”

BTW, the alien language was inspired by Wolfram who consulted on that movie!

Presumably this just makes the machines quicker at inference, and not (taking the hard Sapir-Worf hypothesis to its extreme) able to see all of time at once, like the heptapods (and the Tralfamadorians before them).

causal weight i'd guess basically none, they tried ten thousand other things (many of those also inspired by many colorful sources) that don't get written up and the ones that worked, stuck. but i'm not a historian of science either

'Blurry JPEG of Your Life' This is perfection

The cost is that now I’m humming the One Direction song “Story of MY Life” but close enough

something is missing - attention is widely acknowledged to be a 2015 innovation - everyone cites Bahdanau - RNNs, I believe, previously took one word at a time and were autoregressive - etc...

otoh, this makes sense. after, all, causality going backwards in time is one of the main themes in the film