Post

Did you know that attention across the whole input span was inspired by the time-negating alien language in Arrival? Crazy anecdote from the latest Hard Fork podcast (by @kevinroose.com and @caseynewton.bsky.social). HT nwbrownboi on Threads for the lead.

Victor Geislinger

@victorsvector.com

Also, here's the link to that part of the podcast: youtu.be/RCNwOQ_9Kg4?... I'm not aware of the 'primary source' of this inspiration but would love it if anyone has a direct source they're referring to

youtu.beHard Fork 100 Most Iconic Technologies | EP 111YouTube video by Hard Fork

1/12/24, 3:12 pm0 reposts2 likes

Victor Geislinger

@victorsvector.com

1 Dec 2024

Linking here the discussion on this thread— it's unlikely that the inspiration actually was from Ted Chiang's 'Story of Your Life' (Arrival movie) bsky.app/profile/tilm... Perhaps a good analogy but almost certainly *not* a direct inspiration for self-attention

Tilman Bayer@tilmanbayer.bsky.social

OK, so maybe what Kevin Roose bungled here was that one of the paper's authors (Polosukhin) had made a comparison to "Arrival" ("believed self-attention was a bit like ...") - but without claiming it had been the actual inspiration www.ft.com/content/37bb...