It's close to the end of the year, so it's an ideal time to start making new years resolutions! I don't know why everyone makes them in January, then they have to stick with them for 12 months. That's why I'm making mine now, I only need to stick with it for a bit over 2 months.

Anyway, my October's new year resolution is to start reading more papers, and writing up some notes on them as a learning exercise. I'm thinking of it as a kind of solo papers club.

I thought I'd start by working through the "Deep learning reading list from Ilya Sutskever" which is a rumored list given to John Carmack that probably didn't exist in real life given the main citation for it I can find is just "some guy on twitter said it exists" but it does seem like a really interesting grounding in a lot of modern deep learning, which I haven't touched in a while.

I started with the Annotated Transformer, and BOY this is a dense piece of work. I mean that in the best possible way though - there's a lot here, and it's not only not-afraid to go into the weeds, but the whole point of it kinda is to be a "let's go into the weeds" version of "Attention is All You Need" (AIAYN).

I remember AIAYN from a while back, and had sort of skim level understood the main idea, which is to provide a way for models parsing tokens to estimate the importance of previous tokens rather than basing themselves entirely on recency. Being reminded of just that central insight was very cool, and it's interesting the example is German/English translation, since I'm sure-ish (I don't speak German) that German puts verbs into the end of sentences, so is an extra clear case of why recency alone doesn't determine token relevance.

In skimming it before though, I'd missed how much creativity there is in the design of the model architecture, the whole thing is full of small decisions with huge amounts of creativity in them: like using sin wave functions with token number to encode a sense of recency into the model still; using label smoothing to prevent model overconfidence (I don't get the impression this was standard at the time); the recursive and multi-layered architecture (as in, it's an architecture that requires being multi-layered and recursive, those aren't just extra steps added into an existing model).

I hadn't read this one before, and really enjoyed the hefty detail - would be great to revisit this one again in six months or so, and see what details jump out then.