Transformers and self-attention: the research breakthroughs behind ChatGPT

What are the research breakthroughs behind ChatGPT? In recent times, LLMs have been all the rage and especially transformers and "self-attention". We got a chance to talk to Noam Shazeer of Character.ai / ex-Google who was a key contributor to several of these.

Noam covered a bunch in our conversation and Aarthi Ramamurthy and I had a blast (links at the bottom)

1. How he got into Google in the early 2000s, including his famous interview with Paul Bucheit that lead to a rewrite of the Google spellchecker.
2. The history of AI research, especially key momemts in the 2000s.
3. What is a "transformer"? And the story behind the legendary "Attention is all you need" paper
4. What is "self-attention"?
5. What are key limitations/step-function changes?
6. What are future apps for AI that are interesting?

Warning: this is dense - but fun. Leave a comment below and we will try and answer!

Watch/listen:

1. Spotify
2. Apple.
3. Youtube (best option for comments!)