LessWrong (30+ Karma)

“Generalized belief states in transformers” by Artemy Kolchinsky

41:01

Audio note: this post contains a lot of mathematical notation. We narrate the simpler formulas and flag the complex ones, so you may want the original text to hand.

Many thanks to Xavier Poncini, Paul Riechers, Fernando Rosas, and Matt Smith for useful conversations.

1. Introduction

Despite the recent successes of transformers and other sequence models, there are still many open questions…