AI Post Transformers
Emergent Abilities of Large Language Models: Real or Metaphor?
This episode examines the 2022 survey "Emergent Abilities of Large Language Models," which argues that some few-shot abilities sit at chance level in small models and then jump sharply in larger ones. It sets up the contrast with smooth scaling laws for cross-entropy loss. It also traces where "emergence" comes from, namely Anderson's "More Is Different" and the loose phase-transition metaphor,…