Skip to content

Instantly share code, notes, and snippets.

@cedrickchee
Created May 16, 2024 05:44
Show Gist options
  • Save cedrickchee/4718a35f79c215b821f7ebed7a6207c8 to your computer and use it in GitHub Desktop.
Save cedrickchee/4718a35f79c215b821f7ebed7a6207c8 to your computer and use it in GitHub Desktop.
The Bitter Lesson by Rich Sutton

The Bitter Lesson by Rich Sutton

From the 2019 essay "The Bitter Lesson" by Rich Sutton:

Summary: AI research shows that leveraging computation through general methods like search and learning is far more effective than incorporating human knowledge. As computational power grows, these methods outperform human-centric approaches in fields like chess, Go, speech recognition, and vision. The key takeaway: focus on scalable computational methods, not mimicking human thought.

I wish every graduate student in AI would read "The Bitter Lesson".1

Arguments

I think people have generally taken this essay far too seriously.2

The point of the Bitter Lesson is that research and clever ideas are important, but people should think about how their ideas scale with data and compute rather than just relying on One Weird Trick to get them a little farther than SOTA.3

I think the objective of Sutton's essay was to advance the science of AI, not so much career trajectories of students.

The real bitter lesson: you need money to train a model, and to make money you need to build good products, and to make good products you need to pick the loss that matters to customers.4


FYI, Rich Sutton is a well-known Reinforcement Learning for AI research scientist.

Category: #TechInnovation, #AI, #MachineLearning

Footnotes

  1. Noam Brown's Tweet

  2. https://twitter.com/jxmnop/status/1789371442158530920

  3. https://twitter.com/polynoamial/status/1789381426187546644

  4. https://twitter.com/yacineMTB/status/1789372639628161332

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment