- Let's use "cost" as a proxy for CO2 emissions, because within an order of magnitude I expect $1 on gas emits as much CO2 as $1 of power for a server, and I believe the bulk of the ongoing cost of operating a server is power budget.
- Lifetime cost of a gas for a car: Let's estimate 100,000 miles, and 20 mpg and $4/gal. That gives us 5,000 gal and $20,000 in lifetime gas cost for a car. These numbers all feel pessimistic so we're maybe high.
- A beefy server in AWS or GCP costs something like $1/hr, probably more for the really big ones. So in order to get to $20,000/run, we need 20,000 instance-hours. Let's ballpark a large training run at 1 day, so that's ~800 instances for one day. That feels within the realm of the numbers I've seen in papers.
Conclusion: Entirely plausible. Other conclusion: Lifetime cost of gasoline for a car is (in round numbers) the same as the cost of a new car. I hadn't realized they were so close. Other other conclusion: Now I understand why OpenAI needs billions o