OK. In an effort to learn this stuff (on a Saturday night of course), I also built the tables, indexes, and created fake data.
Here is the big realization: The join is expensive and pointless - it is not needed for the heavy lifting (counting/sorting). Split up into two queries (or a subquery).
First lets do it the original way:
# explain ANALYZE SELECT races.id, races.title, count(participants.id) AS participant_count
FROM races
INNER JOIN participants ON races.id=participants.race_id