I did a bit of benchmarking using our rspec suite and the most cost effective flavor seems to be "c1.medium". It's almost twice as fast as s1.small but still has the same per-core price. The micro instances were about 4 times slower than c1.medium.
The c1.xlarge instance did not perform better than c1.medium (actually worse) and costs as much per core.
I did not try the cluster nodes as they require you to create your own image to use it.
The spot instance price of c1.medium seems to be stable at $0.06 / hour ($0.17 normal). That would be about $100 a month for 10 instances running 9 hours a day, 20 days a month.
Our ~30-35 minute test suite using:
- 10 x c1.medium instances (20 cores): 2.8 minutes
- 15 x c1.medium instances (30 cores): 2.3 minutes
- 15 x c1.medium, and our ordinary hardware (57 cores): 1.8 minutes
- Just the ordinary hardware (27 cores): 2.4 minutes
A few limiting factors here are:
- Some of our spec files take up to 60 seconds to run
- Testbot could do a better job of balancing test load
- Syncing / communication (actually only about 15 secs)
In case you're curious, our ordinary hardware is:
- 8 "old" computers (core 2 duo, 2.x ghz) that we PXE boot to not have to maintain separate installations.
- 3 virtual computers using spare resources in our office and dev servers.