I managed to recover this post by checking Google's cached copy as it was randomly taken down; Enjoy!
Surpassing 100GB of data in your application requires you to have in-depth knowledge of how to operate and run MongoDB. MongoHQ recommends going through the 100GB Scaling Checklist as you grow. Watch the webinar recording on the subject for the full overview:
- Identify your data behavior: Figure out how your data patterns and how they are working within your application. You will need to link your data to how your application accesses this data. Consider the simple queries and the more complex queries you will need to look up, like multi-range queries.
- Refactor your schema to simplify queries
- Remove data that does not fit MongoDB: remove “unrefactorable” data
- Separate hot and cold data
- Don’t lean on mongodump’: this disrupts RAM and causes performance issues. Consider other Backup options instead, like MMS Backup
- Check your gauges: Monitor, monitor, monitor. Even if you aren’t having performance problems, set this up now so you can keep a history of your
- Avoid queries causing page faults: MongoHQ has run benchmarks against this to prove this. A system running in memory that was running at 7,000 operations per second was cut down by 50% to 3,500 operations per second when adding 1% table scans churning on a disk.
- Track and monitor slow queries: use Dex, MongoProfessor, Mongo-QP or MongoHQ’s Slow Query Tracker.
- Buying time with hardware: Don’t get addicted to buying hardware. Before making a purchase, always consider optimization and investigate separating and pairing data.
Watch the full recording with tips from MongoHQ’s Chris Winslet here.