Tweet photo with #mongoatl to win!
Fur bus will shuttle back & forth to after-party, where there are 2 free drinks
Will cover:
- Data modeling, queries, geospacial, updates, map-reduce
- Using location-based app as an example
- Examples work in MongoDB JS shell
- current location => places near location
- add user-generated content
- Record user checking
- Stats about checkin
Example document
place1 = {
name: '10gen HQ',
address: '134 5th Ave',
city: 'New York',
zip: '10011'
}
db.places.find({zip:'10011'}).limit(10)
Let's add some tags!
place1['tags'] = ['business', 'recommended']
db.places.find({zip:'10011', tags:'business'})
Let's add a location and index it as a geospacial point
places['latlong'] = [40.0,72.0]
db.places.ensureIndex({latlong:'2d'}) // this is a geo-spacial point!
db.places.find({latlong:{$near:[40,70]}})
Let's have some user-generated content too!
places['tips'] = [
{ user: 'nosh', time: 6/26/2010, tip: "stop by for office hours Wednesdays from 4-6pm" },
{ ... }, { ... }
]
Querying this database
- Set up indexes on the attributes you'll be querying
- Find by $near, by regexp for typeahead, by tag
Inserting data
db.places.insert(place1)
Updating tips:
db.places.update({name:"10gen HQ"},
{ $push: { tips:
{ user: "nosh", ... } } } })
We've now done steps 1 and 2!
user1 = {
name: "nosh",
email: "nost@10gen.com",
checkins: [ id, id ]
}
Choices: embed or reference
For checkins, we're referencing them instead of embedding -- this lets us do aggregate statistics on the checkins.
checkin1 = {
place: "10gen HQ",
ts: 9/20/2010 10:12:00,
userId: id
}
Check-in in 2 operations
-
insert checking in collection
-
update ($push) user object
ensureIndex({place: 1, ts: 1}) ensureIndex({ts: 1})
$set, $unset, $rename, $push, $pop, $pull, $addToSet, $inc
We're now done with step 3
Simple stats:
db.checkins.find({place: "10gen HQ"}).sort({ts:-1}).limit(10) // last 10 checkins
db.checkins.find({place: "10gen HQ"}, ts: {$gt: midnight}}).count() // number of checkins since midnight
mapFunc = function(){
emit(this.place, 1); }
reduceFunc = function(key, values){
return Array.sum(values); }
res = db.checkins.mapReduce(mapFunc, reduceFunc,
{ query: { timestamp: { $gt: nowMinus3Hours } } })
//=> res = [{_id: "10gen HQ", value: 17}, {...}, {...}]
res.find({ value: { $gt: 15 },
_id: { $in: [ ..., ..., ... ] } })
Single Master Deployments
- One primary/master server
- n secondary/slave server
- Configure as replica set for automated failover
- Add more secondaries to scale reads
- Good for read-heavy applications and smaller applications
Create a few replica sets and distribue data between them automatically
Config server handles distribution and balancing
MongoS is server that apps talk to, it knows where to find data
This is all transparent to the application
- RDMS replacement for high-traffice web apps
- Content-Management
- Social, mobile
- Lots of stuff!
MapReduce isn't really intended from realtime. Have a job that runs in the background to keep results fresh
MapReduce only uses one core right now, but it can work across shards. Will probably work across multiple cores soon too.
Hadoop plugin (experimental) might help get around this limitation
MongoDB.org-- tutorial, wiki, references, use cases, web shell you can try out, downloads (core & drivers)groups.google.com/groups/mongodb-user- mailing list monitored by 10genjira.mongodb.org-- bug tracking -- questions about roadmaps and feature, request a feature, etcirc.freenode.net #mongodb-- A lot of the 10gen team is on heregithub.com/mongodb-- it's open source! db, drivers, etc10gen.com/events-- we're always looking for speakersStackOverflow and Quora-- there's lots of good info here, but google group will get you quicker response
robert@10gen.com -- C# driver developer, based in Atlanta
Normalization - similar goals (to relational) also apply, but rules are different
Cheap to creake (max 24k)
Don't have a schema -- individual documents have a schema. Common for documents to share schema
Consider using multiple collections tied together by naming conventions: LogData-2011-02-08
- It's easy to clean up old data
- Can help with queries and statistics
BSON used on disk and for wire protocol
JSON used for humans
DateTime in database is always in UTC
ObjectID - binary 12bytes (24char hex is usual representation)
Denormalization (in order example) is good for snapshotting data -- what was the customer's address at the time this order was placed
Rich document is
- holistic representation
- still easy to manipulate
- pre-joined for fast retrival
Document size
- Max 4MB in earlier versions, now 16MB
- Performance considerations long before reaching max size
- GridFS is the way to store really large objects -- break them into chunks of 256K
How can we manipulate it?
What kinds of queries?
What index?
Atomic updates? (document update is always atomic -- can't be atomic across multiple documents)
Access patterns
- read/write ratio
- types up updates and queries
- data life-cycle
Considerations
- You can't join - if used together, keep together (if possible)
- Document writes are atomic
Make them map simply to application data
If you're using a higher level tool that maps objects to documents, a lot of this will be done for you
Mongo will add _id if you don't specify one. Custom one can be anything except array or document
Timestamp is embedded in id -- most drivers let you get at this data. They sort chronologically
Verify index exists
db.books.getIndexes()
{
{ _id: objectId,
ns: 'test.books',
key: { author: 1 },
name: 'author_1'
}
}
Examine the query plan
db.bookds.find({ author: '...'}).explain()
{
cursor: 'BtreeCursor author_1',
nscanned: 1, // how many were looked at
nscannedObjects: 1,
n: 1, // how many were returned
millis: 1,
indexBounds: {
author: [
[ ..., ... ]
]
}
}
Lots of operators:
- equality
({ author: '...'}) - matches (regexp)
({ author: /^e/i }) // beginning with 'e' ne,in,nin,mod,all,size,exists,type,lt,lte,gt,gte,ne
count, distinct and group are available aggregation operators
Extending the schema
db.books.update(
{ title: 'The Old Man ...' }, // the doc to update
{ // the changes to apply
$inc: { comments_count: 1 }, // will create attribute with value 0 first if necessary
$push: { comments: comment } // will create empty array first if necessary
}
$inc and $push will create correctly typed attributes if the attribute doesn't already exist
(Relational) you have to have columns for all the types, some wil be null for each type
In mongo, just add the elements that make sense for that documnet
It's fine to create an index on fields that don't occur in every document
Embedded array
{ author: ...,
comments: [
{ author: ... },
{ author: ...,
replies: [
{ author: ... } // now we have a nested tree!
]
]
}
Normalized
{ author: ...,
comment_ids: [ 1, 2 ]
}
Links can go in both or either direction for normalized
Store references on both sides
{
name: 'baseball bat',
category_ids: [ 1, 2 ]
}
{
name: 'sports equipment',
product_ids: [ 1, ... ]
}
find({ category_ids: 1})
find({ product_ids: 1})
Normalize it a little bit -- Drop product ids from category
find({category_ids: 1})
prod = find({ _id: 1 }
find({ _id: { $in: prod.category_ids }})
Full tree in document (example in one-to-many)
Very hard to search -- data is at multiple levels
Could have documents that are too large
Link from parent -> child, child -> parent, or both
Also: arrays of ancestors
{ _id : 1 }
{ _id : 2, ancestors : [1], parent: 1 }
{ _id : 2, ancestors : [1, 2], parent: 2 }
Lets you do parent queries or ancestory queries farily easily
Lets you do child or desendents very easily
Can also store string representing path to document. Using / or . as delimiter will make regexp queries awkward
Finding ancestors or siblings is difficule
Use findAndModify to get and update the document in one atomic command
Basically, you're competing with Oracle.
- Scaling:
- Buy a bigger DB server
- Mostly vertical
- Single-server durability is great until the server dies
- Schema migrations are hard with so many instances
- Closed-source = more bugs (or at least harder to know about bugs)
- SQL = easy to shoot yourself
- Licensing is hard for cloud deploys
Next-generation Video on Demand
- Premium (high-bandwidth) content to cable, pc, and mobile
- Migration path for current deployments : 50 - 200 installations
Want redundancy
New features shoud be fairly easy to add
Don't want to get backed into a corner on performance
Multi-site Oracle sucks
-
Transactions are required
- ACID only works with single database
- Many hardware systems are not RDMS anyway
- Database of Record
Authoritative source for a certain type of data - Transaction is ACID for each database you touch, but the overall flow is not
-
All data is relational
- Lots of it is hierarchical, lots of one-to-many relationships
- Relational: make it all normalized then denormalize until performance is good enough
- NoSQL tradeoff
- data redundancy
- Better performance because of co-locality of data
- When data access is down through hierarchy, it works really well
-
NoSQL is new and unproven
- Relational DBs have been dominant for quarter century
- NoSQL was around before SQL
- Data retrieval was key/value for long time
- RAIC : redundant array of inexpensive computers
Easy to use: Low friction, lots of language drivers
Avoid ORM mismatch
Get up and running quickly
Advantages
- Fast binary drivers
- C++ - no garbage collection
- Document database - no schema
- Seconday keys
- Multi-server scaling
Challenges
- Maturity - 2 years?! OMGWTFBBQ!
- Expertise - There's not a lot of experts yet
- DB Backups in sharded cluster - it's a little tricky
MongoDB is another tool, not the only tool
Blinq - Socal engagement advertising
- Provide content and technology for advertisers
- Headquarters in NY, tech and operations in Atl
- BAM - Blinq ad manager build on Facebook ads API
- Allows targeting based on OpenGraph info
- Quickly launch and manager compaigns
- Get performance data to optimize
Using: Rails, git, redis (Resque), EngineYard, MySQL for account data, MongoDB
Trying to get performance data on all ads
40 - 50 key indicators
Information needs to be available quickly
4 servers in each environment
- Master (1.6.5)
- Utility: redis and mongo slave
- App server
- MySQL DB
Use this data to
- Make basic graphs
- Pivot using MapReduce in realtime
- Compare common attributes across multiple ads
ObjectID != String
If you're not careful, you can end up with both types and things get weird
Symptoms: no data or incomplete data
Not a perfect fit with Chef
It's hard to set up things like replica sets -- you have to run some commands from the Mongo shell, and that's awkward
Schemaless == you app has to be more careful with your schema
my_field vs. my_feild won't generate errors
Your mapper can sometimes help with this
It's a pain to set up with replication with MySQL and there's not a lot of feedback that it's right
It's dead easy with Mongo
Getting started is quick (compare: PostgreSQL)
2009: FossFor.us black ops project with user-generated content, new design
- Started out using CouchDB
- Adding fields was trival and happened frequently
- Scaling Couch to SF.net level of traffice didn't work
- Tried out some alternative, landed on MongoDB
This experiment was successful, so start moving it to SF.net
Most traffic (90%) is on 3 pages
- Project summary
- File browser
- Download
Pages are read-heavy with infrequent updates from "Develop" side
Original idea is 1 MongoDB document per project
Periodic updates via RSS and AMQP from "Develop"
Ended up moving releases into separate docs because of extremely frequest releases on some projects bumping over 4MB limit
Consume is Python (TurboGears) app
Each server has its own slave
There's a single server that takes all the updates
Mongo's performance was actually so good the slaves weren't necessary
There's a single slave for durability
New service in late 2009 -- allow non-SourceForce projects to use mirror network
Stats calculated in Hadoop and stored/server in MongoDB
Same basic architecture as Consume
Was eventually merged into Consume codebase
Rewrite developer tools (change from PHP/db mash to Python and MongoDB)
More tools: wiki, tracker, discussion, Git, Hg, SVN
Using a single MongoDB replica set manually sharded by project
There's a database per project, but that's a bad idea because there's a lot of of overhead
Release early and often, but move people over slowly
Performance -- 90% of traffic on one database works great
Schemaless server allows fast schema evolution in development
Ususally you don't need any migrations at all
Replication is easy, making scalability and backups easy as well Query language is very nice -- good performance without having to use map/reduce
GridFS is wonderful -- also, it's a filesystem shared between all your webservers without setting up NFS, Samba, etc.
Too-large documents -- 4MB docs = 30 docs/sec (slow)
Ignoring indexing -- it's easy to forget in develpment
It's fast until it's not
Bad queries show up in your server log, so keep an eye on it
Always have an index on your sort field
If there's too much data, Mongo just won't do it if it's not indexed
Ignoring schema can be a problem
Don't use more databases than you need to -- there's a min 200MB of disk space
Using too many queries is bad
Keep an eye on your ORM -- it might be generating too many
sf.net/projects/merciless
Object-Document Mapper for Python
Your data does have a schema
- Your schema lives in your app instead of DB
- It's nice if the schema lives in one place
Sometimes you need a migration
- Changing the structure/meaning of a field
- Adding indexes
- Sometimes lazy, sometimes eager -- Ming supports both
You can define multiple schemas and it will migrate to the newest when you access a record
Modify objects in memory, they'll be automatically updated at the end of the web request
You can use objects instead of dicts
Inspired by SQLAlchemy but not as full-featured
MIM -- "Mongo in Memory" for unit tests
Table scan -- look at every record in the database
Tree -- sorted so you can look at fewer objects
db.setProfilingLevel( level )
0 - off
1 - slow operations (>100ms)
2- all operations
db.system.profile.find({millis:{$gt:5}})
{
ts: timestamp,
info: "query test.foo ntoreturn:0 exception bytes:53",
millis: 88
}
When you use an index, nscscanned will be index objects looked at, nscannedObjects is objects scanned after that
db.coll.find({title:"My Blog"}).explain()
{
cursor: "BasicCursor",
indexBounds: [],
nscanned: 57594, // how many we looked at
nscannedObjects: 57594,
n: 3,
millis: 108
}
db.posts.ensureIndex({name: 1})
1 = ascending
-1 = descending
db.posts.ensureIndex({name: 1, date: -1})
Index above useful for posts from a person, sorted by date
db.posts.ensureIndex({title: 1}, {unique: true})
You can't add a second document with the same title
db.posts.save({
title: "...",
comments: [
{ author: '...' }
]
})
db.posts.ensureIndex({"comments.author": 1})
Multikeys
{ tags: ["mongodb", "cool"] }
db.posts.ensureIndex({tags: 1})
Automatically indexes all the items in the array
New in 1.7.4
Usually, tree really just has ObjectID, then you pull it out of collection
When all the fields desired are in the index itself, it can pull straight from the index instead of looking in collection
Make sure you exclude _id from returned fields
New in 1.7.4
Include documents in index only if the indexed value is present
Limited to a single field
db.people.ensureIndex({title: 1}, {sparse: true})
Set x/y coordinates and then use 2d index type
Get objects near or certain distance from a point
Spherical support added in 1.7.0
Implemented on top of b-tree indexes
Geo-hashing plus grid-by-grid search
db.posts.ensureIndex({ latlong: '2d' })
Can be used in a compound index
Listing your indexes
db.posts.getIndexes()
db.posts.dropIndex( ... )
Creating an index is a blocking operation. However, you can build them in the background
db.posts.ensureIndex(..., {background: true})
How the database decides what index to use
Index will be used for:
- find on key
$inquery for key- range query for key
- count query for key -- done from just the index
- sort on key
- find on key, returning only key -- done from just the index
Partial index use:
db.coll.find({b: 0}).sort({a: -1})uses index for sorting, but not query
If you index on { a: 1, b: -1 }, you can't use the index to:
db.collection.find({b: 0})
But you can use it for a query that's on just a
Starts running all the possibly query plans in parallel
Remembers whichever one returns the quickest
Keeps track of number of updates, redoes query planning every 100 updates
Chooses query plan based on pattern of query -- "equality match on x" vs. "where x = 1"
With shards, query planning runs on each shard independently
Using MongoDB with Deep and Diverse Datasets
Jeremy at Intridia -- @jm
Also wickhamhousebrand.com for bowties!
Working on Document Design for MongoDB from O'Reilly
Pharm MD - Audit medicines prescribed by lots of doctors
- Embedding works!
- But be careful -- you can run into performance problems
- Every write would overrun allocated space so it got really slow
- Transforming data is fun
- Formless data is annoying
- Arbitrary embedding is awesome
- Building to work with schemaless data can lead to really powerful app concepts
Be wary. It can crush you.
- Weird app behavior when you're expecting a value
- Huge, long-running data transformations (eg, turning embedded documents into their own collection)
- Annoying data transforms for dev environment
- Difficult to version data models
Server stats
- opcounters : insert, query, update, delete, getmore, command
- connections: current, available
mongostat is like top for mongo
admin console -- HTML representation of server stats
db._adminCommand({diagLogging: 1})
db.currentOp()
{
insert: [ { opid: 35, op: "query", ns: "fundb.parties" } ]
}
mongo, mongod, and mongos are key binaries
Configurations settings for mongod:
--forkfork the server process--logpathto output log to a file
Mongo has some very basic security, but prefers to run in a trusted environment
Reserved databases: admin, local
Special query to run commands
db.runCommand("isMaster")
db.commandHelp("isMaster")
See what commands are currently in progress
db.currentOp()
Use mongodump to get a binary image of a running database
Most people just back up from slave
Oplog is a capped collection in local database
Mongo will automatically copy initial image from master to slave
Two types of replication:
- Master/slave has no automatic promotion
- Replica sets include automatic promotion
Slave delay option -- keep a slave intentionally behind by some amount of time to protect against fat-fingering
Combining multiple data stores in one application or system
Every database has strengths and weaknesses -- use the best for the situation
Keep in mind that they are separate, and querying across them is difficult
You may lose your speed gains by trying to work across
Good at handling highly relational data
Complex graphs with a lot of edges
Use for
- Inventory records of different product option combinations
- User records
- Financial transaction records
- Order records
Encapuslated models with well-defined boundaries
Concise, finite trees
Well, "documents"
Use for
- Order and onderline details for reporting
- References to uploaded assets
Things you only need to look up one way
Very, very fast access
Use for
- Temporary storage of uploaded binary data
- Cross-cloud configuration states
Different methods depending on the nature of the relationship
Simplest is 1-1 between RDMS and K/V store -- just store the key in a RDMS column
That pretty much also applies to 1-1 relationship to MongoDB record, but in reverse
For 1-to-many, lots of people serialize an array and stick it in a text column
Fast and easy, using $in query in Mongo
Using the $in query won't preserve order
Another 1-to-many option, add a key on the document pointing to RDMS record
Can't use it for bidirectional relationships
Usually the best approach when you know about it up-front, but watch out when implementing after the fact (if you use sharding)
Can also make the relationship into an embedded document to add more info:
{ ...,
owner_relationship: {
owner_id: 5,
sort: 2
}
}
Many-to-many can use an array of ids on the Mongo side
Obviously, index this stuff! You're going to be doing a lot of queries on it!
Replication -- high availability and data safety
Sharding -- scalability
Always asynchronous
Master/slave or replica sets
Consensus election to determine primary server
Automatic failover and recovery
Writes always go to primary
Reads can go to any
Defaults to reading from primary unless you say secondary is OK
A write is truly committed once it's on a majority of members of the set
Use getLastError to check that it's been replicated out
Setting up:
- Give the set a name
- Run initiation command
- Allow one to be set as always a slave (priority = 0)
- Two-node replica set with 1 node down can't elect a primary -- define an arbiter that can vote but doesn't store data
Automatically shards and reshards
It's the resharding that's the tricky part
Shard migration is done in the background
In fact, as far as clients are concerned, the chunks don't exist
There can (and will) be more chuncks than shards
Shards are a place for storing chunks, not a 1-1 mapping
Sharding is similar to BigTable implementation -- everything else is very different
v1.6 -- August 2010
- Sharding, replica sets
v1.8 -- Under development (1-3 weeks away)
- Single-server durability (storage engine journaling for crash safety)
- Enhancements to sharding/replica sets
- Spherical coordinates
- Covered indexes
- $rename field
Later
- Better map/reduce and aggregation capabilities
- Full-text search
- TTL timeout collections
- Collection-level concurrency, eventually document-level