terriyu/diary-9jul2013.md

## diary-9jul2013.md

      
    Raw
  

              diary-9jul2013.md
            
          
    9 Jul 2013

Note on using Markdown in Gist


Mixed nested lists don't work in GitHub flavored Markdown, as seen here:

Mixing
Ordered (numbered) lists
With unordered (bulleted) lists
Doesn't work -- gets rendered as a code block


More on MIM

I asked jd "what is MIM and why is it used?"  This is his answer:

it's a Python library providing the same API as PyMongo, but which instead
of connecting to MongoDB like Pymongo does, implements the API internally
with Python code + a JavaScript via Spidermonkey


so the unit tests thinks we are using Pymongo and talking with MongoDB,
but actually that's not true


why we picked it up, that's because we didn't want to have to run MongoDB
on the CI infrastructure back in time


and for our usage of MongoDB, MIM was good enough


now as you can see, we're having problem in MongoDB we can't see in MIM,
so that's causing major issues


plus the fact that I want to use advanced function in MongoDB like
aggregate() and MIM doesn't implement that

Bug I'm working on, "unable to sort data with MongoDB"

Status


The bug report is here: https://bugs.launchpad.net/ceilometer/+bug/1193906


Successfully reproduced the error in the bug report:
OperationFailure: database error: too much data for sort() with no index. add an index or specify a smaller limit


Attempted to fix the bug by creating an index for timestamp, but so far, it
hasn't worked


My patch so far: https://review.openstack.org/#/c/36159/


Indexing references and notes


MongoDB manual on indexing: http://docs.mongodb.org/manual/core/indexes/


MongoDB API references (createIndex() is deprecated): http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex/


PyMongo API references: http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.create_index and http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.ensure_index


Sorting doesn't work if the sort operation consumes more than 32 MB:
http://docs.mongodb.org/manual/reference/method/cursor.sort/.  In these
cases, you need an index.


You can either create an index or ensure an index.  Ensuring an index only
creates an index if it doesn't already exist.  createIndex() is deprecated
in MongoDB, but you can do either create_index() or ensure_index() in
PyMongo.


For single field indexes, order doesn't matter.  From the MongoDB manual http://docs.mongodb.org/manual/core/indexes/#indexes-with-ascending-and-descending-keys,

For single-field indexes, the order of keys doesn’t matter, because
MongoDB can traverse the index in either direction. However, for compound
indexes, if you need to order results against two fields, sometimes you
need the index fields running in opposite order relative to each other.


You can force MongoDB to use a particular index by passing it a hint

References: http://api.mongodb.org/python/current/api/pymongo/cursor.html#pymongo.cursor.Cursor.hint and http://docs.mongodb.org/manual/reference/method/cursor.hint/#cursor.hint


Attempt to add index and fix the bug


In this section, I'm working on the file
ceilometer/storage/impl_mongodb.py


I inserted a line
self.db.meter.create_index([('timestamp', pymongo.DESCENDING)], name='timestamp_idx')

in __init__() of Class Connection.


Since it's an index on a single field, I don't have to specify the order, so
I changed this to
self.db.meter.create_index('timestamp', name='timestamp_idx')


Tried both create_index() and ensure_index() but neither worked; I still
got the error about too much data to sort properly and needing an index.


Tried providing hints in get_samples() -- the function I'm testing, but
this didn't work either.
samples = self.db.meter.find(q, sort=[("timestamp", pymongo.DESCENDING)], hint='timestamp_idx')

samples = self.db.meter.find(q, sort=[("timestamp", pymongo.DESCENDING)], hint='timestamp')


Debugging


In PyMongo, you use the function index_information() to get information on
a collection's indexes.  It returns a dictionary containing all the defined indexes.  Reference: http://api.mongodb.org/python/current/api/pymongo/collection.html#pymongo.collection.Collection.index_information


I used an assert statement to find out the value of
db.meter.index_information():
assert False, self.db.meter.index_information()

Inside __init__(), the value was
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1}, u'timestamp_idx': {u'key': [(u'timestamp', 1)], u'v': 1}, u'meter_idx': {u'key': [(u'resource_id', 1), (u'user_id', 1), (u'counter_name', 1), (u'timestamp', 1), (u'source', 1)], u'v': 1}}

Inside get_samples(), the value was
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1}}

So it seems like the index is created and visible where it is defined in
__init__(), but it either doesn't exist or isn't visible to
get_samples()
I also tried pulling the latest changes and rebasing, but the value of
db.meter.index_information() inside get_samples() is still
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1}}


I tried going back to the MIM implementation instead of using the MongoDB
instance, by running the command
$ unset CEILOMETER_TEST_MONGODB_URL

This deletes the CEILOMETER_TEST_MONGODB_URL environment variable.  When
Ceilometer can't find this variable, it defaults to using MIM.
Inside __init__(), the value was
{'timestamp_idx': {'key': (('timestamp', 1),)}, 'meter_idx': {'key': (('resource_id', 1), ('project_id', 1), ('counter_name', 1), ('timestamp', 1), ('source', 1))}}

Inside get_samples(), the value of index_information() was
{'timestamp_idx': {'key': (('timestamp', 1),)}, 'meter_idx': {'key': (('resource_id', 1), ('project_id', 1), ('counter_name', 1), ('timestamp', 1), ('source', 1))}}

So interestingly enough, with the MIM implementation, the indexes are
present in get_samples()