diary-21aug2013.md

21 Aug 2013

Gerrit automatic rebase upon commit

I see that in the man git review documentation, Gerrit runs git rebase -i by default every time I commit.

jpich says:

That's something done silently (unless it goes horribly wrong), so that your patch always sits on top of the latest commit. I've noticed sometimes when I looked at the parent for my patch on Gerrit it's different than what I have locally, because I hadn't rebased for a few days.

Group by blueprint

• group by blueprint on Launchpad: https://blueprints.launchpad.net/ceilometer/+spec/api-group-by
• group by blueprint discussion on the OpenStack wiki: https://wiki.openstack.org/wiki/Ceilometer/blueprints/api-group-by

Storage tests / SQL Alchemy group by Patch Set 10

Mike Perez did a code review. He said it looked good to him. His one suggestion:

I noted back in ps4 that impl_sqlalchemy.py had validation of which keys can be used for group_by. I don't think it was answered if we can move that up to another layer so that all implementations can take advantage of the validation, instead of having the duplicated code in each implementation.

jd responded:

Mike: this limitation is implementation specific. I don't envision the same limitation in MongoDB, for example. That's why I'm reluctant to move them a layer above.

MongoDB group by Patch Set 1

https://review.openstack.org/#/c/43043/1

The patch set passed Jenkins and jd did code review and said it looked good.

MongoDB group by Patch Set 2

Link: https://review.openstack.org/#/c/43043/2

Patch Set 2 implements partial group by statistics for the MongoDB driver. It covers the case where the groupby fields are a combination of user-id, resource-id, project-id, and source (metadata fields not implemented) and where period = None. I still need to do the case where period is not None.

• I added an if statement in get_meter_statistics() to catch cases where an unknown parameter is passed to groupby for instance:

  get_meter_statistics(f, groupby=['wtf'])
  

• I modified test_group_by_unknown_field in tests/storage/base.py

  This version passes for SQL Alchemy but not for MongoDB:

  def test_group_by_unknown_field(self):
      f = storage.SampleFilter(
          meter='instance',
      )
      result = self.conn.get_meter_statistics(
          f, groupby=['wtf'])
      self.assertRaises(
          NotImplementedError,
          list,
          result)
  

  For SQL Alchemy, the test passes because

  result = self.conn.get_meter_statistics(f, groupby=['wtf'])
  

  returns a generator and isn't evaluated until you pass it to list().

  However, for MongoDB, a NotImplementedError is already thrown when you evaluate

  result = self.conn.get_meter_statistics(f, groupby=['wtf'])
  

  If you override the base test class with

  def test_group_by_unknown_field(self):
      f = storage.SampleFilter(
          meter='instance',
      )
      self.assertRaises(NotImplementedError,
                        self.conn.get_meter_statistics,
                        f,
                        groupby=['wtf'])
  

  This test works for MongoDB but not SQL Alchemy. If you try to use the above test with the SQL Alchemy driver, you get this traceback:

  Traceback (most recent call last):
    File "/opt/stack/ceilometer/tests/storage/base.py", line 1073, in test_group_by_unknown_field
      lambda: self.conn.get_meter_statistics(f, groupby=['wtf']))
    File "/opt/stack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 394, in assertRaises
      self.assertThat(our_callable, matcher)
    File "/opt/stack/ceilometer/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
      raise MismatchError(matchee, matcher, mismatch, verbose)
  MismatchError: <function <lambda> at 0x4786140> returned <generator object get_meter_statistics at 0x4a0dd20>
  

  My solution for a test that works for both SQL Alchemy and MongoDB is:

  def test_group_by_unknown_field(self):
      f = storage.SampleFilter(
          meter='instance',
      )
      self.assertRaises(
          NotImplementedError,
          lambda: list(self.conn.get_meter_statistics(f, groupby=['wtf'])))
  

• It's a little tricky to write the map functions for map_reduce() in MongoDB because the map functions are written in BSON (very similar to Javascript).

  For example, I found that I need to use null in Javascript instead of None as I would in Python.

MongoDB group by Patch Set 3

Link: https://review.openstack.org/#/c/43043/3

In this patch set, I finished the part I had left undone in Patch Set 2. I implemented group by with period in the MongoDB driver.

So I've implemented everything that jd and I planned for the MongoDB driver.

The Ceilometer test suite passes for Patch Set 3:

https://gist.github.com/terriyu/6303882