Skip to content

Instantly share code, notes, and snippets.

@saurabh-hirani

saurabh-hirani/docstrings-ipynb

Last active November 29, 2015 09:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save saurabh-hirani/9305923 to your computer and use it in GitHub Desktop.
Save saurabh-hirani/9305923 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
docstrings-ipynb
{
"metadata": {
"name": ""
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Lessons learnt while automating docstring checking - Saurabh Hirani\n",
"\n",
"Original blog post link just for comments: <a href=\"http://saurabh-hirani.github.io/writing/2014/02/20/lessons-learnt-while-automating/\" target=\"_blank\">here</a>\n",
"\n",
"The aim of this post is describe the evolution of a program which automates docstring testing of a module and its components (functions, classes, nested classes and their respective methods). It starts out simple and we keep on making incremental changes to address limitations.\n",
"\n",
"Although I am using an ipython notebook, I don't want the end user to copy paste and run the cell source code to try the code examples outside of the notebook. So I am using the %load ipython magic to load the source files from my ipython directory instead of typing code directly in the cells. These files are present in <a href=\"https://github.com/saurabh-hirani/talks/tree/master/docstrings\" target=\"_blank\">this</a> github repo. So to use code separately, clone this repo and you are good to run t1.py, t2.py and so on. Read from here, play from there.\n",
"\n",
"For the sake of simplicity, let us assume that the module which we want to test for docstrings has the structure defined in __mod1.py__:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load mod1.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 1
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"class SpecFile(object):\n",
" class Section(object):\n",
" def __init__(self): pass\n",
" def validate(self): pass\n",
" \n",
" class DynamicSection(Section):\n",
" def __init__(self): pass\n",
" \n",
" class StaticSection(Section):\n",
" def __init__(self): pass\n",
" \n",
" class Section1(StaticSection):\n",
" def __init__(self): pass\n",
" def validate(self): pass\n",
" \n",
" class Section2(DynamicSection):\n",
" def __init__(self): pass\n",
" def validate(self): pass\n",
" \n",
"def main():\n",
" pass\n"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 2
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This represents the structure of a spec file which contains various sections. Some sections have static i.e. pre-defined keys and some have dynamic keys i.e. key-value pairs that are not known in advance. Section1 and Section2 are just instances of 2 such sections. The main function is called when user executes the program from the cmdline. We are not concerned with the internal logic as I am trying to automate docstring testing.\n",
"\n",
"nbcommon.py contains all the common operations. "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load nbcommon.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 3
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"# custom exception class for no docstring\n",
"class NoDocstrError(Exception): pass\n",
"\n",
"def has_docstr(entity):\n",
" \"\"\" Check whether this entity has a docstring \"\"\"\n",
" docstr = entity.__doc__\n",
" return docstr != None and docstr.strip() != ''\n",
"\n",
"import inspect\n",
"\n",
"def get_entity_type(entity):\n",
" \"\"\" Check whether entity is supported for docstring heck \"\"\"\n",
" for entity_type in ['module', 'function', 'class', 'method'] :\n",
" # inspect module has inspect.ismodule, inspect.isfunction - leverage that\n",
" inspect_func = getattr(inspect, 'is' + entity_type)\n",
" if inspect_func(entity): return entity_type\n",
" raise ValueError('Invalid entity: %s passed' % entity)\n"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"inspect module is used to get the members of an entity (a class, module, etc.) By using get_entity_type we are just saying that while getting the members of a module - we will perform docstring tests on only the ones which are supported by this function.\n",
"\n",
"I don't know how to run a py.test code in a notebook. So to keep things simple, I will use self contained examples to test for docstrings.\n",
"\n",
"Also because ipython notebook does not support namespaces as far as I know, I will use classes like TestDocstr1, TestDocstr2, etc. to show the evolution of code.\n",
"\n",
"For the first cut, we will use the code in __t1.py__ which has the following structure:\n",
"\n",
"1. Class __Modstruct1__ which when passed an entity - has a method __get_all_members__ to return the entity's members.\n",
"2. Class __TestDocstr1__ whose class method - __test_docstr__ uses __Modstruct1__ to get an entity's members and raises an exception if any of them don't have docstrings."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load t1.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 4
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from nbcommon import *\n",
"import mod1\n",
"\n",
"import inspect\n",
"from collections import defaultdict\n",
"\n",
"class Modstruct1(object):\n",
" \"\"\" Return a data structure representing all members of the passed\n",
" entity \"\"\"\n",
"\n",
" def __init__(self, base_entity):\n",
" self.base_entity = base_entity\n",
"\n",
" def get_all_members(self):\n",
" \"\"\" Get all the members (nested also) of the passed entity \"\"\"\n",
" return inspect.getmembers(self.base_entity)\n",
"\n",
"\n",
"class TestDocstr1(object):\n",
"\n",
" @classmethod\n",
" def test_docstr(self, entity):\n",
" \"\"\" Test whether the passed in entity and its children have docstring \"\"\"\n",
" entity_type = None\n",
" non_docstr_entities = defaultdict(list)\n",
" all_members = Modstruct1(entity).get_all_members()\n",
"\n",
" # get all the members of the passed entity\n",
" for member in all_members:\n",
" ref = member[1]\n",
" try:\n",
" entity_type = get_entity_type(ref)\n",
" if not has_docstr(ref):\n",
" non_docstr_entities[entity_type].append(ref)\n",
" except ValueError:\n",
" # invalid entity type - skip it\n",
" continue\n",
"\n",
" # if any entities without docstring - consolidate and raise error\n",
" if non_docstr_entities.keys():\n",
" errors = []\n",
" for entity_type, refs in non_docstr_entities.iteritems():\n",
" for ref in refs:\n",
" errors.append('%s %s does not have docstr' % (entity_type,\n",
" ref.__name__))\n",
" raise NoDocstrError('\\n'.join(errors))\n",
"\n",
" return True\n",
"\n",
"TestDocstr1.test_docstr(mod1)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "function main does not have docstr\nclass SpecFile does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-5-4f05e337cab5>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m 48\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 49\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 50\u001b[1;33m \u001b[0mTestDocstr1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-5-4f05e337cab5>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 44\u001b[0m errors.append('%s %s does not have docstr' % (entity_type,\n\u001b[0;32m 45\u001b[0m ref.__name__))\n\u001b[1;32m---> 46\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 47\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 48\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: function main does not have docstr\nclass SpecFile does not have docstr"
]
}
],
"prompt_number": 5
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see __t1.py__ has the following limitations:\n",
"\n",
"1. Passing __mod1__ resulted in checks being performed only on it's immediated children - __main__ and __SpecFile__ - we should've tested __mod1__ also along with the members of __SpecFile__.\n",
"2. It does not prefix member name by it's parent i.e. __main__ should be printed as __mod1.main__ and __SpecFile__ as __mod1.SpecFile__. \n",
"\n",
"So in order to address this limitation we need to:\n",
"\n",
"1. Inspect the module also, and if a member is a class - drill down into the class and get it's nested classes and their respective methods.\n",
"2. Prefix each member by it's parent name. \n",
"\n",
"Using the above pointers, we evolve __t1.py__ to __t2.py__ which has the following structure:\n",
"\n",
"1. Same as t1.py but with Modstruct2 and TestDocstr2 instead of Modstruct1 and Testdocstr1. \n",
"2. Adding a little more intelligence in __get_all_members__ if member is of type class and making each member have the following structure:\n",
"\n",
"<pre>\n",
"{\n",
" 'name': member_name,\n",
" 'ref': object_ref,\n",
" 'type': module|function|class|method,\n",
" 'parent_ref': ref_to_parent\n",
" 'parent_name': parent_name,\n",
"}\n",
"</pre>\n",
"\n",
"Also, an additional check we can do at this stage is to filter out only those members which are defined in this module i.e. if someone does a 'from somemodule import \\*' we don't want to test the members which polluted the module name space."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load t2.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 6
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from nbcommon import *\n",
"import mod1\n",
"\n",
"import inspect\n",
"import sys\n",
"from collections import defaultdict\n",
"\n",
"class Modstruct2(object):\n",
" \"\"\" Return a data structure representing all members of the passed\n",
" entity \"\"\"\n",
"\n",
" def __init__(self, base_entity):\n",
" self.base_entity_type = get_entity_type(base_entity)\n",
" self.base_entity = base_entity\n",
" self.base_module = base_entity\n",
" if self.base_entity_type != 'module':\n",
" # if entity_type is class - know which module it belongs to\n",
" self.base_module = sys.modules[base_entity.__module__]\n",
"\n",
" def get_entity_members(self, entity):\n",
" \"\"\" Get first level members of the passed entity \"\"\"\n",
" members = []\n",
" parent_name = entity.__name__\n",
" for member in inspect.getmembers(entity):\n",
" ref = member[1]\n",
" # member has to be of supported entity type\n",
" try:\n",
" ref_type = get_entity_type(ref)\n",
" except ValueError:\n",
" continue\n",
"\n",
" # we will not inspect modules imported in base module\n",
" if inspect.ismodule(ref): continue\n",
"\n",
" # member has to be defined in base module\n",
" if ref.__module__ != self.base_module.__name__: continue\n",
"\n",
" # valid member - construct member data\n",
" member_data = {\n",
" 'type': ref_type, \n",
" 'ref': ref, \n",
" 'name': entity.__name__ + '.' + ref.__name__,\n",
" 'parent_ref': entity,\n",
" 'parent_name': parent_name,\n",
" }\n",
" members.append(member_data)\n",
" return members\n",
"\n",
" def get_all_members(self):\n",
" \"\"\" Get all the members (nested also) of the passed entity \"\"\"\n",
" # add base module as the first element\n",
" all_members = [{'type': 'module',\n",
" 'ref': self.base_module, \n",
" 'name': self.base_module.__name__,\n",
" 'parent_ref': None,\n",
" 'parent_name': None}]\n",
"\n",
" # get first level members of the main entity\n",
" nested_members = self.get_entity_members(self.base_entity)\n",
" all_members.extend(nested_members)\n",
"\n",
" # call get_entity_members repetitively till you reach a stage where \n",
" # there are no nested members\n",
" while nested_members:\n",
" curr_nested_members = []\n",
" for member_data in nested_members:\n",
" if member_data['type'] == 'class':\n",
" # drill nested members only in a class\n",
" members = self.get_entity_members(member_data['ref'])\n",
" curr_nested_members.extend(members)\n",
" nested_members = curr_nested_members\n",
" all_members.extend(nested_members)\n",
"\n",
" return all_members\n",
"\n",
"class TestDocstr2(object):\n",
"\n",
" @classmethod\n",
" def test_docstr(self, entity):\n",
" all_members = Modstruct2(entity).get_all_members()\n",
"\n",
" non_docstr_entities = defaultdict(list)\n",
"\n",
" # get all the nested members of root entity\n",
" for member_data in all_members:\n",
" # consolidate members based on type\n",
" if not has_docstr(member_data['ref']):\n",
" member_name = member_data['name']\n",
" non_docstr_entities[member_data['type']].append(member_name)\n",
"\n",
" if non_docstr_entities.keys():\n",
" errors = []\n",
" # create error string\n",
" for entity_type, refs in non_docstr_entities.iteritems():\n",
" for refname in refs:\n",
" errors.append('%s: %s does not have docstr' % (entity_type,\n",
" refname))\n",
" raise NoDocstrError('\\n' + '\\n'.join(errors))\n",
" return True\n",
"\n",
"TestDocstr2.test_docstr(mod1)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "\nfunction: mod1.main does not have docstr\nclass: mod1.SpecFile does not have docstr\nclass: SpecFile.DynamicSection does not have docstr\nclass: SpecFile.Section does not have docstr\nclass: SpecFile.Section1 does not have docstr\nclass: SpecFile.Section2 does not have docstr\nclass: SpecFile.StaticSection does not have docstr\nmodule: mod1 does not have docstr\nmethod: DynamicSection.__init__ does not have docstr\nmethod: DynamicSection.validate does not have docstr\nmethod: Section.__init__ does not have docstr\nmethod: Section.validate does not have docstr\nmethod: Section1.__init__ does not have docstr\nmethod: Section1.validate does not have docstr\nmethod: Section2.__init__ does not have docstr\nmethod: Section2.validate does not have docstr\nmethod: StaticSection.__init__ does not have docstr\nmethod: StaticSection.validate does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-7-a2b07a2dae29>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m 99\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 100\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 101\u001b[1;33m \u001b[0mTestDocstr2\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-7-a2b07a2dae29>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 96\u001b[0m errors.append('%s: %s does not have docstr' % (entity_type,\n\u001b[0;32m 97\u001b[0m refname))\n\u001b[1;32m---> 98\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 99\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 100\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: \nfunction: mod1.main does not have docstr\nclass: mod1.SpecFile does not have docstr\nclass: SpecFile.DynamicSection does not have docstr\nclass: SpecFile.Section does not have docstr\nclass: SpecFile.Section1 does not have docstr\nclass: SpecFile.Section2 does not have docstr\nclass: SpecFile.StaticSection does not have docstr\nmodule: mod1 does not have docstr\nmethod: DynamicSection.__init__ does not have docstr\nmethod: DynamicSection.validate does not have docstr\nmethod: Section.__init__ does not have docstr\nmethod: Section.validate does not have docstr\nmethod: Section1.__init__ does not have docstr\nmethod: Section1.validate does not have docstr\nmethod: Section2.__init__ does not have docstr\nmethod: Section2.validate does not have docstr\nmethod: StaticSection.__init__ does not have docstr\nmethod: StaticSection.validate does not have docstr"
]
}
],
"prompt_number": 7
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This code, although prints more info than __t1.py__ has a glaring limitation:\n",
"\n",
"1. __Section.validate__ name should have been __mod1.SpecFile.Section.validate__ - names of all members should be fully qualified - parent name prefix is not enough - we need the entire ancestry starting from the module name. \n",
"2. Sometimes we just need to check for the docstrings of a specific subset of the module, instead of the whole module i.e. check if __mod1.SpecFile.Section1__ members have docstrings or not.\n",
"\n",
"##### Addressing limtiation 1 - get fully qualified name:\n",
"If we were using Python 3.3 - we wouldn't need to write code to fix this because as per <a href=\"http://stackoverflow.com/questions/2020014/get-fully-qualified-class-name-of-an-object-in-python\" target=\"_blank\">this SO answer</a> an additional attribute \\_\\_qualname\\_\\_ has been added to functions and classes in Python 3.3 - which would give you the fully qualified name.\n",
"\n",
"Python tutor gives online shell for executing Python 2.7/3.3 code - <a href=\"http://goo.gl/BZn9Kp\" target=\"_blank\">this link</a> runs a qualname checking code snippet in Python 3.3 - try running it in 2.7 also to see the difference.\n",
"\n",
"Fortunately I am using Python 2.7.3 - so we march forth :) \n",
"\n",
"One way to do address this limitation is to create a dictionary - __id_name_map__ - which maps an object's id to it's name and have each newly added id look up to it's parent name before adding it. \n",
"\n",
"To illustrate, the various stages of __id_name_map__ can be:\n",
"\n",
"<pre>\n",
"# add mod1\n",
"{\n",
" id_of_mod1: 'mod1',\n",
"}\n",
"\n",
"# inspect mod1 - returns SpecFile, main\n",
"# class SpecFile - is the parent present in id_name_map - if yes - prefix it with its parent name\n",
"# function main - is the parent present in id_name_map - if yes - prefix it with its parent name\n",
"{\n",
" id_of_mod1: 'mod1',\n",
" id_of_main: 'mod1.main',\n",
" id_of_SpecFile: 'mod1.SpecFile',\n",
"}\n",
"\n",
"# inspect SpecFile - returns Section, StaticSection, DynamicSection, Section1, Section2\n",
"# for each of the above - is the parent present in id_name_map - if yes - prefix it with its parent name\n",
"{\n",
" id_of_mod1: 'mod1',\n",
" id_of_main: 'mod1.main',\n",
" id_of_SpecFile: 'mod1.SpecFile',\n",
" id_of_Section: 'mod1.SpecFile.Section',\n",
" id_of_StaticSection: 'mod1.SpecFile.StaticSection',\n",
" id_of_DynamicSection: 'mod1.SpecFile.DynamicSection',\n",
" id_of_Section1: 'mod1.SpecFile.Section1',\n",
" id_of_Section2: 'mod1.SpecFile.Section2',\n",
"}\n",
"\n",
"and so on .... \n",
"</pre>\n",
"\n",
"##### Addressing limtiation 2 - extract members of a specific subset of the module:\n",
"Overcoming this limitation is merely a matter of selecting a subset of members from __all_members__ (created by __get_all_members__).\n",
"\n",
"We cycle through __all_members__ and extract out only those members whose name starts with the fully qualified name of the user specified entity i.e. beginning with __mod1.SpecFile__ or __mod1.SpecFile.Section1__ and so on.\n",
"\n",
"And so we come up with __t3.py__: "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load t3.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 8
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from nbcommon import *\n",
"import mod1\n",
"\n",
"import inspect\n",
"import sys\n",
"from collections import defaultdict\n",
"\n",
"class Modstruct3(object):\n",
" \"\"\" Return a data structure representing all members of the passed\n",
" entity \"\"\"\n",
"\n",
" def __init__(self, base_entity):\n",
" self.base_entity_type = get_entity_type(base_entity)\n",
" self.base_entity = base_entity\n",
" self.base_module = base_entity\n",
" self.id_name_map = {}\n",
" self.all_members = []\n",
" if self.base_entity_type != 'module':\n",
" # if entity_type is class - know which module it belongs to\n",
" self.base_module = sys.modules[base_entity.__module__]\n",
"\n",
" def get_entity_name(self, entity):\n",
" \"\"\" Return fully qualified name of entity \"\"\"\n",
" return self.id_name_map.get(id(entity), None)\n",
"\n",
" def build_id_name_map(self, entity, parent=None):\n",
" \"\"\" Map entity id to its fully qualified name \"\"\"\n",
" entity_name = entity.__name__\n",
" if not parent is None:\n",
" id_parent = id(parent)\n",
" if id_parent in self.id_name_map:\n",
" parent_name = self.id_name_map[id_parent]\n",
" entity_name = '.'.join([parent_name, entity.__name__])\n",
" self.id_name_map[id(entity)] = entity_name\n",
"\n",
" def extract_entity_members(self):\n",
" \"\"\" From all the members extract out member tree of the base \n",
" entity \"\"\"\n",
" if self.base_entity_type == 'module':\n",
" self.base_entity_members = self.all_members\n",
" return self.base_entity_members\n",
"\n",
" base_entity_name = self.get_entity_name(self.base_entity)\n",
"\n",
" base_entity_members = []\n",
" for member in self.all_members:\n",
" if member['name'].startswith(base_entity_name):\n",
" base_entity_members.append(member)\n",
" self.base_entity_members = base_entity_members\n",
"\n",
" def get_entity_members(self, entity):\n",
" \"\"\" Get first level members of the passed entity \"\"\"\n",
" members = []\n",
" parent_name = self.get_entity_name(entity)\n",
" for member in inspect.getmembers(entity):\n",
" ref = member[1]\n",
" # member has to be of supported entity type\n",
" try:\n",
" ref_type = get_entity_type(ref)\n",
" except ValueError:\n",
" continue\n",
"\n",
" # we will not inspect modules imported in base module\n",
" if inspect.ismodule(ref): continue\n",
"\n",
" # member has to be defined in base module\n",
" if ref.__module__ != self.base_module.__name__: continue\n",
"\n",
" # valid member - construct member data\n",
" member_data = {\n",
" 'type': ref_type, \n",
" 'ref': ref, \n",
" 'name': parent_name + '.' + ref.__name__,\n",
" 'parent_ref': entity,\n",
" 'parent_name': parent_name\n",
" }\n",
" members.append(member_data)\n",
" self.build_id_name_map(ref, entity)\n",
" return members\n",
"\n",
" def get_all_members(self):\n",
" \"\"\" Get all the members (nested also) of the passed entity \"\"\"\n",
"\n",
" # add base module as the first element\n",
" all_members = [{'type': 'module',\n",
" 'ref': self.base_module, \n",
" 'name': self.base_module.__name__,\n",
" 'parent_ref': None,\n",
" 'parent_name': None}]\n",
"\n",
" # add base module as first entry to id_name_map - root of all names\n",
" self.build_id_name_map(self.base_module, None)\n",
"\n",
" # get first level members of the module\n",
" nested_members = self.get_entity_members(self.base_module)\n",
" all_members.extend(nested_members)\n",
"\n",
" # call get_entity_members repetitively till you reach a stage where \n",
" # there are no nested members\n",
" while nested_members:\n",
" curr_nested_members = []\n",
" # for member_type, member_ref, member_name in nested_members:\n",
" for member_data in nested_members:\n",
" if member_data['type'] == 'class':\n",
" # drill nested members only in a class\n",
" members = self.get_entity_members(member_data['ref'])\n",
" curr_nested_members.extend(members)\n",
" nested_members = curr_nested_members\n",
" all_members.extend(nested_members)\n",
"\n",
" self.all_members = all_members\n",
"\n",
" # extract subset of members in case base_entity is not a module\n",
" self.extract_entity_members()\n",
"\n",
" return self.base_entity_members\n",
"\n",
"class TestDocstr3(object):\n",
"\n",
" @classmethod\n",
" def test_docstr(self, entity):\n",
" all_members = Modstruct3(entity).get_all_members()\n",
"\n",
" non_docstr_entities = defaultdict(list)\n",
"\n",
" # get all the nested members of root entity\n",
" for member_data in all_members:\n",
" # consolidate members based on type\n",
" if not has_docstr(member_data['ref']):\n",
" member_name = member_data['name']\n",
" non_docstr_entities[member_data['type']].append(member_name)\n",
"\n",
" if non_docstr_entities.keys():\n",
" errors = []\n",
" # create error string\n",
" for entity_type, refs in non_docstr_entities.iteritems():\n",
" for refname in refs:\n",
" errors.append('%s: %s does not have docstr' % (entity_type,\n",
" refname))\n",
" raise NoDocstrError('\\n' + '\\n'.join(errors))\n",
" return True\n",
"\n",
"TestDocstr3.test_docstr(mod1)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "\nfunction: mod1.main does not have docstr\nclass: mod1.SpecFile does not have docstr\nclass: mod1.SpecFile.DynamicSection does not have docstr\nclass: mod1.SpecFile.Section does not have docstr\nclass: mod1.SpecFile.Section1 does not have docstr\nclass: mod1.SpecFile.Section2 does not have docstr\nclass: mod1.SpecFile.StaticSection does not have docstr\nmodule: mod1 does not have docstr\nmethod: mod1.SpecFile.DynamicSection.__init__ does not have docstr\nmethod: mod1.SpecFile.DynamicSection.validate does not have docstr\nmethod: mod1.SpecFile.Section.__init__ does not have docstr\nmethod: mod1.SpecFile.Section.validate does not have docstr\nmethod: mod1.SpecFile.Section1.__init__ does not have docstr\nmethod: mod1.SpecFile.Section1.validate does not have docstr\nmethod: mod1.SpecFile.Section2.__init__ does not have docstr\nmethod: mod1.SpecFile.Section2.validate does not have docstr\nmethod: mod1.SpecFile.StaticSection.__init__ does not have docstr\nmethod: mod1.SpecFile.StaticSection.validate does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m 141\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 142\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 143\u001b[1;33m \u001b[0mTestDocstr3\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 138\u001b[0m errors.append('%s: %s does not have docstr' % (entity_type,\n\u001b[0;32m 139\u001b[0m refname))\n\u001b[1;32m--> 140\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 141\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 142\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: \nfunction: mod1.main does not have docstr\nclass: mod1.SpecFile does not have docstr\nclass: mod1.SpecFile.DynamicSection does not have docstr\nclass: mod1.SpecFile.Section does not have docstr\nclass: mod1.SpecFile.Section1 does not have docstr\nclass: mod1.SpecFile.Section2 does not have docstr\nclass: mod1.SpecFile.StaticSection does not have docstr\nmodule: mod1 does not have docstr\nmethod: mod1.SpecFile.DynamicSection.__init__ does not have docstr\nmethod: mod1.SpecFile.DynamicSection.validate does not have docstr\nmethod: mod1.SpecFile.Section.__init__ does not have docstr\nmethod: mod1.SpecFile.Section.validate does not have docstr\nmethod: mod1.SpecFile.Section1.__init__ does not have docstr\nmethod: mod1.SpecFile.Section1.validate does not have docstr\nmethod: mod1.SpecFile.Section2.__init__ does not have docstr\nmethod: mod1.SpecFile.Section2.validate does not have docstr\nmethod: mod1.SpecFile.StaticSection.__init__ does not have docstr\nmethod: mod1.SpecFile.StaticSection.validate does not have docstr"
]
}
],
"prompt_number": 9
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"TestDocstr3.test_docstr(mod1.main) # Testing if the function has docstring"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "\nfunction: mod1.main does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-10-acc9704b548d>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mTestDocstr3\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmain\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# Testing if the function has docstring\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 138\u001b[0m errors.append('%s: %s does not have docstr' % (entity_type,\n\u001b[0;32m 139\u001b[0m refname))\n\u001b[1;32m--> 140\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 141\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 142\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: \nfunction: mod1.main does not have docstr"
]
}
],
"prompt_number": 10
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"TestDocstr3.test_docstr(mod1.SpecFile) # Testing if the class and its members have docstring"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "\nclass: mod1.SpecFile does not have docstr\nclass: mod1.SpecFile.DynamicSection does not have docstr\nclass: mod1.SpecFile.Section does not have docstr\nclass: mod1.SpecFile.Section1 does not have docstr\nclass: mod1.SpecFile.Section2 does not have docstr\nclass: mod1.SpecFile.StaticSection does not have docstr\nmethod: mod1.SpecFile.DynamicSection.__init__ does not have docstr\nmethod: mod1.SpecFile.DynamicSection.validate does not have docstr\nmethod: mod1.SpecFile.Section.__init__ does not have docstr\nmethod: mod1.SpecFile.Section.validate does not have docstr\nmethod: mod1.SpecFile.Section1.__init__ does not have docstr\nmethod: mod1.SpecFile.Section1.validate does not have docstr\nmethod: mod1.SpecFile.Section2.__init__ does not have docstr\nmethod: mod1.SpecFile.Section2.validate does not have docstr\nmethod: mod1.SpecFile.StaticSection.__init__ does not have docstr\nmethod: mod1.SpecFile.StaticSection.validate does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-11-67ece24d1001>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mTestDocstr3\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSpecFile\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;31m# Testing if the class and its members have docstring\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 138\u001b[0m errors.append('%s: %s does not have docstr' % (entity_type,\n\u001b[0;32m 139\u001b[0m refname))\n\u001b[1;32m--> 140\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 141\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 142\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: \nclass: mod1.SpecFile does not have docstr\nclass: mod1.SpecFile.DynamicSection does not have docstr\nclass: mod1.SpecFile.Section does not have docstr\nclass: mod1.SpecFile.Section1 does not have docstr\nclass: mod1.SpecFile.Section2 does not have docstr\nclass: mod1.SpecFile.StaticSection does not have docstr\nmethod: mod1.SpecFile.DynamicSection.__init__ does not have docstr\nmethod: mod1.SpecFile.DynamicSection.validate does not have docstr\nmethod: mod1.SpecFile.Section.__init__ does not have docstr\nmethod: mod1.SpecFile.Section.validate does not have docstr\nmethod: mod1.SpecFile.Section1.__init__ does not have docstr\nmethod: mod1.SpecFile.Section1.validate does not have docstr\nmethod: mod1.SpecFile.Section2.__init__ does not have docstr\nmethod: mod1.SpecFile.Section2.validate does not have docstr\nmethod: mod1.SpecFile.StaticSection.__init__ does not have docstr\nmethod: mod1.SpecFile.StaticSection.validate does not have docstr"
]
}
],
"prompt_number": 11
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"But see what happens when we try to run the docstring test on a method: "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"TestDocstr3.test_docstr(mod1.SpecFile.Section.validate)"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "TypeError",
"evalue": "startswith first arg must be str, unicode, or tuple, not NoneType",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-12-91e4aba87ed2>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mTestDocstr3\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSpecFile\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSection\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 120\u001b[0m \u001b[1;33m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 121\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mentity\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 122\u001b[1;33m \u001b[0mall_members\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mModstruct3\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mentity\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_all_members\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 123\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 124\u001b[0m \u001b[0mnon_docstr_entities\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mdefaultdict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mget_all_members\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m 112\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 113\u001b[0m \u001b[1;31m# extract subset of members in case base_entity is not a module\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 114\u001b[1;33m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextract_entity_members\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 115\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 116\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbase_entity_members\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;32m<ipython-input-9-faf23dd8dd0b>\u001b[0m in \u001b[0;36mextract_entity_members\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m 45\u001b[0m \u001b[0mbase_entity_members\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 46\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mmember\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mall_members\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 47\u001b[1;33m \u001b[1;32mif\u001b[0m \u001b[0mmember\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'name'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mbase_entity_name\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 48\u001b[0m \u001b[0mbase_entity_members\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmember\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 49\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mbase_entity_members\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mbase_entity_members\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mTypeError\u001b[0m: startswith first arg must be str, unicode, or tuple, not NoneType"
]
}
],
"prompt_number": 12
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As per the error - member['name'] is None, which implies that we could not find __mod1.SpecFile.Section.validate__'s id key in __id_name_map__. But why did it fail only for the method, and not while checking docstring for a module, class or function? To know more let's run __Modstruct3__ insteance method __get_all_members__ separately."
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"m1 = Modstruct3(mod1)\n",
"members = m1.get_all_members()\n",
"target_keys = ['mod1', 'mod1.main', 'mod1.SpecFile.Section', 'mod1.SpecFile.Section.validate']\n",
"target_members = {}\n",
"for member in members:\n",
" # map id => name for target_keys members\n",
" if member['name'] in target_keys:\n",
" target_members[member['name']] = id(member['ref'])"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 13
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"id(mod1), target_members['mod1'] # same"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 14,
"text": [
"(149022228, 149022228)"
]
}
],
"prompt_number": 14
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"id(mod1.main), target_members['mod1.main'] # same"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 15,
"text": [
"(149019812, 149019812)"
]
}
],
"prompt_number": 15
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"id(mod1.SpecFile.Section), target_members['mod1.SpecFile.Section'] # same"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 16,
"text": [
"(149522052, 149522052)"
]
}
],
"prompt_number": 16
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"id(mod1.SpecFile.Section.validate), target_members['mod1.SpecFile.Section.validate'] # diff"
],
"language": "python",
"metadata": {},
"outputs": [
{
"metadata": {},
"output_type": "pyout",
"prompt_number": 17,
"text": [
"(148864572, 148861892)"
]
}
],
"prompt_number": 17
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see id of __mod1.SpecFile.Section.validate__ and that stored in the dictionary returned by __get_all_members__ instance method is different. This is because, as per <a href=\"http://stackoverflow.com/questions/13348031/python-bound-and-unbound-method-object\" target=\"_blank\">this SO answer</a> whenever you look up a method via class.name or instance.name, the method object is created a-new. We can illustrate this by a simple example:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load test_meth_id.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 18
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"def f1(): pass\n",
"\n",
"class A():\n",
" def m1(): pass\n",
"\n",
"x = f1\n",
"y = f1\n",
"z = f1\n",
"print \"\\n==== id for function f1 ====\"\n",
"print 'id(f1) = ' + str(id(f1))\n",
"print 'id(x) = ' + str(id(x))\n",
"print 'id(y) = ' + str(id(y))\n",
"print 'id(z) = ' + str(id(z))\n",
"\n",
"x = A\n",
"y = A\n",
"z = A\n",
"print \"\\n==== id for class A ====\"\n",
"print 'id(A) = ' + str(id(A))\n",
"print 'id(x) = ' + str(id(x))\n",
"print 'id(y) = ' + str(id(y))\n",
"print 'id(z) = ' + str(id(z))\n",
"\n",
"x = A.m1\n",
"y = A.m1\n",
"z = A.m1\n",
"print \"\\n==== id for method A.m1 ====\"\n",
"print 'id(A.m1) = ' + str(id(A.m1))\n",
"print 'id(x) = ' + str(id(x))\n",
"print 'id(y) = ' + str(id(y))\n",
"print 'id(z) = ' + str(id(z))\n",
"\n",
"print 'x is y ' + str(x is y)\n",
"print 'x == y ' + str(x == y)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"==== id for function f1 ====\n",
"id(f1) = 150503124\n",
"id(x) = 150503124\n",
"id(y) = 150503124\n",
"id(z) = 150503124\n",
"\n",
"==== id for class A ====\n",
"id(A) = 150346556\n",
"id(x) = 150346556\n",
"id(y) = 150346556\n",
"id(z) = 150346556"
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"\n",
"==== id for method A.m1 ====\n",
"id(A.m1) = 148863252\n",
"id(x) = 148864572\n",
"id(y) = 148862892"
]
},
{
"output_type": "stream",
"stream": "stdout",
"text": [
"\n",
"id(z) = 148861852\n",
"x is y False\n",
"x == y True\n"
]
}
],
"prompt_number": 19
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see, the class/function id stays the same no matter how many references are created to it, but the method id changes for each assignment. Hence if the entity is a method, then instead of looking it up via the id, we could just cycle through all members and do a __member == method__ check as shown by introducing the new method __get_base_entity_name__ in __t4.py__: "
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"%load t4.py"
],
"language": "python",
"metadata": {},
"outputs": [],
"prompt_number": 22
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"from nbcommon import *\n",
"import mod1\n",
"\n",
"import inspect\n",
"import sys\n",
"from collections import defaultdict\n",
"\n",
"class Modstruct4(object):\n",
" \"\"\" Return a data structure representing all members of the passed\n",
" entity \"\"\"\n",
"\n",
" def __init__(self, base_entity, **options):\n",
" self.base_entity_type = get_entity_type(base_entity)\n",
" self.base_entity = base_entity\n",
" self.base_module = base_entity\n",
" self.id_name_map = {}\n",
" self.all_members = []\n",
" self.options = {'categorize': False}\n",
" self.options.update(options)\n",
" if self.base_entity_type != 'module':\n",
" # if entity_type is class - know which module it belongs to\n",
" self.base_module = sys.modules[base_entity.__module__]\n",
"\n",
" def get_entity_name(self, entity):\n",
" \"\"\" Return fully qualified name of entity \"\"\"\n",
" return self.id_name_map.get(id(entity), None)\n",
"\n",
" def get_base_entity_name(self):\n",
" \"\"\" Return the name of the base entity passed in by the user \"\"\"\n",
" # if base entity is not a method - just look up its id\n",
" if self.base_entity_type != 'method':\n",
" return self.get_entity_name(self.base_entity)\n",
"\n",
" # else as method id does not stay constant, cycle through all members \n",
" # and return the member matching the base entity ref\n",
" for member in self.all_members:\n",
" if self.base_entity == member['ref']:\n",
" return self.get_entity_name(member['ref'])\n",
"\n",
" def build_id_name_map(self, entity, parent=None):\n",
" \"\"\" Map entity id to its fully qualified name \"\"\"\n",
" entity_name = entity.__name__\n",
" if not parent is None:\n",
" id_parent = id(parent)\n",
" if id_parent in self.id_name_map:\n",
" parent_name = self.id_name_map[id_parent]\n",
" entity_name = '.'.join([parent_name, entity.__name__])\n",
" self.id_name_map[id(entity)] = entity_name\n",
"\n",
" def extract_entity_members(self):\n",
" \"\"\" From all the members extract out member tree of the base \n",
" entity \"\"\"\n",
" if self.base_entity_type == 'module':\n",
" self.base_entity_members = self.all_members\n",
" return self.base_entity_members\n",
"\n",
" base_entity_name = self.get_base_entity_name()\n",
"\n",
" base_entity_members = []\n",
" for member in self.all_members:\n",
" if member['name'].startswith(base_entity_name):\n",
" base_entity_members.append(member)\n",
" self.base_entity_members = base_entity_members\n",
"\n",
" def get_entity_members(self, entity):\n",
" \"\"\" Get first level members of the passed entity \"\"\"\n",
" members = []\n",
" parent_name = self.get_entity_name(entity)\n",
" for member in inspect.getmembers(entity):\n",
" ref = member[1]\n",
" # member has to be of supported entity type\n",
" try:\n",
" ref_type = get_entity_type(ref)\n",
" except ValueError:\n",
" continue\n",
"\n",
" # we will not inspect modules imported in base module\n",
" if inspect.ismodule(ref): continue\n",
"\n",
" # member has to be defined in base module\n",
" if ref.__module__ != self.base_module.__name__: continue\n",
"\n",
" # valid member - construct member data\n",
" member_data = {\n",
" 'type': ref_type, \n",
" 'ref': ref, \n",
" 'name': parent_name + '.' + ref.__name__,\n",
" 'parent_ref': entity,\n",
" 'parent_name': parent_name\n",
" }\n",
" members.append(member_data)\n",
" self.build_id_name_map(ref, entity)\n",
" return members\n",
"\n",
" def get_all_members(self):\n",
" \"\"\" Get all the members (nested also) of the passed entity \"\"\"\n",
"\n",
" # add base module as the first element\n",
" all_members = [{'type': 'module',\n",
" 'ref': self.base_module, \n",
" 'name': self.base_module.__name__,\n",
" 'parent_ref': None,\n",
" 'parent_name': None}]\n",
"\n",
" # add base module as first entry to id_name_map - root of all names\n",
" self.build_id_name_map(self.base_module, None)\n",
"\n",
" # get first level members of the module\n",
" nested_members = self.get_entity_members(self.base_module)\n",
" all_members.extend(nested_members)\n",
"\n",
" # call get_entity_members repetitively till you reach a stage where \n",
" # there are no nested members\n",
" while nested_members:\n",
" curr_nested_members = []\n",
" # for member_type, member_ref, member_name in nested_members:\n",
" for member_data in nested_members:\n",
" if member_data['type'] == 'class':\n",
" # drill nested members only in a class\n",
" members = self.get_entity_members(member_data['ref'])\n",
" curr_nested_members.extend(members)\n",
" nested_members = curr_nested_members\n",
" all_members.extend(nested_members)\n",
"\n",
" self.all_members = all_members\n",
"\n",
" # extract subset of members in case base_entity is not a module\n",
" self.extract_entity_members()\n",
"\n",
" # categorize members if required\n",
" if self.options['categorize']:\n",
" return self.categorize()\n",
"\n",
" return self.base_entity_members\n",
"\n",
"class TestDocstr4(object):\n",
"\n",
" @classmethod\n",
" def test_docstr(self, entity):\n",
" all_members = Modstruct4(entity).get_all_members()\n",
"\n",
" non_docstr_entities = defaultdict(list)\n",
"\n",
" # get all the nested members of root entity\n",
" for member_data in all_members:\n",
" # consolidate members based on type\n",
" if not has_docstr(member_data['ref']):\n",
" member_name = member_data['name']\n",
" non_docstr_entities[member_data['type']].append(member_name)\n",
"\n",
" if non_docstr_entities.keys():\n",
" errors = []\n",
" # create error string\n",
" for entity_type, refs in non_docstr_entities.iteritems():\n",
" for refname in refs:\n",
" errors.append('%s: %s does not have docstr' % (entity_type,\n",
" refname))\n",
" raise NoDocstrError('\\n' + '\\n'.join(errors))\n",
" return True\n",
"\n",
"TestDocstr4.test_docstr(mod1.SpecFile.Section1.validate)\n"
],
"language": "python",
"metadata": {},
"outputs": [
{
"ename": "NoDocstrError",
"evalue": "\nmethod: mod1.SpecFile.Section1.validate does not have docstr",
"output_type": "pyerr",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[1;31mNoDocstrError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-23-691f6c3a5810>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m 159\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 160\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 161\u001b[1;33m \u001b[0mTestDocstr4\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtest_docstr\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmod1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSpecFile\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mSection1\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;32m<ipython-input-23-691f6c3a5810>\u001b[0m in \u001b[0;36mtest_docstr\u001b[1;34m(self, entity)\u001b[0m\n\u001b[0;32m 156\u001b[0m errors.append('%s: %s does not have docstr' % (entity_type,\n\u001b[0;32m 157\u001b[0m refname))\n\u001b[1;32m--> 158\u001b[1;33m \u001b[1;32mraise\u001b[0m \u001b[0mNoDocstrError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'\\n'\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m'\\n'\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0merrors\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 159\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 160\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mNoDocstrError\u001b[0m: \nmethod: mod1.SpecFile.Section1.validate does not have docstr"
]
}
],
"prompt_number": 23
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And that works. So that's about it - now __Modstruct__ can be used to find all the members of an entity and various checks can be performed on them.\n",
"\n",
"As __Modstruct__ provides a more generic functionality, it can be abstracted out as a separate utility. I've made some changes to enhance its usability and uploaded the code at <a href=\"https://github.com/saurabh-hirani/mod_struct\" target=\"_blank\">mod_struct github repo</a>\n",
" \n",
"Hope this post was useful to you. Do share your thoughts and insights in the blog comment section as describe initially."
]
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment