Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Using pyCLI to create some commands and subcommands.

Hello world python subcommand pattern

Using pyCLI vs commando to create some commands and subcommands.

https://gist.github.com/1202975.git

git clone git@gist.github.com:1202975.git python-hello-subcommand

How to write a well behaved console application? The ones with the best UIs allow configuration of core commands in a flexible config file, can override important defaults from the environment, and use the subcommand pattern to logically organize commands.

It should be tab-completable, with nice help output.

Circa 2013, there's still not a good way to do this. Here are a few attempts.

pycli

In terms of frameworks, this one is the only one with the needed pre-setup hook needed to do an initial pass over the arguments to sniff any parameters which might be used to then initialize the rest of the arguments. This happens, for example, when determining which configuration file to use. If we know there is a configuration file, why not make the configured values default?

Goal is to get a combination of tab completion + subcommands in such a way that will allow me to implement independent python modules as a main command, and then re-use the implementation as a subcommand in my main app.

simple

This attempt just uses argparse to iterate over a bunch of example subcommands. The behavior is pretty close to what we want; it's clear where to add reading from environment variables, and the set_parser method offers a nice way to set arguments, regardless of where the parser came form. argparse does not allow any way of re-using parser logic for subcommands.

At one point I tried separating the parsing into two phases, one for the core to find the subcommand, and then another to pass the remaining args to the subcommand as a main parser, but autocompletion failed to cope with this setup.

These examples on the other hand are tab completable subcommands with default configuration read from a config file.

$ ./simple
usage: simple [-h] [-d] [-c CONFIG] {qux,fux,bux,lux} ...

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug
  -c CONFIG, --config CONFIG

Title Group A:
  This is a long description on a simple command.

  {qux,fux,bux,lux}     some help on group A
$ ./simple 
bux       --config  --debug   -h        lux       
-c        -d        fux       --help    qux       
$ ./simple lux 
final pass Namespace(command='lux', config='example.conf', debug=False, foo='XXX', run=<bound method F.main of <__main__.F object at 0x2417d50>>) None
Hello, I'm lux.
['lux',
 'params',
 <__main__.F object at 0x2417d50>,
 Namespace(command='lux', config='example.conf', debug=False, foo='XXX', run=<bound method F.main of <__main__.F object at 0x2417d50>>)]
$ ./simple qux
final pass Namespace(command='qux', config='example.conf', debug=False, foo='baz', run=<bound method F.main of <__main__.F object at 0x1c632d0>>) None
Hello, I'm qux.
['qux',
 'params',
 <__main__.F object at 0x1c632d0>,
 Namespace(command='qux', config='example.conf', debug=False, foo='baz', run=<bound method F.main of <__main__.F object at 0x1c632d0>>)]
$ ./simple qux --help
usage: simple qux [-h] [--foo FOO]

long desc of qux

optional arguments:
  -h, --help  show this help message and exit
  --foo FOO

So the only thing missing is the default values from the help output.

import argparse
import ConfigParser
import os
"""
add_config.py - http://stackoverflow.com/questions/6133517/parse-config-file-environment-and-command-line-arguments-to-get-a-single-coll
"""
def _identity(x):
return x
_SENTINEL = object()
class AddConfigFile(argparse.Action):
def __call__(self,parser,namespace,values,option_string=None):
#I can never remember if `values` is a list all the time or if it
#can be a scalar string ... -- This takes care of both.
if isinstance(values,basestring):
parser.config_files.append(values)
else:
parser.config_files.extend(values)
class ArgumentConfigEnvParser(argparse.ArgumentParser):
def __init__(self,*args,**kwargs):
"""
Added 2 new keyword arguments to the ArgumentParser constructor:
config --> List of filenames to parse for config goodness
default_section --> name of the default section in the config file
"""
self.config_files = kwargs.pop('config',[]) #Must be a list
self.default_section = kwargs.pop('default_section','MAIN')
self._action_defaults = {}
argparse.ArgumentParser.__init__(self,*args,**kwargs)
def add_argument(self,*args,**kwargs):
"""
works like ArgumentParser.add_argument except that we've added an action:
config: add a config file to the parser
and we've added the ability to specify the section to pull the data from in the
config file via the `section` keyword. Relies on the undocumented fact that
`ArgumentParser.add_argument` actually returns the `Action` object that it creates.
We need this to reliably get `dest`, although we could probably write a simple
function to do this for us.
"""
if 'action' in kwargs and kwargs['action'] == 'config':
kwargs['action'] = AddConfigFile
kwargs['default'] = argparse.SUPPRESS
#argparse won't know what to do with the section, so we'll pop it out and add it back in
#later.
#Also, we have to prevent any type conversion by argparse. We'll do it explicitly
#in parse_known_args. This way, we can reliably check whether argparse has replaced the default.
section = kwargs.pop('section',self.default_section)
type = kwargs.pop('type',_identity)
default = kwargs.pop('default',_SENTINEL)
kwargs.update(default=_SENTINEL if default is not argparse.SUPPRESS else argparse.SUPPRESS)
action = argparse.ArgumentParser.add_argument(self,*args,**kwargs)
kwargs.update(section=section,type=type,default=default)
self._action_defaults[action.dest] = (args,kwargs)
return action
def parse_known_args(self,args=None, namespace=None):
#parse_args calls parse_known_args, so we should be OK with this...
ns, argv = argparse.ArgumentParser.parse_known_args(self,args=args,namespace=namespace)
config_parser = ConfigParser.SafeConfigParser()
config_files = [os.path.expanduser(os.path.expandvars(x)) for x in self.config_files]
config_parser.read(config_files)
for dest,(args,init_dict) in self._action_defaults.items():
type_converter = init_dict['type']
default = init_dict['default']
obj = default
if getattr(ns,dest,_SENTINEL) is not _SENTINEL: #found on command line
obj = getattr(ns,dest)
else: #not found on commandline
try: #get from config file
obj = config_parser.get(init_dict['section'],dest)
except (ConfigParser.NoSectionError,ConfigParser.NoOptionError): #Nope, not in config file
try: #get from environment
obj = os.environ[dest.upper()]
except KeyError:
pass
if obj is _SENTINEL:
setattr(ns,dest,None)
elif obj is argparse.SUPPRESS:
pass
else:
setattr(ns,dest,type_converter(obj))
return ns, argv
if __name__ == '__main__':
fake_config = """
[MAIN]
foo:bar
bar:1
"""
with open('_config.file','w') as fout:
fout.write(fake_config)
parser = ArgumentConfigEnvParser()
parser.add_argument('--config-file',action='config',help="location of config file")
parser.add_argument('--foo',type=str,action='store',default="grape",help="don't know what foo does ...")
parser.add_argument('--bar',type=int,default=7,action='store',help="This is an integer (I hope)")
parser.add_argument('--baz',type=float,action='store',help="This is an float(I hope)")
parser.add_argument('--qux',type=int,default='6',action='store',help="this is another int")
ns = parser.parse_args([])
parser_defaults = {'foo':"grape",'bar':7,'baz':None,'qux':6}
config_defaults = {'foo':'bar','bar':1}
env_defaults = {"baz":3.14159}
#This should be the defaults we gave the parser
print ns
assert ns.__dict__ == parser_defaults
#This should be the defaults we gave the parser + config defaults
d = parser_defaults.copy()
d.update(config_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
os.environ['BAZ'] = "3.14159"
#This should be the parser defaults + config defaults + env_defaults
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
ns = parser.parse_args(['--config-file','_config.file'])
print ns
assert ns.__dict__ == d
#This should be the parser defaults + config defaults + env_defaults + commandline
commandline = {'foo':'3','qux':4}
d = parser_defaults.copy()
d.update(config_defaults)
d.update(env_defaults)
d.update(commandline)
ns = parser.parse_args(['--config-file','_config.file','--foo=3','--qux=4'])
print ns
assert ns.__dict__ == d
os.remove('_config.file')
complete -F _optcomplete b.py
# Copyright 2012-2013, Andrey Kislyuk and argcomplete contributors.
# Licensed under the Apache License. See https://github.com/kislyuk/argcomplete for more info.
_python_argcomplete_global() {
local ARGCOMPLETE=0
if [[ "$1" == python* ]] || [[ "$1" == pypy* ]]; then
if [[ -f "${COMP_WORDS[1]}" ]] && (head -c 1024 "${COMP_WORDS[1]}" | grep --quiet "PYTHON_ARGCOMPLETE_OK") >/dev/null 2>&1; then
local ARGCOMPLETE=2
set -- "${COMP_WORDS[1]}"
fi
elif (which "$1" && head -c 1024 $(which "$1") | grep --quiet "PYTHON_ARGCOMPLETE_OK") >/dev/null 2>&1; then
local ARGCOMPLETE=1
elif (which "$1" && head -c 1024 $(which "$1") | egrep --quiet "(EASY-INSTALL-SCRIPT|EASY-INSTALL-ENTRY-SCRIPT)" \
&& python-argcomplete-check-easy-install-script $(which "$1")) >/dev/null 2>&1; then
local ARGCOMPLETE=1
fi
if [[ $ARGCOMPLETE == 1 ]] || [[ $ARGCOMPLETE == 2 ]]; then
local IFS=$(echo -e '\v')
COMPREPLY=( $(_ARGCOMPLETE_IFS="$IFS" \
COMP_LINE="$COMP_LINE" \
COMP_POINT="$COMP_POINT" \
_ARGCOMPLETE=$ARGCOMPLETE \
"$1" 8>&1 9>&2 1>/dev/null 2>&1) )
if [[ $? != 0 ]]; then
unset COMPREPLY
fi
fi
}
complete -o nospace -o default -D -F _python_argcomplete_global
#!/usr/bin/python
import sys
import commando
from pprint import pprint
import argcomplete # PYTHON_ARGCOMPLETE_OK
__version__ = '0.0.1-dev'
"""
class ParamGroup(commando.param):
__params__ = ( ('-d', '--debug'), dict(action='store_true' ) )
def __init__(self):
params = self.params( )
args, kwds = params[0], params[1]
self.values = commando.application.values._make( (args, kwds) )
def params(self):
return self.__params__
"""
## base_val = commando.values
class TaggedMemo(object):
# subcommand = base_val._make(((), dict()))
commando = True
params = [ ]
def __init__(self, name=None):
# default = base_val._make(((name, ), dict( )))
self.subcommand = getattr(self.main, 'subcommand', default)
self.params = [ ]
def __call__(self, app, params):
self.main(params)
def main(self, params):
print "I'm an unsubclassed Command"
pprint([self, params])
class DemoRecurse(TaggedMemo):
@commando.subcommand('far', help='demo far recurse')
def main(self, params):
print "used decorator on a subclass"
class FooBar(commando.Application):
@commando.subcommand('one', help="one")
def one(self, params):
print "one"
@commando.subcommand('two', help="two")
def two(self, params):
print "two"
#@commando.command(description="my command description")
@commando.command('far', description='demo far recurse')
def main(self, params):
print "used decorator on a simple class"
class Root(commando.Application):
def print_usage(self):
self.__parser__.print_help( )
@commando.command(description="b - my command description")
@commando.param('-v', '--version', action='version',
version='%(prog)s ' + __version__)
@commando.param('-foo', dest='foo', action='store', type=str)
def main(self, app, params):
pprint([self, params])
@commando.subcommand('init', help="main command")
def init(self, params):
self.main(self, params)
@commando.subcommand('help', help="This help")
def help(self, params):
self.print_usage( )
@commando.subcommand('foo', help="foo command")
@commando.param('extra', action='store', type=str, nargs="*")
def foo(self, params):
pprint([self, params])
print "HAHA", params
f = FooBar( )
f.run(params.extra)
def __init__(self):
super(type(self), self).__init__( )
# print self.__parser__
argcomplete.autocomplete(self.__parser__)
print "foo.parser"
f = FooBar( )
pprint(self.foo.parser)
print "foo.parser.subcommand"
#pprint(self.foo.subcommand)
# class Application(Root, commando.WeirdApp):
class Application(Root):
# __subcommands__ = [ FooBar( ), DemoRecurse('demo') ]
def __init__(self):
super(Application, self).__init__()
pass
def main( ):
app = Root( )
app.run( )
#tree = commando.TreeApp(root=Root( ))
# tree.run( )
if __name__ == '__main__':
main( )
#####
# EOF
[qux]
foo=baz
#!/usr/bin/python
from cli.log import LoggingApp
import argcomplete # PYTHON_ARGCOMPLETE_OK
from pprint import pprint
"""
Very rough.
"""
class Subcommand(object):
name = ''
def __init__(self, handler):
self.session = handler
def options(self):
return [ ]
def setup(self, parser):
self.parser = parser
for args, kwds in self.options( ):
parser.add_argument(*args, **kwds)
def help(self):
return self.__doc__
def main(self, app):
pprint([self, app])
print self.name
class QuxApp(Subcommand):
"""Qux Does several special things"""
name = "qux"
class FuxApp(Subcommand):
"""Fux is accidently different."""
name = "fux"
class BuxApp(Subcommand):
"""Bux seems special, but it's a trick."""
name = "bux"
class BaxApp(Subcommand):
"""Bax seems special, bat it's a trick."""
name = "bax"
class FaxApp(Subcommand):
"""Fax is accidently different."""
name = "fax"
class QaxApp(Subcommand):
"""Qax Does several special things"""
name = "qax"
class Command(object):
"Fake help"
subcommands = { }
def __init__(self, name, subcommands=None):
self.subcommands = { }
self.name = name
if subcommands is not None:
for Flow in subcommands:
self.addFlow(Flow)
def __repr__(self):
return "%s:%s" % (self.name, type(self))
def subcommand_factory(self, Flow):
return Flow(self)
def addFlow(self, Flow):
flow = self.subcommand_factory(Flow)
self.subcommands[flow.name] = flow
def base(self, params):
pprint(self, params)
print "Without a subcommand!!"
def setup(self, parser):
n = self.name
self.parser = parser
pprint(['before', parser._positionals])
self.commands = parser.add_subparsers(dest='subcommand',
title="my commands title",
help=self.help( ),
prog="command")
pprint(['after', parser._positionals])
parser.set_defaults(run=self.base)
#p = self.commands.add_parser(None, help='basic help')
for flow in self.subcommands.values( ):
#flow = Flow( )
p = self.commands.add_parser(flow.name, help=flow.help())
flow.setup(p)
def help(self):
return self.__doc__
def main(self, app):
subcommand = self.subcommands[app.params.subcommand]
subcommand.main(self)
def get_commands():
devices = [ ]
a = Command('AAA', [ FuxApp, BuxApp ] )
b = Command('BBB', [ FaxApp, QaxApp, ] )
c = Command('CCC', [ FuxApp, BaxApp, QuxApp, BuxApp ] )
return [ a, b, c ]
class GlobalOptions(object):
def setup(self):
print "setting up global options"
class Application(LoggingApp, GlobalOptions):
"""Test Hello World
"""
name = "dispatcher"
commands = { }
def setup(self):
# just after wrapping argument during __call__
# !? or during __init__
super(Application, self).setup( )
GlobalOptions.setup(self)
argcomplete.autocomplete(self.argparser)
self.add_param("bar", help="fake option", action='store_true')
self.subparsers = self.argparser.add_subparsers(dest='command', help='fake help on this command')
for dev in get_commands():
self.add_command( dev )
def pre_run(self):
# called just before main, updates params, parses args
super(Application, self).pre_run()
def add_command(self, command):
self.commands[command.name] = command
parser = self.subparsers.add_parser(command.name, help=command.help())
#parser.set_defaults(run=self.main)
command.setup(parser)
def main(self):
self.log.warn("hello world warn")
self.log.debug("hello world debug")
self.log.info("hello world info")
self.log.error("hello world error")
self.log.critical("hello world critical")
self.log.fatal("hello world fatal")
command = self.commands[self.params.command]
pprint(self.params)
command.main(self)
if __name__ == '__main__':
app = Application()
app.run( )
#####
# EOF
#!/usr/bin/python
"""
Simple app using argparse.
How can I provide a bunch of subcommands, but make one of them a default?
"""
from pprint import pprint, pformat
import argparse
import argcomplete # PYTHON_ARGCOMPLETE_OK
import logging
import ConfigParser
log = logging.getLogger(__name__)
class F(object):
def __init__(self, name):
self.name = name
self.parser = argparse.ArgumentParser(prog=name, description=self.descr( ), add_help=True)
def descr(self):
desc = "A very long description of %s" % self.name
return desc
def set_parser(self, parser, options):
self.parser = parser
parser.add_argument('--foo',
type=str,
default=options.get('foo', 'XXX'))
parser.set_defaults(**options)
def parse_args(self, argv):
return self.parser.parse_args(argv)
def main(self, app, params):
print "Hello, I'm %s." % self.name
pprint([ self.name, 'params', self, params ])
class Simple(object):
"""
This is a very simple description.
We'll simulate it very long here.
Very long indeed.
"""
def global_parser(self):
desc = type(self).__doc__
parser = argparse.ArgumentParser(description=desc, add_help=False,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('-d', '--debug', action='store_true')
parser.add_argument('-c', '--config', default="example.conf", type=str)
return parser
def __init__(self):
desc = type(self).__doc__
conf_parser = self.global_parser( )
self.pre_parser = argparse.ArgumentParser(add_help=False, parents=[conf_parser])
# first sniff for basic debuggery, help fill in defaults
self.known, args = self.pre_parser.parse_known_args( )
# then set up the real parser, cloning the initial one
self.parser = argparse.ArgumentParser(parents=[conf_parser], add_help=True,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
self.read_config(self.known.config)
self.commands = { }
self.setup_a(self.parser)
if not args:
self.parser.print_help( )
argcomplete.autocomplete(self.parser)
def read_config(self, path):
self.config = config = getattr(self, 'config', ConfigParser.ConfigParser( ))
config.read(path)
return config
def setup_a(self, parser, name='lalala'):
desc = """This is a long description on a simple command."""
subparsers = parser.add_subparsers(title='Title Group A',
dest='command',
# default='qux',
description = desc,
help='some help on group A')
config = self.config
choices = [ ]
for x in [ 'qux', 'fux', 'bux', 'lux' ]:
f = F(x)
kwds = dict( description = "long desc of %s" % x,
)
# default= 'qux')
p = subparsers.add_parser(f.name, **kwds)
cfg = config.has_section(f.name) and dict(config.items(f.name)) or { }
f.set_parser(p, cfg)
self.commands[f.name] = f
p.set_defaults(run=f.main)
choices.append(f.name)
self.pre_parser.add_argument('command', type=str, choices=choices, default='help')
def parse(self, argv):
return self.parser.parse_args(argv)
def run(self, args=None):
remain = None
if args is None:
import sys
args = self.parse(sys.argv[1:])
print 'final pass', pformat(args), pformat(remain)
if hasattr(args, 'run'):
args.run(self, args)
else:
pass
# self.main(self.tool, args)
def main(self, app, params):
pass
# self.tool.main(self, params)
if __name__ == '__main__':
app = Simple()
app.run( )
#####
# EOF
#!/usr/bin/python
import unittest
import argparse
from pprint import pprint
class F(object):
def __init__(self, name):
self.name = name
def main(self, app, params):
pprint([ self.name, self, params ])
print "Hello, I'm %s." % self.name
def get_simple_parser( ):
parser = argparse.ArgumentParser()
# flesh it out a bit with a no-op option
parser.add_argument('-d', '--debug', action='store_true')
return parser
class TestSubparserDefaultFun(unittest.TestCase):
def setUp(self):
self.parser = get_simple_parser( )
setup_a(self.parser)
def parse_missing_subcommand_args(self):
pprint(parser.parse_args(prep('')))
def donttest_required_subcommand_behavior(self):
self.parser.parse_args( )
def test_optional_subcommand_behavior(self):
self.parser.parse_args(prep(' '))
def setup_a(parser, name='lalala'):
desc = """This is a long description on a simple command."""
# goal, make qux the default
# default = 'qux'
kwds = dict( title='Title Group A',
dest='command',
description = desc,
prog= "my program",
# XXX: PATCH
default = 'qux',
help='some help on group A')
subparsers = parser.add_subparsers(**kwds)
for x in [ 'qux', 'fux', 'bux', 'lux' ]:
f = F(x)
kwds = dict( description = "long desc of %s" % x,
)
# default= 'qux')
p = subparsers.add_parser(x, **kwds)
p.set_defaults(run=f.main)
def prep(args):
return args.split( )
def main( ):
parser = get_simple_parser( )
#sub = parser.add_subparsers()
#sub.add_parser("info")
#parser.add_argument("paths", "+")
#pprint(parser.parse_args(["foo", "bar"]))
# add some subcommands
setup_a(parser)
#
pprint(parser.parse_args(prep('')))
pprint(parser.parse_args(prep('')))
if __name__ == '__main__':
unittest.main( )
#####
# EOF
You can’t perform that action at this time.