Command line: create dictionary from key-value pairs

key-pairs-cmdline.md

Converting an arbitrary list of key-value pairs from the command-line into a Python dictionary

Problem

You want to create a series of key-value pairs from the command line, using the argparse library, e.g.:

command par1 par2 --set foo=hello bar="hello world" baz=5

This is typically useful when you want to clearly distinguish":

1. Ordinary arguments for the command-line utility itself (output, input, format, etc.) from
2. A set of key-value pairs you want to pass to the python application. This is especially valid when you do not want that set of values to be predetermined, as this can save a lot of code.

This possibility changes somewhat the way we look at argparse.

Solution

Standard declaration with argparse:

import argparse
parser = argparse.ArgumentParser(description="...")
...
parser.add_argument("--set",
                        metavar="KEY=VALUE",
                        nargs='+',
                        help="Set a number of key-value pairs "
                             "(do not put spaces before or after the = sign). "
                             "If a value contains spaces, you should define "
                             "it with double quotes: "
                             'foo="this is a sentence". Note that '
                             "values are always treated as strings.")
args = parser.parse_args()

The argument is optional and multivalued, with a minimum of one occurrence (nargs='+').

The result is a list of strings e.g. ["foo=hello", "bar=hello world", "baz=5"] in args.set, which we now need to parse (note how the shell has processed and removed the quotes!).

Parsing the result

def parse_var(s):
    """
    Parse a key, value pair, separated by '='
    That's the reverse of ShellArgs.

    On the command line (argparse) a declaration will typically look like:
        foo=hello
    or
        foo="hello world"
    """
    items = s.split('=')
    key = items[0].strip() # we remove blanks around keys, as is logical
    if len(items) > 1:
        # rejoin the rest:
        value = '='.join(items[1:])
    return (key, value)


def parse_vars(items):
    """
    Parse a series of key-value pairs and return a dictionary
    """
    d = {}

    if items:
        for item in items:
            key, value = parse_var(item)
            d[key] = value
    return d

# parse the key-value pairs
values = parse_vars(args.set)

Now the variable valuescontains a dictionary with the key-value pairs defined on the command line:

values = {'foo':'hello', 'bar':'hello world', 'baz':'5'}

Note that the values are always returned as strings.

Nice work. Note that you can avoid the rejoining of a value if you limit the string splitting, as in items = s.split('=', 1).

Another approach on this using an argparse.Action:

class ParseDict(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        d = getattr(namespace, self.dest) or {}

        if values:
            for item in values:
                split_items = item.split("=", 1)
                key = split_items[
                    0
                ].strip()  # we remove blanks around keys, as is logical
                value = split_items[1]

                d[key] = value

        setattr(namespace, self.dest, d)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--set",
        metavar="KEY=VALUE",
        nargs="+",
        help="Set a number of key-value pairs "
        "(do not put spaces before or after the = sign). "
        "If a value contains spaces, you should define "
        "it with double quotes: "
        'foo="this is a sentence". Note that '
        "values are always treated as strings.",
        action=ParseDict,
    )
    args = parser.parse_args()

    print(args.set)

Added the fix from https://gist.github.com/fralau/061a4f6c13251367ef1d9a9a99fb3e8d?permalink_comment_id=3748428#gistcomment-3748428

ParseDict solution needs a fix, as it's overriding dict on each parameter, so it's not acumulating.

def __call__(self, parser, namespace, values, option_string=None):
        -d = {}
        +d = getattr(namespace, self.dest, {})

it needs a check if d is not already self.default

ParseDict solution needs a fix, as it's overriding dict on each parameter, so it's not acumulating.

def __call__(self, parser, namespace, values, option_string=None):
        -d = {}
        +d = getattr(namespace, self.dest, {})

Updated, thanks for the note and not sure how I missed that!

@jsmolina and @f0ff886f : Nice additions, thanks! 👍

ParseDict solution needs a fix, as it's overriding dict on each parameter, so it's not acumulating.

def __call__(self, parser, namespace, values, option_string=None):
        -d = {}
        +d = getattr(namespace, self.dest, {})

This line crashes, because namespace has an attribute called set, but its value is None - so it is not replaced by the default {}
Then, you have d=None instead of d={} and the code crashes later

Instead, this line should be:
d = getattr(namespace, self.dest) or {}

applied above patching, it looks like:

class ParseDict(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        d = getattr(namespace, self.dest) or {}

        if values:
            for item in values:
                split_items = item.split("=", 1)
                key = split_items[
                    0
                ].strip()  # we remove blanks around keys, as is logical
                value = split_items[1]

                d[key] = value

        setattr(namespace, self.dest, d)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--set",
        metavar="KEY=VALUE",
        nargs="+",
        help="Set a number of key-value pairs "
        "(do not put spaces before or after the = sign). "
        "If a value contains spaces, you should define "
        "it with double quotes: "
        'foo="this is a sentence". Note that '
        "values are always treated as strings.",
        action=ParseDict,
    )
    args = parser.parse_args()

    print(args.set)

Thanks for this! I wanted something a little different, but this got me where I wanted to go.

The snippet below does the same thing as yours, but allows specifying the same argument multiple times with a single key/value pair each time.

class ParseKVPair(argparse.Action):
    def __call__(self, parser, namespace, value, option_string=None):
        d = getattr(namespace, self.dest) or {}

        if values:
            split_items = item.split("=", 1)
            key = split_items[
                0
            ].strip()  # we remove blanks around keys, as is logical
            value = split_items[1]
            d[key] = value
         setattr(namespace, self.dest, d)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--label",
        dest="labels",
        metavar="KEY=VALUE",
        help="Set a number of key-value pairs "
        "(do not put spaces before or after the = sign). "
        "If a value contains spaces, you should define "
        "it with double quotes: "
        'foo="this is a sentence". Note that '
        "values are always treated as strings.",
        action=ParseKVPair,
    )
    args = parser.parse_args()

    print(args.labels)

I found @jsolbrig idea more suitable in my case, just fixed some typos and put a bit more safeguards:

class store_dict(argparse.Action):
    def __call__(self, parser, namespace, value, option_string=None):
        d = getattr(namespace, self.dest) or {}
        if value:
            kv = value.split('=', 1)
            k = kv[0].strip() # we remove blanks around keys, as it's logical
            d[k] = kv[1] if len(kv) > 1 else None
        setattr(namespace, self.dest, d)

if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument('--label', '-l',
            action=store_dict,
            type=str,
            default={ "grafana_dashboard": "1" },
            dest="labels",
            metavar="LABEL=VALUE",
            help="Set a number of key-value pairs "
            "(do not put spaces before or after the = sign). "
            "If a value contains spaces, you should define "
            "it with double quotes: "
            'foo="this is a sentence". Note that '
            "values are always treated as strings.",
        )

It's questionable, what to do if the user-supplied --label foo without actually providing the value. I decided that I don't want to raise an exception here, so in such a case you'll get { "foo": None }. Alternatives could be { "foo": "" } or an exception.