{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Functional Programming Techniques in Python Part 1\n",
"\n",
"Author:[mkocher](https://github.com/mpkocher)\n",
"\n",
"This is introduction to Functional Programming Techniques (FPT) using Python. \n",
"\n",
"Topics Covered\n",
"\n",
"- Function basics\n",
"- Closures\n",
"- Partial Application (from functools)\n",
"- Reduce (from functools)\n",
"- Function Composition\n",
"- Map and Filter from stdlib\n",
"- Creating Data Pipelines using FPT\n",
"- Conclusion and Summary"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import datetime\n",
"import types\n",
"import string\n",
"import random\n",
"from collections.abc import Iterable, Callable\n",
"# Requires Python >= 3.7\n",
"from dataclasses import dataclass, fields\n",
"\n",
"import functools\n",
"import itertools\n",
"import operator as op"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Today is 2019-02-15 03:19:07.472785\n"
]
}
],
"source": [
"print(\"Today is {}\".format(datetime.datetime.now()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Demo Data\n",
"\n",
"Define some utils for generating test data. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"XS = range(10)\n",
"\n",
"@dataclass(frozen=True)\n",
"class User(object):\n",
" user_id:int\n",
" age:int \n",
" first_name:str\n",
" favorite_color:str \n",
"\n",
" \n",
"def to_users():\n",
" colors = ['red', \"blue\", \"black\", \"orange\", \"yellow\"]\n",
" def to_random(xs):\n",
" return random.choice(xs)\n",
" to_color = functools.partial(to_random, colors)\n",
" \n",
" names = [(\"Ralph\", 13), (\"Joe\", 43),(\"Steve\", 66), (\"Rebecca\", 41), \n",
" (\"Sam\", 4), (\"Richard\", 32), (\"Paul\", 87), \n",
" (\"Stephen\", 55), (\"Sean\", 2)]\n",
"\n",
" for i, (name, age) in enumerate(names):\n",
" yield User(i + 1, age, name, to_color())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Python Function Basics\n",
"\n",
"There are several different models for defining functions in Python.\n",
"\n",
"The first is `lambda` functions. This model should primarly be used within function calls. \n",
"\n",
"A good discussion [in the Python docs](https://docs.python.org/3.7/howto/functional.html#small-functions-and-the-lambda-expression) provides some general tips on when use the `lambda` style. "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"9\n",
"<function <lambda> at 0x10749cd08>\n",
"<lambda>\n"
]
}
],
"source": [
"f0 = lambda n: n * n\n",
"\n",
"f0(3) == 9\n",
"\n",
"print(f0(3))\n",
"print(f0)\n",
"print(f0.__name__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The most common model to write functions in Python show below. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"9\n",
"('f1', 'A simple function to square a number')\n",
"(<function f1 at 0x10749ce18>, <function f2 at 0x10749cea0>)\n"
]
}
],
"source": [
"def f1(n):\n",
" \"\"\"A simple function to square a number\"\"\"\n",
" return n * n\n",
"\n",
"# Or using type annotations in Python 3\n",
"def f2(n:int) -> int:\n",
" return n * n\n",
"\n",
"assert f1(3) == f2(3)\n",
"print(f1(3))\n",
"print((f1.__name__, f1.__doc__))\n",
"print((f1, f2))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, classes can be used to create `Callables`. This can be useful when you need to initialize a class with state."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Callable class <__main__.F3 object at 0x1074eaf60> 9\n",
"<F4 n:2>\n",
"<bound method F4.compute of <F4 n:2>>\n"
]
}
],
"source": [
"# Or using a class style. \n",
"class F3(Callable):\n",
" def __call__(self, n):\n",
" return n * n\n",
" \n",
"f3 = F3()\n",
"print((\"Callable class {} {}\".format(f3, f3(3))))\n",
"\n",
"assert f3(3) == f1(3)\n",
"\n",
"# Perhaps a better example. \n",
"class F4(Callable):\n",
" def __init__(self, n):\n",
" self.n = n\n",
" \n",
" def __repr__(self):\n",
" _d = dict(k=self.__class__.__name__, n=self.n)\n",
" return \"<{k} n:{n}>\".format(**_d)\n",
" \n",
" def __call__(self, n):\n",
" return self.n * n\n",
" \n",
" def compute(self, n):\n",
" return self.n * n\n",
"\n",
"f4 = F4(2)\n",
"print(f4)\n",
"\n",
"assert f4(3) == 6\n",
"\n",
"# You can also define a bound function\n",
"f5 = f4.compute\n",
"print(f5)\n",
"\n",
"assert f5(3) == 6"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This adds quite a bit of boilerplate. There's a cleaner technique leveraging the first class nature of functions in Python. \n",
"\n",
"# 2. Closures \n",
"\n",
"In Python we can define functions that return functions. \n",
"\n",
"For example:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Func <function multipler.<locals>.func at 0x1074ee400> out=6\n"
]
}
],
"source": [
"def multipler(x):\n",
" def func(y):\n",
" return x * y\n",
" return func\n",
"\n",
"f6 = multipler(2)\n",
"print(\"Func {} out={}\".format(f6, f6(3)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is an extremely value pattern to leverage, specifically if you have \"pure\" (i.e. side-effect free) functions. This can be thought of as a \"factory\" that returns a unit computation. It's also information hidding and echos the features of the class approach shown above. \n",
"\n",
"Here's a few examples to help demonstrate the pattern. Let's filter out all the values of list using the list comprehension style."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 1, 2, 3]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def lte_3(n):\n",
" return n <= 3\n",
"\n",
"[x for x in XS if lte_3(x)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's generalize our filter func so that we can leverage this util in other areas of our code base."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 1, 2, 3]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def to_filter(min_value):\n",
" def func(x):\n",
" return x <= min_value\n",
" return func\n",
"\n",
"filter_3 = to_filter(3)\n",
"[x for x in XS if filter_3(x)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is very powerful technique. The examples shown here are minimal to demonstrate the core idea. \n",
"\n",
"\n",
"## Digression\n",
"\n",
"Let's look at the `lru_cache` from the Python standard lib to look at some real world examples using closures as a core design pattern.\n",
"\n",
"https://github.com/python/cpython/blob/3.7/Lib/functools.py#L445\n",
"\n",
"There's a few interesting points to note.\n",
"\n",
"1. The use of bound functions to avoid the extra dictionary lookup (the `lrc_cache` is often used in a tight loop).\n",
"\n",
"https://github.com/python/cpython/blob/3.7/Lib/functools.py#L494\n",
"\n",
"```python\n",
"cache_get = cache.get\n",
"```\n",
"\n",
"2. The use of `nonlocal` to enable mutating the number of hits and misses within a closure. \n",
"\n",
"https://github.com/python/cpython/blob/3.7/Lib/functools.py#L513\n",
"\n",
"\n",
"More details on the use of `nonlocal` in the [official docs](https://docs.python.org/3/reference/simple_stmts.html#nonlocal).\n",
"\n",
"https://github.com/python/cpython/blob/3.7/Lib/functools.py#L597\n",
"\n",
"\n",
"```python\n",
"nonlocal hits, misses\n",
"```\n",
"\n",
"In Python 2.7.x, you can hack around this example by using a mutable container. \n",
"\n",
"For example:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"calling func=3\n",
"total times called=1\n",
"calling func=3\n",
"total times called=2\n",
"calling func=3\n",
"total times called=3\n",
"calling func=3\n",
"total times called=4\n",
"calling func=3\n",
"total times called=5\n"
]
}
],
"source": [
"def two_seven(n):\n",
" # DONT do this in Python 3. Use nonlocal.\n",
" total = [0]\n",
" \n",
" def f(m):\n",
" total[0] += 1\n",
" return n + m\n",
" \n",
" def get_total():\n",
" return total[0]\n",
" \n",
" # see comments below about use of this pattern\n",
" f.get_total = get_total\n",
" return f\n",
"\n",
"def two_seven_example():\n",
" f27 = two_seven(2)\n",
" def fx():\n",
" return f27(1)\n",
" for _ in range(5):\n",
" for name, func in ((\"calling func\", fx), (\"total times called\", f27.get_total)):\n",
" print(\"{}={}\".format(name, func()))\n",
"\n",
"two_seven_example()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3. Another design choice used in `lru_cache` is attach functions to the wrapper function used in the closure. \n",
"\n",
"https://github.com/python/cpython/blob/3.7/Lib/functools.py#L597\n",
"\n",
"```python\n",
"wrapper.cache_info = cache_info\n",
"wrapper.cache_clear = cache_clear\n",
"return wrapper\n",
"```\n",
"\n",
"I suspect the choice of using functions instead of class here is for performance reasons. \n",
"\n",
"In general, this pattern should be used judiciously. Often it's probably better to migrate to use classes for these cases instead of functions and closures. \n",
"\n",
"The often quoted `objects are truly a poor man's closures .... Closures are a poor man's object.` is an interesting read.\n",
"\n",
"http://wiki.c2.com/?ClosuresAndObjectsAreEquivalent\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Other Examples of API Design Leveraging Closures\n",
"\n",
"[More involved examples from production code are shown here](https://github.com/mpkocher/pbcommand/blob/master/pbcommand/cli/quick.py#L249).\n",
"\n",
"[And here's a snippet using argparse's Action](https://github.com/mpkocher/pbcommand/blob/master/pbcommand/common_options.py#L109)."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"import argparse\n",
"\n",
"def _to_print_message_action(msg):\n",
"\n",
" class PrintMessageAction(argparse.Action):\n",
"\n",
" \"\"\"Print message and exit\"\"\"\n",
"\n",
" def __call__(self, parser, namespace, values, option_string=None):\n",
" sys.stdout.write(msg + \"\\n\")\n",
" sys.exit(0)\n",
"\n",
" return PrintMessageAction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 3. Partial Application and Functools.partial\n",
"\n",
"Using closures is a useful pattern, however, we often want to write our functions without this extra function returning layer. This can be addressed using a techinque known as *Partial Application*. This is accessible in [Python's standard library](https://docs.python.org/3/library/functools.html#functools.partial) using `functools.partial`. \n",
"\n",
"Let's revisiting the `to_filter` example."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"min=2 x=4 False\n",
"min=4 x=3 True\n"
]
}
],
"source": [
"def to_filter(min_value):\n",
" def func(x):\n",
" return x <= min_value\n",
" return func\n",
"\n",
"# Let's rewrite this in a more \"natural\" form.\n",
"\n",
"def filter_min_value(min_value, x):\n",
" return x <= min_value\n",
"\n",
"for x, y in [(2, 4),(4, 3)]:\n",
" print(\"min={} x={} {}\".format(x, y, filter_min_value(x, y)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now let's use `functools.partial` to bind the first val in the function and return a new function. "
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"functools.partial(<function filter_min_value at 0x1074eeb70>, 3)\n"
]
},
{
"data": {
"text/plain": [
"[0, 1, 2, 3]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"filter_lte_3 = functools.partial(filter_min_value, 3)\n",
"print(filter_lte_3)\n",
"\n",
"[x for x in XS if filter_lte_3(x)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For some cases the partial application approach can reduce the boilerplate. \n",
"\n",
"However, there is one important note to be aware of. When using `functools.partial`, the returned func will not have `__doc__` defined. Sometimes it might be useful to explicitly set `__doc__` on the func to have closer parity to \"standard\" functions. "
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"filter_min_value\n"
]
}
],
"source": [
"print(filter_min_value.__name__)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"ename": "AttributeError",
"evalue": "'functools.partial' object has no attribute '__name__'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-15-76c5bd9cb71f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilter_lte_3\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m: 'functools.partial' object has no attribute '__name__'"
]
}
],
"source": [
"print(filter_lte_3.__name__)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Filter lte 3\n"
]
}
],
"source": [
"filter_lte_3.__name__ = \"Filter lte 3\"\n",
"print(filter_lte_3.__name__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One more important util from `functools` is `reduce`. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 4. Reduce using functools.reduce\n",
"\n",
"This is used to take a `list[T]` and reduce (or 'fold') to a single result (which is not necessarily type `T`).\n",
"\n",
"General Form:\n",
"\n",
"`f(acc, value, init_value=None) -> value`\n",
"\n",
"The init value allows you to set the initial value to reduce on. Note this fundamentally differs from `map` and `filter` which return an iterable. \n",
"\n",
"See the [official docs](https://docs.python.org/3/library/functools.html#functools.reduce) for more details.\n",
"\n",
"Here's a few examples:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"45"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def adder(a, b):\n",
" return a + b\n",
"\n",
"functools.reduce(adder, XS)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"45"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Or using `operator` from the standard lib https://docs.python.org/3/library/operator.html\n",
"functools.reduce(op.add, XS)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"56"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"init_value = 11\n",
"functools.reduce(op.add, range(10), init_value)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"362880"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"functools.reduce(op.mul, range(1, 10))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note this can be used with custom classes as well. \n",
"\n",
"Here's a bit of a contrived example to determine the `User` with the max age. However, note the use of generators and O(1) space complexity usage."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"def reduce_max_age_user(acc, value):\n",
" if value.age < acc.age:\n",
" return acc\n",
" return value "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"User(user_id=7, age=87, first_name='Paul', favorite_color='red')"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"functools.reduce(reduce_max_age_user, to_users())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 5. Composing Functions: One Final Tool to add to our toolbox\n",
"\n",
"Often there is a need to chain to smaller functions together to create a new function. This is particularly useful for `pure` functions. \n",
"\n",
"For example we don't want to have to do this."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"def squarer(a):\n",
" return a * a\n",
"\n",
"def doubler(a):\n",
" return 2 * a\n",
"\n",
"def incrementer(a):\n",
" return a + 1\n",
"\n",
"def forgetting_basics_of_programming():\n",
" # This is terrible for so many reasons\n",
" a = squarer(doubler(1))\n",
" b = squarer(doubler(2))\n",
" c = squarer(doubler(3))\n",
"\n",
" abc = [a,b,c]\n",
" return abc"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[4, 16, 36]"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"forgetting_basics_of_programming()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There's a lot of chatter and duplication that want to avoid in the example above. \n",
"\n",
"We could define a new func to do compose `squarer` and `doubler`. However, this is a very general need. Composing also often requires `N` number of functions. \n",
"\n",
"First, let's define `compose` to compose `N` functions using our closure technique combined with the `reduce` technique we've just added to our toolbox."
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [],
"source": [
"def compose(*funcs):\n",
" \"\"\"\n",
" Functional composition\n",
" \n",
" [f, g, h] will be f(g(h(x)))\n",
" \"\"\"\n",
" def compose_two(f, g):\n",
" def c(x):\n",
" return f(g(x))\n",
" return c\n",
" return functools.reduce(compose_two, funcs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's test it out"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[3, 9, 99]\n"
]
}
],
"source": [
"fx = compose(incrementer, doubler, squarer)\n",
"\n",
"print(list(map(fx, (1, 2, 7))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. Let's revisit the `forgetting_basics_of_programming` example. "
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"def getting_better():\n",
" f = compose(squarer, doubler)\n",
" a = f(1)\n",
" b = f(2)\n",
" c = f(3)\n",
" return [a, b, c]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is better, but still has duplication that we can improve on."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"def to_better():\n",
" f = compose(squarer, doubler)\n",
" return [f(x) for x in range(1, 4)]"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"([4, 16, 36], [4, 16, 36])"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"getting_better(), to_better()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example \n",
"\n",
"These `helloworld`-esque examples are useful to demonstrate the idea, however, let's demonstrate the pattern in another real world context.\n",
"\n",
"Let's say we have a CLI tool that has 3 different subparsers or separate commandline tools (e.g, alpha, beta, gamma). Each subparser or tool shares a common set of base commandline arguments, while `alpha` and `gamma` share a common subset of args. \n",
"\n",
"Let's use our composition model to construct an argparse.ArgumentParser.\n",
"\n",
"The general structure will be to create functions that take our parser instance and return a parser. These aren't pure functions, however, we can still use the composition model to centralize functionality in our application."
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"def _to_log_opt(p):\n",
" p.add_argument('--level', help=\"Logging Level\", default=\"INFO\")\n",
" return p\n",
"\n",
"def _to_opt_xy(p):\n",
" p.add_argument('x', type=int, default=10, help=\"X Value\")\n",
" p.add_argument('y', type=int, default=20, help=\"Y Value\")\n",
" return p\n",
"\n",
"def _to_opt_z(p):\n",
" p.add_argument('z', type=int, default=30, help=\"Z Value\")\n",
" return p\n",
"\n",
"def to_alpha_parser(name):\n",
" p = argparse.ArgumentParser(description=\"Test Tool {}\".format(name))\n",
" f = compose(_to_opt_z, _to_opt_xy, _to_log_opt)\n",
" return f(p) "
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"p = to_alpha_parser(\"Alpha\")"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Namespace(level='DEBUG', x=1, y=2, z=3)"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"p.parse_args(\"1 2 3 --level=DEBUG\".split())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instead of using OO patterns we're using function composition as core model for code reuse. \n",
"\n",
"For completeness, let's build out the the parser for the other three tools and remove some duplication along the way."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"def _to_opt(achar, default, help):\n",
" def func(p):\n",
" p.add_argument(achar, type=int, default=default, help=help)\n",
" return p\n",
" return func\n",
"\n",
"def _to_opt_xy(p):\n",
" opts = [('X', 10, \"X Value\"), ('Y', 20, \"Y Value\")]\n",
" f_opt = lambda x: _to_opt(*x)\n",
" opt_funcs = map(f_opt, opts)\n",
" f = compose(*opt_funcs)\n",
" return f(p)\n",
"\n",
"def to_parser(name, funcs):\n",
" p = argparse.ArgumentParser(description=\"Test Tool {}\".format(name))\n",
" f = compose(*funcs)\n",
" return f(p) \n",
"\n",
"def to_alpha_parser():\n",
" funcs = (_to_opt_xy, _to_log_opt)\n",
" return to_parser(\"Alpha\", funcs)\n",
"\n",
"def to_beta_parser():\n",
" #let define this inline since it's not shared\n",
" def f(p):\n",
" p.add_argument('--min-radius', help=\"min radius\", default=10)\n",
" return p\n",
" funcs = (f, _to_log_opt)\n",
" return to_parser(\"Beta\", funcs)\n",
"\n",
"def to_gamma_parser():\n",
" _to_opt_z = _to_opt('Z', 30, \"Z Value\")\n",
" funcs = (_to_opt_z, _to_opt_xy, _to_log_opt)\n",
" return to_parser(\"Gamma\", funcs)"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"usage: ipykernel_launcher.py [-h] [--level LEVEL] Y X\n",
"\n",
"Test Tool Alpha\n",
"\n",
"positional arguments:\n",
" Y Y Value\n",
" X X Value\n",
"\n",
"optional arguments:\n",
" -h, --help show this help message and exit\n",
" --level LEVEL Logging Level\n"
]
},
{
"ename": "SystemExit",
"evalue": "0",
"output_type": "error",
"traceback": [
"An exception has occurred, use %tb to see the full traceback.\n",
"\u001b[0;31mSystemExit\u001b[0m\u001b[0;31m:\u001b[0m 0\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/mkocher/anaconda2/envs/core37/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3275: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.\n",
" warn(\"To exit: use 'exit', 'quit', or Ctrl-D.\", stacklevel=1)\n"
]
}
],
"source": [
"alpha_parser = to_alpha_parser()\n",
"# Note, argparse is kinda brutal with it's use of sys.exit. \n",
"# this doesn't really play well with the notebook\n",
"alpha_parser.parse_args([\"--help\"])"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Namespace(X=2, Y=1, level='WARN')"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alpha_parser.parse_args(\"1 2 --level=WARN\".split())"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Namespace(level='INFO', min_radius='11')"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"beta_parser = to_beta_parser()\n",
"beta_parser.parse_args(\"--min-radius 11\".split())"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Namespace(X=2, Y=1, Z=3, level='ERROR')"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"gamma_parser = to_gamma_parser()\n",
"gamma_parser.parse_args(\"1 2 3 --level=ERROR\".split())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. We've constructed our parser for our 3 commandline tools using simple functions and a bit composition to share code and remove duplication along the way."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Check Point\n",
"\n",
"- We've gone over a few different models of defining a function in Python\n",
"- We've added closures to our toolbox of functional programming techniques\n",
"- We've added `functools.partial` from the Python standard library to provide us build up computation\n",
"- Constructed `compose` function that enables us to compose `N` different functions\n",
"\n",
"We've got the basics functions and few new techniques in our toolbox, let's move to 2 remaining core funcs in the python standard lib to round out the fundaments of functional programming in Python."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6. Core Builtin Functions Map and Filter\n",
"\n",
"Just to state the obvious. Every Python programming should be extremely familiar with the core builtin functions. \n",
"\n",
"https://docs.python.org/3/library/functions.html#built-in-functions\n",
"\n",
"In the context of functional programming in Python, we're going to focus on `map` and `filter`. \n",
"\n",
"## Map \n",
"\n",
"https://docs.python.org/3/library/functions.html#map\n",
"\n",
"This enables us to iterate over a list and apply a function and **return an Iterable**. The returning of an `iterable` enables composition. \n",
"\n",
"A few examples show below."
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<map at 0x10758d4e0>"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def f(x): \n",
" return x * x\n",
"\n",
"rx = map(f, XS)\n",
"rx"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"isinstance(rx, Iterable)"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1, 4, 9, 16, 25, 36, 49, 64, 81]"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(map(f, range(1, 10)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Sometimes we'll have to sort out the impedence mismatch of the function signature to leverage `map`"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [],
"source": [
"import math\n",
"def to_distance(x, y, x0=0, y0=0):\n",
" def f(a, a0):\n",
" return math.pow(a - a0, 2)\n",
" return math.sqrt(f(x, x0) + f(y, y0))\n",
"\n",
"def example():\n",
" points = [(3, 4), (5, 12)]\n",
" f = lambda x: to_distance(*x)\n",
" return map(f, points)"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"5.0"
]
},
"execution_count": 42,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"to_distance(3, 4)"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[5.0, 13.0]"
]
},
"execution_count": 43,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(example())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or to drag around context. \n",
"\n",
"For example, for a given set of points, determine the closest point to the origin. "
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"def example():\n",
" points = ((5, 12), (6, 8), (3, 4))\n",
" # let's return a tuple so we have the context of that data points\n",
" # were used. \n",
" f = lambda x: (to_distance(*x), x)\n",
" mx = map(f, points)\n",
" # this is arguably a bit sloppy and abusing tuple sorting. \n",
" return functools.reduce(min, mx)"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(5.0, (3, 4))"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filter\n",
"\n",
"The next core util from the standard lib is `filter`. \n",
"\n",
"https://docs.python.org/3/library/functions.html#filter\n",
"\n",
"This enables us to filter out specific data with a custom function. Similar to `map`, this returns an `Iterable`. \n",
"\n",
"Let's define a custom filter."
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"def custom_filter(x):\n",
" return (x % 2) == 0 and x >= 2"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"fx = filter(custom_filter, range(0, 10))\n",
"isinstance(fx, Iterable)\n"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[2, 4, 6, 8]"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(fx)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's try to make the original filter configurable with the min value "
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"def to_filter(min_value):\n",
" def f(x):\n",
" return (x % 2) == 0 and x >= min_value\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[2, 4, 6, 8]"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"f = to_filter(2)\n",
"list(filter(f, range(-2, 10)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, we can write the function in a more \"natural\" form and create a function using partial application (as described in the previous section)."
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [],
"source": [
"def custom_filter(min_value, x):\n",
" return (x % 2) == 0 and x >= min_value"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[2, 4, 6, 8]"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"f = functools.partial(custom_filter, 2)\n",
"\n",
"list(filter(f, XS))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Defining Multiple Filters\n"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<filter object at 0x10759c048>\n"
]
},
{
"data": {
"text/plain": [
"[4, 6, 8]"
]
},
"execution_count": 53,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def f1(x):\n",
" return (x % 2) == 0\n",
"\n",
"def example():\n",
" f = functools.partial(custom_filter, 4)\n",
" \n",
" # Steps in the pipeline\n",
" s1 = filter(f, range(1, 10))\n",
" s2 = filter(f1, s1)\n",
" return s1\n",
"\n",
"print(example())\n",
"list(example())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively we can define a new custom compose filter function so that there's a single filter applied."
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[4, 6, 8]"
]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def compose_two_filters(f, g):\n",
" def func(x):\n",
" if g(x):\n",
" return f(x)\n",
" return False\n",
" return func\n",
"\n",
"def example():\n",
" filter_min_4 = functools.partial(custom_filter, 4)\n",
" f = compose_two_filters(f1, filter_min_4)\n",
" return filter(f, XS)\n",
"\n",
"list(example())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As an exercise, it's useful to define a compose filter function that takes `N` filter functions. "
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [],
"source": [
"def compose_filter(funcs):\n",
" def compose_two_filters(f, g):\n",
" def func(x):\n",
" if g(x):\n",
" return f(x)\n",
" return False\n",
" return func\n",
" return functools.reduce(compose_two_filters, funcs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### CheckPoint\n",
"\n",
"- We've added the `map` and `filter` core stdlib builtin functions to our toolbox.\n",
"- With `map` sometimes it's necessary to sort out impedence mismatches and write small translation funcs.\n",
"- Leveraging the iterable nature of `map` and `filter` we can compose computation. \n",
"\n",
"Now that we've got the fundamentals down and we've added `map` and `filter` to our toolbox, we can now start building data processing pipelines!\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 7. Creating Pipelines\n",
"\n",
"### Iteration 1:\n",
"\n",
"Let's construct a pipeline that takes a stream (i.e., iterable) of `User` instances and filters the users by an attribute of the `User` and computes the user with largest age. Let's also make our pipeline configurable such that the user filter can be passed as input. \n",
"\n",
"This translates to this pipeline:\n",
"\n",
"```\n",
"(iterable of users) -> (filter:by user metadata)\n",
" -> (reduce: to max user by age)\n",
" -> User\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"User(user_id=3, age=66, first_name='Steve', favorite_color='orange')"
]
},
"execution_count": 56,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"def reduce_max_age_user(acc, value):\n",
" if value.age < acc.age:\n",
" return acc\n",
" return value \n",
"\n",
"def filter_user_by_name(achar):\n",
" def func(user):\n",
" return user.first_name.startswith(achar)\n",
" return func\n",
"\n",
"\n",
"def to_pipeline(users_it, user_filter):\n",
" # Simple pipeline will only one step. \n",
" # Note, this doesn't store the entire user's list in memory.\n",
" s1 = filter(user_filter, to_users())\n",
" result = functools.reduce(reduce_max_age_user, s1)\n",
" return result\n",
" \n",
"\n",
"filter_s_users = filter_user_by_name('S') \n",
"to_pipeline(to_users(), filter_s_users)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. We've leveraged several tools in our toolbox and have a pretty useful data pipeline to process the stream of `User`s. Let's expand on this in next iteration."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Iteration 2:\n",
"\n",
"Let's expand the first pipeline to return the Top `N` User sorted by age. \n",
"\n",
"There's several mechanisms to get the top `N` values. We'll use a `max heap` to do this by leveraging `heapq` in the Python standard lib. To get the `heap` sorting to work as expected, we'll have to add a few transformations to `tuples`. This is very similar to the sorting mechanism used in the closest points to the origin problem demonstrated in a previous section.\n",
"\n",
"The new pipeline will approximately be translated to this pipeline:\n",
"\n",
"```\n",
"(iterable of users) -> (filter:by user metadata)\n",
" -> (map: (age, user)) \n",
" -> (reduce: to N users by age)\n",
" -> list[(int, User)] \n",
" -> map( (age, user) -> user) \n",
" -> list[User]\n",
"```\n",
"\n",
"Note that due to the use of the `heap` we've got to some extra `map` calls to transform the data to work with the heap. Similarly to 'untuple' the data using a `map` call to get the list of `User` instances. \n",
"\n",
"Also note the \"reduce\" step is not calling `functools.reduce` directly, but providing a similar interface in the pipeline. \n",
"\n",
"First let's implement the a mechanism to get the max N users. This is our \"reduce\". The \"reduce\" will take an iterable and return a list of top N `User`s."
]
},
{
"cell_type": "code",
"execution_count": 57,
"metadata": {},
"outputs": [],
"source": [
"import heapq as H\n",
"\n",
"# https://docs.python.org/3.7/library/heapq.html\n",
"\n",
"def compute_max_n(n):\n",
" def f(it):\n",
" h = []\n",
" H.heapify(h)\n",
" for item in it:\n",
" if len(h) < n:\n",
" H.heappush(h, item)\n",
" else:\n",
" # this will keep the heap to size `n`\n",
" # Therefore keeping the space complexity to approximately O(1)\n",
" _ = H.heappushpop(h, item)\n",
" return H.nlargest(n, h)\n",
" return f"
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [],
"source": [
"def user_to_tuple(u):\n",
" # for the heap\n",
" return u.age, u\n",
"\n",
"def from_tuple(x):\n",
" return x[1]\n",
"\n",
"def to_pipeline(users_it, user_filter, max_users):\n",
" \n",
" reducer = compute_max_n(max_users)\n",
" \n",
" # Note, the map and filter are Iterables\n",
" p1 = filter(user_filter, users_it)\n",
" p2 = map(user_to_tuple, p1)\n",
" \n",
" # this is effectively the reduce step,\n",
" # however, it results a list\n",
" p3 = reducer(p2)\n",
" \n",
" # transform back to list[User]\n",
" p4 = map(from_tuple, p3)\n",
" return p4\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 59,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[User(user_id=3, age=66, first_name='Steve', favorite_color='orange'),\n",
" User(user_id=8, age=55, first_name='Stephen', favorite_color='blue'),\n",
" User(user_id=5, age=4, first_name='Sam', favorite_color='blue'),\n",
" User(user_id=9, age=2, first_name='Sean', favorite_color='blue')]"
]
},
"execution_count": 59,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(to_pipeline(to_users(), filter_s_users, 10))"
]
},
{
"cell_type": "code",
"execution_count": 60,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[User(user_id=4, age=41, first_name='Rebecca', favorite_color='blue'),\n",
" User(user_id=6, age=32, first_name='Richard', favorite_color='blue'),\n",
" User(user_id=1, age=13, first_name='Ralph', favorite_color='blue')]"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(to_pipeline(to_users(), filter_user_by_name('R'), 3))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Nice. Our pipeline appears to be working as expected and is quite configurable. Want to filter by all Users who's age is `>= 21` and name length is `< 10`? This only requires writing a custom function to do the user filtering. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 8. Conclusion and Summary\n",
"\n",
"We've built up a toolbox of useful techinques using Functional Techinques using Python.\n",
"\n",
"Recap:\n",
"\n",
"- We've gone over a few different models of defining a function in Python\n",
"- We've added closures to our toolbox of functional programming techniques\n",
"- We've added `functools.partial` from the Python standard library to provide us build up computation\n",
"- Constructed `compose` function that enables us to compose `N` different functions\n",
"- We've added the `map` and `filter` core stdlib builtin functions to our toolbox.\n",
"- With `map` sometimes it's necessary to sort out impedence mismatches and write small translation funcs.\n",
"- Leveraging the iterable nature of `map` and `filter` we can compose computation. \n",
"- Created extensible pipelines leveraging the `map`, `filter` and `reduce`\n",
"\n",
"In Part 2, we'll go through an example of building a client to REST API using Functional style design. \n",
"\n",
"Hopefully you've picked up some tricks or solidified current your knowledge of Functional Techniques using Python.\n",
"\n",
"Best to you and your Python-ing!\n",
"\n",
"### Further Reading and Resources\n",
"\n",
"- https://docs.python.org/3/library/functions.html \n",
"- https://docs.python.org/3.7/howto/functional.html\n",
"- https://toolz.readthedocs.io/en/latest/index.html"
]
},
{
"cell_type": "code",
"execution_count": 61,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Today is 2019-02-15 03:20:20.292023\n"
]
}
],
"source": [
"print(\"Today is {}\".format(datetime.datetime.now()))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}