Python Package Structure Notebook

package_structure_meeting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Package Structure\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Namespaces\n",
    "\n",
    "> _A namespace is a mapping from names to objects. Most namespaces are currently implemented as Python dictionaries._\n",
    "<br> <br> https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [0, 1, 2, 3]\n",
    "y = [0, 1, 2, 3]\n",
    "z = x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "[hex(id(i)) for i in [x, y, z]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['In', 'Out', '_', '_2', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_i', '_i1', '_i2', '_i3', '_ih', '_ii', '_iii', '_oh', 'exit', 'get_ipython', 'quit', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "print(dir())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError', 'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'ModuleNotFoundError', 'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError', 'ProcessLookupError', 'RecursionError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopAsyncIteration', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__IPYTHON__', '__build_class__', '__debug__', '__doc__', '__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'breakpoint', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'display', 'divmod', 'enumerate', 'eval', 'exec', 'filter', 'float', 'format', 'frozenset', 'get_ipython', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']\n"
     ]
    }
   ],
   "source": [
    "print(dir(__builtins__))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hi\n"
     ]
    }
   ],
   "source": [
    "print('Hi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Modules\n",
    "\n",
    "> _\"A module is a file containing Python definitions and statements. The file name is the module name with the suffix `.py` appended.\"_ \n",
    "<br> <br> https://docs.python.org/3/tutorial/modules.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing farm.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile farm.py\n",
    "\"\"\" This is the farm module. \"\"\"\n",
    "\n",
    "def make_sound(animal):\n",
    "    animals = {'cow': 'moo',\n",
    "               'donkey': 'hee-haw',\n",
    "               'sheep': 'baa'}\n",
    "    \n",
    "    return f\"The {animal} makes `{animals.get(animal, '?')}`\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import farm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The cow makes `moo`'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "farm.make_sound('cow')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A module is a single namespace! Creating a Python module automatically creates an additional namespace for that module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__builtins__',\n",
       " '__cached__',\n",
       " '__doc__',\n",
       " '__file__',\n",
       " '__loader__',\n",
       " '__name__',\n",
       " '__package__',\n",
       " '__spec__',\n",
       " 'make_sound']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(farm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'farm'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "farm.__name__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/epassaro/Desktop/tardis-sn/package_structure_meeting/farm.py'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "farm.__file__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' This is the farm module. '"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "farm.__doc__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'make_sound' in dir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from farm import make_sound"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'make_sound' in dir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The donkey makes `hee-haw`'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "make_sound('donkey')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from farm import make_sound as toy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The sheep makes `baa`'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy('sheep')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['In', 'Out', '_', '_10', '_11', '_12', '_13', '_15', '_16', '_18', '_2', '_8', '_9', '__', '___', '__builtin__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', '_dh', '_i', '_i1', '_i10', '_i11', '_i12', '_i13', '_i14', '_i15', '_i16', '_i17', '_i18', '_i19', '_i2', '_i3', '_i4', '_i5', '_i6', '_i7', '_i8', '_i9', '_ih', '_ii', '_iii', '_oh', 'exit', 'farm', 'get_ipython', 'make_sound', 'quit', 'toy', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "print(dir())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"namespaces.png\" alt=\"drawing\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "When a module named `farm` is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named `farm.py` in a list of directories given by the variable `sys.path`. `sys.path` is initialized from these locations:\n",
    "\n",
    "1. The directory containing the input script (or the current directory when no file is specified).\n",
    "\n",
    "2. `PYTHONPATH` (a list of directory names, with the same syntax as the shell variable `PATH`).\n",
    "\n",
    "3. The installation-dependent default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 2. Packages\n",
    "\n",
    "> _Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A._\n",
    "<br> <br> https://docs.python.org/3/tutorial/modules.html#packages\n",
    "\n",
    "The `__init__.py` files are required to make Python treat directories containing the file as packages. In the simplest case, `__init__.py` can just be an empty file, but it can also execute initialization code for the package or set the `__all__` variable, which defines what's imported when you type:\n",
    "\n",
    "\n",
    "\n",
    "```python\n",
    "__all__ = ['make_sound', 'function_2']\n",
    "```\n",
    "\n",
    "```python\n",
    "from package import *\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir -p my_package\n",
    "mv farm.py my_package\n",
    "touch my_package/__init__.py\n",
    "echo \"from farm import make_sound\" >> my_package/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When a regular package is imported, this `__init__.py` file is implicitly executed, and the objects it defines are bound to names in the package’s namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import my_package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['__builtins__',\n",
       " '__cached__',\n",
       " '__doc__',\n",
       " '__file__',\n",
       " '__loader__',\n",
       " '__name__',\n",
       " '__package__',\n",
       " '__path__',\n",
       " '__spec__',\n",
       " 'make_sound']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(my_package)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Subpackages\n",
    "\n",
    "Let's create a subpackage for `my_package`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir -p my_package/subpackage\n",
    "touch my_package/subpackage/__init__.py\n",
    "echo \"print('Initializing subpackage.')\" >> my_package/subpackage/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relative imports works, but is not recommended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"from ..farm import make_sound\" >> my_package/subpackage/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always use absolute imports for intra-package references."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "echo \"from my_package.farm import make_sound\" >> my_package/subpackage/__init__.py\n",
    "echo \"print(make_sound('cow'))\" >> my_package/subpackage/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing `my_package.subpackage` will implicitly execute `my_package/__init__.py` and `my_package/subpackage/__init__.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing subpackage.\n",
      "The cow makes `moo`\n"
     ]
    }
   ],
   "source": [
    "import my_package.subpackage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Packaging (the hard way)\n",
    "\n",
    "A minimum project template would be:\n",
    "\n",
    "```\n",
    "my_package_project\n",
    "├── LICENSE\n",
    "├── README.md\n",
    "├── my_package\n",
    "│   └── __init__.py\n",
    "│   └── farm.py\n",
    "│   └── subpackage\n",
    "│       └── __init___.py\n",
    "├── setup.py\n",
    "└── tests\n",
    "```\n",
    "\n",
    "... where the most important thing is the `setup.py` file, the build script for `setuptools`:\n",
    "\n",
    "> `setuptools` (which includes easy_install) is a collection of enhancements to the Python `distutils` that allow you to more easily build and distribute Python distributions, especially ones that have dependencies on other packages.\n",
    "<br><br> https://packaging.python.org/key_projects/#setuptools\n",
    "\n",
    "Basically allows you to do `python setup.py install` and `python setup.py develop`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Cookiecutters (the easy way)\n",
    "\n",
    "- https://cookiecutter.readthedocs.io/en/1.7.2/\n",
    "- https://docs.astropy.org/projects/package-template/en/latest/"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "conda install -c conda-forge cookiecutter gitpython\n",
    "\n",
    "cookiecutter gh:astropy/package-template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. See also\n",
    "\n",
    "- More on namespaces: https://medium.com/better-programming/namespacing-with-python-79574d125564\n",
    "- Python Packaging Tutorial: https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html\n",
    "- (not covered) Pip and Wheels: https://pydist.com/blog/pip-install\n",
    "- (not covered) Usage of `pyproject.toml` with `poetry` or `flit`: https://www.scivision.dev/pyproject-toml-vs-setup-py/\n",
    "- (not covered) Conda vs. Pip: https://www.anaconda.com/blog/understanding-conda-and-pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "rm -r my_package"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}