br3ndonland/type-annotations.md

## type-annotations.md

      
    Raw
  

              type-annotations.md
            
          
    Type annotations

Adventures in the land of Python type annotations
Table of Contents 


Introduction
Adding type annotations to source code

Basics
Challenges


Distributing type information

py.typed
Stubs


Checking type annotations

Mypy
Pyright and Pylance
Pyre and Pysa


Summary

Introduction

Static typing is becoming increasingly popular in dynamically typed languages. PEP 484 added type annotations to Python. For some background on static typing, check out episode 151 of the Talk Python to Me podcast with Lukasz Langa, author of PEP 484 (and the Black autoformatter).
I have found type annotations to be extremely useful. I was skeptical at first, because dynamic typing is supposed to be a feature of Python, not a bug. However, I have found that type-annotating my Python code helps me understand my code better, by encouraging me to think about the inputs and outputs of methods more thoroughly.
Type annotations are also useful for data validation tools like Pydantic, which is used by FastAPI to generate an OpenAPI-compliant API schema.
The most widely used tool to check static types is Mypy. Other options include Pyright and the Pylance VSCode extension from Microsoft, and Pyre from Facebook.
Adding type annotations to source code

Basics

Real Python has provided a helpful tutorial on Python type checking, so I would refer you there to get started. Here is some basic syntax:
from typing import Dict


def greeting(name: str) -> str:
    return f"Hello {name}!"

hello_dict: Dict[str, str] = {"Hello": "World"}
The docs for the typing module are less clear than Real Python's resources, though it can be useful to skim through them. Note that, after PEP 585 and Python 3.9, developers will no longer have to import types from the typing module.
As mentioned above, type-annotating Python code encourages developers to think about data flow through their methods and applications, reducing type errors. Developers may also understand dependencies more thoroughly, because classes from other modules are frequently used as type annotations.
For example, say you want to create a filepath from an input string:
from pathlib import Path


def return_a_path_from_a_string(input_string: str) -> Path:
    """Construct a Path object from an input string."""
    return Path(input_string)
The process of writing this code encourages the developer to delve into the pathlib docs to learn about how paths work and how to use the Path class.
Challenges

Class confusion

It can be confusing and challenging to type-annotatate class methods and decorators.
Class instance methods accept self as the first argument. Class methods (decorated with @classmethod) accept cls as the first argument. Mypy doesn't understand self or cls, so the developer has to help Mypy by providing user-defined generic types with TypeVar().
See br3ndonland/algorithms decorators_real_python.py for an example. Here's an abbreviated version:
from typing import Tuple, Type, TypeVar

T = TypeVar("T", bound="MyClass")


class MyClass:
    def method(self: T) -> Tuple[str, T]:
        """Class instance method
        ---
        - Class instance methods accept `self` as an argument.
        - Can access the class itself (the original class object definition).
        """
        return "instance method called", self

    @classmethod
    def class_method(cls: Type[T]) -> Tuple[str, Type[T]]:
        """Class method
        ---
        - `@classmethod`s accept `cls`, but not `self`.
        - Can modify class state (applies to other instances of the class)
        - Cannot modify object/instance state (the original class object definition).
        """
        return "class method called", cls
See the Mypy docs on declaring new generic classes for more examples.
I'm hoping it's eventually unnecessary to explicitly declare TypeVar(). The type checker is supposed to help me, not the other way around.
I would hope to eventually see something like this:
# No need for typing module imports after Python 3.9
# from typing import Tuple, Type, TypeVar

# No need to explicitly declare TypeVar
# T = TypeVar("T", bound="MyClass")


# Intuitive typing of instance and class methods
class MyClass:
    def method(self) -> tuple[str, self]:
        """Class instance method
        ---
        - Class instance methods accept `self` as an argument.
        - Can access the class itself (the original class object definition).
        """
        return "instance method called", self

    @classmethod
    def class_method(cls) -> tuple[str, cls]:
        """Class method
        ---
        - `@classmethod`s accept `cls`, but not `self`.
        - Can modify class state (applies to other instances of the class)
        - Cannot modify object/instance state (the original class object definition).
        """
        return "class method called", cls
The most useful escape hatches to avoid dealing with this:

Set the type to Any (dynamic typing). See the Mypy docs on implications of using Any.
Append a # type: ignore comment to the line. See the Mypy docs on common issues.

Update on type-annotating classes: Python 3.11 adds a Self type. See PEP 673 and what's new in Python 3.11.
Distributing type information

py.typed

PEP 561 introduced distribution methods for type information. As explained in the mypy docs, an empty file named py.typed can be supplied with a package to indicate that the package supplies type information. Distribution of the py.typed file is now supported by Poetry (python-poetry/poetry#1338).
Stubs

TODO: intro to stubs
TODO: provide details on each of these

MonkeyType
PyAnnotate
Pyre automatic type inference

Checking type annotations

Mypy

Intro


Mypy can be challenging to use, but it is the most helpful Python static type checking tool that I have tried.
Mypy does not check dynamic types, so you will need to add some type annotations to get started.
Configuration is done with an old-style mypy.ini file, though eventually pyproject.toml will be supported.
Here's a mypy.ini with some sensible defaults:
[mypy]
disallow_untyped_defs = True
files = **/*.py

Note that, in order for Mypy to pick up the configuration file, it should be run from the same directory in which the configuration file is located.
The VSCode Python extension has built-in support for Mypy. It doesn't always read configuration files, so some settings may need to be added to the VSCode settings.json.

Specifying files to check

Mypy's command-line file checking behavior is confusing. The docs say:

Note that directories are checked recursively.

This doesn't appear to be entirely correct, for me at least. Running mypy . doesn't recursively check all sub-directories. A glob pattern is needed, like **/*.py. Here's an example from br3ndonland/template-python@cbdbab4:
~/dev/template-python
.venv ❯ mypy . -v

LOG:  Mypy Version:           0.782
LOG:  Config File:            mypy.ini
LOG:  Source Paths:
      ./templatepython/__init__.py
      ./templatepython/examples/__init__.py
      ./templatepython/examples/fibonacci.py
      ./templatepython/examples/fizzbuzz.py
      ./templatepython/examples/palindrome.py
...

~/dev/template-python
.venv ❯ mypy **/*.py -v

LOG:  Mypy Version:           0.782
LOG:  Config File:            mypy.ini
LOG:  Source Paths:
      templatepython/__init__.py
      templatepython/examples/__init__.py
      templatepython/examples/fibonacci.py
      templatepython/examples/fizzbuzz.py
      templatepython/examples/palindrome.py
      tests/examples/test_fibonacci.py
      tests/examples/test_fizzbuzz.py
      tests/examples/test_palindrome.py
      tests/test_version.py
...


The config file used above was simply:
[mypy]
disallow_untyped_defs = True
Pyright and Pylance

I would not recommend Pyright or Pylance.

Doesn't warn about untyped defs, even with strict mode on.
Doesn't appear to have compatibility with Mypy stubs.
Pyright configuration has to be in a JSON file, which is unconventional for Python tooling. User requests for pyproject.toml support have been declined. The lack of pyproject.toml support is particularly ironic considering that this tool comes from Microsoft. One of the people who originally introduced pyproject.toml to Python, via PEP 518 (Brett Cannon), also works at Microsoft. If you're not familiar with pyproject.toml, Brett Cannon wrote a helpful blog post about it.
VSCode support requires the proprietary Pylance extension.

Pyre and Pysa

I tried out Pyre, which also includes the security analysis tool Pysa, after seeing Facebook's blog post announcing Pysa. So far, I don't think Pyre is mature enough to be useful.

Pyre requires installation of a separate file watcher called Watchman. In comparison, Mypy does not require a separate file watcher. As a side note, it would be advisable to find a gender-neutral name for this tool.
Adding configuration files with pyre init adds a number of hard-coded absolute file paths to the .pyre-configuration file. It's unclear how to generalize the config file for other machines.
Running pyre infer generates a small number of type stubs in ./.pyre/types. However, it misses most of the functions, and doesn't actually add type annotations like monkeytype apply does.
Pysa throws tons of errors unrelated to the project (may be some sort of default definitions that are included):
~/path/to/repo
.venv ❯ pyre analyze
ƛ Invalid model for `django.db.models.manager.Manager.get` defined in
  `/path/to/repo/.venv/lib/pyre_check/taint/django_sources_sinks.pysa:139`:
  Modeled entity is not part of the environment!


If you see this issue, consider adding some of the following settings to a .pyre_configuration file:
{
  "binary": "/usr/local/bin/pyre.bin",
  "source_directories": ["."],
  "taint_models_path": ["pyre-check/stubs/taint"],
  "search_path": "pyre-check/stubs",
  "typeshed": "pyre_check/typeshed"
}


Summary


Type annotations are useful. They help developers understand their code, and they help reduce type errors.
Type checking tools help verify type annotations, but can be difficult to work with. Mypy is the most useful, but requires some configuration.