Skip to content

Instantly share code, notes, and snippets.

@br3ndonland
Last active June 24, 2023 20:55
Show Gist options
  • Save br3ndonland/987bdc6263217895d4bf03d0a5ff114c to your computer and use it in GitHub Desktop.
Save br3ndonland/987bdc6263217895d4bf03d0a5ff114c to your computer and use it in GitHub Desktop.
Python type annotations

Type annotations

Adventures in the land of Python type annotations

Table of Contents

Introduction

Static typing is becoming increasingly popular in dynamically typed languages. PEP 484 added type annotations to Python. For some background on static typing, check out episode 151 of the Talk Python to Me podcast with Lukasz Langa, author of PEP 484 (and the Black autoformatter).

I have found type annotations to be extremely useful. I was skeptical at first, because dynamic typing is supposed to be a feature of Python, not a bug. However, I have found that type-annotating my Python code helps me understand my code better, by encouraging me to think about the inputs and outputs of methods more thoroughly.

Type annotations are also useful for data validation tools like Pydantic, which is used by FastAPI to generate an OpenAPI-compliant API schema.

The most widely used tool to check static types is Mypy. Other options include Pyright and the Pylance VSCode extension from Microsoft, and Pyre from Facebook.

Adding type annotations to source code

Basics

Real Python has provided a helpful tutorial on Python type checking, so I would refer you there to get started. Here is some basic syntax:

from typing import Dict


def greeting(name: str) -> str:
    return f"Hello {name}!"

hello_dict: Dict[str, str] = {"Hello": "World"}

The docs for the typing module are less clear than Real Python's resources, though it can be useful to skim through them. Note that, after PEP 585 and Python 3.9, developers will no longer have to import types from the typing module.

As mentioned above, type-annotating Python code encourages developers to think about data flow through their methods and applications, reducing type errors. Developers may also understand dependencies more thoroughly, because classes from other modules are frequently used as type annotations.

For example, say you want to create a filepath from an input string:

from pathlib import Path


def return_a_path_from_a_string(input_string: str) -> Path:
    """Construct a Path object from an input string."""
    return Path(input_string)

The process of writing this code encourages the developer to delve into the pathlib docs to learn about how paths work and how to use the Path class.

Challenges

Class confusion

It can be confusing and challenging to type-annotatate class methods and decorators.

Class instance methods accept self as the first argument. Class methods (decorated with @classmethod) accept cls as the first argument. Mypy doesn't understand self or cls, so the developer has to help Mypy by providing user-defined generic types with TypeVar().

See br3ndonland/algorithms decorators_real_python.py for an example. Here's an abbreviated version:

from typing import Tuple, Type, TypeVar

T = TypeVar("T", bound="MyClass")


class MyClass:
    def method(self: T) -> Tuple[str, T]:
        """Class instance method
        ---
        - Class instance methods accept `self` as an argument.
        - Can access the class itself (the original class object definition).
        """
        return "instance method called", self

    @classmethod
    def class_method(cls: Type[T]) -> Tuple[str, Type[T]]:
        """Class method
        ---
        - `@classmethod`s accept `cls`, but not `self`.
        - Can modify class state (applies to other instances of the class)
        - Cannot modify object/instance state (the original class object definition).
        """
        return "class method called", cls

See the Mypy docs on declaring new generic classes for more examples.

I'm hoping it's eventually unnecessary to explicitly declare TypeVar(). The type checker is supposed to help me, not the other way around.

I would hope to eventually see something like this:

# No need for typing module imports after Python 3.9
# from typing import Tuple, Type, TypeVar

# No need to explicitly declare TypeVar
# T = TypeVar("T", bound="MyClass")


# Intuitive typing of instance and class methods
class MyClass:
    def method(self) -> tuple[str, self]:
        """Class instance method
        ---
        - Class instance methods accept `self` as an argument.
        - Can access the class itself (the original class object definition).
        """
        return "instance method called", self

    @classmethod
    def class_method(cls) -> tuple[str, cls]:
        """Class method
        ---
        - `@classmethod`s accept `cls`, but not `self`.
        - Can modify class state (applies to other instances of the class)
        - Cannot modify object/instance state (the original class object definition).
        """
        return "class method called", cls

The most useful escape hatches to avoid dealing with this:

Update on type-annotating classes: Python 3.11 adds a Self type. See PEP 673 and what's new in Python 3.11.

Distributing type information

py.typed

PEP 561 introduced distribution methods for type information. As explained in the mypy docs, an empty file named py.typed can be supplied with a package to indicate that the package supplies type information. Distribution of the py.typed file is now supported by Poetry (python-poetry/poetry#1338).

Stubs

TODO: intro to stubs

TODO: provide details on each of these

Checking type annotations

Mypy

Intro

  • Mypy can be challenging to use, but it is the most helpful Python static type checking tool that I have tried.
  • Mypy does not check dynamic types, so you will need to add some type annotations to get started.
  • Configuration is done with an old-style mypy.ini file, though eventually pyproject.toml will be supported.
  • Here's a mypy.ini with some sensible defaults:
    [mypy]
    disallow_untyped_defs = True
    files = **/*.py
  • Note that, in order for Mypy to pick up the configuration file, it should be run from the same directory in which the configuration file is located.
  • The VSCode Python extension has built-in support for Mypy. It doesn't always read configuration files, so some settings may need to be added to the VSCode settings.json.

Specifying files to check

Mypy's command-line file checking behavior is confusing. The docs say:

Note that directories are checked recursively.

This doesn't appear to be entirely correct, for me at least. Running mypy . doesn't recursively check all sub-directories. A glob pattern is needed, like **/*.py. Here's an example from br3ndonland/template-python@cbdbab4:

~/dev/template-python
.venv ❯ mypy . -v

LOG:  Mypy Version:           0.782
LOG:  Config File:            mypy.ini
LOG:  Source Paths:
      ./templatepython/__init__.py
      ./templatepython/examples/__init__.py
      ./templatepython/examples/fibonacci.py
      ./templatepython/examples/fizzbuzz.py
      ./templatepython/examples/palindrome.py
...

~/dev/template-python
.venv ❯ mypy **/*.py -v

LOG:  Mypy Version:           0.782
LOG:  Config File:            mypy.ini
LOG:  Source Paths:
      templatepython/__init__.py
      templatepython/examples/__init__.py
      templatepython/examples/fibonacci.py
      templatepython/examples/fizzbuzz.py
      templatepython/examples/palindrome.py
      tests/examples/test_fibonacci.py
      tests/examples/test_fizzbuzz.py
      tests/examples/test_palindrome.py
      tests/test_version.py
...

The config file used above was simply:

[mypy]
disallow_untyped_defs = True

Pyright and Pylance

I would not recommend Pyright or Pylance.

  • Doesn't warn about untyped defs, even with strict mode on.
  • Doesn't appear to have compatibility with Mypy stubs.
  • Pyright configuration has to be in a JSON file, which is unconventional for Python tooling. User requests for pyproject.toml support have been declined. The lack of pyproject.toml support is particularly ironic considering that this tool comes from Microsoft. One of the people who originally introduced pyproject.toml to Python, via PEP 518 (Brett Cannon), also works at Microsoft. If you're not familiar with pyproject.toml, Brett Cannon wrote a helpful blog post about it.
  • VSCode support requires the proprietary Pylance extension.

Pyre and Pysa

I tried out Pyre, which also includes the security analysis tool Pysa, after seeing Facebook's blog post announcing Pysa. So far, I don't think Pyre is mature enough to be useful.

  • Pyre requires installation of a separate file watcher called Watchman. In comparison, Mypy does not require a separate file watcher. As a side note, it would be advisable to find a gender-neutral name for this tool.
  • Adding configuration files with pyre init adds a number of hard-coded absolute file paths to the .pyre-configuration file. It's unclear how to generalize the config file for other machines.
  • Running pyre infer generates a small number of type stubs in ./.pyre/types. However, it misses most of the functions, and doesn't actually add type annotations like monkeytype apply does.
  • Pysa throws tons of errors unrelated to the project (may be some sort of default definitions that are included):
    ~/path/to/repo
    .venv ❯ pyre analyze
    ƛ Invalid model for `django.db.models.manager.Manager.get` defined in
      `/path/to/repo/.venv/lib/pyre_check/taint/django_sources_sinks.pysa:139`:
      Modeled entity is not part of the environment!
    
  • If you see this issue, consider adding some of the following settings to a .pyre_configuration file:
    {
      "binary": "/usr/local/bin/pyre.bin",
      "source_directories": ["."],
      "taint_models_path": ["pyre-check/stubs/taint"],
      "search_path": "pyre-check/stubs",
      "typeshed": "pyre_check/typeshed"
    }

Summary

  • Type annotations are useful. They help developers understand their code, and they help reduce type errors.
  • Type checking tools help verify type annotations, but can be difficult to work with. Mypy is the most useful, but requires some configuration.
@br3ndonland
Copy link
Author

br3ndonland commented Oct 22, 2020

Hi, i get same error as you by use Pyre,Modeled entity is not part of the environment!,I don't know why this mistake happened.
Have you solved the problem?

Not yet, but it might have to do with the file path passed in. See facebook/pyre-check#306. The next release might have a fix for this issue.

Overall, I don't think Pyre is mature or dependable enough for regular use yet. For type checking, I'm happy with Mypy for now. Mypy is difficult to configure and use, but it works much better than Pyright or Pysa. For security scanning, I'm relying on CodeQL now.

@SPuerBRead
Copy link

Hi, i get same error as you by use Pyre,Modeled entity is not part of the environment!,I don't know why this mistake happened.
Have you solved the problem?

Not yet, but it might have to do with the file path passed in. See facebook/pyre-check#306. The next release might have a fix for this issue.

Overall, I don't think Pyre is mature or dependable enough for regular use yet. For type checking, I'm happy with Mypy for now. Mypy is difficult to configure and use, but it works much better than Pyright or Pysa. For security scanning, I'm relying on CodeQL now.

Thanks! I found the way solved the problem.
I use pyre for security scan, and i test python code by codeql, result is not very satisfactory.

this is my way to solved the problem, i found in pyre source code has django directory pyre-check/stubs/django and pysa_tutorial code can run well, so i test change my project .pyre_configuration like project in pysa_tutorial such as this

  "binary": "/usr/local/bin/pyre.bin",
  "source_directories": [
    "."
  ],
  "taint_models_path": ["pyre-check/stubs/taint"],
  "search_path": "pyre-check/stubs",
  "typeshed": "pyre_check/typeshed"
}

it works and no error, maybe pyre-check/stubs/django and rest_framework code for default scan rules is necessary, try set taint_models_path and search_path to source code stubs/taint and stubs path

Thank you for your reply, my english not good,hope to describe the solution is no problem

@br3ndonland
Copy link
Author

Thanks for sharing your solution @SPuerBRead! I added the example configuration file to the Gist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment