python-type-files.md

Python Types, why you should use stubs

TLDR;

Types are great, and so are stub files. However there's a major issue at the moment which means your implementation doesn't have to behave the same as what your stub file say. Don't use stub files yet.

The Background

When I write python, I insist on using types. There are 2 main reasons I do this:

• Autocomplete is fantastic when you are in the middle of writing out a component
• It forces me to use sensible yet minimal interfaces

Both of these amount to being able to write a library, then pick up where I left off 6 months later, something that is normally a common issue with dynamic languages.

The monster under the bed

While it's easy enough to do type hinting for the above points, I actually like relying upon the type checker to catch mistakes. This often results in some pretty gnarly looking code.

Lets take a look at a decorator:

def some_decorator(func=None, *, param=False):
    def inner(func):
        nonlocal param
        func.my_param = param
        return func
    if func is None:
        return inner
    return inner(func)

@some_decorator
def foo(bar: int) -> int: ...

reveal_type(foo)  # Revealed type is 'Any'
    

This decorator can be used on a function. It can be used in a few different ways:

@some_decorator
def _(): ...

@some_decorator()
def _(): ...

@some_decorator(param=None)
def _(): ...

Here is how you can do type hinting in a basic fashion:

from typing import Callable, Any

def some_decorator(
    func: Callable = None,
    *,
    param: Any = False,
) -> Callable:
    def inner(func: Callable) -> Callable:
        nonlocal param
        func.my_param = param
        return func
    if func is None:
        return inner
    return inner(func)


@some_decorator
def foo(bar: int) -> int: ...

reveal_type(foo)  # Revealed type is 'def (*Any, **Any) -> Any'
    

While this makes it pass type checking, it doesn't really add any valuable information.

Let's add some context to the types:

from typing import Callable, Any, TypeVar, Optional, Union

# TypeVar allows the input type to be passed around, essentially a Generic.
# typescript has a good guide on generics: https://www.typescriptlang.org/docs/handbook/generics.html
T_Func = TypeVar('T_Func', bound=Callable)

def some_decorator(
    func: Optional[T_Func] = None,
    *,
    param: Any = False,
) -> Union[
    T_Func,
    Callable[[T_Func], T_Func],
]:
    def inner(func: T_Func) -> T_Func:
        nonlocal param
        func.my_param = param
        return func
    if func is None:
        return inner
    return inner(func)

@some_decorator
def foo(bar: int) -> int: ...


reveal_type(foo)  # Revealed type is 'Union[def (bar: builtins.int) -> builtins.int, def (def (bar: builtins.int) -> builtins.int) -> def (bar: builtins.int) -> builtins.int]'

This works, and gives us more context when dealing with static checking. However the checks are pretty loose at the moment. The decorator can return either a function wrapping your function, or your function.

Reading the code, it's obvious that when you don't pass a function, you get a wrapper. but when you do pass a function, you'll get your function back.

Let's tell the type checker this:

from typing import Callable, Any, TypeVar, Optional, Union, overload

T_Func = TypeVar('T_Func', bound=Callable)


@overload
def some_decorator(
    func: None = None,
    *,
    param: Any = False,
) -> Callable[[T_Func], T_Func]: ...

@overload
def some_decorator(
    func: T_Func,
    *,
    param: Any = False,
) -> T_Func: ...

def some_decorator(
    func: Optional[T_Func] = None,
    *,
    param: Any = False,
) -> Union[
    T_Func,
    Callable[[T_Func], T_Func],
]:
    def inner(func: T_Func) -> T_Func:
        nonlocal param
        func.my_param = param
        return func
    if func is None:
        return inner
    return inner(func)

@some_decorator
def foo(bar: int) -> int: ...


reveal_type(foo)  # Revealed type is 'def (bar: builtins.int) -> builtins.int'

This is a lot of code for a fairly simple decorator, and it heavily pollutes the logic.

But now I hate types!

Sure, I get it. it's not pretty, and it makes the actual logic hard to grok.

So let's split it up:

# some_decorator.py
def some_decorator(func=None, *, param=False):
    def inner(func):
        nonlocal param
        func.my_param = param
        return func

    if func is None:
        return inner
    return inner(func)

# some_decorator.pyi
from typing import Callable, Any, TypeVar, overload

T_Func = TypeVar('T_Func', bound=Callable)

@overload
def some_decorator(
    func: T_Func,
    *,
    param: Any = False,
) -> T_Func: ...

@overload
def some_decorator(
    func: None = None,
    *,
    param: Any = False,
) -> Callable[[T_Func], T_Func]: ...

# foo.py
## Must be a different file for the .pyi files to pickup
from .some_decorator import some_decorator

@some_decorator
def foo(bar: int) -> int: ...

reveal_type(foo)  # Revealed type is 'def (bar: builtins.int) -> builtins.int'

Note that now we have a .pyi file, we don't need to use type hints in the main .py file.

There's now a clean divide between types and logic.

The pitfall

There's currently a bug (even though it's classed as a feature) that makes it so .pyi files are not cross-checked against their counterpart .py file. python/mypy#5028

This means that the function defined in .py doesn't have to align with .pyi.

This is a massive hole, and means you can't actually do type checking using just stub files. While you can promise an implementation will behave, it doesn't mean the implementation will behave.