Annotating Python code with types (for mypy-checking)

TypePatcher.md

I just finished adding type annotations to a 10-year old python codebase (for type-checking with the magnificent mypy).

I had to do stuff like this:

find . -type f -iname \*A_mapper.py | while read ANS ; do
    cat "$ANS" | \
        sed 's/def OnBasic(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnBasic(\1: str, \2: AsnBasicNode, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnChoice(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnChoice(\1: str, \2: AsnChoice, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnSequence(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnSequence(\1: str, \2: AsnSequenceOrSet, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnSet(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnSet(\1: str, \2: AsnSequenceOrSet, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnSequenceOf(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnSequenceOf(\1: str, \2: AsnSequenceOrSetOf, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnSetOf(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnSetOf(\1: str, \2: AsnSequenceOrSetOf, \3: AST_Leaftypes) -> None:/' | \
        sed 's/def OnEnumerated(\([^,:]*\), \([^,:]*\), \([^,:]*\)):$/def OnEnumerated(\1: str, \2: AsnEnumerated, \3: AST_Leaftypes) -> None:/' \
        > /tmp/foo && mv /tmp/foo "$ANS"
done

Fun! (for various definitions of "fun"). Felt a bit like writing Lisp macros :-)

On a more serious note - worth it. REALLY worth it. Found many bugs after I annotated with types...

I particularly enjoyed using Generics:

TSrc = TypeVar('TSrc')
TDest = TypeVar('TDest')
class RecursiveMapperGeneric(Generic[TSrc, TDest]):
    def MapInteger(self, srcVHDL: TSrc, ...
...
class FromVHDLToASN1SCC(RecursiveMapperGeneric[List[int], str]):  # pylint: disable=invalid-sequence-index
    def MapInteger(self, srcVHDL: List[int], ...

And taming dynamic creation while limiting the typeset was awesome:

U = TypeVar('U', int, float)
def GetRange(newModule: Module, lineNo: int, nodeWithMinAndMax: Element, valueType: Type[U]) -> Tuple[U, U]:
    ...
    rangel = valueType(mmin)
    rangeh = valueType(mmax)
    return (rangel, rangeh)

# Example use of 'tamed' creation of the type I want (in this case, 'int'):
def CreateOctetString(newModule: Module, lineNo: int, xmlOctetString: Element) -> AsnOctetString:
    return AsnOctetString(
        asnFilename=newModule._asnFilename,
        lineno=lineNo,
        range=GetRange(newModule, lineNo, xmlOctetString, int))

There are warts, of course - e.g. notice the 'pylint, shutup' comment in the first example, since pylint is not aware of typing constructs (yet) - but I must use it, of course; it's part of the "mandatory weapons" set :-) flake8, pylint, now mypy, and of course good old py.test-driven tests combined with coverage checking.

Anyway - mypy has raised Python back to being my go-to language for everything (when the decision is mine to make, that is). And I honestly think it's the best reason to migrate to Python 3. (No, the comment-driven stuff in Py2 are not as good-looking. And the way the code looks matters!)

Cheers, Jukka!

Discussion in /r/Python is here.