Adding Python 3 compatibility and mypy annotations to Zulip
Hi everyone! I'm Eklavya, a Google Summer of Code student working on Zulip. This summer, I mostly worked on 2 things for Zulip:
- Making Zulip's code Python 3 compatible.
- Adding mypy type annotations to Zulip's python code.
It was a great experience and I loved working on Zulip. I got to learn a lot about python, mypy, several libraries (both standard and third-party) and many other things. I started contributing in March 2016 and from then to 20 August 2016, I have the second highest number of commits (commit graph, commit list) in Zulip!
I'm very grateful to my mentor, Tim Abbott. He has been very responsive and helpful and it was great working with him. Zulip has a vibrant community and I love being a part of it.
Mypy type annotations
To understand what mypy is and what purpose it serves in Zulip, it's best to read Zulip's documentation on mypy - http://zulip.readthedocs.io/en/latest/mypy.html
95% of the files we're checking are covered by mypy. See https://coveralls.io/jobs/17452042 for details.
For mypy and check-py3 (check-py3 is explained in the python 3 section), we needed to list all python files. So it was decided that we should write a module whose function is to list files of a particular type. I wrote that module and named it
I wrote a script which runs mypy in python 2 mode on most of our python files. I added to the exclude list all the files which didn't pass the mypy check. I also made
tools/run-mypy run in Travis CI.
Shrink mypy exclude list
Even when we don't annotate our code, mypy can still infer a lot about our code and sometimes find a few bugs. So our strategy was to first make as many files pass the mypy check (in python 2 mode) as possible and then start annotating them.
This constitutes a big part of my work. I learned a lot about python while doing this. Many people added mypy type annotations during PyCon sprints and I was one of the reviewers of pull requests sent by them.
I mostly focused on the core of Zulip, that is
zerver/lib. While annotating, I also fixed a few bugs I encountered. Sometimes mypy also revealed some bugs in Zulip. Most of these bugs were about fixing string types. This helped us a lot when we were adding python 3 support because most of the compatibility issues were because of incorrect string handling.
Run mypy in python 3 mode
- #1164 - After we had done some work towards python 3 compatibility, we thought it would make sense to run mypy in python 3 mode.
- #1172 - Unannotated files had to be edited a bit to make them pass mypy check in python 3 mode.
This mostly helped detect bugs where we were using standard library functions which were no longer supported in python 3. I also found out some bugs in typeshed itself and fixed those by sending pull requests to typeshed.
Run mypy on extensionless python scripts
Add Python 3 compatibility
Zulip's development environment now works on python 3, with the small exceptions of
tools/run-dev.py (because it uses twisted). We also have all tests running on python 3 (except the production deployment test suite).
- Strings needed to be handled correctly. See http://zulip.readthedocs.io/en/latest/mypy.html#annotating-strings for more info.
- Dependencies needed to be upgraded because older versions of dependencies were not python 3 compatible.
- Other python 3 compatibility issues, like using standard library functions no longer supported by python 3.
This document explains what is py3k. py3k was not run on all python files; I fixed that. I also made it faster. This made our code syntactically python 3 compatible.
Make tools/lint-all runnable on python 3
Pyflakes was able to detect some python 3 compatibility issues.
Use a virtualenv in production
Issue #717 explains why we need this.
Split requirements based on python version
A big thanks to Umair Khan for upgrading django-pipeline, replacing Django templates by Jinja2 templates and moving from apns-client to PyAPNs. These changes helped add python 3 compatibility.
Most dependencies were easy to upgrade. Some of them, like python-twitter, required rewriting some parts of Zulip. We also plan to replace twisted by other libraries because twisted is not fully python 3 compatible. I replaced twisted by imaplib in email-mirror. But we're still finding it difficult to replace twisted in
Make frontend and backend tests pass in python 3 mode
This involved running tests in python 3 and analyzing the tracebacks. This was the most interesting part.
tools/run-dev.py run in python 3 mode
This was a major milestone, because apart from all tests (except production test suite) passing in python 3, people can try out the development version of Zulip in python 3.
There's a catch though.
tools/run-dev.py uses twisted, so it itself runs in python 2. But if a python 3 virtualenv is active, the Django and Tornado processes are run in python 3.
We plan to eventually replace twisted by tornado or something else.