Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
GSoC16 Output

Adding Python 3 compatibility and mypy annotations to Zulip

by Eklavya Sharma

Hi everyone! I'm Eklavya, a Google Summer of Code student working on Zulip. This summer, I mostly worked on 2 things for Zulip:

  • Making Zulip's code Python 3 compatible.
  • Adding mypy type annotations to Zulip's python code.

It was a great experience and I loved working on Zulip. I got to learn a lot about python, mypy, several libraries (both standard and third-party) and many other things. I started contributing in March 2016 and from then to 20 August 2016, I have the second highest number of commits (commit graph, commit list) in Zulip!

I'm very grateful to my mentor, Tim Abbott. He has been very responsive and helpful and it was great working with him. Zulip has a vibrant community and I love being a part of it.

Mypy type annotations

To understand what mypy is and what purpose it serves in Zulip, it's best to read Zulip's documentation on mypy - http://zulip.readthedocs.io/en/latest/mypy.html

95% of the files we're checking are covered by mypy. See https://coveralls.io/jobs/17452042 for details.

tools/lister.py

#569

Some of our scripts require listing files of a particular type. For example, pyflakes requires listing all python files and our linters for plain text files, markdown, CSS, JavaScript, etc. require listing files of appropriate types. We usually have to include extensionless scripts in the list (based on shebang line) as well and we need a mechanism for excluding certain files which we don't want listed for some reason. There didn't used to be a centralized way to do this listing.

For mypy and check-py3 (check-py3 is explained in the python 3 section), we needed to list all python files. So it was decided that we should write a module whose function is to list files of a particular type. I wrote that module and named it lister.py.

Add tools/run-mypy

#680

I wrote a script which runs mypy in python 2 mode on most of our python files. I added to the exclude list all the files which didn't pass the mypy check. I also made tools/run-mypy run in Travis CI.

Shrink mypy exclude list

#803, #834

Even when we don't annotate our code, mypy can still infer a lot about our code and sometimes find a few bugs. So our strategy was to first make as many files pass the mypy check (in python 2 mode) as possible and then start annotating them.

Annotate files

This constitutes a big part of my work. I learned a lot about python while doing this. Many people added mypy type annotations during PyCon sprints and I was one of the reviewers of pull requests sent by them.

I mostly focused on the core of Zulip, that is zerver/lib. While annotating, I also fixed a few bugs I encountered. Sometimes mypy also revealed some bugs in Zulip. Most of these bugs were about fixing string types. This helped us a lot when we were adding python 3 support because most of the compatibility issues were because of incorrect string handling.

Run mypy in python 3 mode

  • #1164 - After we had done some work towards python 3 compatibility, we thought it would make sense to run mypy in python 3 mode.
  • #1172 - Unannotated files had to be edited a bit to make them pass mypy check in python 3 mode.

This mostly helped detect bugs where we were using standard library functions which were no longer supported in python 3. I also found out some bugs in typeshed itself and fixed those by sending pull requests to typeshed.

Run mypy on extensionless python scripts

#1354, #1392

Add Python 3 compatibility

Zulip's development environment now works on python 3, with the small exceptions of tools/run-dev.py (because it uses twisted). We also have all tests running on python 3 (except the production deployment test suite).

Main challenges

  • Strings needed to be handled correctly. See http://zulip.readthedocs.io/en/latest/mypy.html#annotating-strings for more info.
  • Dependencies needed to be upgraded because older versions of dependencies were not python 3 compatible.
  • Other python 3 compatibility issues, like using standard library functions no longer supported by python 3.

Improved py3k

This document explains what is py3k. py3k was not run on all python files; I fixed that. I also made it faster. This made our code syntactically python 3 compatible.

Make tools/lint-all runnable on python 3

#833

Pyflakes was able to detect some python 3 compatibility issues.

Use a virtualenv in production

#1100

Issue #717 explains why we need this.

Split requirements based on python version

#1163

Upgrade dependencies

A big thanks to Umair Khan for upgrading django-pipeline, replacing Django templates by Jinja2 templates and moving from apns-client to PyAPNs. These changes helped add python 3 compatibility.

Most dependencies were easy to upgrade. Some of them, like python-twitter, required rewriting some parts of Zulip. We also plan to replace twisted by other libraries because twisted is not fully python 3 compatible. I replaced twisted by imaplib in email-mirror. But we're still finding it difficult to replace twisted in tools/run-dev.py .

Make frontend and backend tests pass in python 3 mode

This involved running tests in python 3 and analyzing the tracebacks. This was the most interesting part.

Make tools/run-dev.py run in python 3 mode

This was a major milestone, because apart from all tests (except production test suite) passing in python 3, people can try out the development version of Zulip in python 3.

There's a catch though. tools/run-dev.py uses twisted, so it itself runs in python 2. But if a python 3 virtualenv is active, the Django and Tornado processes are run in python 3.

We plan to eventually replace twisted by tornado or something else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment