GSoC project page: https://summerofcode.withgoogle.com/archive/2016/projects/6309614853292032/
Hi everyone! I'm Eklavya, a Google Summer of Code student working on Zulip. This summer, I mostly worked on 2 things for Zulip:
- Making Zulip's code Python 3 compatible.
- Adding mypy type annotations to Zulip's python code.
It was a great experience and I loved working on Zulip. I got to learn a lot about python, mypy, several libraries (both standard and third-party) and many other things. I started contributing in March 2016 and from then to 20 August 2016, I have the second highest number of commits (commit graph, commit list) in Zulip!
I'm very grateful to my mentor, Tim Abbott. He has been very responsive and helpful and it was great working with him. Zulip has a vibrant community and I love being a part of it.
To understand what mypy is and what purpose it serves in Zulip, it's best to read Zulip's documentation on mypy - http://zulip.readthedocs.io/en/1.7.2/mypy.html
95% of the files we're checking are covered by mypy. See https://coveralls.io/jobs/17452042 for details.
Some of our scripts require listing files of a particular type. For example, pyflakes requires listing all python files and our linters for plain text files, markdown, CSS, JavaScript, etc. require listing files of appropriate types. We usually have to include extensionless scripts in the list (based on shebang line) as well and we need a mechanism for excluding certain files which we don't want listed for some reason. There didn't used to be a centralized way to do this listing.
For mypy and check-py3 (check-py3 is explained in the python 3 section), we needed to list all python files. So it was decided that we should write a module whose function is to list files of a particular type. I wrote that module and named it lister.py.
I wrote a script which runs mypy in python 2 mode on most of our python files. I added to the exclude list all the files which didn't pass the mypy check. I also made tools/run-mypy run in Travis CI.
Even when we don't annotate our code, mypy can still infer a lot about our code and sometimes find a few bugs. So our strategy was to first make as many files pass the mypy check (in python 2 mode) as possible and then start annotating them.
This constitutes a big part of my work. I learned a lot about python while doing this. Many people added mypy type annotations during PyCon sprints and I was one of the reviewers of pull requests sent by them.
I mostly focused on the core of Zulip, that is zerver/lib. While annotating, I also fixed a few bugs I encountered. Most of these bugs were about fixing string types. This helped us a lot when we were adding python 3 support because most of the compatibility issues were because of incorrect string handling.
- #1164 - After we had done some work towards python 3 compatibility, we thought it would make sense to run mypy in python 3 mode.
- #1172 - Unannotated files had to be edited a bit to make them pass mypy check in python 3 mode.
This mostly helped detect bugs where we were using standard library functions which were no longer supported in python 3. I also found out some bugs in typeshed itself and fixed those by sending pull requests to typeshed.
Zulip's development environment now works on python 3, with the small exceptions of tools/run-dev.py (because it uses twisted). We also have all tests running on python 3 (except the production deployment test suite).
- Strings needed to be handled correctly. See http://zulip.readthedocs.io/en/1.7.2/mypy.html#annotating-strings for more info.
- Dependencies needed to be upgraded because older versions of dependencies were not python 3 compatible.
- Other python 3 compatibility issues, like using standard library functions no longer supported by python 3.
This document explains what is py3k. py3k was not run on all python files; I fixed that. I also made it faster. This made our code syntactically python 3 compatible.
Pyflakes was able to detect some python 3 compatibility issues.
Issue #717 explains why we need this.
A big thanks to Umair Khan for upgrading django-pipeline, replacing Django templates by Jinja2 templates and moving from apns-client to PyAPNs. These changes helped add python 3 compatibility.
Most dependencies were easy to upgrade. Some of them, like python-twitter, required rewriting some parts of Zulip. We also plan to replace twisted by other libraries because twisted is not fully python 3 compatible. I replaced twisted by imaplib in email-mirror. But we're still finding it difficult to replace twisted in tools/run-dev.py .
This involved running tests in python 3 and analyzing the tracebacks. This was the most interesting part.
This was a major milestone, because apart from all tests (except production test suite) passing in python 3, people can try out the development version of Zulip in python 3.
There's a catch though. tools/run-dev.py uses twisted, so it itself runs in python 2. But if a python 3 virtualenv is active, the Django and Tornado processes are run in python 3.
We plan to eventually replace twisted by tornado or something else.