Work Product: Google Summer of Code 2021 with Zulip
I'm Suyash Vardhan Mathur, and I have been working on the project Enhancing Zulip API Documentation Framework under the GSoC'21 program at Zulip. As GSoC comes to an end, I would like to thank Tim Abbott, Steve Howell, Anders Kaseorg, my mentors Aditya Verma and Kartik Srivastava for their continuous guidance, reviews and for helping me out whenever I was stuck. I am also thankful to the entire Zulip community and Google for the amazing learning opportunity of participating in Google Summer of Code. This journey would've been impossible without the continuous support of the Zulip community, that helped me grow throughout the summer and has helped me become a better developer.
The goal of my project was to enhance the Zulip API documentation framework and work in related areas to improve Zulip. As GSoC concludes, I am elated to have 86 merged commits in zulip/zulip, having contributed in various areas like writing markdown extensions, dabbling with frontend, understanding and writing code for Zulip's event management system and backend, writing documentation, and much more.
Summary
Github profile: MSurfer20
Respositories contributed to: zulip/zulip & zulip/python-zulip-api
Pull Requests opened in zulip/zulip during GSoC
Commits made in zulip/zulip during GSoC
Pull Requests opened in zulip/python-zulip-api during GSoC
I achieved a lot of goals during the summer, such as:
- Removed the redundant templates for API docs pages: Despite having its REST API documented using OpenAPI standard data, Zulip's actual docs were generated through different Markdown templates for each endpoint. These templates had different information hardcoded, and the testing and example generating systems too were dependent on these Markdown files. As a part of this project, the framework was modified to generate the docs directly from OpenAPI data and delete these 60 redundant templates. At each migration step, the diff of HTML pages was checked to ensure the migration happened properly. This also reduced the effort of adding a Markdown template when documenting a new endpoint. The culmination of this work happened in #18573.
- Integrated OpenAPI Generator: Wrote code to integrate OpenAPI generator into Github Actions and auto-push the bindings for 137 languages in the to-be-created zulip/zulip-openapi repo. The Github Actions is configured to make commits to the repo only when changes happen in the auto-generated docs. The PR related to this is yet to be merged.
- Worked on Zulip's backend & API: I worked on some backend issues, the highlight of which was writing the backend for the feature to mark messages as unread in #19307. This work involved writing code in Zulip's real-time events system, and the event was then applied on the client-side to sync the unread messages. I also worked on adding support for an API endpoint to fetch user-status for a given user ID.
- Increased endpoint coverage by docs: Added the OpenAPI data and python examples for various high-priority endpoints which increased the endpoint coverage by Zulip's API docs.
- Improved Zulip's frontend: I also managed to do some work in the direction of improving the frontend and UI of API docs in #19595 and #19175. This work involved changing the UI of
get-events
page to make the individual events collapsible, which made the documentation page much more organized and navigable. Also, I changed the title of responses to be auto-generated using thecode
of the response, and to have the response codes visually represented in the docs. - Along with the above-mentioned work, I also fixed various bugs I discovered along the way as well as made different enhancements to Zulip.
During the Application period, I was able to fix various bugs in Zulip's OpenAPI data and its framework, added the feature of parsing and rendering the data type of each variable in the docs, and made other enhancements to Zulip.
Pull Requests merged during Coding Period
- Make a common template for /api pages: This was the first PR that moved towards the goal of deleting the 60 redundant templates that existed for Zulip's API Documentation pages. The first half of this PR involved adjusting code to auto-generate the title of the page as well as the HTML title from a newly added OpenAPI parameter using a Markdown extension, which was merged first. This also involved some bug fixes which were discovered while working on the task. After this PR grew unwieldy due to the mammoth nature of the task, I split the work in this direction into smaller and more atomic PRs.
- minor: Fix typo in summary.: This PR fixed a minor typo introduced during the merging of #18573.
- apidocs: Fix description of topic parameter in update_message.: This PR involved fixing incorrect OpenAPI parameters present in stream data.
- apidocs: Fix invalid API page bug.: This PR fixed a bug in HTML title generation for invalid pages that was resulting in 500 Internal Server Error whenever a doc page was non-existent.
- curl_examples: Fix testing with new apidocs template.: This PR restructured the curl testing system for endpoints, which was dependent on individual Markdown files, and those were to be deleted as a goal of this project.
- apidocs: Migrate response descriptions into OpenAPI data: This PR added a Markdown extension to automatically render response descriptions, as well as created an OpenAPI component for frequently used descriptions.
- apidocs: Automatically add javascript example tabs: This PR modified the Javascript example code such that standard code in the template could be used whether the example existed or not for an endpoint.
- openapi: Auto-render heading with return values: This PR involved removing the hardcoded
Return Values
heading and made it auto-generated such that the same code could be used whether Return values existed or not for the endpoint. - apidocs: Migrate admin_config out of templates.: This PR added an OpenAPI parameter to render python's admin configuration automatically, and adjusted the code to use it as required. This admin configuration was needed for the endpoints which were accessible exclusively to the admins, and hence the need for the parameter.
- openapi: Migrate curl examples' configs into OpenAPI data.: This PR was a better implementation of the discarded #18868. This PR changed the curl example generation logic and allowed all possible examples to be represented in OpenAPI data along with descriptions, and be rendered using them.
- openAPI: Auto render parameter and response descriptions.: This PR involved adding markdown extensions to automatically render parameter and response descriptions from OpenAPI data.
- apidocs: Move stray descriptions to OpenAPI.: This PR involved migrating dozen of stray API data into appropriate OpenAPI parameters. This was the final prep PR before removing the redundant Markdown files for API docs pages.
- Make a common template for /api pages: After all the prep PRs, this was the final PR, which involved changing the code to auto-render documentation pages from OpenAPI data and deleted 59 Markdown templates. The new page generation logic uses a template if it exists, which allowed to cover the endpoints whose docs don't follow the standard docs template. In the other case when the template for an endpoint doesn't exist, its API page is auto-generated using the common template of API docs directly from the OpenAPI data.
- apidocs: Remove template of get-events.: This PR involved fixing the Python bindings for
get-events
page, which allowed its template to be deleted as well. - Fix certain responses in API docs: This was a bug-fix where certain responses weren't appropriately migrated into OpenAPI data and as a result weren't being rendered correctly.
- docs: Document the new APIDocs template system.: This PR updated the Zulip's Developer Documentation with the new OpenAPI system that was merged, as well as updated the instructions on how to write OpenAPI documentation for Zulip's endpoints.
- openapi: Make endpoint operationId dash-separated.: As a result of #18573, both dash-separated and underscore_separated endpoint names could be used in API docs' URL. This PR migrated the operation IDs in OpenAPI data to dash-separated to fix the URL issue, only allowing dash-separated endpoint names in the URL.
- openapi: Render all responses of an operation.: This PR changed the response rendering logic to render all possible responses for all response codes of an endpoint, while previously, one would need to write each response code to render the specific response.
- openapi: Make base class for Markdown Preprocessors.: This PR involved deduplicating the code for Markdown extensions by creating a base class from which each extension could inherit and extend the functionality of a basic markdown extension.
- openapi: Document delete-topic endpoint.: This PR added Python example and OpenAPI data to document the
delete-topic
endpoint. - openapi: Document get-subscribers endpoint.: This PR added OpenAPI data to document the
get-subscribers
endpoint. - openapi: Document update-self-status documentation.: This PR added Python example and OpenAPI data to document the
update-self-status
endpoint. - apidocs: Remove automatically added : at end of response.: This PR removed the automatically added
:
at the end of response descriptions and adds:
wherever needed in the OpenAPI data.
Unmerged Pull Requests opened during Coding Period
- apidocs: Display response codes in example responses.: This PR added support for auto-generated heading for responses as per their
code
value and visually representing the response codes in the docs. - openapi: Fix function names in python examples.: This PR standardizes the python-examples' function names as well as
python-zulip-api
's binding names with the Operation IDs defined in OpenAPI data, but it first requires the merging of #712. - Mark as unread backend: This PR adds the backend functionality for addition of the feature to mark messages as unread. The work in this PR is continued by @showell in #18743, which adds the frontend and client-side component for this feature on top of this PR's work.
- openapi: Document send-invites and get-invites endpoints.: This PR added Python example and OpenAPI data to document the
send-invites
andget-invites
endpoints. - openapi: Document remove-attachment endpoint.: This PR added Python example and OpenAPI data to document the
remove-attachment
endpoint. - Document add-apns-device token and remove-apns-device-token endpoints.: This PR added Python example and OpenAPI data to document the
add-apns-device token
andremove-apns-device-token
endpoints. - api: Add User status endpoint.: This PR adds the API endpoint to fetch a user's status given the user ID.
- Test openapigenerator: This PR integrates OpenAPI generator with Github Actions to automatically generate documentation using OpenAPI generator and auto-push(if the documentation changes) into the to-be-created
zulip-openapi
repo. - zulip: Standardize the names of bindings with OperationIDs.: This PR standardizes the name of the python bindings with the operation IDs of the respective endpoints, and deprecates the older binding names.
- [WIP]Make events collapsible in API Docs.: This is a WIP PR that changes the UI to make the events in the
get-events
API docs page collapsible, which makes the page more organized and navigable.
Open/Merged Pull Requests made during Application Period
- streams: Fixed live update of Stream Name during Stream Rename. Merged
- APIDocumentation: Display Data Type of Parameters in API Documentation. Merged
- api docs: Improve readability of code and response blocks. Merged
- apidocs: Add documentation of deactivate-my-account endpoint. Merged
- apidocs: Display data type of responses in API Documentation and minor API Documentation fixes. Merged
- api docs: Removed duplicacy in CSS. Merged
- api docs: Fix documentation of update-subscription-settings. Merged
- api docs: Fix non-rendering response parameter data types. Merged
- api docs: Expand condition for deprecated fields. Merged
- api docs: Fix blank line at end of Code Blocks. Merged
- api docs: Fix id and type fields of events and display them. Merged
- api docs: Add missing ID fields for event responses. Merged
- api docs: Sort response keys in /get-events. Merged
- openapi: Fix schema for unread_msgs in /register. Merged
- middleware: Add client version to log and other logging improvements. Merged
- openapi: Fix non-checking of /events and /register schema and yaml fixes. Merged
- api: Add dataclass for update_message events. Open
- [WIP]apidocs: Document Changes in /api as per API changelog. Open
Closing Notes
GSoC was a great learning experience, as I worked with various kinds of technologies throughout the summer. The most satisfying bit was seeing my work getting merged, knowing that the code that I wrote would be useful to the numerous users and developers of Zulip.
In future, I would continue contributing to Zulip and exploring new and exciting areas in the project, and would also try my utmost to help newcomers get involved with Zulip and Open Source in general. I look forward to continuing association with Zulip and Open Source.
~ Suyash Vardhan Mathur