SageMath Project Development Quickstart Guide v1.1

quickstart-1.1.md

SageMath Development Quickstart Guide

This document aims to provide a "shortest path" introduction to contributing code to the Sage project, assembling all essential details of the development process in a single place for a streamlined experience getting started.

Instructions are presented for the most common use case; if you find that you're having trouble at a particular step, check with linked reference documents or ask a question at the sage-support email list at https://groups.google.com/group/sage-support/.

Set up development environment

The first step in contributing code to Sage is to set up the development environment needed to work on the development branch of Sage and interact with the Sage source repository and issue tracking system. This is one-time setup that is only needed the first time you develop for Sage; once the necessary accounts and software are configured, the development process is fortunately quite streamlined.

Install dependencies

First install the local software dependencies needed to access and compile the Sage developer distribution. A basic list of package dependencies for Linux systems is the following.

• Required packages: binutils gcc make m4 perl tar git openssl libssl-dev
• Recommended additional packages: g++ gfortran dvipng ffmpeg imagemagick texlive

To install dependencies on Debian or Ubuntu systems, the following should be sufficient:

sudo apt-get install binutils gcc g++ gfortran make m4 perl tar git openssl \
libssl-dev dvipng ffmpeg imagemagick texlive

For additional documentation of software dependencies, and for installation tips for other distributions and operating systems, see https://doc.sagemath.org/html/en/installation/source.html#prerequisites.

Build Sage developer branch

Now configure git global variables for development:

git config --global user.name 'Your Name'
git config --global user.email 'you@yourdomain.example.com'

The name and email you specify here will appear in any commits you make to the Sage project, so use your full name and your preferred professional contact information. Next clone the Sage source code into a local repository, and start the compilation process:

cd /desired/repository/parent/directory/
git clone git://github.com/sagemath/sage.git
cd sage
git checkout develop
MAKE='make -jNUM' make

The last command above compiles sage from source code using NUM processor threads. Note that Sage takes multiple hours (say 4 to 8) to compile from scratch in a single thread on modern systems; if your hardware supports it, compilation is much faster using multiple processor threads. In general, it is fastest to set NUM to be the number of parallel processor threads supported by your CPU, but consider using one less than this number to maintain responsiveness of your computer if you still plan to use it while compiling Sage.

Set up trac account and git trac utility

There are two ways to interact with the Sage project's issue tracking software: using a GitHub account, or using a separate "trac account". Both account types are sufficient for developing software for Sage, but the separate trac account has the advantage of allowing you to modify the Sage wiki (https://wiki.sagemath.org/) which is used for event organization and discussion of long-term projects.

To sign up for a separate trac account, send an email to sage-trac-account@googlegroups.com containing:

• your full name
• preferred username
• contact email
• reason for needing a trac account

Next, generate SSH public and private keys:

ssh-keygen -t rsa -b 4096

Your public key is found at ~/.ssh/id_rsa.pub and can be shared broadly as a token of your identity.

Your private key is found at ~/.ssh/id_rsa and should NEVER be shared.

Now set up SSH authentication in your trac account:

• Go to https://trac.sagemath.org
• Log in with your trac username/password, or with your Github account
• Click on "Preferences"
• Go to the "SSH Keys" tab
• Paste the contents of your public key file
• Click on "Save changes"

Next install the git trac utility, which simplifies interactions with the Sage trac system. Clone the git-trac-command repository and install locally:

git clone git://github.com/sagemath/git-trac-command.git
cd git-trac-command
python setup.py install --user

This installs the most up-to-date version of the utility in your ~/.local/bin/ directory, which contains scripts installed by the user. This directory will be in your terminal's PATH variable by default in most distributions, so that the terminal will find and execute scripts installed there. For more information on installation, see: https://github.com/sagemath/git-trac-command/

Finally, configure the Sage repository for development using your trac or GitHub account. First find your trac account's account token, which allows an application to log in to your account (so it should be kept private, like a password):

• Go to https://trac.sagemath.org
• Log in with your trac username/password, or with your Github account
• Click on "Preferences"
• Go to the "Token" tab
• Click on the button labeled "Copy Token"

Your token should now be in your system clipboard so that you can paste it into a terminal command in the next step. Finally, install the token in your local copy of the sage repository, and lock down the resulting configuration file (which includes the token in plaintext).

cd /path/to/sage/repository/
git trac config --user=USERNAME --token=TOKEN
chmod 0600 .git/config

And that's it! Once Sage has finished compiling, you're ready to start contributing code.

Development loop

The following sections describe the complete process of contributing code and other modifications to the Sage project, starting with how to vet an idea for suitability for the Sage codebase, continuing with an overview of style and content requirements for contributed code, and concluding with the details of Sage's peer review process.

Find the right place for your code

The first step of developing code for an open source software project is to determine what you want to change, and whether and how this change fits into the project. The following is the typical process for accomplishing this in the Sage project.

To begin, identify a specific, precisely formulated change that you would like to implement in Sage. Note that the Sage development community is generally very open to including new useful functionality (for instance, that code you wrote for your research), so don't be bashful! Once you've decided on a specific change, consider the following questions.

• Is the change suitable for inclusion in Sage?
  • If the change is to fix a bug:
    • Are you sure that it is actually a bug?
    • Do you sufficiently understand the relevant mathematics?
    • Is it possible that what you're observing is intended behavior?
    • Make sure to read through the documentation for the code in question.
  • If the change is to implement a new feature:
    • Is this feature appropriate and useful?
    • If modifying existing behavior, is there a reason that the code currently behaves the way it does? Will the change break any other code?
    • If implementing new functionality, is the functionality already possible in Sage by other means?
  • If you're not sure, or if the change you want to implement is significant, start a discussion on the sage-devel mailing list, or in an appropriate subject-specific mailing list; see: https://www.sagemath.org/development-groups.html
• Is the scope appropriate?
  • Smaller incremental changes are preferred to giant "patch bombs" because they are easier to understand and review.
  • If the change is very large, try to break it into several smaller subtasks which can be addressed individually.
• Has a ticket already been created which addresses the issue or something closely related?
  • Search trac for relevant keywords; see: https://trac.sagemath.org/search
  • Also feel free to ask the mailing lists, or search the mailing list history
• See also: https://doc.sagemath.org/html/en/developer/trac.html#guidelines-for-opening-tickets

If your change is appropriate and corresponds with an existing trac ticket, make a comment on the ticket to check on the status of the issue and express your interest in contributing. If your desired change isn't completely addressed by the ticket currently, feel free to discuss the direction of development and give suggestions. To check out the most up-to-date version of the code for the ticket, use

git trac checkout NUM

where NUM is the ticket number of the existing ticket, and recompile Sage. This takes only a small fraction of the amount of time it took to originally compile Sage since only files that have changed from the developer branch need to be recompiled. See the section on "Compiling Sage" below for more details.

If your change is appropriate and warrants a new trac ticket, create it using

git trac create 'Short summary of ticket'

This creates a new ticket, automatically assigns it a ticket number NUM, and creates the git branch /t/NUM/short_summary_of_ticket. Next, visit the web interface of the ticket at https://trac.sagemath.org/ticket/NUM (shortcut: git trac browse NUM) and update the fields of the ticket to appropriately describe the change. If you plan to work on the code yourself, make a note to this effect in the comments. See: https://doc.sagemath.org/html/en/developer/trac.html#the-ticket-fields

Write your code

It's time to write some code! Do your best to follow the standards and conventions for coding which are described at https://doc.sagemath.org/html/en/developer/coding_basics.html

The following summarizes the basics:

1. Use standard style conventions for the Python programming language when coding in Sage.
2. Write detailed and rigorous documentation for all code you write, using Python docstrings and a heading comment at the top of each code file.
3. In your documentation, include doctests, which are code examples that are included in the Sage reference manual and are executed by the Sage test suite to check for correctness as the code base evolves over time.
4. Update the files __init__.py and all.py in the relevant source directory if you've added new source files to Sage or if there are new methods that should be included in Sage's top level namespace. Update files index.rst or module_list.rst to include new files in the Sage Reference Manual. See the section "Adding new files to Sage" below for more information.
5. Look at existing source files for plentiful examples of source code style, especially for examples of documentation and doctests.

If this is your first time writing code for Sage, give the coding conventions document a skim, but try not to get bogged down in the details. The most important thing is to write readable and mathematically correct code with documentation; if there are stylistic errors or any pieces missing, other developers will point things out for you to fix in the review phase of the ticket.

Once you've made self-contained changes to the Sage codebase, test them out. First recompile sage; for changes to Python or Cython code in a subdirectory of src/sage/, this only requires executing sage -br, which only recompiles new and updated files. For more extensive changes, see the section below on compiling for more details.

After recompiling, run Sage and play with your code to make sure it's working. Run the doctests of new or updated files using sage -t --long source_file.py. See below for more options for running doctests.

Once you are happy with your changes, include them in a git commit, as follows.

git status
git add src/sage/path/to/source_file_1.py src/sage/path/to/source_file_2.py ...
git commit
git trac push

Feel free to do this multiple times during development, whenever you feel like you've reached a good checkpoint in your work. For the git commit step, your default text editor should come up to allow you to write a commit message. The first line is a short summary (50-80 characters or less) of the contents of the commit, followed by a longer description of the changes if further information is needed. The goal of the commit message is to make it clear to other developers (and yourself) who look back on this commit in the future exactly what was done. Here's an example:

Improve the representation of Hasse diagrams for FiniteLattice objects

Implements the representation for Hasse diagrams explored in the 1997 paper
"Space-efficient Representations of Hasse Diagrams" by Gauss and Newton.
Decreases the space usage of Hasse diagrams for finite lattices by 97% while
preserving the time efficiency of existing algorithms.

Save the file in your editor and close it to finalize the commit. The git trac push command pushes your code to the trac server, which makes it visible to other developers who are interested in your ticket, and also lets them know that you're actively working on the issue.

At this point you've made a complete, visible checkpoint for your ticket. If there's more work needed, write some more code, retest, and recommit. Once you feel that you've addressed the revision described by your ticket, take an extra precaution of making sure you haven't broken anything else in Sage by running the entire Sage test suite, using:

make ptestlong

This takes some time, but runs on parallel processes to speed things up. If the test suite succeeds, you're in the clear! Time to move on to the review process.

Initiate code review

You now have code that you think is ready to be included in Sage. The final step is to get confirmation from other Sage developers that your branch is mathematically and stylistically correct.

To indicate to other developers that your ticket is ready to go and needs review, log in to trac and visit your ticket, and set the ticket status to needs_review. Other developers will eventually take a look and will either confirm that the code looks good to go, or raise issues or concerns for you to address. Be patient if this takes some time (e.g. possibly days or weeks), remembering that:

• Sage development is conducted by volunteers in their free time, and there is always a lot to do
• For mathematical contributions, not all Sage developers may have the mathematical expertise to deal with your particular contribution

Once a reviewer takes a look at the ticket, if they encounter any problems they will change the ticket status to:

• needs_info if they need additional information before continuing; do your best to answer their questions in the ticket comments
• needs_work if they think there are some issues in the code that need improvement

If the reviewer indicates needs_work, they will give a list of specific issues that need to be addressed in the ticket comments. Don't despair! This means you now have a collaborator for getting the ticket ready for prime time. In this case:

• Make sure you understand the issues that were raised; ask for clarification and discuss in the comments if you're uncertain
• Write additional code as usual to address the concerns; then test, commit, and push the updated code to the trac server
• Once you're finished, set the ticket status back to needs_review

It may take a couple of back-and-forth exchanges with the reviewer before they are satisfied with your code. Once everything looks good to go, the ticket status will be set to positive_review, and your changes should be included in the next Sage release.

Code review

Code that is contributed to the Sage project goes through a peer review process before being included in the public release version of Sage. This is to double-check that new code is mathematically correct and follows the stylistic and documentation requirements of the Sage project. A fresh set of eyes is crucially important for ensuring good code quality!

The review process is also taken care of by volunteers, and in particular is not restricted to a special inner circle of Sage developers --- you can help with code review too! This is a very effective way to contribute to Sage, as there is always a large queue of features and bug fixes that have been carefully implemented, but are sitting in limbo with the needs_review status tag.

After your developer environment is set up, visit the following page to see a listing of tickets in need of review: https://trac.sagemath.org/report/30.

The main background needed for reviewing a ticket is to understand the mathematical and software changes which are implemented. Be bold! Note that many tickets are simple fixes in documentation or additions of doctests, so frequently the technical requirements of reviewing a ticket are fairly low. In addition, if you start reviewing a ticket and realize part way through that you don't have the expertise needed to effectively complete the review, you can still contribute to the review process in a number of ways.

Once you've found a ticket you want to review, say ticket number NUM, start by obtaining the code for the ticket and compiling it on your system. This can be done as usual by checking out the branch using git trac checkout NUM, but this can be slow to compile since the ticket might be based on an earlier release of Sage which may have significantly different code than the current developer branch.

The streamlined approach to testing the changes in a ticket is to create an "anonymous merge" of the ticket. To do so, check out the developer branch, and then use git trac try, as follows:

git checkout develop
git trac try NUM

This puts your repository in a mode called "headless" mode, which means that your code is not associated with a named branch of the repository. Once you're finished reviewing the ticket, just use git checkout develop again to revert to the main Sage development branch.

Next start compiling the source code, either using sage -b if only Python files are changed from your current build, or make if more fundamental changes need to be compiled.

While the code is compiling, go back to the trac ticket, carefully read through the ticket statement and comments, and look at the ticket's diff (the readout of changes or differences introduced by the ticket) by clicking on the green branch name. Get an idea of what changes are introduced in the ticket so that you know where to look in the source code when you're reviewing.

Next, read the new code and take it out for a test-drive! The following should generally be checked:

• Purpose: Does the code address the ticket's stated purpose? Does it introduce any new problems? Does the code work as expected on input besides the examples included in the documentation, providing expected and robust output without errors or crashes?
• User documentation: Is the usage of new code clear to a user? Are all mathematical notions involved standard, or is explanation provided or linked to? Can a user find the new code easily if they need it?
• Code documentation: Does the code include enough comments to make the purpose clear?
• Conventions: Does the code respect coding conventions for Sage, Python, and Cython as appropriate?
• Doctest coverage: Do all functions contain doctests? (Even private functions which start with an underscore!) Use sage -coverage <files> to check this.
• Bugfixes: For any bugfix, does the ticket add a doctest illustrating the behavior which was fixed? Such a doctest should reference the ticket number in comment.
• Performance: Does the ticket degrade the performance of any code? Does it speed up any code? If so, does the ticket include any code examples to demonstrate the improvement?
• Manuals: Do the Sage manuals build? See below for how to build documentation.
• Doctests: Do all doctests pass without errors? Run doctests for the entire Sage library, including "long" doctests, as unrelated components of Sage may be affected by localized changes. The quick command to run these tests using parallel processes is make ptestlong, but see below for other ways to run the tests.

As you work through this checklist, if something is not clear to you which prevents you from proceeding, ask for clarification in the ticket comments, and set the ticket's status to needs_info. One of the ticket authors should write back to clarify, and should set the ticket status back to needs_review.

If you realize while you're reviewing that you're in over your head and don't feel that you have the experience or expertise needed to make a final decision on the ticket, write a comment to explain what you already did and ask if someone else could take a look as well. This is very useful input, as it can significantly reduce the amount of work another reviewer has to do to complete the review.

Once you've finished the review, change the ticket's status:

• positive_review if everything looks good and you believe that the ticket is ready to be included in a future Sage release. In this case, add your full name to the "reviewer" field of the ticket.
• needs_work if there are issues that need to be addressed. Write a comment listing the problems you've identified in your review, so the ticket authors can implement appropriate fixes.

Make sure any suggestions from the review are clear and actionable, and refrain from open-ended discussion or further feature requests. If you notice issues of this sort that could use additional attention, open a new ticket for them!

Additionally, if an issue is small enough that you know how to fix it yourself, you can make a commit in your own name and mark in the commit message that it is a "reviewer's patch". In this case, the new addition must also be reviewed, for instance by the author of the original patch, so set the ticket status back to needs_review.

As a last note, be kind, and be helpful! The author of the patch you're reviewing is also doing their best to improve Sage, so don't belittle their contributions or get annoyed at any problems you find. Do your best to work together to make the most of the time and effort you are both putting in to a good cause. :)

Sage commands cheat sheet

The following sections provide a brief reference for the most important commands and protocols for different parts of Sage development.

Compiling Sage

First time after cloning Sage repository:

• Single processor: make
• Multiple processors (replace NUM with desired number): MAKE='make -jNUM' make

Recompiling after making changes to source:

• If only Python and Cython files in src/sage/... are changed: ./sage -br
• If external libraries are changed: make
• To completely recompile from scratch: make distclean && make

Compiling documentation:

• Build docs from source: ./sage -b && ./sage --docbuild reference html
• General format: ./sage --docbuild <document-name> <format>
• document-name can be one of:
  • installation: Installation guide
  • tutorial: Sage tutorial
  • constructions: Mathematical constructions
  • reference: Reference manual
  • developer: Developer's guide
  • Others found in: src/doc/en
• format can be any format allowed by the Sphinx software package, e.g. html, pdf, ...

See also: doc.sagemath.org/html/en/developer/sage_manuals.html

Navigating trac tickets

Ticket development workflow:

• Create new ticket: git trac create 'Short summary of ticket'
• Check out existing ticket: git trac checkout NUM
• Push current commits to trac server: git trac push
• Pull current commits (= fetch + merge) from ticket: git trac pull or git trac pull NUM

Check out a ticket for code review:

• Create "anonymous merge" branch: git checkout develop && git trac try NUM
  • Often quicker to compile
• Check out named ticket branch: git trac checkout NUM
  • Slower to compile, but allows contributing code to ticket

Navigation and other utilities:

• View ticket in browser: git trac browse NUM
• Print ticket info: git trac print NUM
• View log of commits for a ticket: git trac log NUM

See also: https://github.com/sagemath/git-trac-command/

Running doctests

Test a single file or package:

• Single file: ./sage -t <source-file>
• All files in directory: ./sage -t <source-directory>

Testing options:

• Run tests for entire library: --all
• Include long tests, labeled with comment # long time: --long
• Test with NUM parallel processors: -pNUM
• List all available testing options: ./sage -th

Run entire Sage test suite using parallel processes, one of:

• make ptestlong
• ./sage -tpNUM --all --long
• ./sage -tpNUM --long src/

See also: doc.sagemath.org/html/en/developer/doctesting.html

Adding new files to Sage

Update the following configuration files to make Sage load the new source files appropriately:

• Update __init__.py in the package directory which includes the file
• Update all.py in the package directory which includes the file to add classes or methods to Sage's top level namespace

Incorporate new source files into the reference manual:

• If not in the combinat package, update index.rst in the appropriate reference manual directory
  • e.g. for a file in sage/matroids/, update src/doc/en/reference/matroids/index.rst
• If in the combinat package, update src/doc/en/reference/combinat/module_list.rst