Instantly share code, notes, and snippets.

Embed
What would you like to do?
Quicker Travis builds that rely on numpy and scipy using Miniconda

For ETS's SKLL project, we found out the hard way that Travis-CI's support for numpy and scipy is pretty abysmal. There are pre-installed versions of numpy for some versions of Python, but those are seriously out of date, and scipy is not there are at all. The two most popular approaches for working around this are to (1) build everything from scratch, or (2) use apt-get to install more recent (but still out of date) versions of numpy and scipy. Both of these approaches lead to longer build times, and with the second approach, you still don't have the most recent versions of anything. To circumvent these issues, we've switched to using Miniconda (Anaconda's lightweight cousin) to install everything.

A template for installing a simple Python package that relies on numpy and scipy using Miniconda is provided below. Since it's a common setup, I've also included steps to use Coveralls for calculating test coverage.

All you need to do is replace the YOUR_PACKAGE_NAME_HERE parts with your package's name in both .coveragerc and .travis.yml, and include both files in your project's root directory. Then just activate Travis and you should be set.

You may also consider adding the latest Miniconda shell script to your repository, and modifying your .travis.yml to just use that instead of downloading it every time. It can make building slightly faster, and is more resistent to server issues on Continuum's side (although you still are downloading the packages from them).

Additional files (only show up on roughdraft.io)

.coveragerc

.coveragerc

.travis.yml

.travis.yml

[run]
source = YOUR_PACKAGE_NAME_HERE
omit =
*/python?.?/*
*/lib-python/?.?/*.py
*/lib_pypy/_*.py
*/site-packages/ordereddict.py
*/site-packages/nose/*
*/unittest2/*
language: python
python:
- 2.7
- 3.3
notifications:
email: false
# Setup anaconda
before_install:
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- chmod +x miniconda.sh
- ./miniconda.sh -b
- export PATH=/home/travis/miniconda/bin:$PATH
- conda update --yes conda
# The next couple lines fix a crash with multiprocessing on Travis and are not specific to using Miniconda
- sudo rm -rf /dev/shm
- sudo ln -s /run/shm /dev/shm
# Install packages
install:
- conda install --yes python=$TRAVIS_PYTHON_VERSION atlas numpy scipy matplotlib nose dateutil pandas statsmodels
# Coverage packages are on my binstar channel
- conda install --yes -c dan_blanchard python-coveralls nose-cov
- python setup.py install
# Run test
script:
- nosetests --with-cov --cov YOUR_PACKAGE_NAME_HERE --cov-config .coveragerc --logging-level=INFO
# Calculate coverage
after_success:
- coveralls --config_file .coveragerc
@Jorge-C

This comment has been minimized.

Show comment
Hide comment
@Jorge-C

Jorge-C Jan 3, 2014

This is great! Notice that you don't need to use miniconda3 to run python3 code (I modified your approach so that it creates a conda environment with the right python version).

Jorge-C commented Jan 3, 2014

This is great! Notice that you don't need to use miniconda3 to run python3 code (I modified your approach so that it creates a conda environment with the right python version).

@danielballan

This comment has been minimized.

Show comment
Hide comment
@danielballan

danielballan Feb 11, 2014

A warning to others, miniconda is not always updated with conda, which can cause builds to break. Add conda update --yes conda to the before_install.

danielballan commented Feb 11, 2014

A warning to others, miniconda is not always updated with conda, which can cause builds to break. Add conda update --yes conda to the before_install.

@dan-blanchard

This comment has been minimized.

Show comment
Hide comment
@dan-blanchard

dan-blanchard Mar 19, 2014

Wow, I didn't know people were commenting on this. (Weird that GitHub doesn't send notifications for Gists.)

Anyway, I should probably update this to use the latest version of miniconda.

Oh, and the reason I make it explicitly download miniconda3 for Python 3 code is that it's less to download (and therefore slightly faster).

Owner

dan-blanchard commented Mar 19, 2014

Wow, I didn't know people were commenting on this. (Weird that GitHub doesn't send notifications for Gists.)

Anyway, I should probably update this to use the latest version of miniconda.

Oh, and the reason I make it explicitly download miniconda3 for Python 3 code is that it's less to download (and therefore slightly faster).

@dan-blanchard

This comment has been minimized.

Show comment
Hide comment
@dan-blanchard

dan-blanchard Mar 21, 2014

Just updated to:

  • include the conda update suggestion from @danielballan
  • use my binstar channel for installing coverage tools
  • use latest version of Miniconda
Owner

dan-blanchard commented Mar 21, 2014

Just updated to:

  • include the conda update suggestion from @danielballan
  • use my binstar channel for installing coverage tools
  • use latest version of Miniconda
@zonca

This comment has been minimized.

Show comment
Hide comment
@zonca

zonca Apr 3, 2014

@dan-blanchard, you can directly use:

http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh

to get the latest version of miniconda!

zonca commented Apr 3, 2014

@dan-blanchard, you can directly use:

http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh

to get the latest version of miniconda!

@moconnell

This comment has been minimized.

Show comment
Hide comment
@moconnell

moconnell Apr 9, 2014

@dan-blanchard thanks for this. found one erratum found, should be:

  • export PATH=/home/travis/miniconda/bin:$PATH

moconnell commented Apr 9, 2014

@dan-blanchard thanks for this. found one erratum found, should be:

  • export PATH=/home/travis/miniconda/bin:$PATH
@rowanc1

This comment has been minimized.

Show comment
Hide comment
@rowanc1

rowanc1 commented May 17, 2014

Thanks!

@dan-blanchard

This comment has been minimized.

Show comment
Hide comment
@dan-blanchard

dan-blanchard Jul 12, 2014

Thanks for the catch @moconnell. I've fixed that.

I've also switched to using @zonca's URL.

Owner

dan-blanchard commented Jul 12, 2014

Thanks for the catch @moconnell. I've fixed that.

I've also switched to using @zonca's URL.

@vr2262

This comment has been minimized.

Show comment
Hide comment
@vr2262

vr2262 Aug 8, 2014

I was having issues with the environment when using this as well as pip. In order to fix it, in .travis.yml I had to put:

- conda update --yes conda
- conda create --yes -n condaenv python=2.7
- conda install --yes -n condaenv pip
- source activate condaenv

That allowed me to use pip without any problem.

vr2262 commented Aug 8, 2014

I was having issues with the environment when using this as well as pip. In order to fix it, in .travis.yml I had to put:

- conda update --yes conda
- conda create --yes -n condaenv python=2.7
- conda install --yes -n condaenv pip
- source activate condaenv

That allowed me to use pip without any problem.

@KelSolaar

This comment has been minimized.

Show comment
Hide comment
@KelSolaar

KelSolaar Aug 25, 2014

Is there a version of that recipe for Python 2.6?

KelSolaar commented Aug 25, 2014

Is there a version of that recipe for Python 2.6?

@metakermit

This comment has been minimized.

Show comment
Hide comment
@metakermit

metakermit Sep 1, 2014

Thanks @dan-blanchard & @vr2262. Just to clarify, the addition would go into before_install as well and if you don't want to hardcode the Python version you can write:

- conda create --yes -n condaenv python=$TRAVIS_PYTHON_VERSION

metakermit commented Sep 1, 2014

Thanks @dan-blanchard & @vr2262. Just to clarify, the addition would go into before_install as well and if you don't want to hardcode the Python version you can write:

- conda create --yes -n condaenv python=$TRAVIS_PYTHON_VERSION
@metakermit

This comment has been minimized.

Show comment
Hide comment
@metakermit

metakermit Sep 1, 2014

Also, the coverage flags didn't work for me, so I set them to:

script: nosetests --with-coverage --cover-package=packagename --logging-level=INFO

and I added to install:

- pip install coveralls

Now coveralls.io is not showing my organisation's repos, so I can't really test this, but that's another problem...

metakermit commented Sep 1, 2014

Also, the coverage flags didn't work for me, so I set them to:

script: nosetests --with-coverage --cover-package=packagename --logging-level=INFO

and I added to install:

- pip install coveralls

Now coveralls.io is not showing my organisation's repos, so I can't really test this, but that's another problem...

@xanderdunn

This comment has been minimized.

Show comment
Hide comment
@xanderdunn

xanderdunn May 25, 2015

Many thanks for this! My Travis CI build was actually timing out trying to install scipy with pip! This saved me.

xanderdunn commented May 25, 2015

Many thanks for this! My Travis CI build was actually timing out trying to install scipy with pip! This saved me.

@douglasrizzo

This comment has been minimized.

Show comment
Hide comment
@douglasrizzo

douglasrizzo Feb 7, 2016

If anyone is interested, there is a tutorial on the conda website teaching how to configure Miniconda with Travis.

douglasrizzo commented Feb 7, 2016

If anyone is interested, there is a tutorial on the conda website teaching how to configure Miniconda with Travis.

@estroz

This comment has been minimized.

Show comment
Hide comment
@estroz

estroz Nov 14, 2016

The - export PATH=/home/travis/miniconda/bin:$PATH step should be updated to - export PATH=/home/travis/miniconda2/bin:$PATH to reflect install path changes in the miniconda install script.
esheldon/fitsio#71

estroz commented Nov 14, 2016

The - export PATH=/home/travis/miniconda/bin:$PATH step should be updated to - export PATH=/home/travis/miniconda2/bin:$PATH to reflect install path changes in the miniconda install script.
esheldon/fitsio#71

@AlJohri

This comment has been minimized.

Show comment
Hide comment
@AlJohri

AlJohri Nov 15, 2016

@dan-blanchard can you update the script as per @estroz's last comment?

AlJohri commented Nov 15, 2016

@dan-blanchard can you update the script as per @estroz's last comment?

@johnyf

This comment has been minimized.

Show comment
Hide comment
@johnyf

johnyf Aug 24, 2017

No need to use Miniconda.

pip install --upgrade pip setuptools wheel
pip install --only-binary=numpy,scipy numpy scipy

travis-ci/travis-ci#2650 (comment)

johnyf commented Aug 24, 2017

No need to use Miniconda.

pip install --upgrade pip setuptools wheel
pip install --only-binary=numpy,scipy numpy scipy

travis-ci/travis-ci#2650 (comment)

@Seanny123

This comment has been minimized.

Show comment
Hide comment
@Seanny123

Seanny123 Apr 27, 2018

@johnyf based off my experience, the pip installed version is still way slower than conda. Will continue to monitor the situation as it evolves.

Seanny123 commented Apr 27, 2018

@johnyf based off my experience, the pip installed version is still way slower than conda. Will continue to monitor the situation as it evolves.

@mroberge

This comment has been minimized.

Show comment
Hide comment
@mroberge

mroberge May 11, 2018

I also had a great improvement to my build time by using the --only-binary option for numpy and scipy. Travis has recently put numpy into its Python 3.5 and 3.6 environments, so these build in just a minute. However, testing against 3.4 was taking 9 minutes. The pip install --only-binary=numpy,scipy numpy scipy cut the running time down to seven minutes, so I added some of my other dependencies like pandas, and now the Python 3.4 build only takes one minute.
My final .travis.yml file
Travis already uses virtual environments to test against different python versions, so using conda essentially creates an environment inside an environment.

mroberge commented May 11, 2018

I also had a great improvement to my build time by using the --only-binary option for numpy and scipy. Travis has recently put numpy into its Python 3.5 and 3.6 environments, so these build in just a minute. However, testing against 3.4 was taking 9 minutes. The pip install --only-binary=numpy,scipy numpy scipy cut the running time down to seven minutes, so I added some of my other dependencies like pandas, and now the Python 3.4 build only takes one minute.
My final .travis.yml file
Travis already uses virtual environments to test against different python versions, so using conda essentially creates an environment inside an environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment