Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Comprehensive Python testing on Travis CI
language: python
# ===== Linux ======
os: linux
dist: xenial
python:
- 2.7
- 3.6
- 3.7
- 3.8
- 3.9
jobs:
include:
# ======= OSX ========
# ----- changes in Travis images means this doesn't work for versions before 3.7.5 ---
# - name: "Python 2.7.14 on macOS 10.13"
# os: osx
# osx_image: xcode9.3 # Python 2.7.14_2 running on macOS 10.13
# language: shell # 'language: python' is an error on Travis CI macOS
# before_install:
# - python --version
# # - pip install -U pip
# # - python -m pip install --upgrade pip
# - pip install pytest --user
# - pip install codecov --user
# install: pip install ".[test]" --user
# script: python -m pytest
# after_success: python -m codecov
# - name: "Python 3.6.5 on macOS 10.13"
# os: osx
# osx_image: xcode9.4 # Python 3.6.5 running on macOS 10.13
# language: shell # 'language: python' is an error on Travis CI macOS
# before_install:
# - python3 --version
# - pip3 install -U pip
# - pip3 install -U pytest
# - pip3 install codecov
# script: python3 -m pytest
# after_success: python3 -m codecov
- name: "Python 3.7.5 on macOS 10.14"
os: osx
osx_image: xcode10.2 # Python 3.7.5 running on macOS 10.14.3
language: shell # 'language: python' is an error on Travis CI macOS
before_install:
- python3 --version
- python3 -m pip install --upgrade pip
- python3 -m pip install --upgrade codecov pytest
script: python3 -m pytest
after_success: python3 -m codecov
- name: "Python 3.8.0 on macOS 10.14"
os: osx
osx_image: xcode11.3 # Python 3.8.0 running on macOS 10.14.6
language: shell # 'language: python' is an error on Travis CI macOS
before_install:
- python3 --version
- python3 -m pip install --upgrade pip
- python3 -m pip install --upgrade codecov pytest
script: python3 -m pytest
after_success: python3 -m codecov
# ====== WINDOWS =========
- name: "Python 2.7 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' errors Travis CI Windows
before_install:
- choco install python2
- python --version
- python -m pip install --upgrade pip
- pip install --upgrade pytest
- pip install codecov
env: PATH=/c/Python27:/c/Python27/Scripts:$PATH
- name: "Python 3.5.4 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' is an error on Travis CI Windows
before_install:
- choco install python --version 3.5.4
- python --version
- python -m pip install --upgrade pip
- python -m pip install --upgrade codecov pytest
env: PATH=/c/Python35:/c/Python35/Scripts:$PATH
- name: "Python 3.6.8 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' is an error on Travis CI Windows
before_install:
- choco install python --version 3.6.8
- python --version
- python -m pip install --upgrade pip
- python -m pip install --upgrade codecov pytest
env: PATH=/c/Python36:/c/Python36/Scripts:$PATH
- name: "Python 3.7.9 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' is an error on Travis CI Windows
before_install:
- choco install python --version 3.7.9
- python --version
- python -m pip install --upgrade pip
- python -m pip install --upgrade codecov pytest
env: PATH=/c/Python37:/c/Python37/Scripts:$PATH
- name: "Python 3.8.8 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' is an error on Travis CI Windows
before_install:
- choco install python --version 3.8.8
- python --version
- python -m pip install --upgrade pip
- python -m pip install --upgrade codecov pytest
env: PATH=/c/Python38:/c/Python38/Scripts:$PATH
- name: "Python 3.9.2 on Windows"
os: windows # Windows 10.0.17134 N/A Build 17134
language: shell # 'language: python' is an error on Travis CI Windows
before_install:
- choco install python --version 3.9.2
- python --version
- python -m pip install --upgrade pip
- python -m pip install --upgrade codecov pytest
env: PATH=/c/Python39:/c/Python39/Scripts:$PATH
before_install:
- python --version
- python3 -m pip install --upgrade pip
- python3 -m pip install --upgrade codecov pytest
install:
- pip install ".[test]"
script: pytest
after_success:
- codecov # submit coverage
@cclauss
Copy link

cclauss commented Apr 9, 2020

Would it be possible to update for Python 3.8 and osx_image: xcode11.3 (and drop legacy Python)?

Also pip will be getting a real dependency resolver soon so python3 -m pip install --upgrade codecov pytest might be better.

@shaypal5
Copy link
Author

Well, I've added the 3.8.0 job for macOS, but it doesn't make since to remove jobs for legacy Python, since this is a template, meant for people to take and remove what they don't need. Some projects still test on legacy Python, even Python 2 after its deprecation.

About the dependency resolver - when will this happen? And why add this before this happens?

@shaypal5
Copy link
Author

Also, thank you for the input! :)

@cclauss
Copy link

cclauss commented Apr 26, 2020

What about Python 3.8 on Linux just after line 8?

Why prepare now for pip dependency resolution? Because it is now in beta and preparing has no negative effects but not preparing will mean that anyone who grabs this template before you change modify it will not benefit from pip having true dependency resolution.

@pmeier
Copy link

pmeier commented Apr 27, 2020

Thanks for this awesome template 💯


@shaypal5 Could you have a look if the macOs python versions are still correct? I've tried your trick with the differnt osx_images but they resolve as follows:

osx_image python3 --version
xcode9.4 Python 3.7.5
xcode10.2 Python 3.7.5
xcode11.3 Python 3.7.7

Here is a build of a sample project of mine.

@shaypal5
Copy link
Author

shaypal5 commented Apr 27, 2020

Hey @pmeier,

Unfortunately, changes in Travis images means this doesn't work for versions before 3.7.5 anymore.

his used to work - Travis even put my blog post on this up on their own official blog - but I guess they keep rebuilding the images, and so now even old xcode images come with Python 3.7.5.

This is the image that gets pulled by the job, not something we can control. At the time, when I was figuring this out, this was the only way I found to easily test different Python versions on macOS. I think building it from source each time is too expensive an overhead, so we can bare to not test this on earlier Pythons specifically on macOS (they ARE tested on linux, after all).

I've encountered this for one of my projects as well:
python-cachier/cachier#23

I had to move to only testing Python 3.7.5 and Python 3.8.0 on macOS. :\

@pmeier
Copy link

pmeier commented Apr 27, 2020

Hm, that is unfortunate. Heads up for you: even with xcode11.3 you are only testing against 3.7. See for example one of your latest builds.


Travis even pulled my blog post on this to their own official blog

Are you sure? I came to this gist by this

https://blog.travis-ci.com/2019-08-07-extensive-python-testing-on-travis-ci

@shaypal5
Copy link
Author

shaypal5 commented Apr 27, 2020

Sorry, this was an incorrect use of American English. I'm a non-native speaker. I meant they even put it on their official blog up back in the day (August 2019), but that this has become obsolete since (I guess rather recently, but I can't tell). :)

@ozancaglayan
Copy link

Aren't after_success's supposed to be python3 ... instead of python 3 ... ?

@cclauss
Copy link

cclauss commented Mar 31, 2021

  • The comment on line 2 should be replaced by os: linux.
    • See View config on the Travis run page for Build config validation hints.
  • matrix: on line 9 should be replaced with jobs:. matrix has been removed from Travis CI docs.
  • Python 3.8 and 3.9 should be added and Python 3.5 should be dropped. Same for Windows/choco.
  • pip has options for both a --upgrade and --user so the -U should be replaced with something more explicit.
  • Now that pip has a real dependency resolver, once pip itself has been upgraded, all remaining dependencies should be bundled into a single command.
    • python3 -m pip install --upgrade pip
    • python3 -m pip install --upgrade codecov pytest # Allow the pip dependency resolver to do its work.

@shaypal5
Copy link
Author

shaypal5 commented Apr 1, 2021

Aren't after_success's supposed to be python3 ... instead of python 3 ... ?

Definitely, that's a great catch. Thank you!
I've now fixed this.

@shaypal5
Copy link
Author

shaypal5 commented Apr 1, 2021

  • The comment on line 2 should be replaced by os: linux.

    • See View config on the Travis run page for Build config validation hints.
  • matrix: on line 9 should be replaced with jobs:. matrix has been removed from Travis CI docs.

  • Python 3.8 and 3.9 should be added and Python 3.5 should be dropped. Same for Windows/choco.

  • pip has options for both a --upgrade and --user so the -U should be replaced with something more explicit.

  • Now that pip has a real dependency resolver, once pip itself has been upgraded, all remaining dependencies should be bundled into a single command.

    • python3 -m pip install --upgrade pip
    • python3 -m pip install --upgrade codecov pytest # Allow the pip dependency resolver to do its work.

OK. I've amended almost everything, but someone will have to lookup the right image for Python 3.9 on OSX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment