Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Building ROS Melodic with python3 support
# quick gist to accompany post on building ROS Melodic w/python3 support
# https://www.miguelalonsojr.com/blog/robotics/ros/python3/2019/08/20/ros-melodic-python-3-build.html
# Warning: don't try to run this as a script. It probably won't work.
# remove all things python (optional)
sudo apt-get remove python-*
# remove previous installations of ROS
sudo apt-get remove ros-*
sudo apt-get remove ros-melodic-*
sudo apt-get autoremove
# setup python3 dependencies
sudo apt update
sudo apt install -y python3 python3-dev python3-pip build-essential
sudo -H pip3 install rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg
# initializ catkin build environment
sudo rosdep init
rosdep update
cd ~
mkdir ros_catkin_ws
cd ros_catkin_ws
# TODO: get rqt_rviz rviz_plugin_tutorials librviz_tutorial building w/python3
catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install
# setup ROS install
rosinstall_generator desktop_full --rosdistro melodic --deps --tar > melodic-desktop-full.rosinstall
wstool init -j8 src melodic-desktop-full.rosinstall
# if packages fail to download, run the following
wstool update -j4 -t src
# setup environment and install dependencies
export ROS_PYTHON_VERSION=3
pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython
# create the following script in the ros_catkin_ws directory and call it install_skip
#!/bin/bash
#Check whether root
if [ $(whoami) != root ]; then
echo You must be root or use sudo to install packages.
return
fi
#Call apt-get for each package
for pkg in "$@"
do
echo "Installing $pkg"
sudo apt-get -my install $pkg >> install.log
done
# and chmod it
chmod +x install_skip
# sed magic
sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/python/python3/g"`
rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`"
find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*python/\/usr\/bin\/env python3/g' {} +
# build ROS
catkin build
@zzh112119

This comment has been minimized.

Copy link

zzh112119 commented Oct 22, 2019

(Moved this from the blog for history preservation purpose)
Two errors were encountered during setting up with this tutorial:

  1. When catkin_make:
File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

This will not affect launching roscore just as you posted before.

  1. When rosrun somepacakge somecode.py:
rosrun topic_demo pytalker.py
Traceback (most recent call last):
File "/root/catkin_ws/src/ROS-Academy-for-Beginners/topic_demo/scripts/pytalker.py", line 3, in <module>
import rospy
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/rospy/__init__.py", line 47, in <module>
from std_msgs.msg import Header
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/std_msgs/msg/__init__.py", line 1, in <module>
from ._Bool import *
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/std_msgs/msg/_Bool.py", line 5, in <module>
import genpy
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/genpy/__init__.py", line 34, in <module>
from . message import Message, SerializationError, DeserializationError, MessageException, struct_I
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/genpy/message.py", line 44, in <module>
import yaml
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 284
class YAMLObject(metaclass=YAMLObjectMetaclass):
^
SyntaxError: invalid syntax

This is an interesting error.. Python seems to be complaining about the python3's metaclass syntax here. And with the traceback log, ROS is trying to run with python3 here but somehow ended up using python2 when the error happens.

Does anyone have any insight?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 23, 2019

@zzh112119 Do you have another version of rosrun installed, perhaps the standard melodic install? You may want to check your paths to make sure which rosrun is being used if that's the case. Also, is the shebang in pytalker.py using python3? Also check genpy's message.py. It should be using python3 not python2. I recall seeing this issue in the past and it was related to the shebang.

@antietam76

This comment has been minimized.

Copy link

antietam76 commented Oct 24, 2019

Everything worked great until I tried to build it!

[build] Summary: 0 of 204 packages succeeded.                                  
[build]   Ignored:   33 packages were skipped or are blacklisted.              
[build]   Warnings:  None.                                                     
[build]   Abandoned: 203 packages were abandoned.                              
[build]   Failed:    1 packages failed.                                        
[build] Runtime: 2.7 seconds total.                                            
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/trollius/base_events.py", line 395, in __del__
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 65, in close
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 166, in remove_signal_handler
  File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

What could have happened? this is on a fresh install of ubuntu.
Thanks!

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 24, 2019

@Antietam , please post the error received for the package that failed to build. Also, be sure to read the blog post https://www.miguelalonsojr.com/blog/robotics/ros/python3/2019/08/20/ros-melodic-python-3-build.html and follow the instructions there.

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 24, 2019

@zzh112119 Also, if you post the python script that you are trying to run, I can try to see what's going on.

@antietam76

This comment has been minimized.

Copy link

antietam76 commented Oct 24, 2019

@drmaj The link you shared is broken.

I re-fresh-installed ubuntu and have only followed your guide. I get to catkin build and it seems to fail on different packages everytime. the most common on it fails on it the dynamic_reconfigure

Errors     << dynamic_reconfigure:make /home/jacobksu/ros_catkin_ws/logs/dynamic_reconfigure/build.make.004.log
Traceback (most recent call last):
  File "/home/jacobksu/ros_catkin_ws/src/dynamic_reconfigure/cfg/Test.cfg", line 37, in <module>
    from dynamic_reconfigure.parameter_generator_catkin import *
  File "/home/jacobksu/ros_catkin_ws/devel/.private/dynamic_reconfigure/lib/python3/dist-packages/dynamic_reconfigure/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/home/jacobksu/ros_catkin_ws/install/lib/python3/dist-packages/roslib/__init__.py", line 50, in <module>
    from roslib.launcher import load_manifest
  File "/home/jacobksu/ros_catkin_ws/install/lib/python3/dist-packages/roslib/launcher.py", line 42, in <module>
    import rospkg
ImportError: No module named rospkg
make[2]: *** [/home/jacobksu/ros_catkin_ws/devel/.private/dynamic_reconfigure/include/dynamic_reconfigure/TestConfig.h] Error 1
make[1]: *** [CMakeFiles/dynamic_reconfigure_gencfg.dir/all] Error 2
make: *** [all] Error 2
cd /home/jacobksu/ros_catkin_ws/build/dynamic_reconfigure; catkin build --get-env dynamic_reconfigure | catkin env -si  /usr/bin/make --jobserver-fds=3,4 -j; cd -
...............................................................................
Failed     << dynamic_reconfigure:make                        [ Exited with code 2 ]

thanks again!

@zzh112119

This comment has been minimized.

Copy link

zzh112119 commented Oct 24, 2019

@drmaj I was actually able to find out the problematic shebang in my own code..

And yes! ROS seems to be working properly with python3 env and code! (I installed ros_base instead of desktop_full version tho)

But I noticed that during the step sudo ./install_skip ......., python2.7, python2.7-minimal and libpython2.7:amd64 were installed and python was linked to python2.7 by default. And when I try to dpkg --purge the python2.7 out of my system, I found out there's a dependency tree that originated from libboost-all-dev. Do you know if there's a good way to totally get rid out python2 in the system during installation of ros?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 25, 2019

@zzh112119 It seems like the only way is to a) build boost against python3 from scratch b) ignore the dependencies when installing boost, or c) use equivs to trick apt. Here's a more detailed description: https://askubuntu.com/questions/944035/installing-libboost-python-dev-for-python3-without-installing-python2-7 . Even then, any of these solutions may not work with my build steps. I haven't explored this since the python2 dependencies don't cause any trouble for me.

@MYComodo

This comment has been minimized.

Copy link

MYComodo commented Nov 1, 2019

When i use
$ rosdep update

I get this error :

"

komodo@komodo-desktop:~/ros_catkin_ws$ rosdep update
bash: /home/komodo/.local/bin/rosdep: /usr/bin/python: bad interpreter: No such file or directory

"

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Nov 1, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:

rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""

where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

@MYComodo

This comment has been minimized.

Copy link

MYComodo commented Nov 1, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:

rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""

where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

i tried this code but it did not work. In my OS i am still using python3 and i have python2 too. This python3 can be the problem for this issue. Should i delete python3 too?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Nov 4, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:
rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""
where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

i tried this code but it did not work. In my OS i am still using python3 and i have python2 too. This python3 can be the problem for this issue. Should i delete python3 too?

No, this technique requires a python3 installation. If anything, you should remove python2, not python3. the problem seems to be the version of rosdep that you have installed. You should remove any old setup for ros, remove python2, and then try again.

@kiselevsky

This comment has been minimized.

Copy link

kiselevsky commented Nov 11, 2019

I'm trying to create a docker image using your instructions. And I have several problems:

  1. I can't use "SED magic" when I create image from Dockerfile due to file access issues:
sed: couldn't open temporary file ./sys/kernel/sedSaprdt: Read-only file system
sed: couldn't open temporary file ./sys/devices/pci0000:00/0000:00:05.0/ata17/host16/scsi_host/host16/sedJXIaWt: Read-only file system
sed: couldn't open temporary file ./sys/devices/system/clockevents/clockevent1/power/sed8nct7s: Read-only file system
sed: couldn't open temporary file ./sys/devices/virtual/block/nbd8/power/sedvpMyXO: Read-only file system
sed: couldn't open temporary file ./proc/sedLPWOHY: No such file or directory
sed: cannot rename ./etc/sedBGZwH9: Device or resource busy
  1. it seems that during the execution of the ./install_skip ... there is a duplication of 3 in the name of some packages, because of which their installation is skipped:
Installing python3-rospkg
Installing python33-lxml
E: Unable to locate package python33-lxml
Installing python33-yaml
E: Unable to locate package python33-yaml
Installing python33-mock
E: Unable to locate package python33-mock
Installing python33-catkin-pkg
E: Unable to locate package python33-catkin-pkg
Installing python33-rosdep
E: Unable to locate package python33-rosdep
Installing python33-coverage
E: Unable to locate package python33-coverage
Installing python33-rospkg
etc ...

But after checking the logs and manual installation it seems that everything is installed correctly... Is this a mistake?

  1. I managed to successfully build the ROS melodic in a running container, but roscore and other default pkgs work correctly only with default PYTHONPATH=/root/ros_catkin_ws/install/lib/python3/dist-packages/. Exporting the PYTHONPATH=/usr/lib/python3/dist-packages results in the following error:
Traceback (most recent call last):
  File "/root/ros_catkin_ws/install/bin/roscore", line 36, in <module>
    from rosmaster.master_api import NUM_WORKERS
ModuleNotFoundError: No module named 'rosmaster'
  1. But these were all solvable problems compared to the following. When trying to call catkin_make in my catkin_ws project directory, the following error occurs:
/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em:23: error: <class 'AssertionError'>: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/em.py", line 3302, in <module>
    if __name__ == '__main__': main()
  File "/usr/lib/python3/dist-packages/em.py", line 3300, in main
    invoke(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/em.py", line 3283, in invoke
    interpreter.wrap(interpreter.file, (file, name))
  File "/usr/lib/python3/dist-packages/em.py", line 2295, in wrap
    self.fail(e)
  File "/usr/lib/python3/dist-packages/em.py", line 2284, in wrap
    callable(*args)
  File "/usr/lib/python3/dist-packages/em.py", line 2359, in file
    self.safe(scanner, done, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2401, in safe
    self.parse(scanner, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2421, in parse
    token.run(self, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 1425, in run
    interpreter.execute(self.code, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2595, in execute
    _exec(statements, self.globals, locals)
  File "<string>", line 17, in <module>
  File "/usr/lib/python3/dist-packages/catkin_pkg/topological_order.py", line 124, in topological_order
    for path, package in find_packages(space).items():
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 86, in find_packages
    packages = find_packages_allowing_duplicates(basepath, exclude_paths=exclude_paths, exclude_subspaces=exclude_subspaces, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 146, in find_packages_allowing_duplicates
    xml, filename=filename, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/package.py", line 457, in parse_package_string
    assert pkg.package_format in [1, 2], "Unable to handle package.xml format version '%d', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)" % pkg.package_format
AssertionError: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
CMake Error at /root/ros_catkin_ws/install/share/catkin/cmake/safe_execute_process.cmake:11 (message):
  execute_process(/home/test_ws/build/catkin_generated/env_cached.sh
  "/usr/bin/python3" "/usr/lib/python3/dist-packages/em.py" "--raw-errors"
  "-F" "/home/test_ws/build/catkin_generated/order_packages.py" "-o"
  "/home/test_ws/build/catkin_generated/order_packages.cmake"
  "/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em")
  returned error code 1
Call Stack (most recent call first):
  /root/ros_catkin_ws/install/share/catkin/cmake/em_expand.cmake:25 (safe_execute_process)
  /root/ros_catkin_ws/install/share/catkin/cmake/catkin_workspace.cmake:35 (em_expand)
  CMakeLists.txt:67 (catkin_workspace)


-- Configuring incomplete, errors occurred!
See also "/home/test_ws/build/CMakeFiles/CMakeOutput.log".
See also "/home/test_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

All actions were performed in a container based on pure ubuntu:latest image. And I'm trying to build ROS-Comm (Bare Bones) This is my script:

    apt-get update && \
    apt-get install -y apt-utils && \
    apt-get install -y python3 python3-dev python3-pip build-essential && \
    pip3 install rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg && \
    rosdep init && \
    rosdep update && \
    cd ~ && \
    mkdir ros_catkin_ws && \
    cd ros_catkin_ws
    catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install && \
    rosinstall_generator ros_comm --rosdistro melodic --deps --tar > melodic-ros_comm.rosinstall && \
    wstool init -j8 src melodic-ros_comm.rosinstall && \
    wstool update -j4 -t src && \
    export ROS_PYTHON_VERSION=3 && \
    pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython && \
    echo '#!/bin/bash' >> install_skip && \
    echo 'for pkg in "$@"' >> install_skip && \
    echo 'do' >> install_skip && \
    echo '    echo "Installing $pkg"' >> install_skip && \
    echo '    DEBIAN_FRONTEND=noninteractive apt-get -my install $pkg >> install.log' >> install_skip && \
    echo 'done' >> install_skip && \
    chmod +x install_skip && \
    ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/python/python3/g"` && \
    rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`" && \
    find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*python/\/usr\/bin\/env python3/g' {} + && \
    catkin build

I will be very grateful for help in this matter!

@erelson

This comment has been minimized.

Copy link

erelson commented Nov 18, 2019

I too ran into the python33 errors like in the previous comment. Without digging fulling into what strings are being replaced, perhaps tweaking sed -e "s/python/python3/g" with added dashes: sed -e "s/python-/python3-/g" makes sense?

@kiselevsky: I believe your issue (1) may be due to ~ in a docker likely pointing to / instead of /home/$username as it would be on a regular install.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.