Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Building ROS Melodic with python3 support
# quick gist to accompany post on building ROS Melodic w/python3 support
# https://www.miguelalonsojr.com/blog/robotics/ros/python3/2019/08/20/ros-melodic-python-3-build.html
# Warning: don't try to run this as a script. It probably won't work.
# remove all things python (optional)
sudo apt-get remove python-*
# remove previous installations of ROS
sudo apt-get remove ros-*
sudo apt-get remove ros-melodic-*
sudo apt-get autoremove
# setup python3 dependencies
sudo apt update
sudo apt install -y python3 python3-dev python3-pip build-essential
sudo -H pip3 install rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg
# initializ catkin build environment
sudo rosdep init
rosdep update
cd ~
mkdir ros_catkin_ws
cd ros_catkin_ws
# TODO: get rqt_rviz rviz_plugin_tutorials librviz_tutorial building w/python3
catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install
# setup ROS install
rosinstall_generator desktop_full --rosdistro melodic --deps --tar > melodic-desktop-full.rosinstall
wstool init -j8 src melodic-desktop-full.rosinstall
# if packages fail to download, run the following
wstool update -j4 -t src
# setup environment and install dependencies
export ROS_PYTHON_VERSION=3
pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython
# create the following script in the ros_catkin_ws directory and call it install_skip
#!/bin/bash
#Check whether root
if [ $(whoami) != root ]; then
echo You must be root or use sudo to install packages.
return
fi
#Call apt-get for each package
for pkg in "$@"
do
echo "Installing $pkg"
sudo apt-get -my install $pkg >> install.log
done
# and chmod it
chmod +x install_skip
# sed magic
sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/python/python3/g"`
rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`"
find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*python/\/usr\/bin\/env python3/g' {} +
# build ROS
catkin build
@zzh112119

This comment has been minimized.

Copy link

zzh112119 commented Oct 22, 2019

(Moved this from the blog for history preservation purpose)
Two errors were encountered during setting up with this tutorial:

  1. When catkin_make:
File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

This will not affect launching roscore just as you posted before.

  1. When rosrun somepacakge somecode.py:
rosrun topic_demo pytalker.py
Traceback (most recent call last):
File "/root/catkin_ws/src/ROS-Academy-for-Beginners/topic_demo/scripts/pytalker.py", line 3, in <module>
import rospy
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/rospy/__init__.py", line 47, in <module>
from std_msgs.msg import Header
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/std_msgs/msg/__init__.py", line 1, in <module>
from ._Bool import *
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/std_msgs/msg/_Bool.py", line 5, in <module>
import genpy
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/genpy/__init__.py", line 34, in <module>
from . message import Message, SerializationError, DeserializationError, MessageException, struct_I
File "/root/ros_catkin_ws/install/lib/python3/dist-packages/genpy/message.py", line 44, in <module>
import yaml
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 284
class YAMLObject(metaclass=YAMLObjectMetaclass):
^
SyntaxError: invalid syntax

This is an interesting error.. Python seems to be complaining about the python3's metaclass syntax here. And with the traceback log, ROS is trying to run with python3 here but somehow ended up using python2 when the error happens.

Does anyone have any insight?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 23, 2019

@zzh112119 Do you have another version of rosrun installed, perhaps the standard melodic install? You may want to check your paths to make sure which rosrun is being used if that's the case. Also, is the shebang in pytalker.py using python3? Also check genpy's message.py. It should be using python3 not python2. I recall seeing this issue in the past and it was related to the shebang.

@antietam76

This comment has been minimized.

Copy link

antietam76 commented Oct 24, 2019

Everything worked great until I tried to build it!

[build] Summary: 0 of 204 packages succeeded.                                  
[build]   Ignored:   33 packages were skipped or are blacklisted.              
[build]   Warnings:  None.                                                     
[build]   Abandoned: 203 packages were abandoned.                              
[build]   Failed:    1 packages failed.                                        
[build] Runtime: 2.7 seconds total.                                            
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/trollius/base_events.py", line 395, in __del__
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 65, in close
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 166, in remove_signal_handler
  File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

What could have happened? this is on a fresh install of ubuntu.
Thanks!

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 24, 2019

@Antietam , please post the error received for the package that failed to build. Also, be sure to read the blog post https://www.miguelalonsojr.com/blog/robotics/ros/python3/2019/08/20/ros-melodic-python-3-build.html and follow the instructions there.

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 24, 2019

@zzh112119 Also, if you post the python script that you are trying to run, I can try to see what's going on.

@antietam76

This comment has been minimized.

Copy link

antietam76 commented Oct 24, 2019

@drmaj The link you shared is broken.

I re-fresh-installed ubuntu and have only followed your guide. I get to catkin build and it seems to fail on different packages everytime. the most common on it fails on it the dynamic_reconfigure

Errors     << dynamic_reconfigure:make /home/jacobksu/ros_catkin_ws/logs/dynamic_reconfigure/build.make.004.log
Traceback (most recent call last):
  File "/home/jacobksu/ros_catkin_ws/src/dynamic_reconfigure/cfg/Test.cfg", line 37, in <module>
    from dynamic_reconfigure.parameter_generator_catkin import *
  File "/home/jacobksu/ros_catkin_ws/devel/.private/dynamic_reconfigure/lib/python3/dist-packages/dynamic_reconfigure/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/home/jacobksu/ros_catkin_ws/install/lib/python3/dist-packages/roslib/__init__.py", line 50, in <module>
    from roslib.launcher import load_manifest
  File "/home/jacobksu/ros_catkin_ws/install/lib/python3/dist-packages/roslib/launcher.py", line 42, in <module>
    import rospkg
ImportError: No module named rospkg
make[2]: *** [/home/jacobksu/ros_catkin_ws/devel/.private/dynamic_reconfigure/include/dynamic_reconfigure/TestConfig.h] Error 1
make[1]: *** [CMakeFiles/dynamic_reconfigure_gencfg.dir/all] Error 2
make: *** [all] Error 2
cd /home/jacobksu/ros_catkin_ws/build/dynamic_reconfigure; catkin build --get-env dynamic_reconfigure | catkin env -si  /usr/bin/make --jobserver-fds=3,4 -j; cd -
...............................................................................
Failed     << dynamic_reconfigure:make                        [ Exited with code 2 ]

thanks again!

@zzh112119

This comment has been minimized.

Copy link

zzh112119 commented Oct 24, 2019

@drmaj I was actually able to find out the problematic shebang in my own code..

And yes! ROS seems to be working properly with python3 env and code! (I installed ros_base instead of desktop_full version tho)

But I noticed that during the step sudo ./install_skip ......., python2.7, python2.7-minimal and libpython2.7:amd64 were installed and python was linked to python2.7 by default. And when I try to dpkg --purge the python2.7 out of my system, I found out there's a dependency tree that originated from libboost-all-dev. Do you know if there's a good way to totally get rid out python2 in the system during installation of ros?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Oct 25, 2019

@zzh112119 It seems like the only way is to a) build boost against python3 from scratch b) ignore the dependencies when installing boost, or c) use equivs to trick apt. Here's a more detailed description: https://askubuntu.com/questions/944035/installing-libboost-python-dev-for-python3-without-installing-python2-7 . Even then, any of these solutions may not work with my build steps. I haven't explored this since the python2 dependencies don't cause any trouble for me.

@MYComodo

This comment has been minimized.

Copy link

MYComodo commented Nov 1, 2019

When i use
$ rosdep update

I get this error :

"

komodo@komodo-desktop:~/ros_catkin_ws$ rosdep update
bash: /home/komodo/.local/bin/rosdep: /usr/bin/python: bad interpreter: No such file or directory

"

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Nov 1, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:

rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""

where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

@MYComodo

This comment has been minimized.

Copy link

MYComodo commented Nov 1, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:

rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""

where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

i tried this code but it did not work. In my OS i am still using python3 and i have python2 too. This python3 can be the problem for this issue. Should i delete python3 too?

@drmaj

This comment has been minimized.

Copy link
Owner Author

drmaj commented Nov 4, 2019

@MYComodo It seems like you have a different version of rosdep on your path. My rosdep with this build is under /usr/local/bin. Also, you can't just run rosdep. It will try to use the default melodic apt packages. You should use the following:
rosdep [COMMAND] --from-paths src --ignore-src -y --skip-keys="rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g""
where [COMMAND] is what you want to do with rosdep, e.g. install, update, check, etc. This will skip all of the python based package dependencies. You'll then have to satisfy those dependencies manually by cloning those ros packages into your workspace and building them along with your packages. It's the only way I've found to reliably use ros with python3. Hopefully, with the next ros release, this won't be an issue any longer, since python is reaching end of life in 2020.

i tried this code but it did not work. In my OS i am still using python3 and i have python2 too. This python3 can be the problem for this issue. Should i delete python3 too?

No, this technique requires a python3 installation. If anything, you should remove python2, not python3. the problem seems to be the version of rosdep that you have installed. You should remove any old setup for ros, remove python2, and then try again.

@kiselevsky

This comment has been minimized.

Copy link

kiselevsky commented Nov 11, 2019

I'm trying to create a docker image using your instructions. And I have several problems:

  1. I can't use "SED magic" when I create image from Dockerfile due to file access issues:
sed: couldn't open temporary file ./sys/kernel/sedSaprdt: Read-only file system
sed: couldn't open temporary file ./sys/devices/pci0000:00/0000:00:05.0/ata17/host16/scsi_host/host16/sedJXIaWt: Read-only file system
sed: couldn't open temporary file ./sys/devices/system/clockevents/clockevent1/power/sed8nct7s: Read-only file system
sed: couldn't open temporary file ./sys/devices/virtual/block/nbd8/power/sedvpMyXO: Read-only file system
sed: couldn't open temporary file ./proc/sedLPWOHY: No such file or directory
sed: cannot rename ./etc/sedBGZwH9: Device or resource busy
  1. it seems that during the execution of the ./install_skip ... there is a duplication of 3 in the name of some packages, because of which their installation is skipped:
Installing python3-rospkg
Installing python33-lxml
E: Unable to locate package python33-lxml
Installing python33-yaml
E: Unable to locate package python33-yaml
Installing python33-mock
E: Unable to locate package python33-mock
Installing python33-catkin-pkg
E: Unable to locate package python33-catkin-pkg
Installing python33-rosdep
E: Unable to locate package python33-rosdep
Installing python33-coverage
E: Unable to locate package python33-coverage
Installing python33-rospkg
etc ...

But after checking the logs and manual installation it seems that everything is installed correctly... Is this a mistake?

  1. I managed to successfully build the ROS melodic in a running container, but roscore and other default pkgs work correctly only with default PYTHONPATH=/root/ros_catkin_ws/install/lib/python3/dist-packages/. Exporting the PYTHONPATH=/usr/lib/python3/dist-packages results in the following error:
Traceback (most recent call last):
  File "/root/ros_catkin_ws/install/bin/roscore", line 36, in <module>
    from rosmaster.master_api import NUM_WORKERS
ModuleNotFoundError: No module named 'rosmaster'
  1. But these were all solvable problems compared to the following. When trying to call catkin_make in my catkin_ws project directory, the following error occurs:
/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em:23: error: <class 'AssertionError'>: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/em.py", line 3302, in <module>
    if __name__ == '__main__': main()
  File "/usr/lib/python3/dist-packages/em.py", line 3300, in main
    invoke(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/em.py", line 3283, in invoke
    interpreter.wrap(interpreter.file, (file, name))
  File "/usr/lib/python3/dist-packages/em.py", line 2295, in wrap
    self.fail(e)
  File "/usr/lib/python3/dist-packages/em.py", line 2284, in wrap
    callable(*args)
  File "/usr/lib/python3/dist-packages/em.py", line 2359, in file
    self.safe(scanner, done, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2401, in safe
    self.parse(scanner, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2421, in parse
    token.run(self, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 1425, in run
    interpreter.execute(self.code, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2595, in execute
    _exec(statements, self.globals, locals)
  File "<string>", line 17, in <module>
  File "/usr/lib/python3/dist-packages/catkin_pkg/topological_order.py", line 124, in topological_order
    for path, package in find_packages(space).items():
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 86, in find_packages
    packages = find_packages_allowing_duplicates(basepath, exclude_paths=exclude_paths, exclude_subspaces=exclude_subspaces, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 146, in find_packages_allowing_duplicates
    xml, filename=filename, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/package.py", line 457, in parse_package_string
    assert pkg.package_format in [1, 2], "Unable to handle package.xml format version '%d', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)" % pkg.package_format
AssertionError: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
CMake Error at /root/ros_catkin_ws/install/share/catkin/cmake/safe_execute_process.cmake:11 (message):
  execute_process(/home/test_ws/build/catkin_generated/env_cached.sh
  "/usr/bin/python3" "/usr/lib/python3/dist-packages/em.py" "--raw-errors"
  "-F" "/home/test_ws/build/catkin_generated/order_packages.py" "-o"
  "/home/test_ws/build/catkin_generated/order_packages.cmake"
  "/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em")
  returned error code 1
Call Stack (most recent call first):
  /root/ros_catkin_ws/install/share/catkin/cmake/em_expand.cmake:25 (safe_execute_process)
  /root/ros_catkin_ws/install/share/catkin/cmake/catkin_workspace.cmake:35 (em_expand)
  CMakeLists.txt:67 (catkin_workspace)


-- Configuring incomplete, errors occurred!
See also "/home/test_ws/build/CMakeFiles/CMakeOutput.log".
See also "/home/test_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

All actions were performed in a container based on pure ubuntu:latest image. And I'm trying to build ROS-Comm (Bare Bones) This is my script:

    apt-get update && \
    apt-get install -y apt-utils && \
    apt-get install -y python3 python3-dev python3-pip build-essential && \
    pip3 install rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg && \
    rosdep init && \
    rosdep update && \
    cd ~ && \
    mkdir ros_catkin_ws && \
    cd ros_catkin_ws
    catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install && \
    rosinstall_generator ros_comm --rosdistro melodic --deps --tar > melodic-ros_comm.rosinstall && \
    wstool init -j8 src melodic-ros_comm.rosinstall && \
    wstool update -j4 -t src && \
    export ROS_PYTHON_VERSION=3 && \
    pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython && \
    echo '#!/bin/bash' >> install_skip && \
    echo 'for pkg in "$@"' >> install_skip && \
    echo 'do' >> install_skip && \
    echo '    echo "Installing $pkg"' >> install_skip && \
    echo '    DEBIAN_FRONTEND=noninteractive apt-get -my install $pkg >> install.log' >> install_skip && \
    echo 'done' >> install_skip && \
    chmod +x install_skip && \
    ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/python/python3/g"` && \
    rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`" && \
    find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*python/\/usr\/bin\/env python3/g' {} + && \
    catkin build

I will be very grateful for help in this matter!

@erelson

This comment has been minimized.

Copy link

erelson commented Nov 18, 2019

I too ran into the python33 errors like in the previous comment. Without digging fulling into what strings are being replaced, perhaps tweaking sed -e "s/python/python3/g" with added dashes: sed -e "s/python-/python3-/g" makes sense?

@kiselevsky: I believe your issue (1) may be due to ~ in a docker likely pointing to / instead of /home/$username as it would be on a regular install.

@saifullah3396

This comment has been minimized.

Copy link

saifullah3396 commented Jan 3, 2020

@zzh112119 It seems like the only way is to a) build boost against python3 from scratch b) ignore the dependencies when installing boost, or c) use equivs to trick apt. Here's a more detailed description: https://askubuntu.com/questions/944035/installing-libboost-python-dev-for-python3-without-installing-python2-7 . Even then, any of these solutions may not work with my build steps. I haven't explored this since the python2 dependencies don't cause any trouble for me.

Installing boost with python3 from source works!

A few additional errors that I faced along with their solutions are given in following:
caktin_tools trollius error on build: trollius has a dependency on python-2 so installing caktin_tools from pip3 is still dependent on python3-trollius which has bugs of its own. The easiest solution for this was just to build latest catkin-tools from source which replaces trollius with asyncio.

AttributeError: 'bytes' object has no attribute 'read' runtime error on running roscore. I'm not sure why this happens, but I believe the previous version of shlex required bytestrings to be passed to its functions which now require simple strings. So I just removed the conversion from string to bytestring wherever shlex split() function was used raising this error. The files that I updated are following:

  1. ros_comm/roslaunch/src/roslaunch/loader.py. (Just remove the conversion of the command string to byte string)
  2. ros_comm/roslaunch/src/roslaunch/node_args.py.

One addition that I did while building the source files was that I didn't only replace the shebangs but used the 2to3 automated python script for converting all the source files to python3. There were a few files with wrong print syntaxes that were easily fixed with the script. The script can be called as follows.

#!/bin/bash
# File: convert.sh
find src/ -name '*.py' -type f -exec 2to3 -w -n {} \;
@anthonysimeonov

This comment has been minimized.

Copy link

anthonysimeonov commented Feb 5, 2020

@kiselevsky

  1. But these were all solvable problems compared to the following. When trying to call catkin_make in my catkin_ws project directory, the following error occurs:
/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em:23: error: <class 'AssertionError'>: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/em.py", line 3302, in <module>
    if __name__ == '__main__': main()
  File "/usr/lib/python3/dist-packages/em.py", line 3300, in main
    invoke(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/em.py", line 3283, in invoke
    interpreter.wrap(interpreter.file, (file, name))
  File "/usr/lib/python3/dist-packages/em.py", line 2295, in wrap
    self.fail(e)
  File "/usr/lib/python3/dist-packages/em.py", line 2284, in wrap
    callable(*args)
  File "/usr/lib/python3/dist-packages/em.py", line 2359, in file
    self.safe(scanner, done, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2401, in safe
    self.parse(scanner, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2421, in parse
    token.run(self, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 1425, in run
    interpreter.execute(self.code, locals)
  File "/usr/lib/python3/dist-packages/em.py", line 2595, in execute
    _exec(statements, self.globals, locals)
  File "<string>", line 17, in <module>
  File "/usr/lib/python3/dist-packages/catkin_pkg/topological_order.py", line 124, in topological_order
    for path, package in find_packages(space).items():
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 86, in find_packages
    packages = find_packages_allowing_duplicates(basepath, exclude_paths=exclude_paths, exclude_subspaces=exclude_subspaces, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/packages.py", line 146, in find_packages_allowing_duplicates
    xml, filename=filename, warnings=warnings)
  File "/usr/lib/python3/dist-packages/catkin_pkg/package.py", line 457, in parse_package_string
    assert pkg.package_format in [1, 2], "Unable to handle package.xml format version '%d', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)" % pkg.package_format
AssertionError: Unable to handle package.xml format version '3', please update catkin_pkg (e.g. on Ubuntu/Debian use: sudo apt-get update && sudo apt-get install --only-upgrade python-catkin-pkg)
CMake Error at /root/ros_catkin_ws/install/share/catkin/cmake/safe_execute_process.cmake:11 (message):
  execute_process(/home/test_ws/build/catkin_generated/env_cached.sh
  "/usr/bin/python3" "/usr/lib/python3/dist-packages/em.py" "--raw-errors"
  "-F" "/home/test_ws/build/catkin_generated/order_packages.py" "-o"
  "/home/test_ws/build/catkin_generated/order_packages.cmake"
  "/root/ros_catkin_ws/install/share/catkin/cmake/em/order_packages.cmake.em")
  returned error code 1
Call Stack (most recent call first):
  /root/ros_catkin_ws/install/share/catkin/cmake/em_expand.cmake:25 (safe_execute_process)
  /root/ros_catkin_ws/install/share/catkin/cmake/catkin_workspace.cmake:35 (em_expand)
  CMakeLists.txt:67 (catkin_workspace)


-- Configuring incomplete, errors occurred!
See also "/home/test_ws/build/CMakeFiles/CMakeOutput.log".
See also "/home/test_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

Any updates here on a resolution? We're trying to do the same thing and ran into the same problem, except we're building with catkin build. Will appreciate any help!

@erelson

This comment has been minimized.

Copy link

erelson commented Feb 12, 2020

Unfortunatley I don't have a dockerfile handy, but I did get to the point where I had a better understanding of how to get a python3 package to build. The key thing that I did was to do it in docker: (1) started from the ubuntu:18.04 base image (2) figure out what dependencies were needed while (3) working towards building the base ros packages from source with the -DPYTHON_VERSION=3 flag, (4) then sourcing this python3-built workspace and building my workspace of interest, again with the -DPYTHON_VERSION=3 flag.

@Karthik-Ranga

This comment has been minimized.

Copy link

Karthik-Ranga commented Feb 12, 2020

I completed all the ROS build, when catkin_make was called, I got the below error.

  File "/home/karthik/ros_catkin_ws/install/bin/catkin_make", line 13, in <module>
    from catkin.terminal_color import disable_ANSI_colors, fmt
  File "/home/karthik/ros_catkin_ws/install/lib/python3/dist-packages/catkin/terminal_color.py", line 2, in <module>
    from catkin_pkg.terminal_color import *  # noqa
ModuleNotFoundError: No module named 'catkin_pkg.terminal_color'

has anyone solved this problem?

@jaypee92

This comment has been minimized.

Copy link

jaypee92 commented Feb 14, 2020

build success.
but roscore does not start

setting /run_id to a19b38b4-4f0e-11ea-95bf-e8d8d16102f5
ERROR: cannot launch node of type [rosout/rosout]: can't locate node [rosout] in package [rosout]
RLException: failed to start core service [/rosout]
The traceback for the exception was written to the log file
[master] killing on exit

@saifullah3396

This comment has been minimized.

Copy link

saifullah3396 commented Feb 15, 2020

I know this goes against the thread but for anyone who wants to use ROS with python3 its better to just build your own project with python3 instead of building the whole ROS. You can build any project with -DPYTHON_VERSION=3 and ROS_PYTHON_VERSION=3. If you want to use anything that depends on python3, just rebuild that specific package with python3 as well. I tried this solution and it worked but I had to rebuild every project every time which was not worth it. Instead, when I just built my own project with python3 it worked completely well with original ros, meaning there are no conflicts running python3 and ros together since internally ros is already updated to work with python3 when you set the flag ROS_PYTHON_VERSION=3.

Cheers.

@Karthik-Ranga

This comment has been minimized.

Copy link

Karthik-Ranga commented Feb 18, 2020

I know this goes against the thread but for anyone who wants to use ROS with python3 its better to just build your own project with python3 instead of building the whole ROS. You can build any project with -DPYTHON_VERSION=3 and ROS_PYTHON_VERSION=3. If you want to use anything that depends on python3, just rebuild that specific package with python3 as well. I tried this solution and it worked but I had to rebuild every project every time which was not worth it. Instead, when I just built my own project with python3 it worked completely well with original ros, meaning there are no conflicts running python3 and ros together since internally ros is already updated to work with python3 when you set the flag ROS_PYTHON_VERSION=3.

Cheers.

Can you elaborate by what you mean? I am not sure of the process.

@saifullah3396

This comment has been minimized.

Copy link

saifullah3396 commented Feb 18, 2020

I know this goes against the thread but for anyone who wants to use ROS with python3 its better to just build your own project with python3 instead of building the whole ROS. You can build any project with -DPYTHON_VERSION=3 and ROS_PYTHON_VERSION=3. If you want to use anything that depends on python3, just rebuild that specific package with python3 as well. I tried this solution and it worked but I had to rebuild every project every time which was not worth it. Instead, when I just built my own project with python3 it worked completely well with original ros, meaning there are no conflicts running python3 and ros together since internally ros is already updated to work with python3 when you set the flag ROS_PYTHON_VERSION=3.
Cheers.

Can you elaborate by what you mean? I am not sure of the process.

I will just list down the steps (tested on ROS/melodic Ubuntu 18.04):

  1. Make a virtual env and install python3 within it. This is important since without the virtual env ubuntu (that comes with prebuilt python3 can cause conflicts with python2 used by ros).
  2. Activate the virtual env.
  3. Make a catkin workspace as usual for your custom project based on python3.
  4. Set the environment variable ROS_PYTHON_VERSION=3
  5. Build your project with catkin_make --cmake-args -DPYTHON_VERSION=3
  6. Call launch files, nodes, etc as usual.
@Karthik-Ranga

This comment has been minimized.

Copy link

Karthik-Ranga commented Feb 19, 2020

Thanks for your time. I did try the above steps, and I got the below error

Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Invoking "make -j12 -l12" failed

I guess the package has lot of dependencies which could not be resolved. I tried tracing back to all the dependency but it is a huge mess.

@sharpe-developer

This comment has been minimized.

Copy link

sharpe-developer commented Feb 28, 2020

@drmaj Thank you for the detailed explanation.

The ROS team has started migrating towards python3 (for melodic on Ubuntu 18.04) which has caused an error in your sed command. The ROS team now has some instances of python3 within the install. Using sed to find the string python without using exact match causes sed to find instances of python which are in the string python3. Then when python is replaced with python3 the result is python33 which is not the desired result.

Changing the sed command to use exact match \<python\> fixes the issue.

In the sed magic section on your blog you state:

Be sure to only run this once. If you run this twice by mistake, you might end up with shebangs that have python33 instead of python3.

Using exact match also allows the command to be run multiple times without any problems. No need to warn users to "Be sure to only run this once."

The modified sed commands are shown below.

sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/\<python\>/python3/g"`

find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*\<python\>/\/usr\/bin\/env python3/g' {} +
@yuanzhedong

This comment has been minimized.

Copy link

yuanzhedong commented Mar 11, 2020

@sharp-developer, thank you for the script, it helps!

By the way do you have a timeline about the ROS python2->python3 migration? I'm trying to use the TimeSynchronizer which works on python2 but somehow failed on python3.

@sharpe-developer

This comment has been minimized.

Copy link

sharpe-developer commented Mar 11, 2020

@yuanzhedong Glad I could help!

I ended up reverting back as I also ran into various issues. It's my understanding that the ROS Noetic will target only Python 3. It is scheduled to be released on May 23rd, 2020.

@AabedSolayman

This comment has been minimized.

Copy link

AabedSolayman commented Mar 11, 2020

@sharp-developer
Hi! your solution helped, but still I get the same error when I build.
"TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object"
anyway after I used your solution 13 of 204 packages were succeded before that all the packages were abandoned or skipped

@nkueng

This comment has been minimized.

Copy link

nkueng commented Mar 11, 2020

Noob here. Trying to run catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install, I get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/catkin", line 6, in <module>
    from catkin_tools.commands.catkin import main
  File "/usr/local/lib/python3.7/dist-packages/catkin_tools/commands/catkin.py", line 28, in <module>
    from catkin_tools.common import is_tty
  File "/usr/local/lib/python3.7/dist-packages/catkin_tools/common.py", line 23, in <module>
    import trollius as asyncio
  File "/usr/local/lib/python3.7/dist-packages/trollius/__init__.py", line 21, in <module>
    from .base_events import *
  File "/usr/local/lib/python3.7/dist-packages/trollius/base_events.py", line 42, in <module>
    from . import tasks
  File "/usr/local/lib/python3.7/dist-packages/trollius/tasks.py", line 565
    def async(coro_or_future, loop=None):
            ^
SyntaxError: invalid syntax

I am trying to follow the instructions on a Raspberry Pi running Raspbian Buster Lite and had previously installed Ros Melodic from source according to this tutorial. Before trying the instructions above, I simply deleted my existing ros_catkin_ws and catkin_ws.

Thank you very much for any help!

@sharpe-developer

This comment has been minimized.

Copy link

sharpe-developer commented Mar 12, 2020

@AabedSolayman @nkueng I was able to get through the entire process with the following changes (if my memory serves me correctly)

  • sed commands as described above
  • I skipped the first couple sections (removing Python and ROS) and started at the section # setup python3 dependencies

I started with a clean install of Ubuntu 18.04 (virtual machine)

That being said I ended up abandoning this build because I ran into more issues after finishing the build. Given the fact that Python 3 support will be available in just a few months in the next ROS release it wasn't worth the trouble.

@yev-d

This comment has been minimized.

Copy link

yev-d commented Mar 31, 2020

I got the same errors as @antietam76, when performing catkin build. Are there any solutions to this?


Errors     << catkin:cmake /home/jek/ros_catkin_ws/logs/catkin/build.cmake.000.log                  
CMake Error at /home/jek/ros_catkin_ws/src/catkin/cmake/empy.cmake:29 (message):
  Unable to find either executable 'empy' or Python module 'em'...  try
  installing the package 'python-empy
Call Stack (most recent call first):
  cmake/all.cmake:164 (include)`
  CMakeLists.txt:8 (include)
cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/cmake /home/jek/ros_catkin_ws/src/catkin --no-warn-unused-cli -DCATKIN_DEVEL_PREFIX=/home/jek/ros_catkin_ws/devel/.private/catkin -DCMAKE_INSTALL_PREFIX=/home/jek/ros_catkin_ws/install -DCMAKE_BUILD_TYPE=Release; cd -
....................................................................................................
Failed     << catkin:cmake                                    [ Exited with code 1 ]                
Failed    <<< catkin                                          [ 3.7 seconds ]                       
Abandoned <<< genmsg                                          [ Unrelated job failed ]

and so on for all 204 packages until the error summary:

[build] Summary: 0 of 204 packages succeeded.                                                       
[build]   Ignored:   33 packages were skipped or are blacklisted.                                   
[build]   Warnings:  None.                                                                          
[build]   Abandoned: 203 packages were abandoned.                                                   
[build]   Failed:    1 packages failed.                                                             
[build] Runtime: 6.7 seconds total.                                                                 
[build] Note: Workspace packages have changed, please re-source setup files to use them.
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/trollius/base_events.py", line 395, in __del__
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 65, in close
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 166, in remove_signal_handler
  File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

So after installing python-empy and also catkin_pkg, the errors I get are a bit different. Since this time it doesn't ask to install anything, I'm at a loss.

Errors     << catkin:install /home/jek/ros_catkin_ws/logs/catkin/build.install.000.log               
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --install-layout not recognized
CMake Error at catkin_generated/safe_execute_install.cmake:4 (message):
  
  execute_process(/home/jek/ros_catkin_ws/build/catkin/catkin_generated/python_distutils_install.sh)
  returned error code
Call Stack (most recent call first):
  cmake_install.cmake:171 (include)

make: *** [install] Error 1
cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/make install; cd -
.....................................................................................................

I'm on Ubuntu 18.04.4 LTS
Python 3.7.3
ROS-Melodic

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

I got the same errors as @antietam76, when performing catkin build. Are there any solutions to this?

Errors     << catkin:cmake /home/jek/ros_catkin_ws/logs/catkin/build.cmake.000.log                  
CMake Error at /home/jek/ros_catkin_ws/src/catkin/cmake/empy.cmake:29 (message):
  Unable to find either executable 'empy' or Python module 'em'...  try
  installing the package 'python-empy
Call Stack (most recent call first):
  cmake/all.cmake:164 (include)`
  CMakeLists.txt:8 (include)
cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/cmake /home/jek/ros_catkin_ws/src/catkin --no-warn-unused-cli -DCATKIN_DEVEL_PREFIX=/home/jek/ros_catkin_ws/devel/.private/catkin -DCMAKE_INSTALL_PREFIX=/home/jek/ros_catkin_ws/install -DCMAKE_BUILD_TYPE=Release; cd -
....................................................................................................
Failed     << catkin:cmake                                    [ Exited with code 1 ]                
Failed    <<< catkin                                          [ 3.7 seconds ]                       
Abandoned <<< genmsg                                          [ Unrelated job failed ]

and so on for all 204 packages until the error summary:

[build] Summary: 0 of 204 packages succeeded.                                                       
[build]   Ignored:   33 packages were skipped or are blacklisted.                                   
[build]   Warnings:  None.                                                                          
[build]   Abandoned: 203 packages were abandoned.                                                   
[build]   Failed:    1 packages failed.                                                             
[build] Runtime: 6.7 seconds total.                                                                 
[build] Note: Workspace packages have changed, please re-source setup files to use them.
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/trollius/base_events.py", line 395, in __del__
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 65, in close
  File "/usr/local/lib/python3.6/dist-packages/trollius/unix_events.py", line 166, in remove_signal_handler
  File "/usr/lib/python3.6/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

So after installing python-empy and also catkin_pkg, the errors I get are a bit different. Since this time it doesn't ask to install anything, I'm at a loss.

Errors     << catkin:install /home/jek/ros_catkin_ws/logs/catkin/build.install.000.log               
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: option --install-layout not recognized
CMake Error at catkin_generated/safe_execute_install.cmake:4 (message):
  
  execute_process(/home/jek/ros_catkin_ws/build/catkin/catkin_generated/python_distutils_install.sh)
  returned error code
Call Stack (most recent call first):
  cmake_install.cmake:171 (include)

make: *** [install] Error 1
cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/make install; cd -
.....................................................................................................

I'm on Ubuntu 18.04.4 LTS
Python 3.7.3
ROS-Melodic

I am at the same step. What I did here was removed all occurrences of "--install-layout=deb " in the "python_distutils_install.sh" files.
I managed to get 48 of the 200 packages to build, but the rest then fails on failed imports. Let me know if you managed to get past this point.
I think I did something like this:

find ./ -name 'python_distutils_install.sh' -exec sed -i 's/--install-layout=deb//g' {} \;
@sharpe-developer

This comment has been minimized.

Copy link

sharpe-developer commented Apr 1, 2020

@yev-d

So after installing python-empy and also catkin_pkg

Just a guess but are you installing the python3 versions? If using apt to install python-empy should be python3-empy. Maybe you are using pip3?

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

# ensure there are no ros packages
sudo apt-get remove --autoremove ros-*

# check for updates
sudo apt update

# ensure /etc/ros removal
sudo rm -rf /etc/ros/

# install the python3 libraries
sudo apt install -y python3 python3-dev python3-pip build-essential

# Remove python2
sudo apt purge -y python2.7-minimal

# link python -> python3
sudo ln -s /usr/bin/python3 /usr/bin/python

# Same for pip
sudo ln -s /usr/bin/pip3 /usr/bin/pip

# install the ros dependencies
sudo -H pip3 install --no-cache-dir --ignore-installed rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg

# initialize catkin build environment
sudo rosdep init && rosdep update

# create catkin workspace
mkdir -p ~/ros_catkin_ws/src && cd "$_/.."

# initialize catkin workspace (will show warning about Extending... ignore that)
catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install

# generate a ros melodic install
rosinstall_generator desktop_full --rosdistro melodic --deps --tar > melodic-desktop-full.rosinstall

# initialize the install
wstool init -j8 src melodic-desktop-full.rosinstall

# setup environment and isntall dependencies
export ROS_PYTHON_VERSION=3

# install wxPython
pip3 install -U -f --no-cache-dir --ignore-installed https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython

# create install_skip file
printf '#/bin/bash\nif [ $(whoami) != root ]; then\n    echo You must be root or use sudo to install packages.\n    return\nfi\n\nfor pkg in "$@"\ndo\n    echo "Installing $pkg"\n    sudo apt-get -my install $pkg >> install.log\ndone' > install_skip

# make file executable
chmod +x install_skip

# install python 3 packages
sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/\<python\>/python3/g"`

# skip python 2 packages
rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`"

# rename all old python links to python3
find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*\<python\>/\/usr\/bin\/env python3/g' {} +

# build ros
catkin build

# export python path
export PYTHONPATH=/usr/lib/python3/dist-packages

# source setup
source ~/ros_catkin_ws/install/setup.bash

Running this worked for me.

@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

@sharpe-developer, yes you were right, but when I installed python3-empy, and it didn't change anything.

@metalglove, this line gave this error:
~/ros_catkin_ws$ pip3 install -U -f --no-cache-dir --ignore-installed https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython

 Collecting https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04
   Downloading https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04
      \ 20kB 159kB/s
   Cannot unpack file /tmp/pip-zxfu_1fn-unpack/ubuntu-18.04 (downloaded from /tmp/pip-fz1jq3ce-build, content-type: text/html;charset=UTF-8); cannot detect archive format
 Cannot determine archive format of /tmp/pip-fz1jq3ce-build

Not sure what that was I continued. Everything was fine up to catkin build. It didn't, with the following:

 Errors     << catkin:check /home/jek/ros_catkin_ws/logs/catkin/build.check.001.log                  
 /home/jek/anaconda3/bin/python3: can't open file '/usr/bin/empy': [Errno 2] No such file or directory
 CMake Error at /home/jek/ros_catkin_ws/src/catkin/cmake/safe_execute_process.cmake:11 (message):
 
   execute_process(/home/jek/ros_catkin_ws/build/catkin/catkin_generated/env_cached.sh
   "/home/jek/anaconda3/bin/python3" "/usr/bin/empy" "--raw-errors" "-F"
   "/home/jek/ros_catkin_ws/build/catkin/catkin_generated/pkg.develspace.context.pc.py"
   "-o"
   "/home/jek/ros_catkin_ws/devel/.private/catkin/lib/pkgconfig/catkin.pc"
   "/home/jek/ros_catkin_ws/src/catkin/cmake/em/pkg.pc.em") returned error
   code 2
 Call Stack (most recent call first):
   cmake/em_expand.cmake:25 (safe_execute_process)
   cmake/catkin_package.cmake:327 (em_expand)
   cmake/catkin_package.cmake:102 (_catkin_package)
   CMakeLists.txt:11 (catkin_package)

 make: *** [cmake_check_build_system] Error 1
 cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/make cmake_check_build_system; cd -

It says it can't find python3-empy, but it loks like it is installed and sits comfortably in usr/share/doc
Also, I'm not sure why it looks into conda environment when I execute all of this outside of conda.

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

@yev-d I could have tempered with that line last-minute... remove --no-cache-dir and --ignore-installed.

pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython
@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

@metalglove, that line worked, thanks! But the catkin build still fails to find empy. Do you have any thoughts on that?

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

@yev-d Did you run my complete script? what does the catkin build error show?

@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

@metalglove, yup. Everything was fine except catkin build and this little thing.
When running this line:

 sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/\<python\>/python3/g"

it doesn't find python3-wxtools, though other packages are ok:

 Installing python3-rospkg
 Installing python3-mock
 Installing python3-wxtools
 E: Unable to locate package python3-wxtools
 Installing python3-yaml
 Installing python3-matplotlib
 Installing python3-numpy
 Installing python3-psutil
 Installing python3-catkin-pkg
 Installing python3-opengl
 Installing python3-opencv
 Installing python3-cairo

Then as I come to catkin build command, this comes out:

 ----------------------------------------------------------------------------
 Profile:                     default
 Extending:                   None
 Workspace:                   /home/jek/ros_catkin_ws
 ----------------------------------------------------------------------------
 Build Space:        [exists] /home/jek/ros_catkin_ws/build
 Devel Space:        [exists] /home/jek/ros_catkin_ws/devel
 Install Space:      [exists] /home/jek/ros_catkin_ws/install
 Log Space:          [exists] /home/jek/ros_catkin_ws/logs
 Source Space:       [exists] /home/jek/ros_catkin_ws/src
 DESTDIR:            [unused] None
 ----------------------------------------------------------------------------
 Devel Space Layout:          linked
 Install Space Layout:        merged
 ----------------------------------------------------------------------------
 Additional CMake Args:       -DCMAKE_BUILD_TYPE=Release
 Additional Make Args:        None
 Additional catkin Make Args: None
 Internal Make Job Server:    True
 Cache Job Environments:      False
 ----------------------------------------------------------------------------
 Whitelisted Packages:        None
 Blacklisted Packages:        rqt_rviz rviz_plugin_tutorials librviz_tutorial
 ----------------------------------------------------------------------------
 
 
 ----------------------------------------------------------------------------
 WARNING: Your workspace is not extending any other result space, but it is
 set to use a `linked` devel space layout. This requires the `catkin` CMake
 package in your source space in order to be built.
 ----------------------------------------------------------------------------
 
 [build] Found '237' packages in 0.0 seconds.                                                        
 Starting  >>> catkin                                                                                
 ____________________________________________________________________________________________________
 Errors     << catkin:check /home/jek/ros_catkin_ws/logs/catkin/build.check.002.log                  
 /home/jek/anaconda3/bin/python3: can't open file '/usr/bin/empy': [Errno 2] No such file or directory
 CMake Error at /home/jek/ros_catkin_ws/src/catkin/cmake/safe_execute_process.cmake:11 (message):
 
   execute_process(/home/jek/ros_catkin_ws/build/catkin/catkin_generated/env_cached.sh
   "/home/jek/anaconda3/bin/python3" "/usr/bin/empy" "--raw-errors" "-F"
   "/home/jek/ros_catkin_ws/build/catkin/catkin_generated/pkg.develspace.context.pc.py"
   "-o"
   "/home/jek/ros_catkin_ws/devel/.private/catkin/lib/pkgconfig/catkin.pc"
   "/home/jek/ros_catkin_ws/src/catkin/cmake/em/pkg.pc.em") returned error
   code 2
 Call Stack (most recent call first):
   cmake/em_expand.cmake:25 (safe_execute_process)
   cmake/catkin_package.cmake:327 (em_expand)
   cmake/catkin_package.cmake:102 (_catkin_package)
   CMakeLists.txt:11 (catkin_package)
 
 
 make: *** [cmake_check_build_system] Error 1
 cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/make cmake_check_build_system; cd -
 ....................................................................................................
 Failed     << catkin:check                                    [ Exited with code 2 ]                
 Failed    <<< catkin                                          [ 0.9 seconds ]                       
@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

@yev-d Ah. "E: Unable to locate package python3-wxtools" makes sense, since we install it using the pip3 command above. And, the script changes all the python references to python3 hence it trying to install python3-wxtools. This can be ignored.

Try sudo apt install python-empy. This should work now.

I am currently not at my Ubuntu machine. Tomorrow I will spin up a new VM and run the complete script and check if I missed something in the script.

@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

@metalglove, yeah, after installing python-empy (and not python3-empy), the error changes! Whoo-hoo!

 Errors     << catkin:install /home/jek/ros_catkin_ws/logs/catkin/build.install.002.log
 usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
    or: setup.py --help [cmd1 cmd2 ...]
    or: setup.py --help-commands
    or: setup.py cmd --help
 
 error: option --install-layout not recognized
 CMake Error at catkin_generated/safe_execute_install.cmake:4 (message):
   
   execute_process(/home/jek/ros_catkin_ws/build/catkin/catkin_generated/python_distutils_install.sh)
   returned error code
 Call Stack (most recent call first):
   cmake_install.cmake:171 (include)
 
 
 make: *** [install] Error 1
 cd /home/jek/ros_catkin_ws/build/catkin; catkin build --get-env catkin | catkin env -si  /usr/bin/make install; cd -
 ...............................................................................

Yeah, no worries, take your time. I'm gonna look for a support group for Emotionally-Defeated-by-ROS-Installation-Anonymous group.

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 1, 2020

Oh, I know how to fix that error!
cd ~/ros_catkin_ws
find ./ -name 'python_distutils_install.sh' -exec sed -i 's/--install-layout=deb//g' {} \;
Now it should work ;-)

--install-layout=deb needs to be removed from all the generated installs scripts.

@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

@metalglove, nice! it solved THAT problem, and even started installing bunch of packages. But it failed miserably with 592 lines of errors for bunch of packages.
At this point, I'm going to try installing it on the new fresh Linux OS.

@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 1, 2020

I've installed a new OS with Ubuntu 18.04.4, and I ran all the code that @metalglove commented above
The only line that I've changed is this one:

 # install wxPython
 pip3 install -U -f --no-cache-dir --ignore-installed https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython

To this one:

pip3 install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython

catkin build installed all the packages, though with occasional warnings about missing "()" on "print". All 204 packages were installed, and 33 ignored.
I think when installing conda it does something that later on it is in conflict with the rest of the world.

So now that this is done, how can I make sure that it works? After so many failed attempts I have my doubts. Also, roscore doesn't work.

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 2, 2020

@yev-d conda uses its own python environment. Hence you will not be able to call roscore.
And, if you want to call python that uses the ros stuff do this:
make sure you are out of anaconda
conda deactivate
and source the ros setup:
source ~/ros_catkin_ws/install/setup.bash

you have to do this each time. You can add that last line to your .bashrc and comment the anaconda lines there.

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 2, 2020

The complete script for installing ROS with python3.

# ensure there are no ros packages
sudo apt-get remove --autoremove ros-*

# check for updates
sudo apt update

# ensure /etc/ros removal
sudo rm -rf /etc/ros/

# install the python3 libraries 
sudo apt install -y python3 python3-dev python3-pip build-essential

# Remove python2
sudo apt purge -y python2.7-minimal

# link python -> python3
sudo ln -s /usr/bin/python3 /usr/bin/python

# Same for pip
sudo ln -s /usr/bin/pip3 /usr/bin/pip

# install the ros dependencies
sudo -H pip3 install rosdep rospkg rosinstall_generator rosinstall wstool vcstools catkin_tools catkin_pkg

# initialize catkin build environment
sudo rosdep init && rosdep update

# create catkin workspace
mkdir -p ~/ros_catkin_ws/src && cd "$_/.."

# initialize catkin workspace (will show warning about Extending... ignore that)
catkin config --init -DCMAKE_BUILD_TYPE=Release --blacklist rqt_rviz rviz_plugin_tutorials librviz_tutorial --install

# generate a ros melodic install
rosinstall_generator desktop_full --rosdistro melodic --deps --tar > melodic-desktop-full.rosinstall

# initialize the install
wstool init -j8 src melodic-desktop-full.rosinstall

# setup environment and install dependencies
export ROS_PYTHON_VERSION=3

# install wxPython
pip install -U -f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04 wxPython

# create install_skip file
printf '#/bin/bash\nif [ $(whoami) != root ]; then\n    echo You must be root or use sudo to install packages.\n    return\nfi\n\nfor pkg in "$@"\ndo\n    echo "Installing $pkg"\n    sudo apt-get -my install $pkg >> install.log\ndone' > install_skip

# make file executable
chmod +x install_skip

# install python 3 packages
sudo ./install_skip `rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g" | sed -e "s/\<python\>/python3/g"`

# skip python 2 packages
rosdep install --from-paths src --ignore-src -y --skip-keys="`rosdep check --from-paths src --ignore-src | grep python | sed -e "s/^apt\t//g" | sed -z "s/\n/ /g"`"

# rename all old python links to python3
find . -type f -exec sed -i 's/\/usr\/bin\/env[ ]*\<python\>/\/usr\/bin\/env python3/g' {} +

# remove all depricated install-layout=deb arguments
find ./ -name 'python_distutils_install.sh' -exec sed -i 's/--install-layout=deb//g' {} \;

# install python-empy
sudo apt install -y python-empy

# build ros
catkin build

# export python path
export PYTHONPATH=/usr/lib/python3/dist-packages

# source setup
source ~/ros_catkin_ws/install/setup.bash
@yev-d

This comment has been minimized.

Copy link

yev-d commented Apr 2, 2020

@metalglove, Just to clarify... so after all of this, should roscore work, or are there any other steps for it?
(P.s. if you will assume that you are talking to a noob, I think your answer will go a long way ;) )

@metalglove

This comment has been minimized.

Copy link

metalglove commented Apr 2, 2020

@yev-d It should work. If everything installed fine. You should be able to open a new terminal and make sure that you have deactivated anaconda environment
conda deactivate
followed by a source of the ros setup
source ~/ros_catkin_ws/install/setup.bash
If you then run python and try to import rospy it should import just fine.
also, check if python -V shows python 3.
and roscore should run as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.