Install script of caffe2 and detectron on AWS EC2 instance with Deep Learning Base AMI

install_caffe2_detectron.sh

# Install script of Caffe2 and Detectron on AWS EC2
#
# Tested environment:
#   - AMI: Deep Learning Base AMI (Ubuntu) Version 3.0 - ami-38c87440  (CUDA is already installed)
#   - Instance: p3.2xlarge (V100 * 1)
#   - Caffe2: https://github.com/caffe2/caffe2/commit/e1f614a5f8ae92f4ecb828e1d5f84d2cd1fe12bd
#   - Detectron: https://github.com/facebookresearch/Detectron/commit/a22302de27f9004422a96414ed4088d05c664978
#
# Usage:
#   Launch a fresh EC2 instance, put this script on the /home/ubuntu/, and run the following command.
#   $ cd ~
#   $ source install_caffe2_detectron.sh
#
# Test:
#   $ cd ~/work/detectron
#   $ python2 tests/test_spatial_narrow_as_op.py
#
# Run samples:
#   Run the following commands and see the results in /tmp/detectron-visualizations
#   $ cd ~/work/detectron
#   $ python2 tools/infer_simple.py \
#    --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml \
#    --output-dir /tmp/detectron-visualizations \
#    --image-ext jpg \
#    --wts https://s3-us-west-2.amazonaws.com/detectron/35861858/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_2x.yaml.02_32_51.SgT4y1cO/output/train/coco_2014_train:coco_2014_valminusminival/generalized_rcnn/model_final.pkl \
#    demo
#
# Note that:
#   - In the Deep Learning AMI (Version 4.0), CUDA and caffe2 are already installed. But the caffe2 in the AMI is
#     a bit old version and does not include some modules required for Detectron.
#     So this script supposes that an AMI is Deep Learning "Base" AMI (Version 3.0), where only CUDA is installed.
#   - Manual configuration of setup.py is not a recommended way. Any suggestions are welcome.



INSTALL_DIR=~/work

### Install Caffe2
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
      build-essential \
      cmake \
      git \
      libgoogle-glog-dev \
      libgtest-dev \
      libiomp-dev \
      libleveldb-dev \
      liblmdb-dev \
      libopencv-dev \
      libopenmpi-dev \
      libsnappy-dev \
      libprotobuf-dev \
      openmpi-bin \
      openmpi-doc \
      protobuf-compiler \
      python-dev \
      python-pip                          
sudo pip install \
      future \
      numpy \
      protobuf
sudo apt-get install -y --no-install-recommends libgflags-dev

mkdir -p $INSTALL_DIR && cd $INSTALL_DIR
git clone --recursive https://github.com/abrichr/caffe2.git && cd caffe2
mkdir -p build && cd build
cmake ..
sudo make install -j6

# Export paths
echo "export PYTHONPATH=/usr/local:\$PYTHONPATH" >> ~/.bashrc
echo "export PYTHONPATH=\$PYTHONPATH:${INSTALL_DIR}/caffe2/build" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=/usr/local/lib:\$LD_LIBRARY_PATH" >> ~/.bashrc
source ~/.bashrc


### Install Detectron
sudo pip install numpy pyyaml matplotlib opencv-python setuptools Cython mock scipy

# First, install coco api
COCOAPI=$INSTALL_DIR/cocoapi
git clone https://github.com/cocodataset/cocoapi.git $COCOAPI
cd $COCOAPI/PythonAPI

# https://github.com/facebookresearch/Detectron/issues/105
# https://github.com/cocodataset/cocoapi/issues/94
# Based on the comments above, insert `extra_link_args=['-L/usr/lib/x86_64-linux-gnu/']` in setup.py
sed -i -e "/extra_compile_args/a \        extra_link_args=['-L/usr/lib/x86_64-linux-gnu/']," setup.py

# Install into global site-packages
sudo make install

# Next, install detectron
DETECTRON=$INSTALL_DIR/detectron
git clone https://github.com/facebookresearch/detectron $DETECTRON
cd $DETECTRON/detectron
# Add `extra_link_args ... ` in setup.py in the same manner as COCOAPI
sed -i -e "/name='utils.cython/a \        extra_link_args=['-L/usr/lib/x86_64-linux-gnu/']," setup.py
make

Hi,

I tried script above, but it ends as below. Any ideas? thanks.

creating build/temp.linux-x86_64-2.7/pycocotools
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I../common -I/usr/include/python2.7 -c ../common/maskApi.c -o build/temp.linux-x86_64-2.7/../common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I../common -I/usr/include/python2.7 -c pycocotools/_mask.c -o build/temp.linux-x86_64-2.7/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
x86_64-linux-gnu-gcc: error: pycocotools/_mask.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Makefile:7: recipe for target 'install' failed
make: *** [install] Error 1
Cloning into '/home/ubuntu/work/detectron'...
remote: Counting objects: 856, done.
remote: Total 856 (delta 0), reused 0 (delta 0), pack-reused 856
Receiving objects: 100% (856/856), 4.06 MiB | 0 bytes/s, done.
Resolving deltas: 100% (528/528), done.
Checking connectivity... done.
sed: can't read setup.py: No such file or directory
make: *** No targets specified and no makefile found. Stop.