Skip to content

Instantly share code, notes, and snippets.

@jhillix

jhillix/archlinux-ami-setup-with-xfce4-for-photogrammetry.md

Last active November 24, 2023 21:45
Show Gist options
  • Save jhillix/aa543e675ad6e917cecd48d960f7f41a to your computer and use it in GitHub Desktop.
Save jhillix/aa543e675ad6e917cecd48d960f7f41a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
archlinux-ami-setup-with-xfce4-for-photogrammetry.md

Arch Linux AMI Setup with XFCE4 for Photogrammetry

Purpose

The purpose for this Gist is to document for myself the steps to setup Arch Linux AMI in AWS with XFCE4 as the desktop environment. As well as, setting up a RDP server. The goal is to have an instance in AWS with GPU resources for the purposes of photogrammetry.

NOTE: Pay attention to $ and # when commands are shown. # indicates that the command needs to be run as root - - OR - - with elevated priveleges. $ indicates that the command is invoked as a regular user.

Steps Overview

  1. Install Arch Linux AMI
  2. Update system
  3. Install Helpful Software
  4. User setup
  5. Install NVIDIA Driver
  6. Install XFCE4
  7. Install display manager
  8. XFCE Configuration
  9. Install Remote Desktop Server
  10. Connect with RDP Client
  11. Photogrammetry Software Install (COLMAP)

Step 1 - Install Arch Linux AMI

  1. I like to stick with the std kernel: Arch Linux AMIs for Amazon Web Services
  2. You will need to select an EC2 Instance Type that supports GPUs. I use the p2.xlarge
  3. Ensure that you open ports 22 - SSH - and 3389 - RDP

Step 2 - Update System

  1. SSH with the default user arch: ssh -i </path/to/.pem file> arch@<EC2 IP>
  2. # pacman-key --init
  3. # pacman-key --populate
  4. # pacman -Syu

Step 3 - Install Helpful Software

  1. # pacman -S base-devel git nano xdg-user-dirs

Step 4 - User Setup

arch user already exists and it has sudo priveleges but things work better if a new user is created and HAS a password.

  1. # useradd -m pg - PG is an acrovym for Photogrammetry
  2. # passwd pg
  3. # usermod -aG wheel,video,audio,storage pg
  4. # EDITOR=nano visudo
  5. Remove # from %wheel ALL=(ALL) ALL
  6. Save and Exit
  7. Create the usual suspects for user pg:
    1. In pg home directory run: $ xdg-user-dirs-update
  8. Setup AUR:
    1. $ mkdir /home/pg/.aur
  9. Setup SSH for pg:
    1. # cp -R /home/arch/.ssh /home/pg/
    2. # chown "pg:pg" /home/pg/.ssh
    3. # chown "pg:pg" /home/pg/.ssh/authorized_keys

Step 5 - Install NVIDIA Driver

NOTE: Identify the driver you need with $ lspci | grep -E 'VGA|3D' or $ lspci -k | grep -A 2 -E "(VGA|3D)" and NVIDIA website.

As user pg:

  1. For p2 Instance Types (as of the time of this writing):
    1. $ cd .aur
    2. $ git clone https://aur.archlinux.org/nvidia-470xx-utils.git
    3. $ cd nvidia-470xx-utils/
    4. $ makepkg -si
  2. For all others (p3, g3, etc.):
    1. # pacman -S nvidia-dkms
  3. # nano /etc/mkinitcpio.conf
  4. Remove kms from the HOOKS array
  5. Save and Exit
  6. Regenerate the initramfs: # mkinitcpio -P
  7. # reboot
  8. Verify driver is working: $ nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02   Driver Version: 470.223.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   26C    P0    73W / 149W |      0MiB / 11441MiB |     98%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Step 6 - Install XFCE4

  1. # pacman -S xfce4 xfce4-goodies xorg-server xorg-apps

Step 7 - Install Display Manager

  1. # pacman -S sddm
  2. $ cd /home/pg/.aur
  3. $ git clone https://aur.archlinux.org/archlinux-themes-sddm.git - theme for SDDM
  4. $ cd archlinux-themes-sddm/
  5. $ makepkg -si
  6. # systemctl enable sddm
  7. # systemctl start sddm

Step 8 - XFCE Configuration

Enable Audio

  1. # pacman -S pipewire pipewire-alsa pipewire-pulse
  2. # reboot

Install Archive Utility

  1. # pacman -S file-roller

Step 9 - Install Remote Desktop Server

  1. $ cd /home/pg/.aur
  2. $ git clone https://aur.archlinux.org/xorgxrdp.git
  3. $ cd xorgxrdp/
  4. $ makepkg -si
  5. If you see: ==> ERROR: 'pacman' failed to install missing dependencies. then just install them manually:
    1. # pacman -S nasm xorg-server-devel
    2. $ cd ..
    3. $ git clone https://aur.archlinux.org/xrdp.git
    4. $ cd xrdp/
    5. $ makepkg -si
    6. $ cd ../xorgxrdp/
    7. $ makepkg -si
  6. If you see: ==> ERROR: One or more PGP signatures could not be verified!:
    1. $ gpg --recv-key <The key that shows in the error message, most likely: 03993B4065E7193B>
    2. $ makepkg -si
  7. # systemctl enable xrdp
  8. # systemctl start xrdp
  9. # nano /etc/xrdp/sesman.ini
  10. Change param=Xorg to param=/usr/lib/Xorg, otherwise X server will error: [ERROR] There is no X server active on display
  11. Save and Exit
  12. Add xinitrc file: $ cp /etc/X11/xinit/xinitrc ~/.xinitrc
  13. Modify xinitrc:
    1. $ nano .xinitrc
    2. Add exec startxfce4 under the LAST if block in file
    3. Save and Exit
  14. # reboot

Step 10 - Connect with RDP Client

Use your preferred RDP client to connect to the RDP server you just installed above. I use Microsoft Remote Desktop to connect from a Mac to the EC2 instance.

Step 11 - Photogrammetry Software Install (COLMAP)

For this step you can use any photogrammetry software - within reason. I have better luck with COLMAP and that is what I'll be showing here. Although there's an AUR for COLMAP I'll build from source for simplicity.

  1. $ git clone https://github.com/colmap/colmap
  2. Install dependencies:
    1. For p2 Instance Types (as of the time of this writing):
      1. # pacman -S cgal ceres-solver glew freeimage flann cmake ninja python-sphinx
      2. $ cd ~/.aur
      3. $ git clone https://aur.archlinux.org/cuda-11.0.git
      4. $ cd cuda-11.0/
      5. $ makepkg -si
    2. For all others (p3, g3, etc.):
      1. # pacman -S cgal ceres-solver glew freeimage flann cmake ninja python-sphinx cuda
    3. Verify NVIDIA compiler exists and is functional $ /opt/cuda/bin/nvcc -V: 
      nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:24_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0
    4. $ cd /home/pg/.aur
    5. $ git clone https://aur.archlinux.org/metis.git
    6. $ git clone https://aur.archlinux.org/gklib.git
    7. $ cd gklib/
    8. $ makepkg -si
    9. $ cd ../metis/
    10. $ makepkg -si
  3. $ cd /<path to>/colmap/
  4. $ mkdir build
  5. $ cd build/
  6. We need to ensure cmake finds nvcc so that the build is configured for CUDA: $ export PATH=/opt/cuda/bin:$PATH
  7. $ cmake .. -DCMAKE_CUDA_ARCHITECTURES=all
  8. $ make
  9. colmap executable is located at build/src/colmap/exe/
  10. In the exe directory run ./colmap gui to launch the GUI and have fun :-)

NOTE: To verify that colmap supports CUDA run ./colmap -h and look for:

COLMAP 3.9 -- Structure-from-Motion and Multi-View Stereo
              (Commit f6b94fc2 on 2023-11-21 with CUDA)

Use the directions to get going. I keep the defaults in Automatic reconstruction modal box and check the Dense model checkbox. To ensure colmap is using the GPU you can run nvidia-smi -a | grep Utilization -A2. If Gpu > 0% then you're good.

References:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment