Skip to content

Instantly share code, notes, and snippets.

@evertrol

evertrol/photometry-software-installation.md

Last active April 14, 2016 02:36
Show Gist options
  • Save evertrol/37d912a32eadda4bf080 to your computer and use it in GitHub Desktop.
Save evertrol/37d912a32eadda4bf080 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
photometry-software-installation.md

Installation of the astronomical photometry packages.

This describes the installation of photometry pipeline external dependencies, starting from a minimal CentOS 7.1 (1503) system. (Depending on the selected ISO image, you may want to select 'minimal'.)

Note that the minimal installations comes without a default (X) window manager, nor will it have a network connection.

I have not enabled the Extra Packages for Enterprise Linux here (EPEL); all software that can't be found in the standard package list is installed from source.

Quite a few downloads have their version number in the URL. You could simply use that one (if still available), or adjust the URL to the latest version. Just beware of it if you copy-paste a lot.

The full installation has been tested on a CentOS system running in a virtual machine (VirtualBox); this is fairly easy to set up, in case you'd like to try this out yourself this way first.

The overall order of the steps is important, in the sense that some installation step may install a library that is also used by another step. So if you only install some steps, read all the previous steps to be usre you have all the required libraries installed.

Installation steps

  1. Networking

    From http://lintut.com/how-to-setup-network-after-rhelcentos-7-minimal-installation/

    Find your ethernet card(s):

    $ nmcli d

    Mine is enp0s3.

    Open the network manager:

    $ nmtui

    Chose "Edit a connection", then tab to "<Edit...>" for the relevant ethernet card. In the next window, IPv4 and IPv6 should be "". Tick "Automatically connect" and "", then "" when returned to the previous screen.

    Restart the network services:

    $ service network restart

    And test, e.g.:

    $ ip a

$ ping www.google.com

Basic development packages

Install gcc (4.8) and g++:

$ yum install gcc gcc-c++

Python 2 packages

Astrometry.net uses Python 2 with numpy:

$ yum install python-devel numpy

Python 3

Python 3 is not available through the normal packages. We install it manually.

Install some extra libraries, so that we get Python with batteries fully enabled (except for Tkinter).

Side-note: we need bzip2 (non-development) for the bunzip2 executable, which is run during Python's installation testing (make test).

$ yum install sqlite-devel readline-devel ncurses-devel bzip2-devel \
zlib-devel openssl-devel xz-devel gdbm-devel bzip2
$ mkdir sources
$ cd sources

Download the current version of Python 3:

$ curl https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tgz -O
$ tar -zxvf Python-3.5.0.tgz
$ cd Python-3.5.0

Install it out of the normal system stuff, but still in PATH:

$ ./configure --prefix=/usr/local
$ make
$ make test
$ make altinstall

The last command (make altinstall) will install Python 3 as python3.5, not as python. So the default system Python remains Python 2.7. Ditto for 2to3, easy_install, idle, pip pydoc and pyvenv.

On a new system, you can look in /usr/local/bin and verify that there are only Python 3 executables.

Python 3 packages

Install packages for Python 3, and the libraries they need. matplotlib wants libpng, libjpeg and libfreetype2.

$ yum install libpng-devel libjpeg-turbo-devel freetype-devel

$ pip3.5 install Cython numpy scipy astropy matplotlib

GOTO Python packages

Install the GOTO Python packages and dependencies. These are installed directly from Github.

$ pip3.5 install git+https://github.com/GOTO-OBS/gotoflow.git

Photometry tools

Install the photometry tools: astrometry.net, sextractor, swarp, ...

These are installed in a separate /usr/local/astro directory, and some (astrometry, cdsclient) in their own separate subdirectory.

To use them, this will mean setting LD_LIBRARY_PATH to /usr/local/astro/lib; but see further down on the -rpath flag.

$ mkdir /usr/local/astro

For convenience, you may want to change the ownership of /usr/local/astro (or the complete /usr/local even) to whoever is normally administrating the machine. For example:

$ chown -R evert:evert /usr/local/astro

Astrometry.net

a. cfitsio:

$ cd ~/sources
$ curl ftp://heasarc.gsfc.nasa.gov/software/fitsio/c/cfitsio_latest.tar.gz -O
$ tar -zxvf cfitsio_latest.tar.gz
$ cd cfitsio
$ ./configure --prefix=/usr/local/astro
$ make
$ make shared
$ make testprog

# If you didn't make a shared library, there is no need for the
# temporary LD_LIBRARY_PATH

$ LD_LIBRARY_PATH=. ./testprog > testprog.output
$ diff testprog.output testprog.out
$ cmp testprog.fit testprog.std
$ make install

b. wcslib:

$ cd ~/sources
$ curl ftp://ftp.atnf.csiro.au/pub/software/wcslib/wcslib.tar.bz2 -O
$ tar -jxvf wcslib.tar.bz2
$ cd wcslib-5.9
$ LDFLAGS="-Wl,-rpath,/usr/local/astro/lib" ./configure \
     --prefix=/usr/local/astro \
     --without-pgplot \
     --with-cfitsiolib=/usr/local/astro/lib \
     --with-cfitsioinc=/usr/local/astro/include
$ make
$ make check
$ make install

A quick explanation of the LDFLAGS: the -Wl flag passes any next (comma-separated) flags to the linker, in this case -rpath /usr/local/astro/lib. This will hardcode that directory for (additional) shared libraries to be found, in this case libcfitsio. This avoid using LD_LIBRARY_PATH. This option is not always recommended, but since we're here dealing with a single system (and probably single user) for a few libraries in that directory, it is probably easier. Alternatively, we could also update /etc/ld.so.conf.d/ and add /usr/local/astro/lib to the default loader path.

c. libcairo, gsl, netpbm; swig for the Python bindings:

   $ yum install cairo-devel
   $ yum install netpbm-devel
   $ yum install gsl-devel

   $ yum install swig

and pyfits for Python 2.7:

   $ cd ~/sources
   $ curl https://pypi.python.org/packages/source/p/pyfits/pyfits-3.3.tar.gz -O
   $ tar -zxvf pyfits-3.3.tar.gz
   $ cd pyfits-3.3
   $ python setup.py install

d. astrometry.net itself:

   $ cd ~/sources

   # Update the URL to current
   $ curl http://astrometry.net/downloads/astrometry.net-0.63.tar.gz -O

   $ tar -jxvf astrometry.net-0.63.tar.gz
   $ cd astrometry.net-0.63

Note: there is no configure step, just make. Instead, astrometry.net relies on pkg-config to find libcairo and libcfitsio, and since we installed some libraries in /usr/local/astro, we'll need to point it there.

   $ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

   $ LDFLAGS="-Wl,-rpath,/usr/local/astro/lib" make

   # note: the astrometry.net Python bindings are with Python 2,
   # not Python 3
   $ LDFLAGS="-Wl,-rpath,/usr/local/astro/lib" make py

   $ LDFLAGS="-Wl,-rpath,/usr/local/astro/lib" make extra

   $ mkdir /usr/local/astro/astrometry
   $ make install INSTALL_DIR=/usr/local/astro/astrometry

e. astrometry.net index files

The index files contain the 2MASS and Tycho catalogue positions. The first is for small field of view, the second for large field of view. We're probably somewhere in the middle, thus we simply grab both. In total, about 10 GB.

   # We could also `yum install wget` below, but it's more
   # fun to use sed once in a while

   $ cd /usr/local/astro/astrometry/data
   $ curl http://data.astrometry.net/4200/wget.sh -O
   $ sed 's/wget \-\-continue/curl -O/i' wget.sh > curl.sh
   $ sh curl.sh

   $ for i in {4107..4119}; \
        do curl -O http://data.astrometry.net/4100/index-$i.fits; done

SExtractor

SExtractor uses fftw and the atlas implementations of BLAS and LAPACK routines.

$ yum install atlas-devel
$ yum install fftw-devel

Unfortunately, configuring and compiling SExtractor with atlas is a bit of a pain: SExtractor expects atlas to also provide a cblas and lapack library (libcblas.so and liblapack.so), and accordingly links those libraries. Instead, the routines from these two libraries are bundled together on CentOS (and various other systems) in libatlas.so (note btw, that the atlas implementation of lapack is incomplete, but sufficient for SExtractor).

In addition, CentOS provides two atlas libraries: libsatlas.so and libtatlas.so: one serial, and one threaded. But SExtractor's configure script will look for libatlas.so instead.

Thus, we do a bit of symlinking:

$ cd /usr/local/lib

The atlas subdirectory is not necessary; putting the libraries in /usr/local/lib directly will avoid the need for LDFLAGS below.

$ mkdir atlas
$ cd atlas
$ ln -s /usr/lib64/atlas/libtatlas.so libatlas.so
$ ln -s /usr/lib64/atlas/libtatlas.so libcblas.so
$ ln -s /usr/lib64/atlas/libtatlas.so liblapack.so

We install fftw and atlas ourselves, so we can configure it using multi-threading. It also avoids issues with the SExtractor configuration step not being able to find the (C-implementation) of LAPACK routines.

$ cd ~/sources
$ curl http://www.astromatic.net/download/sextractor/sextractor-2.19.5.tar.gz -O
$ tar -zxvf sextractor-2.19.5.tar.gz
$ cd sextractor-2.19.5

Now configure SExtractor. As per above, we use the libraries in /usr/local/atlas (don't use --with-atlas-libdir; this will fail to pick up the linked cblas and lapack libraries):

$ LDFLAGS=-L/usr/local/atlas ./configure --prefix=/usr/local/astro
$ make
$ make check
$ make install
$ mv /usr/local/astro/bin/sex /usr/local/astro/bin/sextractor

(Yes, that last step is not necessary, but some humour just escapes me.)

On Ubuntu, you can install sextractor directly using a package manager. If you want a source installation, you can still install the various dependencies (fftw, lapack, atlas) using the package manager. There is a catch: you need to install liblapacke-dev for the C-interface (clapack). But even with that installed, I kept getting complaints from the SExtractor configure tool that it couldn't find some symbols: apparently, it picks up liblapack instead. So I remove liblapack-dev (in the hope that it doesn't mess with numpy dependencies) and installed liblapacke-dev. Now, with also libfftw3-dev, libblas3-dev and libatlas-dev installed, I could configure SExtractor:

./configure --prefix=/usr/local/astro --with-fftw-libdir=/usr/lib/x86_64-linux-gnu/ --with-fftw-incdir=/usr/include --with-atlas-libdir=/usr/lib/atlas-base --with-atlas-incdir=/usr/include/atlas

Scamp

Scamp requires the cdsclient, which is in my opinion a horrible package: it consists of 81 shell scripts, 3 compiled binaries and 1 awk script (for a package that is, unpacked, less than 1GB), and puts that all straight in the bin/ directory. Additionally, it puts a versions file at your --prefix value, e.g. /usr/local/versions. Therefore, we put cdsclient safely away in its own little subdirectory:

$ cd ~/sources
$ curl http://cdsarc.u-strasbg.fr/ftp/pub/sw/cdsclient.tar.gz -O
$ tar -zxvf cdsclient.tar.gz
$ cd cdsclient-3.80
$ mkdir /usr/local/astro/cdsclient
$ ./configure --prefix=/usr/local/astro/cdsclient
$ make
$ make install

Scamp has the same atlas problem as SExtractor, so we use the same configure flag. We also need it to tell the location of the cdsclient executable:

$ cd ~/sources
$ curl http://www.astromatic.net/download/scamp/scamp-2.0.4.tar.gz -O
$ tar -zxvf scamp-2.0.4.tar.gz
$ cd scamp-2.0.4
$ LDFLAGS=-L/usr/local/lib/atlas ./configure --prefix=/usr/local/astro \
  --with-cdsclient-dir=/usr/local/astro/cdsclient/bin
$ make
$ make check
$ make install

Swarp

As easy as:

 $ cd ~/sources
 $ curl http://www.astromatic.net/download/swarp/swarp-2.38.0.tar.gz -O
 $ tar -zxvf swarp-2.38.0.tar.gz
 $ cd swarp-2.38.0
 $ ./configure --prefix=/usr/local/astro
 $ make
 $ make check
 $ make install

Hotpants

$ cd ~/sources
$ curl http://www.astro.washington.edu/users/becker/v2.0/software/hotpants_v5.1.10.tar.gz -O
$ tar -zxvf hotpants_v5.1.10.tar.gz
$ cd hotpants_v5.1.10b    # Note the 'b'

Hotpants does not have a configure step, just a Makefile. Unfortunately, the compiler options contains -ansi and --pedantic-error, which means gethostname() is not recognised and will cause the compilation to crash. Using e.g. gnu99 as a C standard is more lenient and avoids this. Thus:

$ sed -i 's/ansi/std=gnu99/' Makefile

And since we put libcfitsio in /usr/local/astro:

$ make CFITSIOINCDIR=/usr/local/astro/include LIBDIR=/usr/local/astro/lib

There is no make install command, but that is easily solved:

$ cp hotpants extractkern maskim /usr/local/astro/bin/

PostgreSQL

Install and configure PostgreSQL

Install the PostgreSQL database, the Python binding, and configure it.

$ yum install postgresql-devel postgresql-server postgresql-contrib

$ pip3.5 install psycopg2

Initialize the database:

$ sudo postgresql-setup initdb

Update the access right:

$ sed -i 's/ident$/md5/' /var/lib/pgsql/data/pg_hba.conf

Start and enable postgres:

$ sudo systemctl start postgresql
$ sudo systemctl enable postgresql

Create a 'goto' user, and a 'photometry' database This needs to be done through the postgresql root user, 'postgres'

$ sudo -i -u
$ createuser -d -P goto
# Pick a password

Now that we have the goto postgres user, and it can create database, we can logout of the postgres account again:

$ logout

Create the photometry database:

$ createdb photometry -O goto -U goto -h localhost

The options are: -O makes 'goto' the database owner; -U logs in as 'goto'; -h localhost forces host-with-md5 connection, instead of the local-with-peer.

Now that we have the database and user, we can install the tables. With the photometry-database.sql:

$ psql -d photometry -U goto -h localhost < photometry-database.sql
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment