Installing GDAL 3.2.1 on Amazon Linux 2

README.md

Installing GDAL 3.2.1 on Amazon Linux 2

As of this day, this is probably the only and fastest way of installing it.

Package requirements

Based from the GDAL and PROJ build requirements, here is the full list of required packages to install:

• C++11 - cpp version 7.3.1
• PROJ 6 - proj version 6.1.1
  • SQLite3 - sqlite3 version 3.7.17
  • libtiff - version 4.0
  • cmake - version 3.13.1

Installation of these packages are covered in the sections below. However, before you proceed with the installation, make sure you are already logged in to your EC2 Amazon Linux 2 via SSH. After logging in, you just need to issue some statements in the command-line.

1. Installation of required packages

Luckilly, we don't have to install everything from source. At least some can be installed through yum.

sudo yum install gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64 -y

2. Installation of PROJ

Installation from source, currently there's no other way but this.

cd /tmp
wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
tar -xvf proj-6.1.1.tar.gz
cd proj-6.1.1
./configure
sudo make
sudo make install

Important

Be careful not to install PROJ version 6.2.0 or newer because the one from the yum packages is only SQLite version 3.7.17. Installing PROJ version 6.2.0 would require installing SQLite version 3.11 which you may need to install from source and not with yum.

3. Installation of GDAL

Installation from source, currently there's no other way but this.

cd /tmp
wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
tar -xvf gdal-3.2.1.tar.gz
cd gdal-3.2.1
./configure --with-proj=/usr/local --with-python
sudo make
sudo make install

Testing your installation

which gdalinfo; gdalinfo --version

It should show something like

/usr/local/bin/gdalinfo
GDAL 3.2.1, released 2020/12/29

Comments and suggestions

Did I miss something? Are you having some troubles with the installed GDAL? Write them down as comments below. Let's help our selves and others at the same time.

@mahmutbilgen @RichardScottOZ You may also try installing newer versions of GDAL. Essentially doing steps 1 & 2 as it is and then opt for a newer GDAL version on step 3.

Thank you!

Thank you!

Yes, if can get the development libraries to support for that particular repository, or have to add that too.

Thanks so much for the guideline! Though, I ran into some error when trying to make gdal, stating that fatal error: Python.h: No such file or directory. Turns out python-devel and python3-devel also required.

i have this error
install: .libs/libgdal.lai does not exist

This is a elastic beanstalk configuration file. Added within .ebextensions as 01-gdal.config:

commands:
  01_install_gdal:
    test: "[ ! -d /usr/local/gdal ]"
    command: "/tmp/gdal_install.sh"
files:
  "/tmp/gdal_install.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      sudo yum -y update
      sudo yum -y upgrade
      sudo amazon-linux-extras install epel -y
      sudo yum -y install gcc-c++ cpp sqlite-devel libtiff cmake3
      cd /tmp
      wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
      tar -xvf proj-6.1.1.tar.gz
      cd proj-6.1.1
      ./configure
      sudo make
      sudo make install
      cd /tmp
      wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
      tar -xvf gdal-3.2.1.tar.gz
      cd gdal-3.2.1
      ./configure --with-proj=/usr/local --with-python
      sudo make
      sudo make install
      which gdalinfo; gdalinfo --version

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

@filipkubala-th-ey IDK the line which fails... Look deeper within the logs

For the ones that are using this inside a docker image and want a straightforward solution, I strongly recommend you to simply use this image as a builder and simply copy the files inside your own image. Here the tutorial is creating an AWS Lambda layer, that may not be relevant at all for most of you, but if you copy the first steps changing "lambci/lambda:build-python3.8" image for your own base image, that will work magically.

@vrbrito consider 2 limitations of using it on lambda:

1. 500 MB of storage
2. 15 min of execution time
   I was able to overcome the 500 MB limit by sending the tiles directly to the S3 bucket after generation and then, if needed, read it from there, but unfortunately, I was unable to overcome 15 minutes of execution time, for the 5.5 GB tif input.
   Were you able to overcome this problem? Or were you able to chain the lambdas and parallel the generation?
   Thanks :)

BTW. I was able to overcome all the issues, switching from Amazon Linux 2 to Ubuntu virtual machine - if you have this option, maybe it's the way to go.

It worked fine even when replacing with specific newer version "3.4.2" on step 3.

Getting it run on python3 took some trial and error though. Here are the steps as far as I can remember in case anyone attempts the same.

• Had to also run "yum install python3-devel"
• Installed specific module version with "pip3 install gdal==3.4.2"
• Had to append "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PATH since python could not find the relevant .so files otherwise

Hello! Thank you so much for the guidance. But I still have some issues on python3 (I am using python 3.9). Following @JolanenL comment,

• I run "yum install python3-devel"
• I did "pip3.9 install gdal==3.2.1" then got errors 'use_2to3_fixers', saying "error in GDAL setup command: use_2to3 is invalid."
• I appended "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PAT but got the same errors from 2.

Any suggestions? Thank you so much in advance.

Hello again! I wandered for a while, then find
$ export PYTHONPATH="/usr/local/lib/python3.9/site-packages/"
$ python3.9 -c "from osgeo import gdal"
then GDAL worked fine! Thank you so much!

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

Finally I left the idea of using Amazon Elastic Beanstalk and I turned to using AWS ECR and ECS, dockerizing my python/django project. If anybody need help feel free to ask :)

Hello again and again... After threw the comment above, I noticed that gdal did not work fine.

from osgeo import gdal
import numpy as np
ds = gdal.Open('cea.tif', gdal.GA_ReadOnly)
ERROR 4: cea.tif: No such file or directory
ds = gdal.Open('/home/yam-ma/cea.tif', gdal.GA_ReadOnly)
a = np.array([ds.GetRasterBand(i + 1).ReadAsArray() for i in range(ds.RasterCount)])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'osgeo._gdal_array'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal.py", line 3832, in ReadAsArray
from osgeo import gdalnumeric
File "/usr/local/lib/python3.9/site-packages/osgeo/gdalnumeric.py", line 2, in
from osgeo.gdal_array import *
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 17, in
_gdal_array = swig_import_helper()
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 16, in swig_import_helper
return importlib.import_module('_gdal_array')
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal_array'

Does anyone know how to solve the issue? Thank you!

I had to add these to my Dockerfile to get it to work:

RUN pip install "setuptools<58.0.0" (before installing GDAL)
RUN cp /usr/local/lib/libgdal.so.28* /usr/lib64/ (after installing GDAL)

Now I can run "from osgeo import gdal" in a python3 shell.

I needed to install this for some R packages to work, sf, rnaturalearth, and leaflet

@ababaian, How/where did you install the R packages after getting all these dependencies installed? I am trying to run R on EMR but having trouble to install geospatial packages ('sf', 'terra', 'lidR').

@diego-gris here's my docker where it's working (last I checked).

https://github.com/ababaian/palmid/blob/main/Dockerfile

when configuring gdal i had to add the flags

--with-libpng --with-libpng-devel

to make gdal link with libpng and find some png symbols