This guide shows how to create a WMS server using Apache, MapServer and imagery data sets curated by NASA. This WMS server serves as a model for a production MapServer instance to be deployed on an a network.
The guide contains a brief overview of how the WorldWind clients make requests, followed by instructions for setting up the Apache web server, MapServer and the Apache caching.
WorldWind clients consume imagery data in PNG, JPEG and DDS formats obtained from a MapServer instance via OGC WMS requests. The NASA WorldWind imagery server provides Blue Marble and Landsat7 imagery to WorldWind clients as the core imagery services
- BlueMarble (bm2004xx): 12 months of global BlueMarble imagery used by WorldWind Java and Web WorldWind clients.
- Landsat (esat): Landsat 7 imagery from i-cubed.
WorldWind clients obtain the BlueMarble imagery tiles via GetMap requests for a specific month, May for example:
TODO
WorldWind clients obtain the Landsat imagery tiles via a composite GetMap combining BlueMarble and Landsat request, for example:
TODO!
Install the GDAL tools used to prepare your data for MapServer. MapServer itself also
uses GDAL to serve the tiles in the DDS format (image/dds mime-type) used by WorldWind Java.
The DDS format requires a custom version of GDAL that was compiled with DDS support.
- Download the public key file wwdebs.pub https://files.worldwind.arc.nasa.gov/artifactory/list/debs/wwdebs.pub
- Install the public key with the following command:
sudo apt-key add wwdebs.pub
- Download the repo link file for your OS: Ubuntu 16.4 LTS(xenial), 18.4 LTS(bionic)
- Save the file to
/etc/apt/sources.list.d, make sure the file is owned by root, and it has permissions set to 644 with the following commands:
sudo chown root:root gdalww.list
sudo chmod 644 gdalww.list
- Remove the old gdal-bin (Optional, if applicable)
sudo apt remove gdal-bin libgdal1i
- Update the local packages cache:
sudo apt update
- Check which gdal packages are available from which repo, and which will be installed:
sudo apt policy gdal-bin
- Install the gdal-bin
sudo apt install gdal-bin
- To check the version installed:
apt list -a gdal-bin libgdal1i
Once GDAL is installed, we can use the gdal_translate and gdaladdo commands to convert the original data
to tiled GeoTiffs with compression (optional) and overviews.
For best performance, imagery data should be in a tiled format with overviews. If disk space is a
concern, then the imagery can be compressed with a number of formats with JPEG providing the best compression.
The following bash script will covert input files (*.tifs in the example) to GeoTiffs with JPEG compression. The JPEG compression factors used in script (85) are arbitrary, you can use other values as appropriate.
#!/bin/bash
set +x
# Optimize GeoTiff with JPEG compression and overviews
input_path='/mnt/m/data/imagery/i3'
deflate_path='/mnt/j/prod/imagery/i3/deflate'
optimized_path='/mnt/j/prod/imagery/i3/optimized'
mkdir -p $deflate_path
mkdir -p $optimized_path
for file in ${input_path}/*.tif; do
input_file=${file}
base_file=`basename ${file}`
deflate_file=${deflate_path}/${base_file}
optimized_file=${optimized_path}/${base_file}
if [ -s $deflate_file ]
then
continue
fi
echo Converting $input_file to $deflate_file
gdal_translate $input_file $deflate_file -co tiled=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=DEFLATE -co PREDICTOR=2
echo Adding overviews to $deflate_file
gdaladdo -r average ${deflate_file} 2 4 8 16 32 64 128 --config PHOTOMETRIC_OVERVIEW YCBCR --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 85 --config INTERLEAVE_OVERVIEW PIXEL
if [ -s $optimized_file ]
then
continue
fi
echo Building optimized $optimized_file
gdal_translate $deflate_file $optimized_file -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -co JPEG_QUALITY=85 -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 512 &
done
echo All Done!The ubiquitous Apache web server is used to serve imagery via a MapServer CGI integration. Install the Apache web server package (apache2):
sudo apt-get install apache2Subsequently, adjust your firewall accordingly to allow access to port 80.
You can test the Apache configuration with apachectl. It should return Syntax OK.
sudo apachectl configtestCheck the status of your Apache configuration:
sudo systemctl status apache2Start (or restart) Apache if necessary:
sudo systemctl start apache2sudo systemctl restart apache2Finally, test Apache by entering your server's IP address (or domain name) into your browser's address bar:
http://server_ip_address_or_domainYou should see a default Apache web page.
Configure Apache to run MapServer on the /wms endpoint.
Add the following content to your Apache configuration file (e.g., /etc/apache2/sites-enabled/000-default.conf).
Note that the MS_MAPFILE variable below refers to an imagery.map file at /opt/mapserver/map/. We will create that
in the next configuration step.
Alias /wms /usr/lib/cgi-bin/mapserv
<Location /wms>
SetHandler cgi-script
Options ExecCGI
SetEnv MS_MAPFILE /opt/mapserver/map/imagery.map
</Location>
We're going to configure MapServer to serve RASTER data. See the MapServer [Raster Data]{https://mapserver.org/input/raster.html#raster-data) documentation for more information about what we are accomplishing with the following.
Prepare the folders used by MapServer:
sudo mkdir -p /opt/mapserver/map
sudo mkdir -p /opt/mapserver/map/layers
sudo mkdir -p /opt/mapserver/data
sudo mkdir -p /opt/mapserver/tmp
sudo mkdir -p /opt/mapserver/templatesEnsure the MapServer tmp folder can be written to by Apache:
sudo chown www-data:www-data /opt/mapserver/tmp/Create the imagery.map file and the individual layer files (*.lay) for Landsat and Blue Marble
sudo touch /opt/mapserver/map/imagery.map
sudo touch /opt/mapserver/map/layers/landsat.lay
sudo touch /opt/mapserver/map/layers/bm200401.lay
sudo touch /opt/mapserver/map/layers/bm200402.lay
sudo touch /opt/mapserver/map/layers/bm200403.lay
sudo touch /opt/mapserver/map/layers/bm200404.lay
sudo touch /opt/mapserver/map/layers/bm200405.lay
sudo touch /opt/mapserver/map/layers/bm200406.lay
sudo touch /opt/mapserver/map/layers/bm200407.lay
sudo touch /opt/mapserver/map/layers/bm200408.lay
sudo touch /opt/mapserver/map/layers/bm200409.lay
sudo touch /opt/mapserver/map/layers/bm200410.lay
sudo touch /opt/mapserver/map/layers/bm200411.lay
sudo touch /opt/mapserver/map/layers/bm200412.layEdit the imagery.map file...
sudo nano /opt/mapserver/map/imagery.map... and add the following content. Add your contact information as appropriate.
MAP
NAME ""
STATUS ON
SIZE 800 600
#SYMBOLSET "../etc/symbols.txt"
EXTENT -180 -90 180 90
UNITS DD
SHAPEPATH "../data"
IMAGECOLOR 255 255 255
#FONTSET "../etc/fonts.txt"
#DEBUG 5
#CONFIG "MS_ERRORFILE" "/tmp/ms_error.txt"
WEB
IMAGEPATH "/opt/mapserver/tmp/"
IMAGEURL "/ms_tmp/"
METADATA
"ows_title" "WorldWind Imagery Server"
"ows_abstract" "NASA WorldWind WMS server for imagery"
"ows_onlineresource" "https://worldwind25.arc.nasa.gov/wms"
"ows_enable_request" "*"
"ows_srs" "EPSG:4326 EPSG:4269 EPSG:3857"
"ows_updatesequence" "2014-05-30T16:26:00Z"
"ows_sld_enabled" "false"
"wms_contactperson" "<YOUR NAME>"
"wms_contactorganization" "<YOUR ORG>"
"wms_contactPosition" " "
"wms_contactelectronicmailaddress" "<YOUR EMAIL>"
END
TEMPLATE "../templates/blank.html"
END
#define your output projection
PROJECTION
"init=epsg:4326"
END
#define output formats
OUTPUTFORMAT
NAME "png"
DRIVER AGG/PNG
MIMETYPE "image/png"
IMAGEMODE RGB
EXTENSION "png"
FORMATOPTION "GAMMA=0.75"
END
OUTPUTFORMAT
NAME "GTiff"
DRIVER GDAL/GTiff
MIMETYPE "image/tiff"
IMAGEMODE RGB
EXTENSION "tif"
END
OUTPUTFORMAT
NAME "JPEG2000"
DRIVER "GDAL/JPEG2000"
MIMETYPE "image/jp2k"
IMAGEMODE "RGB"
EXTENSION "jp2"
END
# DDS is not supported without a customized build of GDAL with DDS enabled
#OUTPUTFORMAT
# NAME "DDS"
# DRIVER GDAL/dds
# MIMETYPE "image/dds"
# IMAGEMODE RGBA
# EXTENSION "dds"
# FORMATOPTION "QUALITY=NORMAL" # Should be SUPERFAST, FAST, NORMAL (default), BETTER, UBER
# FORMATOPTION "FORMAT=DXT3" # Should be DXT1, DXT1A, DXT3 (default) or DXT5
#END
#
# Start of layer definitions
#
INCLUDE "layers/bm200401.lay"
INCLUDE "layers/bm200402.lay"
INCLUDE "layers/bm200403.lay"
INCLUDE "layers/bm200404.lay"
INCLUDE "layers/bm200405.lay"
INCLUDE "layers/bm200406.lay"
INCLUDE "layers/bm200407.lay"
INCLUDE "layers/bm200408.lay"
INCLUDE "layers/bm200410.lay"
INCLUDE "layers/bm200411.lay"
INCLUDE "layers/bm200412.lay"
INCLUDE "landsat.lay"
#INCLUDE "earth-at-night.lay"
END # Map File
Establish a tile index for each Blue Marble dataset. We place the indexes within the MapServer's data folder instead of where the image data is stored so that the image data remains portable and shareable. If the imagery is ever moved, just delete the tile indexes and regenerate them for each MapServer instance using the data.
The following example builds the tile index for the January data set (bm200401). You'll repeat the process for each month of the year, changing the month digits in the paths and filenames (bm2004xx where xx = month) for each month.
Create (if needed) and change to the folder used for the BlueMarble tile indexes:
sudo mkdir -p /opt/mapserver/data/bluemarble
cd /opt/mapserver/data/bluemarbleRemove the old January index first if you are recreating, otherwise the indexing operation will append to the existing index.
sudo rm bm20401-index.*Now you will build the tile index with the gdaltindex and create a spatial index for it with shptree.
Replace the /data/imagery/bluemarble/bm200401/gtif/*.tif path with the actual path to your January data.
sudo gdaltindex bm200401-index.shp /data/imagery/bluemarble/200401/gtif/*.tif
sudo shptree bm200401-index.shp
Validate the contents of the Blue Marble January data with ogrinfo:
ogrinfo -al -fields=yes bm200401-index.shpEdit the bm200401.lay file...
sudo nano /opt/mapserver/map/layers/bm200401.lay... and add the following content:
LAYER
PROCESSING "RESAMPLE=BILINEAR"
NAME "BlueMarble-200401"
METADATA
"wms_title" "BlueMarble January 2004"
"wms_abstract" "BlueMarble imagery for January 2004"
"wms_keywordlist" "LastUpdate= 2013-12-12T16:26:00Z"
"wms_opaque" "1"
END
TYPE RASTER
STATUS ON
TILEINDEX "bluemarble/bm200401-index.shp" # path is relative to the SHAPEPATH var
TILEITEM "Location"
TYPE RASTER
# MINSCALEDENOM 1785714
PROJECTION
"init=epsg:4326"
END
END
Repeat the process for Feburary through December. Using the preceding example, you will need to change references to "January" to the correct month, and change "bm200401" to "bm2004xx" where xx is the month number.
Establish a tile index for MapServer on the Landsat imagery
Create (if needed) and change to the folder used for the Landsat tile index:
sudo mkdir -p /opt/mapserver/data/landsat
cd /opt/mapserver/data/landsatRemove the old Landsat index first if you are recreating the index. Skip this step if you want to append to an existing index.
sudo rm landsat-index.*Now you will build the tile index with gdaltindex and create a spatial index for it with shptree.
Replace the /data/imagery/i3/gtif/*.tif path with the actual path to your Landsat data.
sudo gdaltindex landsat-index.shp /data/imagery/i3/gtif/*.tif
sudo shptree landsat-index.shp
Validate the contents of the Landsat index data with ogrinfo:
ogrinfo -al -fields=yes landsat-index.shpEdit the landsat.lay file...
sudo nano /opt/mapserver/map/layers/landsat.lay... and add the following content:
LAYER
PROCESSING "RESAMPLE=BILINEAR"
NAME "esat"
METADATA
"wms_title" "ESAT"
"wms_abstract" "I-Cubed ESAT World Landsat7 Mosaic"
"wms_keywordlist" "LastUpdate= 2013-12-12T16:26:00Z"
"wms_opaque" "1"
END
TYPE RASTER
STATUS ON
TILEINDEX "landsat/landsat-index.shp"
TILEITEM "Location"
TYPE RASTER
# MINSCALEDENOM 53571
PROJECTION
"init=epsg:4326"
END
EXTENT -180 -58 180 82
OFFSITE 0 0 0
END
Using the preceeding examples you can craft tile-indexes and layer files for other datasets and add them to your map file.
Generate a WMS GetCapabilities request by entering the following http or https URL in your browser, using your server's ip address or domain:
http://server_ip_address_or_domain/wms?service=WMS&request=GetCapabilities
You should get an XML document similar to the elided example below. Examine the <Capability/> section for the existance of <Layer/> entries for all the layers that you defined in the configuration.
<WMS_Capabilities xmlns="http://www.opengis.net/wms" xmlns:sld="http://www.opengis.net/sld" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ms="http://mapserver.gis.umn.edu/mapserver" version="1.3.0" updateSequence="2015-02-27T16:26:00Z" xsi:schemaLocation="http://www.opengis.net/wms http://schemas.opengis.net/wms/1.3.0/capabilities_1_3_0.xsd http://www.opengis.net/sld http://schemas.opengis.net/sld/1.1.0/sld_capabilities.xsd http://mapserver.gis.umn.edu/mapserver https://worldwind26.arc.nasa.gov/wms?service=WMS&version=1.3.0&request=GetSchemaExtension">
<!--
MapServer version 7.0.0 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE
-->
<Service>
<Name>WMS</Name>
<Title>WorldWind Imagery Server</Title>
<Abstract>NASA WorldWind WMS Service</Abstract>
<OnlineResource xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://worldwind26.arc.nasa.gov/wms?"/>
<ContactInformation>
<ContactPersonPrimary>
<ContactPerson>Randolph Kim</ContactPerson>
<ContactOrganization>NASA</ContactOrganization>
</ContactPersonPrimary>
<ContactPosition> </ContactPosition>
<ContactElectronicMailAddress>rkim@mail.arc.nasa.gov</ContactElectronicMailAddress>
</ContactInformation>
<MaxWidth>2048</MaxWidth>
<MaxHeight>2048</MaxHeight>
</Service>
<Capability>...</Capability>
</WMS_Capabilities>Generate a WMS GetMap request to ensure a response code of 200 and Content-Type of application/bil16. Using a browser with the development tools open on the network tab, enter the following URL and examine the response headers.
TODO
Caching is essential to ensure that costly, common global requests do not hamper system performance.
Select instructions extracted from How To Configure Apache Content Caching on Ubuntu by Digital Ocean.
The HTTP caching logic is available through the mod_cache module. The actual caching is done with one of the caching providers. Typically, the cache is stored on disk using the mod_cache_disk module, but shared object caching is also available through the mod_cache_socache module.
The mod_cache_disk module caches on disk, so it can be useful if you are proxying content from a remote location, generating it from a dynamic process, or just trying to speed things up by caching on a faster disk than your content typically resides on. This is the most well-tested provider and should probably be your first choice in most cases. The cache is not cleaned automatically, so a tool called
htcachecleanmust be run occasionally to slim down the cache. This can be run manually, set up as a regular cron job, or run as a daemon.
In order to enable caching, you'll need to enable the mod_cache module as well as one of its caching providers. As we stated above, mod_cache_disk is well tested, so we will rely on that.
On an Ubuntu system, you can enable these modules by typing:
sudo a2enmod cache
sudo a2enmod cache_disk
The mod_expires module can set both the Expires header and the max-age option in the Cache-Control header. The mod_headers module can be used to add more specific Cache-Control options to tune the caching policy further. You can enable these modules by typing:
sudo a2enmod expires
sudo a2enmod headers
You will also need to install the apache2-utils package, which contains the
htcachecleanutility used to pare down the cache when necessary. You can install this by typing:
sudo apt-get update
sudo apt-get install apache2-utils
Most of the configuration for caching will take place within individual virtual host definitions or location blocks. However, enabling mod_cache_disk also enables a global configuration that can be used to specify some general attributes. Open that file now to take a look:
sudo nano /etc/apache2/mods-enabled/cache_disk.conf
With the comments removed, the file should look like this:
<IfModule mod_cache_disk.c>
CacheRoot /var/cache/apache2/mod_cache_disk
CacheDirLevels 2
CacheDirLength 1
</IfModule>
We'll use the default values for now.
Open your virtual host file(s) for the imagery server. For example:
sudo nano /etc/apache2/sites-enabled/000-default.conf
Add the Apache caching configuration, as follows:
For example:
To start leave the CacheQuickHandler off for complete processing of caching rules:
CacheQuickHandler off
Setup a locking mechanism based on Apache docs:
CacheLock on
CacheLockPath /tmp/mod_cache-lock
CacheLockMaxAge 5
Don't store cookies in the cache to prevent leaking of user-specific cookies
CacheIgnoreHeaders Set-Cookie
Web WorldWind requests require CacheIgnoreCacheControl to be enabled to obtain cache hits. This tells the server to attempt to serve the resource from the cache even if the request contains no-cache header values.
CacheIgnoreCacheControl On
Now we'll enable caching for the /wms endpoint with a number of directives. CacheEnable disk defines the caching implemenation. CacheHeader on enables a reponse header that will indicate whether there was a cache hit or miss. Another directive we'll set is CacheDefaultExpire so that we can set an expiration (in seconds) if neither the Expires nor the Last-Modified headers are set on the content. Similarly, we'll set CacheMaxExpire to cap the amount of time items will be saved. We'll set the CacheLastModifiedFactor so that Apache can create an expiration date if it has a Last-Modified date, but no expiration. The factor is multiplied by the time since modification to set a reasonable expiration.
The ExpiresActive on enables expiration processing. The ExpiresDefault directive sets the default expiration time. These will set the Expires and the Cache-Control "max-age" to the correct values. When you are certain the caching is working as desired, you can extend the expiration time.
Within the <Location /wms> block, add the following cache directives:
CacheEnable disk
CacheHeader on
CacheDefaultExpire 600
CacheMaxExpire 86400
CacheLastModifiedFactor 0.5
ExpiresActive on
ExpiresDefault "access plus 1 week"
Header merge Cache-Control public
Your edited virtual host .conf file should look like this:
<VirtualHost *:80>
ServerAdmin webmaster@localhost
.
.
.
# Apache caching configuration
CacheQuickHandler off
CacheLock on
CacheLockPath /tmp/mod_cache-lock
CacheLockMaxAge 5
CacheIgnoreHeaders Set-Cookie
CacheIgnoreCacheControl On
# MapServer /wms endpoint
Alias /wms /usr/lib/cgi-bin/mapserv
<Location /wms>
CacheEnable disk /wms
CacheHeader on
CacheDefaultExpire 600
CacheMaxExpire 86400
CacheLastModifiedFactor 0.5
ExpiresActive on
ExpiresDefault "access plus 1 week"
Header merge Cache-Control public
SetHandler cgi-script
Options ExecCGI
SetEnv MS_MAPFILE /opt/mapserver/map/imagery.map
</Location>
.
.
.
</VirtualHost>
htcacheclean (installed by apache2-utils) is used to manage the cache. Following are a few examples of its use.
The following command displays the contents of the cache. The -p switch specifies the cache location; the -a (or -A) dumps the contents.
sudo htcacheclean -p /var/cache/apache2/mod_cache_disk/ -aThe following command manually cleans the cache and ensure the size is not larger than 100MB. The -l switch specifies the resulting cache size; the -v displays verbose results.
sudo htcacheclean -p /var/cache/apache2/mod_cache_disk/ -l 100M -vThis command runs the cache cleanup in a daemon:
htcacheclean -d30 -n -t -p /var/cache/apache2/mod_disk_cache -l 100M -i
This will clean our cache directory every 30 minutes and make sure that it will not get bigger than 100MB. To learn more about htcacheclean, take a look at
man htcacheclean
The apache2-utils install may have already installed the apache-htcacheclean service. Examine the status and runtime parameters of the service with systemctl.
sudo systemctl status apache-htcacheclean
To change the service's runtime parameters, tdit the file /etc/default/apache-htcacheclean and change the default values. Start or stop the service with systemctl as required for your installation.
See: https://www.howtoforge.com/caching-with-apache-mod_cache-on-debian-etch
We can use fail2ban to help prevent an inadvertent denial-of-service attacks caused by bulk downloads.
Install fail2ban with the package manager:
sudo apt-get install fail2ban
Create and open the http-get-dos.conf for editing...
sudo touch /etc/fail2ban/filter.d/http-get-dos.conf
sudo nano /etc/fail2ban/filter.d/http-get-dos.conf
... and add the following content:
# Fail2Ban configuration file
#
# Author: http://www.go2linux.org
#
[Definition]
# Option: failregex
# Note: This regex will match any GET entry in your logs, so basically all valid and not valid entries are a match.
# You should set up in the jail.conf file, the maxretry and findtime carefully in order to avoid false positives.
failregex = ^<HOST> -.*GET
# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =
Create a local copy of the fail2ban configuration file and open it for editing:
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local
--- and add the following content:
[http-get-dos]
# Simple attempt to block very basic DOS attacks over GET:
# If [maxRetry] requests occur by an ip within [findtime] secs, then bans the ip for [bantime] secs.
enabled = true
port = http,https
filter = http-get-dos
logpath = /var/log/apache2/access.log
maxRetry = 100
findtime = 30
bantime = 10
The configuration above was used for testing fail2ban. The actual values you use for maxRetry, findtime and bantime will have to be figured out with testing.
You can check the validity of the http-get-doc jail's regex expression against Apache's access.log with this command:
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/http-get-dos.conf
You can check the status of the http-get-dos jail with this command:
sudo fail2ban-client status http-get-dos
You must restart the fail2ban service after making changes to the jail.local file.
sudo systemctl restart fail2ban