Skip to content

Instantly share code, notes, and snippets.

@Beak-man
Forked from emxsys/imagery_server.md
Last active April 17, 2019 16:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Beak-man/d97811068594075cd776c7ac85aab90e to your computer and use it in GitHub Desktop.
Save Beak-man/d97811068594075cd776c7ac85aab90e to your computer and use it in GitHub Desktop.
How to setup a NASA imagery server

How to setup a NASA WorldWind imagery server

Introduction

This guide shows how to create a WMS server using Apache, MapServer and imagery data sets curated by NASA. This WMS server serves as a model for a production MapServer instance to be deployed on a network.

The guide contains a brief overview of how the WorldWind clients make requests, followed by instructions for setting up the Apache web server, MapServer and the Apache caching.

Overview

WorldWind clients consume imagery data in PNG, JPEG and DDS formats obtained from a MapServer instance via OGC WMS requests. The NASA WorldWind imagery server provides Blue Marble and Landsat7 imagery to WorldWind clients as the core imagery services

  • BlueMarble (bm2004xx): 12 months of global BlueMarble imagery used by WorldWind Java and Web WorldWind clients.
  • Landsat (esat): Landsat 7 imagery from i-cubed.

WorldWind clients obtain the BlueMarble imagery tiles via GetMap requests for a specific month, May for example:

https://worldwind25.arc.nasa.gov/wms?service=WMS&request=GetMap&version=1.3.0&transparent=TRUE&layers=BlueMarble-200405&styles=&format=image/jpeg&width=256&height=256&crs=EPSG:4326&bbox=-45,0,0,45

WorldWind clients obtain the Landsat imagery tiles via a composite GetMap combining BlueMarble and Landsat request, for example:

https://worldwind25.arc.nasa.gov/wms?service=WMS&request=GetMap&version=1.3.0&transparent=TRUE&layers=BlueMarble-200405,esat&styles=&format=image/jpeg&width=256&height=256&crs=EPSG:4326&bbox=0,0,45,45

Step 1: Prepare your data

Install GDAL

Install the GDAL tools used to prepare your data for MapServer. MapServer itself also uses GDAL to serve the tiles in the DDS format (image/dds mime-type) used by WorldWind Java. The DDS format requires a custom version of GDAL that was compiled with DDS support.

Install the repo public key

  1. Download the public key file wwdebs.pub https://files.worldwind.arc.nasa.gov/artifactory/list/debs/wwdebs.pub
  2. Install the public key with the following command:
sudo apt-key add wwdebs.pub

Add the Repo link file to apt

  1. Download the repo link file for your OS: Ubuntu 16.4 LTS(xenial), 18.4 LTS(bionic)
  2. Save the file to  /etc/apt/sources.list.d, make sure the file is owned by root, and it has permissions set to 644 with the following commands:
sudo chown root:root gdalww.list
sudo chmod 644 gdalww.list

Example:  Installing WorldWind's DDS-enabled GDAL for Ubuntu 16.4

  1. Remove the old gdal-bin (Optional, if applicable)
sudo apt remove gdal-bin libgdal1i
  1. Update the local packages cache:
sudo apt update
  1. Check which gdal packages are available from which repo, and which will be installed:
sudo apt policy gdal-bin
  1. Install the gdal-bin
sudo apt install gdal-bin
  1. To check the version installed:
apt list -a gdal-bin libgdal1i

Prepare imagery

Once GDAL is installed, we can use the gdal_translate and gdaladdo commands to convert the original data to tiled GeoTiffs with compression (optional) and overviews. For best performance, imagery data should be in a tiled format with overviews. If disk space is a concern, then the imagery can be compressed with a number of formats with JPEG providing the best compression.

The following bash script will covert input files (*.tifs in the example) to GeoTiffs with JPEG compression. The JPEG compression factors used in script (85) are arbitrary, you can use other values as appropriate.

#!/bin/bash
set +x

# Optimize GeoTiff with JPEG compression and overviews

input_path='/mnt/m/data/imagery/i3'
deflate_path='/mnt/j/prod/imagery/i3/deflate'
optimized_path='/mnt/j/prod/imagery/i3/optimized'

mkdir -p $deflate_path
mkdir -p $optimized_path

for file in ${input_path}/*.tif; do 
    input_file=${file}
    base_file=`basename ${file}`
    deflate_file=${deflate_path}/${base_file}
    optimized_file=${optimized_path}/${base_file}
    if [ -s $deflate_file ]
    then
        continue
    fi 
    
    echo Converting $input_file to $deflate_file
    gdal_translate $input_file $deflate_file -co tiled=yes -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=DEFLATE -co PREDICTOR=2   
    
    echo Adding overviews to $deflate_file
    gdaladdo -r average ${deflate_file} 2 4 8 16 32 64 128 --config PHOTOMETRIC_OVERVIEW YCBCR --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 85 --config INTERLEAVE_OVERVIEW PIXEL 

    if [ -s $optimized_file ]
    then
        continue
    fi     

    echo Building optimized $optimized_file
    gdal_translate $deflate_file $optimized_file -co TILED=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co COMPRESS=JPEG -co JPEG_QUALITY=85 -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES --config GDAL_TIFF_OVR_BLOCKSIZE 512 &
       
done
echo All Done!

Step 2: Setup Apache

The ubiquitous Apache web server is used to serve imagery via a MapServer CGI integration. Install the Apache web server package (apache2):

sudo apt-get install apache2

Subsequently, adjust your firewall accordingly to allow access to port 80.

You can test the Apache configuration with apachectl. It should return Syntax OK.

sudo apachectl configtest

Check the status of your Apache configuration:

sudo systemctl status apache2

Start (or restart) Apache if necessary:

sudo systemctl start apache2
sudo systemctl restart apache2

Finally, test Apache by entering your server's IP address (or domain name) into your browser's address bar:

http://server_ip_address_or_domain

You should see a default Apache web page.

Step 3: Setup MapServer

The MapServer CGI is integrated with Apache via the package install.

Install MapServer

Install the MapServer packages for Apache:

sudo apt-get install cgi-mapserver mapserver-bin

Configure Apache to run MapServer CGI

Configure Apache to run MapServer on the /wms endpoint. Add the following content to your Apache configuration file (e.g., /etc/apache2/sites-enabled/000-default.conf).
Note that the MS_MAPFILE variable below refers to an imagery.map file at /opt/mapserver/map/. We will create that in the next configuration step.

Alias /wms /usr/lib/cgi-bin/mapserv
<Location /wms>
	SetHandler cgi-script
	Options ExecCGI
	SetEnv MS_MAPFILE /opt/mapserver/map/imagery.map
</Location>

Configure MapServer to serve imagery

We're going to configure MapServer to serve RASTER data. See the MapServer Raster Data documentation for more information about what we are accomplishing with the following.

Prepare the folders used by MapServer:

sudo mkdir -p /opt/mapserver/map
sudo mkdir -p /opt/mapserver/map/layers
sudo mkdir -p /opt/mapserver/data
sudo mkdir -p /opt/mapserver/tmp
sudo mkdir -p /opt/mapserver/templates

Ensure the MapServer tmp folder can be written to by Apache:

sudo chown www-data:www-data /opt/mapserver/tmp/

Create the imagery.map file and the individual layer files (*.lay) for Landsat and Blue Marble

sudo touch /opt/mapserver/map/imagery.map
sudo touch /opt/mapserver/map/layers/landsat.lay
sudo touch /opt/mapserver/map/layers/bm200401.lay
sudo touch /opt/mapserver/map/layers/bm200402.lay
sudo touch /opt/mapserver/map/layers/bm200403.lay
sudo touch /opt/mapserver/map/layers/bm200404.lay
sudo touch /opt/mapserver/map/layers/bm200405.lay
sudo touch /opt/mapserver/map/layers/bm200406.lay
sudo touch /opt/mapserver/map/layers/bm200407.lay
sudo touch /opt/mapserver/map/layers/bm200408.lay
sudo touch /opt/mapserver/map/layers/bm200409.lay
sudo touch /opt/mapserver/map/layers/bm200410.lay
sudo touch /opt/mapserver/map/layers/bm200411.lay
sudo touch /opt/mapserver/map/layers/bm200412.lay

Edit the imagery.map file...

sudo nano /opt/mapserver/map/imagery.map

... and add the following content. Add your contact information as appropriate.

MAP

  NAME ""
  STATUS ON
  SIZE 800 600
  #SYMBOLSET "../etc/symbols.txt"
  EXTENT -180 -90 180 90
  UNITS DD
  SHAPEPATH "../data"
  IMAGECOLOR 255 255 255
  #FONTSET "../etc/fonts.txt"
  
  #DEBUG 5
  #CONFIG "MS_ERRORFILE" "/tmp/ms_error.txt"


  WEB
    IMAGEPATH "/opt/mapserver/tmp/" 
    IMAGEURL "/ms_tmp/"
    METADATA
      "ows_title"           "WorldWind Imagery Server"
      "ows_abstract"        "NASA WorldWind WMS server for imagery"
      "ows_onlineresource"  "https://worldwind25.arc.nasa.gov/wms"
      "ows_enable_request"  "*"
      "ows_srs"             "EPSG:4326 EPSG:4269 EPSG:3857"
      "ows_updatesequence"  "2014-05-30T16:26:00Z"
      "ows_sld_enabled"     "false"
      "wms_contactperson"   "<YOUR NAME>"
      "wms_contactorganization" "<YOUR ORG>"
      "wms_contactPosition" " "
      "wms_contactelectronicmailaddress" "<YOUR EMAIL>" 
    END
    TEMPLATE "../templates/blank.html"
  END

  #define your output projection
  PROJECTION
    "init=epsg:4326"
  END

  #define output formats
  OUTPUTFORMAT
    NAME "png"
    DRIVER AGG/PNG
    MIMETYPE "image/png"
    IMAGEMODE RGB
    EXTENSION "png"
    FORMATOPTION "GAMMA=0.75"
  END

  OUTPUTFORMAT
    NAME "GTiff"
    DRIVER GDAL/GTiff
    MIMETYPE "image/tiff"
    IMAGEMODE RGB
    EXTENSION "tif"
  END

  OUTPUTFORMAT
    NAME "JPEG2000"
    DRIVER "GDAL/JPEG2000"
    MIMETYPE "image/jp2k"
    IMAGEMODE "RGB"
    EXTENSION "jp2"
  END

  # DDS is not supported without a customized build of GDAL with DDS enabled
  #OUTPUTFORMAT
  #  NAME "DDS"
  #  DRIVER GDAL/dds
  #  MIMETYPE "image/dds"
  #  IMAGEMODE RGBA
  #  EXTENSION "dds"
  #  FORMATOPTION "QUALITY=NORMAL" # Should be SUPERFAST, FAST, NORMAL (default), BETTER, UBER
  #  FORMATOPTION "FORMAT=DXT3" # Should be DXT1, DXT1A, DXT3 (default) or DXT5
  #END

  #
  # Start of layer definitions
  #

  INCLUDE "layers/bm200401.lay"
  INCLUDE "layers/bm200402.lay"
  INCLUDE "layers/bm200403.lay"
  INCLUDE "layers/bm200404.lay"
  INCLUDE "layers/bm200405.lay"
  INCLUDE "layers/bm200406.lay"
  INCLUDE "layers/bm200407.lay"
  INCLUDE "layers/bm200408.lay"
  INCLUDE "layers/bm200410.lay"
  INCLUDE "layers/bm200411.lay"
  INCLUDE "layers/bm200412.lay"

  INCLUDE "landsat.lay"

  #INCLUDE "earth-at-night.lay"

END # Map File

Configure Blue Marble layers

January Imagery

Establish a tile index for each Blue Marble dataset. We place the indexes within the MapServer's data folder instead of where the image data is stored so that the image data remains portable and shareable. If the imagery is ever moved, just delete the tile indexes and regenerate them for each MapServer instance using the data.

The following example builds the tile index for the January data set (bm200401). You'll repeat the process for each month of the year, changing the month digits in the paths and filenames (bm2004xx where xx = month) for each month.

Create (if needed) and change to the folder used for the BlueMarble tile indexes:

sudo mkdir -p /opt/mapserver/data/bluemarble
cd /opt/mapserver/data/bluemarble

Remove the old January index first if you are recreating, otherwise the indexing operation will append to the existing index.

sudo rm bm20401-index.*

Now you will build the tile index with the gdaltindex and create a spatial index for it with shptree. Replace the /data/imagery/bluemarble/bm200401/gtif/*.tif path with the actual path to your January data.

sudo gdaltindex bm200401-index.shp /data/imagery/bluemarble/200401/gtif/*.tif
sudo shptree bm200401-index.shp

Validate the contents of the Blue Marble January data with ogrinfo:

ogrinfo -al -fields=yes bm200401-index.shp

Edit the bm200401.lay file...

sudo nano /opt/mapserver/map/layers/bm200401.lay

... and add the following content:

LAYER
  PROCESSING "RESAMPLE=BILINEAR"
  NAME "BlueMarble-200401"
  METADATA
    "wms_title"          "BlueMarble January 2004"
    "wms_abstract"       "BlueMarble imagery for January 2004"
    "wms_keywordlist"    "LastUpdate= 2013-12-12T16:26:00Z"
    "wms_opaque"         "1"
  END
  TYPE RASTER
  STATUS ON 
  TILEINDEX "bluemarble/bm200401-index.shp"  # path is relative to the SHAPEPATH var
  TILEITEM "Location"
  TYPE RASTER
#  MINSCALEDENOM 1785714
  PROJECTION
    "init=epsg:4326"
  END
END

Repeat for the remaining months

Repeat the process for February through December. Using the preceding example, you will need to change references to "January" to the correct month, and change "bm200401" to "bm2004xx" where xx is the month number.

Configure Landsat layer

Establish a tile index for MapServer on the Landsat imagery

Create (if needed) and change to the folder used for the Landsat tile index:

sudo mkdir -p /opt/mapserver/data/landsat
cd /opt/mapserver/data/landsat

Remove the old Landsat index first if you are recreating the index. Skip this step if you want to append to an existing index.

sudo rm landsat-index.*

Now you will build the tile index with gdaltindex and create a spatial index for it with shptree. Replace the /data/imagery/i3/gtif/*.tif path with the actual path to your Landsat data.

sudo gdaltindex landsat-index.shp /data/imagery/i3/gtif/*.tif
sudo shptree landsat-index.shp

Validate the contents of the Landsat index data with ogrinfo:

ogrinfo -al -fields=yes landsat-index.shp

Edit the landsat.lay file...

sudo nano /opt/mapserver/map/layers/landsat.lay

... and add the following content:

LAYER
  PROCESSING "RESAMPLE=BILINEAR" 
  NAME "esat"
  METADATA
    "wms_title"          "ESAT"
    "wms_abstract"       "I-Cubed ESAT World Landsat7 Mosaic"
    "wms_keywordlist"    "LastUpdate= 2013-12-12T16:26:00Z" 
    "wms_opaque"         "1"
  END
  TYPE RASTER
  STATUS ON 
  TILEINDEX "landsat/landsat-index.shp"
  TILEITEM "Location"
  TYPE RASTER
#  MINSCALEDENOM 53571 
  PROJECTION
    "init=epsg:4326"
  END
  EXTENT -180 -58 180 82
  OFFSITE 0 0 0
END

Add other layers as required

Using the preceding examples you can craft tile-indexes and layer files for other datasets and add them to your map file.

Test the configurations

WMS GetCapabilties Test

Generate a WMS GetCapabilities request by entering the following http or https URL in your browser, using your server's IP address or domain:

http://server_ip_address_or_domain/wms?service=WMS&request=GetCapabilities

You should get an XML document similar to the elided example below. Examine the <Capability/> section for the existence of <Layer/> entries for all the layers that you defined in the configuration.

<WMS_Capabilities xmlns="http://www.opengis.net/wms" xmlns:sld="http://www.opengis.net/sld" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ms="http://mapserver.gis.umn.edu/mapserver" version="1.3.0" updateSequence="2015-02-27T16:26:00Z" xsi:schemaLocation="http://www.opengis.net/wms http://schemas.opengis.net/wms/1.3.0/capabilities_1_3_0.xsd http://www.opengis.net/sld http://schemas.opengis.net/sld/1.1.0/sld_capabilities.xsd http://mapserver.gis.umn.edu/mapserver https://worldwind26.arc.nasa.gov/wms?service=WMS&version=1.3.0&request=GetSchemaExtension">
 <!--
 MapServer version 7.0.0 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS SUPPORTS=GEOS INPUT=JPEG INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE 
-->
	<Service>
		<Name>WMS</Name>
		<Title>WorldWind Imagery Server</Title>
		<Abstract>NASA WorldWind WMS Service</Abstract>
		<OnlineResource xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://worldwind26.arc.nasa.gov/wms?"/>
		<ContactInformation>
			<ContactPersonPrimary>
				<ContactPerson>Randolph Kim</ContactPerson>
				<ContactOrganization>NASA</ContactOrganization>
			</ContactPersonPrimary>
			<ContactPosition> </ContactPosition>
			<ContactElectronicMailAddress>rkim@mail.arc.nasa.gov</ContactElectronicMailAddress>
		</ContactInformation>
		<MaxWidth>2048</MaxWidth>
		<MaxHeight>2048</MaxHeight>
	</Service>
	<Capability>...</Capability>
</WMS_Capabilities>

WMS GetMap Test

Generate a WMS GetMap request to ensure a response code of 200 and Content-Type of application/bil16. Using a browser with the development tools open on the network tab, enter the following URL and examine the response headers.

http://server_ip_address_or_domain/wms?service=WMS&request=GetMap&version=1.3.0&layers=BlueMarble-200405&format=image/jpeg&width=256&height=256&crs=EPSG:4326&bbox=-45,0,0,45

Step 4: Setup Caching

Caching is essential to ensure that costly, common global requests do not hamper system performance.

Select instructions extracted from How To Configure Apache Content Caching on Ubuntu by Digital Ocean.

Standard HTTP Caching

The HTTP caching logic is available through the mod_cache module. The actual caching is done with one of the caching providers. Typically, the cache is stored on disk using the mod_cache_disk module, but shared object caching is also available through the mod_cache_socache module.

The mod_cache_disk module caches on disk, so it can be useful if you are proxying content from a remote location, generating it from a dynamic process, or just trying to speed things up by caching on a faster disk than your content typically resides on. This is the most well-tested provider and should probably be your first choice in most cases. The cache is not cleaned automatically, so a tool called htcacheclean must be run occasionally to slim down the cache. This can be run manually, set up as a regular cron job, or run as a daemon.

Enable Apache cache and cache_disk mods

In order to enable caching, you'll need to enable the mod_cache module as well as one of its caching providers. As we stated above, mod_cache_disk is well tested, so we will rely on that.

On an Ubuntu system, you can enable these modules by typing:

sudo a2enmod cache
sudo a2enmod cache_disk

The mod_expires module can set both the Expires header and the max-age option in the Cache-Control header. The mod_headers module can be used to add more specific Cache-Control options to tune the caching policy further. You can enable these modules by typing:

sudo a2enmod expires
sudo a2enmod headers

You will also need to install the apache2-utils package, which contains the htcacheclean utility used to pare down the cache when necessary. You can install this by typing:

sudo apt-get update
sudo apt-get install apache2-utils

Modifying the Global Configuration

Most of the configuration for caching will take place within individual virtual host definitions or location blocks. However, enabling mod_cache_disk also enables a global configuration that can be used to specify some general attributes. Open that file now to take a look:

sudo nano /etc/apache2/mods-enabled/cache_disk.conf

With the comments removed, the file should look like this:

<IfModule mod_cache_disk.c>
    CacheRoot /var/cache/apache2/mod_cache_disk
    CacheDirLevels 2
    CacheDirLength 1
</IfModule>

We'll use the default values for now.

Modifying the Virtual Server

Open your virtual host file(s) for the imagery server. For example:

sudo nano /etc/apache2/sites-enabled/000-default.conf

Add the Apache caching configuration, as follows:

To start leave the CacheQuickHandler off for complete processing of caching rules:

        CacheQuickHandler off

Setup a locking mechanism based on Apache docs:

        CacheLock on
        CacheLockPath /tmp/mod_cache-lock
        CacheLockMaxAge 5

Don't store cookies in the cache to prevent leaking of user-specific cookies

        CacheIgnoreHeaders Set-Cookie

Web WorldWind requests require CacheIgnoreCacheControl to be enabled to obtain cache hits. This tells the server to attempt to serve the resource from the cache even if the request contains no-cache header values.

        CacheIgnoreCacheControl On

Now we'll enable caching for the /wms endpoint with a number of directives. CacheEnable disk defines the caching implemenation. CacheHeader on enables a reponse header that will indicate whether there was a cache hit or miss. Another directive we'll set is CacheDefaultExpire so that we can set an expiration (in seconds) if neither the Expires nor the Last-Modified headers are set on the content. Similarly, we'll set CacheMaxExpire to cap the amount of time items will be saved. We'll set the CacheLastModifiedFactor so that Apache can create an expiration date if it has a Last-Modified date, but no expiration. The factor is multiplied by the time since modification to set a reasonable expiration.

The ExpiresActive on enables expiration processing. The ExpiresDefault directive sets the default expiration time. These will set the Expires and the Cache-Control "max-age" to the correct values. When you are certain the caching is working as desired, you can extend the expiration time.

Within the <Location /wms> block, add the following cache directives:

	CacheEnable disk 
	CacheHeader on

	CacheDefaultExpire 600
	CacheMaxExpire 86400
	CacheLastModifiedFactor 0.5

	ExpiresActive on
	ExpiresDefault "access plus 1 week"

	Header merge Cache-Control public

Your edited virtual host .conf file should look like this:

<VirtualHost *:80>
    	ServerAdmin webmaster@localhost
	.
	.
	.
	# Apache caching configuration
        CacheQuickHandler off

        CacheLock on
        CacheLockPath /tmp/mod_cache-lock
        CacheLockMaxAge 5

        CacheIgnoreHeaders Set-Cookie
        CacheIgnoreCacheControl On
	
        # MapServer /wms endpoint
        Alias /wms /usr/lib/cgi-bin/mapserv
        <Location /wms>
                CacheEnable disk /wms
                CacheHeader on

                CacheDefaultExpire 600
                CacheMaxExpire 86400
                CacheLastModifiedFactor 0.5

                ExpiresActive on
                ExpiresDefault "access plus 1 week"

                Header merge Cache-Control public

                SetHandler cgi-script
                Options ExecCGI
                SetEnv MS_MAPFILE /opt/mapserver/map/imagery.map
        </Location>	
	.
	.
	.
</VirtualHost>

Cache Maintenance

htcacheclean (installed by apache2-utils) is used to manage the cache. Following are a few examples of its use.

The following command displays the contents of the cache. The -p switch specifies the cache location; the -a (or -A) dumps the contents.

sudo htcacheclean -p /var/cache/apache2/mod_cache_disk/ -a

The following command manually cleans the cache and ensure the size is not larger than 100MB. The -l switch specifies the resulting cache size; the -v displays verbose results.

sudo htcacheclean -p /var/cache/apache2/mod_cache_disk/ -l 100M -v

This command runs the cache cleanup in a daemon:

htcacheclean -d30 -n -t -p /var/cache/apache2/mod_disk_cache -l 100M -i

This will clean our cache directory every 30 minutes and make sure that it will not get bigger than 100MB. To learn more about htcacheclean, take a look at:

man htcacheclean

The apache2-utils install may have already installed the apache-htcacheclean service. Examine the status and runtime parameters of the service with systemctl.

sudo systemctl status apache-htcacheclean

To change the service's runtime parameters, edit the file /etc/default/apache-htcacheclean and change the default values. Start or stop the service with systemctl as required for your installation.

Reference

https://www.digitalocean.com/community/tutorials/web-caching-basics-terminology-http-headers-and-caching-strategies

https://www.digitalocean.com/community/tutorials/how-to-configure-apache-content-caching-on-ubuntu-14-04

https://www.howtoforge.com/caching-with-apache-mod_cache-on-debian-etch

Step 5: Configure fail2ban to prevent DOS (experimental)

We can use fail2ban to help prevent an inadvertent denial-of-service (DOS) attacks caused by bulk downloads.

Install fail2ban with the package manager:

sudo apt-get install fail2ban

Create and open the http-get-dos.conf for editing...

sudo touch /etc/fail2ban/filter.d/http-get-dos.conf 
sudo nano /etc/fail2ban/filter.d/http-get-dos.conf 

... and add the following content:

# Fail2Ban configuration file
#
# Author: http://www.go2linux.org
#
[Definition]

# Option: failregex
# Note: This regex will match any GET entry in your logs, so basically all valid and not valid entries are a match.
# You should set up in the jail.conf file, the maxretry and findtime carefully in order to avoid false positives.

failregex = ^<HOST> -.*GET

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

Create a local copy of the fail2ban configuration file and open it for editing:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

--- and add the following content:

[http-get-dos]
# Simple attempt to block very basic DOS attacks over GET: 
# If  [maxRetry] requests occur by an ip within [findtime] secs, then bans the ip for [bantime] secs.
enabled = true
port = http,https
filter = http-get-dos
logpath   = /var/log/apache2/access.log
maxRetry = 100
findtime = 30
bantime = 10

The configuration above was used for testing fail2ban. The actual values you use for maxRetry, findtime and bantime will have to be figured out with testing.

You can check the validity of the http-get-doc jail's regex expression against Apache's access.log with this command:

fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/http-get-dos.conf

You can check the status of the http-get-dos jail with this command:

sudo fail2ban-client status http-get-dos

You must restart the fail2ban service after making changes to the jail.local file.

sudo systemctl restart fail2ban

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment