Builds Daylight Maps v1.33

daylight maps buildscript

# nohup ./daylight.sh 1.33 >> daylight.log 2>&1 &
(
  echo "Getting daylight $1 planet file"
  wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/planet-v$1.osm.pbf
) &\
(
  (
    echo "Getting daylight $1 buildings file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/ml-buildings-v$1.osm.pbf
  ) &\
  (
    echo "Getting daylight $1 fb roads file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/fb-ml-roads-v$1.osc.gz
  ) &\
  (
    echo "Getting daylight $1 admin file"
    wget -c --quiet https://daylight-map-distribution.s3.amazonaws.com/release/v$1/admin-v$1.osc.gz
  ) &\
  wait
) &\
wait

/* apply-changes admin and roads first */
osmium apply-changes planet-v$1.osm.pbf admin-v$1.osc.gz fb-ml-roads-v$1.osc.gz -o planet-osc-v$1.osm.pbf  --verbose --progress

/* merge buildings second */
osmium merge planet-osc-v$1.osm.pbf ml-buildings-v$1.osm.pbf -o planet-daylight-v$1.osm.pbf --progress


/*
MEMORY USAGE
osmium apply-changes keeps the contents of all the change files in main memory. This will take roughly 10 times as much memory as the files take on disk in .osm.bz2 format.
2688M admin-v1.33.osc.gz
4912M fb-ml-roads-v1.33.osc.gz
= 7600M
* 10 = 76GB

I ran this script on a r5a.2xlarge, 8 x 64 GiB, for $0.452/hr.  

According to the logs only 33983Mb memory used. Might be able to squeek by with 48GB, but 32GB is doutbfull.

[ 0:00] Started osmium apply-changes
[ 0:00]   osmium version 1.14.0
[ 0:00]   libosmium version 2.18.0
[ 0:00] Command line options and default settings:
[ 0:00]   input data file name: planet-v1.33.osm.pbf
[ 0:00]   input change file names:
[ 0:00]     admin-v1.33.osc.gz
[ 0:00]     fb-ml-roads-v1.33.osc.gz
[ 0:00]   data file format:
[ 0:00]   change file format:
[ 0:00]   output options:
[ 0:00]     file name: planet-osc-v1.33.osm.pbf
[ 0:00]     file format:
[ 0:00]     generator: osmium/1.14.0
[ 0:00]     overwrite: no
[ 0:00]     fsync: no
[ 0:00]   reading and writing history file: no
[ 0:00]   locations on ways: no
[ 0:00] Reading change file contents...
[32:46] Opening input file...
[32:46] Opening output file...
[32:46] Sorting change data...
[33:44] Applying changes and writing them to output...
[>                                                                     ]   0% ^M[>                                                                     ]   1% ^M[=>                    >
[85:04] Peak memory used: 33983 MBytes
*/