Skip to content

Instantly share code, notes, and snippets.

@MilesMcBain
Last active March 12, 2021 04:45
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save MilesMcBain/ed51aa8d4c04b2144bb6bd77f7898733 to your computer and use it in GitHub Desktop.
Save MilesMcBain/ed51aa8d4c04b2144bb6bd77f7898733 to your computer and use it in GitHub Desktop.
WSL Setup OSRM

as per: aws_overpass_instance.md

  • docker
  • aws cli
    • aws credentials

get osrm australia data from s3

 aws s3 cp s3://pims-osrm-source-data/australia-latest.osm.bz2 ./australia-latest.osm.bz2

write data to s3

aws s3 cp ./australia-latest.osm.pbf s3://pims-osrm-source-data/australia-latest.osm.pbf

get osrm docker image

sudo docker pull milesmcbainqfes/qfes_osrm

run docker osrm docker

Init data

sudo docker run -t -v ~/Downloads:/data milesmcbainqfes/qfes_osrm osrm-extract /data/australia-latest.osm.pbf -p /opt/truck.lua
sudo docker run -t -v ~/Downloads:/data milesmcbainqfes/qfes_osrm osrm-contract /data/australia-latest.osrm

Took around 25 min and used just under 2 gb RAM

run server

docker run -d -p 80:5000 -v /home/ubuntu/Downloads:/data milesmcbainqfes/qfes_osrm osrm-routed /data/australia-latest.osrm

Look in /home/ubuntu/bin for the launch/init scripts

Docker

install docker:

sudo apt-get update
sudo apt-get install docker.io

Get overpass docker image:

sudo docker pull wiktorn/overpass-api

AWS CLI

from: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

create a folder for our aws config:

mkdir ~/.aws
touch ~/.aws/config
touch ~/.aws/credentials

populate those files with data not listed here since it's a public gist.

Fetch overpass source data:

curl http://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2 --output australia-latest.osm.bz2

Write it to aws bucket

 aws s3 cp ./australia-latest.osm.bz2 s3://pims-osrm-source-data/

Initalise overpass

Launch the overpass server and process the data. Example:

docker run \
          -e OVERPASS_META=yes \
          -e OVERPASS_MODE=init \
          -e OVERPASS_PLANET_URL=file:///input/australia-latest.osm.bz2 \
          -e OVERPASS_RULES_LOAD=1 \
          -v /big/docker/overpass_db/:/db \
          -v /home/ubuntu/Downloads/:/input \
          -p 80:80 \
          -i -t \
          --name overpass_australia wiktorn/overpass-api
          
Now contained in ~/bin/init_overpass_db.sh

run detached using an existing db (clone)

docker run \
          -d \
          -e OVERPASS_META=yes \
          -e OVERPASS_MODE=clone \
          -e OVERPASS_RULES_LOAD=1 \
          -v /big/docker/overpass_db/:/db \
          -v /home/ubuntu/Downloads/:/input \
          -p 80:80 \
          --name overpass_australia wiktorn/overpass-api
          
Now contained in ~/bin/overpass_clone.sh

setup to run on server init

Run sudo crontab -e. add a line like: @reboot /home/ubuntu/bin/start_osrm_router.sh

Apt proxy

from: https://www.serverlab.ca/tutorials/linux/administration-linux/how-to-set-the-proxy-for-apt-for-ubuntu-18-04/

sudo touch /etc/apt/apt.conf.d/proxy.conf

Then add these lines, say with sudo vim /etc/apt/apt.conf.d/proxy.conf:

Acquire::http::Proxy "http://dcsproxy:8080";
Acquire::https::Proxy "http://dcsproxy:8080";

Then test with sudo apt update

System Proxy

Add these two lines to ~/.profile:

export HTTP_PROXY=http://dcsproxy:8080
export HTTPS_PROXY=http://dcsproxy:8080

You'll need to close and reopen your terminal to get a new shell so the settings will take.

Add psba root certificate

For git and curl.

Need to work out how to copy the inspection bundle to the host. But once it is there:

from: https://manuals.gfi.com/en/kerio/connect/content/server-configuration/ssl-certificates/adding-trusted-root-certificates-to-the-server-1605.html

Copy your CA to dir /usr/local/share/ca-certificates/
Use command: sudo cp inspection-bundle.crt /usr/local/share/ca-certificates/inspection-bundle.crt
Update the CA store: sudo update-ca-certificates

Installing OSRM Dependencies

from: https://github.com/Project-OSRM/osrm-backend/wiki/Building-on-Ubuntu

sudo apt install build-essential git cmake pkg-config \
libbz2-dev libstxxl-dev libstxxl1v5 libxml2-dev \
libzip-dev libboost-all-dev lua5.2 liblua5.2-dev libtbb-dev \
libluabind-dev libluabind0.9.1v5 

Cloning OSRM backend

git clone --depth=1 https://github.com/Project-OSRM/osrm-backend.git

Building OSRM

from: https://github.com/Project-OSRM/osrm-backend/wiki/Building-OSRM

From inside osrm repo dir:

mkdir -p build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .
sudo cmake --build . --target install

Test OSRM

Build network

from: https://github.com/Project-OSRM/osrm-backend/wiki/Running-OSRM

Using our truck profile and map from: \analytics\Pims\OSRM

Need to put files in an s3 bucket or something.

Copy the truck profile into the profiles folder in osrm-backend repo:

cp FireTruck.lua osrm-backend/profiles

Run:

osrm-extract australia-latest.osm.pbf -p osrm-backend/profiles/FireTruck.lua

inside the directory where the pbf file was will be an australia-latest.osrm file.

Run:

osrm-contract australia-latest.osrm

(Uses 2.2GB of RAM)

Start server

osrm-routed australia-latest.osrm

Note we plan to use the table service to batch our requests. So we will want to set the --max-table-size <n_locations> option reasonably high. See: https://github.com/Project-OSRM/osrm-backend/blob/master/docs/http.md#table-service

Make api call on server

Request route for Roma Street station to Kedron Park. Google says ~13 min over 10km.

curl "http://127.0.0.1:5000/route/v1/driving/153.0130931,-27.4658784;153.0358785,-27.4147576?steps=false"

with response:

{"code":"Ok","waypoints":[{"hint":"J7Y2gP___38CAAAABAAAABAAAAAAAAAAStBwQMPo_j90HKtBAAAAAAIAAAAEAAAAEAAAAAAAAACXDwAAL8oeCS7mXP5lyx4Jaudc_gIAfxPD7UVd","distance":46.52748,"location":[153.012783,-27.466194],"name":"Petrie Terrace"},{"hint":"ue81gP___38XAAAAGAAAAC8AAAC0AAAAkof5Qd_hID_X23ZCGXhvQxcAAAAYAAAALwAAALQAAACXDwAATyQfCQ2vXf5nJB8JGq9d_gMAnxTD7UVd","distance":2.776107,"location":[153.035855,-27.414771],"name":"Airport Link"}],"routes":[{"legs":[{"steps":[],"weight":799.1,"distance":12237,"summary":"","duration":799.1}],"weight_name":"routability","geometry":"tnsfD{h|d\\}CHyG{ImU[_D~@_FpM`A{@kFyF_Fas@aZ}RcJ}Y}IgH_NsByv@IekAmLcZmEySaJyYab@aMcb@iBuTxBcr@cB`r@fBfT`M`b@dPfLzBtFeAbJoH|E","weight":799.1,"distance":12237,"duration":799.1}]}

Suggesting travel time of 13.3 minutes over 12.2km.

Clone openstreetmap-website

It has the code for the back and front end.

https://github.com/qfes/openstreetmap-website

Build the docker image:

The image contains both the front and back ends.

There are instructions here: https://github.com/qfes/openstreetmap-website/blob/master/DOCKER.md

Load source data into back end

The data that goes into the back end needs to have full metadata to avoid key constraint violations. The public extracts on http://download.geofabrik.de/australia-oceania/australia.html have had the metadata scrubbed. There are full extracts available, but you need to log in with an open street map account: https://osm-internal.download.geofabrik.de/australia-oceania/australia.html.

I am not sure how to automate the download whilst authenticated with OSM. So for now I have downloaded with my browser and put a full extract on S3: https://s3.console.aws.amazon.com/s3/object/pims-osrm-source-data?region=ap-southeast-2&prefix=australia-oceania-latest-internal.osm.pbf

from: http://overpass-api.de/no_frills.html

Get Australia data

https://download.geofabrik.de/australia-oceania/australia-latest.osm.bz2

from https://download.geofabrik.de/australia-oceania/australia.html

Get required packages

sudo apt-get install wget g++ make expat libexpat1-dev zlib1g-dev apache2

Get and build release

wget https://dev.overpass-api.de/releases/osm-3s_v0.7.55.tar.gz
gunzip <osm-3s_v0.7.55.tar.gz | tar xvf -
cd osm-3s_v0.7.55/
./configure
make
chmod 755 bin/*.sh cgi-bin/*

Import data file

inside the osm-3s-v0.7.55 folder:

mkdir -p db
 bin/init_osm3s.sh /mnt/c/data/OSRM/australia-latest.osm.bz2 "db/" "./" --meta

Takes about an hour.

Configure a webserver for cgi

from: https://wiki.openstreetmap.org/wiki/Overpass_API/Installation#WebAPI_using_NGINX_.28Ubuntu.29


@anthonynorth
Copy link

wget config

echo check-certificate = off > $HOME/.wgetrc

@anthonynorth
Copy link

anthonynorth commented Aug 20, 2020

Apt proxy & ssl

Add verify-peer and verify-host. Needed for ppas

Acquire {
  http::proxy "http://dcsproxy:8080";
  https::proxy "http://dcsproxy:8080";
  https::Verify-Peer "false";
}

@anthonynorth
Copy link

Certificates

Store the psba inspection certificate file https://github.com/qfes/environment_config/blob/master/setup/inspection-bundle.crt somewhere (I'll assume $HOME/inspection-bundle.crt). Then add the following environment variables in ~/.profile

export CURL_CA_BUNDLE=$HOME/inspection-bundle.crt
export GIT_SSL_CAINFO=$HOME/inspection-bundle.crt

These variables remove the need for git sslVerify=false and .curlrc insecure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment