libosmscout-maps.ipynb

libosmscout-maps.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "libosmscout-maps.ipynb",
      "provenance": [],
      "private_outputs": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/ross-newman/8634f69e98ac2aded46552e7b0768dbb/libosmscout-maps.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6sqfdqVEyDEp"
      },
      "source": [
        "**libosmscout**\n",
        "\n",
        "This notebook sets up the OSMScout project to process OpenStreetMap files for offline client side processing. The examples below was tested for England but can be adapted for you purposes. Below is an example of libosmscout rndering a map of London in [vivoe-lite](https://github.com/ross-newman/vivoe-lite) over on Github.\n",
        "\n",
        "![Cairo Map Example](https://github.com/ross-newman/vivoe-lite/raw/master/images/Screenshot-OSMScout-Map.png)\n",
        "\n",
        "Firstly we will pick up our OSM maps from gdrive and place the data here when done."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DUXYhYTapNrV"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "syGhN_aQjMu-"
      },
      "source": [
        "%%bash\n",
        "apt update\n",
        "apt install protobuf-compiler qtpositioning5-dev libqt5svg5-dev qttools5-dev unzip g++-8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Okc3Qj-SdgmX"
      },
      "source": [
        "Lets get the basemap first. This contains the global coastlines. https://osmdata.openstreetmap.de/data/coastlines.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tq1Pm2QZdfiu"
      },
      "source": [
        "%%bash\n",
        "wget https://osmdata.openstreetmap.de/download/coastlines-split-4326.zip\n",
        "unzip ./coastlines-split-4326.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GFfjaf2mfMHx"
      },
      "source": [
        "Extract the required file and rename. Tidy up the archive and temporary directory. We will convert this later to water.idx."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lw3d_rcgeBZ-"
      },
      "source": [
        "%%bash\n",
        "cp ./coastlines-split-4326/lines.shp ./coastlines.shp\n",
        "rm -rf coastlines-split-4326\n",
        "rm coastlines-split-4326.zip\n",
        "ls\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qEo9Xjk-qKiO"
      },
      "source": [
        "To process .pbf files in libosmscout we need to install Google Protobuf 3 from github."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aqr4gSeIjmCu"
      },
      "source": [
        "%%bash\n",
        "apt autoremove\n",
        "apt install autoconf automake libtool curl make g++ unzip git -y \n",
        "git clone https://github.com/google/protobuf.git\n",
        "cd protobuf\n",
        "git submodule update --init --recursive\n",
        "./autogen.sh\n",
        "./configure\n",
        "make\n",
        "#make check\n",
        "make install\n",
        "ldconfig"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdTpuIw5qZcP"
      },
      "source": [
        "Now lets install libosmscout."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0v2_Pe17kbtV"
      },
      "source": [
        "%%bash\n",
        "if [ ! -d \"libosmscout\" ]; then\n",
        "  git clone https://github.com/Framstag/libosmscout\n",
        "fi\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B3HGRfGmlUGi"
      },
      "source": [
        "%%bash\n",
        "cd libosmscout\n",
        "if [ -d \"build\" ]; then\n",
        "  rm -rf build\n",
        "fi\n",
        "mkdir build\n",
        "cd build\n",
        "cmake .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MnMEfmagowfi"
      },
      "source": [
        "Now we can build the library needed to process our maps"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eb4GSIWLmeMY"
      },
      "source": [
        "%%bash\n",
        "cd libosmscout/build\n",
        "echo \"Building libosmscout...\"\n",
        "make -j $(nproc) CFLAGS='-lstdc++fs -std=c++17'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WZhGj1-ApN8q"
      },
      "source": [
        "We need some storage space on gdrive to keep the processed maps. Maps are stored on GDrive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ahs_xgC3qnSe"
      },
      "source": [
        "%%bash\n",
        "cd ./gdrive/My\\ Drive\n",
        "\n",
        "# England\n",
        "if [ ! -f \"england-latest.osm.pbf\" ]; then\n",
        "  wget https://download.geofabrik.de/europe/great-britain/england-latest.osm.pbf\n",
        "fi\n",
        "if [ ! -f \"england.poly\" ]; then\n",
        "  wget https://download.geofabrik.de/europe/great-britain/england.poly\n",
        "fi\n",
        "echo \"England OK...\"\n",
        "\n",
        "# Germany\n",
        "if [ ! -f \"germany-latest.osm.pbf\" ]; then\n",
        "  wget https://download.geofabrik.de/europe/germany-latest.osm.pbf\n",
        "fi\n",
        "if [ ! -f \"germany.poly\" ]; then\n",
        "  wget https://download.geofabrik.de/europe/germany.poly\n",
        "fi\n",
        "echo \"Germany OK...\"\n",
        "\n",
        "# Australia\n",
        "if [ ! -f \"australia-latest.osm.pbf\" ]; then\n",
        "  wget https://download.geofabrik.de/australia-oceania/australia-latest.osm.pbf\n",
        "fi\n",
        "if [ ! -f \"australia.poly\" ]; then\n",
        "  wget https://download.geofabrik.de/australia-oceania/australia.poly\n",
        "fi\n",
        "echo \"Australia OK...\"\n",
        "\n",
        "echo \"Map files ready..\"\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iN_qGXzEq8FX"
      },
      "source": [
        "Finally finish the processing of the coastlines and copy the result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UQwaOGYnrCyi"
      },
      "source": [
        "%%bash\n",
        "pwd\n",
        "cd libosmscout/maps\n",
        "mkdir world\n",
        "/content/libosmscout/build/BasemapImport/BasemapImport --destinationDirectory world --coastlines /content/coastlines.shp\n",
        "tar -zcvf world.tar.gz world\n",
        "cp -rf /content/libosmscout/maps/world.tar.gz /content/gdrive/My\\ Drive/.\n",
        "!ls /content/libosmscout/maps/world.tar.gz /content/gdrive/My\\ Drive/*\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFpxLACG1gjf"
      },
      "source": [
        "When the  🌐 maps files are downloaded we can process them. Create symbolic links to the map files and start processing."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ysl0e3aEzjBt"
      },
      "source": [
        "%%bash\n",
        "ln -s /content/gdrive/My\\ Drive/england-latest.osm.pbf /content/libosmscout/maps/england-latest.osm.pbf \n",
        "ln -s /content/gdrive/My\\ Drive/england.poly /content/libosmscout/maps/england.poly \n",
        "ln -s /content/gdrive/My\\ Drive/england-latest.osm.pbf /content/libosmscout/maps/germany-latest.osm.pbf \n",
        "ln -s /content/gdrive/My\\ Drive/england.poly /content/libosmscout/maps/germany.poly \n",
        "ln -s /content/gdrive/My\\ Drive/england-latest.osm.pbf /content/libosmscout/maps/australia-latest.osm.pbf \n",
        "ln -s /content/gdrive/My\\ Drive/england.poly /content/libosmscout/maps/australia.poly \n",
        "\n",
        "cd /content/libosmscout/maps\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjxLpjDxpwVv"
      },
      "source": [
        "Process 🌍󠁧󠁢󠁥󠁮󠁧󠁿 England maps..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N1ECW2eC03p3"
      },
      "source": [
        "%%bash\n",
        "cd ./libosmscout/maps\n",
        "mkdir england-latest\n",
        "../build/Import/Import  \\\n",
        "    --typefile ../stylesheets/map.ost \\\n",
        "    --destinationDirectory england-latest \\\n",
        "    --bounding-polygon england.poly\\\n",
        "    england-latest.osm.pbf \n",
        "tar -zcvf england-latest.tar.gz england-latest\n",
        "mv england-latest.tar.gz /content/gdrive/My\\ Drive/.\n",
        "rm -rf england-latest"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9fxXS3GlqX2k"
      },
      "source": [
        "Process 🌍 German maps..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ArgJ91Lf6clj"
      },
      "source": [
        "%%bash\n",
        "cd ./libosmscout/maps\n",
        "mkdir germany-latest\n",
        "../build/Import/Import  \\\n",
        "    --typefile ../stylesheets/map.ost \\\n",
        "    --destinationDirectory germany-latest \\\n",
        "    --bounding-polygon germany.poly\\\n",
        "    germany-latest.osm.pbf \n",
        "tar -zcvf germany-latest.tar.gz germany-latest\n",
        "mv germany-latest.tar.gz /content/gdrive/My\\ Drive/.\n",
        "rm -rf germany-latest"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u1z79xUZqR2I"
      },
      "source": [
        "Process 🌏 Australian map..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DhgWwc1O6c2C"
      },
      "source": [
        "%%bash\n",
        "cd ./libosmscout/maps\n",
        "mkdir australia-latest\n",
        "../build/Import/Import  \\\n",
        "    --typefile ../stylesheets/map.ost \\\n",
        "    --destinationDirectory australia-latest \\\n",
        "    --bounding-polygon australia.poly\\\n",
        "    australia-latest.osm.pbf \n",
        "tar -zcvf australia-latest.tar.gz australia-latest\n",
        "mv australia-latest.tar.gz /content/gdrive/My\\ Drive/.\n",
        "rm -rf australia-latest"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bSoWV47vphJ9"
      },
      "source": [
        "Finally we clean up all the intermediate files we dont need to reduce our could storage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DQv7davm6ria"
      },
      "source": [
        "%%bash\n",
        "ls -hal /content/gdrive/My\\ Drive/england-latest.tar.gz \n",
        "ls -hal /content/gdrive/My\\ Drive/germany-latest.tar.gz\n",
        "ls -hal /content/gdrive/My\\ Drive/australia-latest.tar.gz\n",
        "ls -hal /content/gdrive/My\\ Drive/\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NcSSKiJPRIka"
      },
      "source": [
        "%%bash\n",
        "cd ./libosmscout/maps\n",
        "ls -hal"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}