Skip to content

Instantly share code, notes, and snippets.

@jimjam-slam

jimjam-slam/prep-tiles-from-osm.md

Last active June 11, 2024 06:03
Show Gist options
  • Save jimjam-slam/fa5371f2d77d37a02c41359e2a5345fa to your computer and use it in GitHub Desktop.
Save jimjam-slam/fa5371f2d77d37a02c41359e2a5345fa to your computer and use it in GitHub Desktop.
Download ZIP
Notes on extracting, processing and hosting vector tiles
Raw
prep-tiles-from-osm.md

tl;dr

Shapefile(s) to vector tile set

graph LR
    A[Vector source - eg. shapefile] --> |ogr2ogr| B[geoJSON]
    B --> |tippecanoe| C[Set of .pbf tiles]
Loading

Sourcing data

Existing processed vector tiles

Getting an OSM Planet or Extract file

Other data

Get a data source you can turn into geoJSON to use with tippecanoe (see below).

For example, convert an ABS shapefile of postcode boudnaries using ogr2ogr, which comes with GDAL (brew install gdal):

ogr2ogr -f GeoJSON postcodes.geojson POA_2021_AUST_GDA2020.shp

⚠️ It looks like ogr2ogr doesn't reproject data into WGS 84, even though that's now the only projection supported by the geoJSON format (and what tippecanoe expects). Might wanna reproject first!

Create a tile archive (.mbtiles)

From an OSM file

From what I understand, MapLibre used MapTiler Desktop to create the tiles, but I'm having trouble getting it to create anything. Let's try and do it ourselves.

First we create the tiles (as an .mbtiles archive) using TileMaker:

# create an .mbtiles from an osm planet file or extract (.pbf)

./tilemaker-macos-10.15/build/tilemaker \
  --input tileset/australia-oceania-latest.osm.pbf \
  --output aus.mbtiles \
  --process ./tilemaker-macos-10.15/resources/process-openmaptiles.lua \
  --config ./tilemaker-macos-10.15/resources/config-openmaptiles.json

Use tippecanoe to turn a vector data source into a tile archive or a hierarchy of .pbf tiles

# .mbtiles output
tippecanoe -o postcodes.mbtiles postcodes.geojson

# folder of .pbf tiles output
tippecanoe --output-to-directory postcodes postcodes.geojson

Some things to note:

  • Tippecanoe has a lot of options for controlling which features are visible at what zoom levels!
  • In the absence of a custom name, Tippecanoe will name the layer in the tile archive after the file. Layer names are important when we start styling!
  • I believe tippecanoe can combine several input files into a tile archive as separate layers. Good to know!

Use planetiler

Alternative: planetiler can also do this and is explciitly designed to ingest the whole OSM planet file if you have the RAM for it... probably more than 32 GB 😬

Extract the archive to a set of .pbf files

Option 0: skip this and just use tippecanoe

Tippecanoe can output either an .mbtiles archive or a folder of .pbf tiles (using the --output-to-directory option).

Option A: Extract .pbf tiles to directory from .mbtiles archive

If we want to bundle our tiles directly with our code (eg. a JAMstack product), we can't put the tile archive directly in the project: it'll likely be bigger than the 50 MB file size limit imposed by GitHub. But we can extract the tiles into a {z}/{x}/{y).pbf directory structure and commit those to our project, provided they don't blow the total repo size.

This might be the better solution for:

  1. Global maps where we don't need detail beyond zoom 6 or so; or
  2. Regional/city maps where we don't need to zoom out much.

To extract the individual tiles .mbtiles archive (which is essentially an SQLite database), we can use Mapbox's mbutil:

mb-util aus.mbtiles aus-tiles

(Apparently mb-util produces gzipped output!, but it looks like you can turn this off with a --do_compression option.)

Option B: serving the archive using Lambda + S3 or Cloudflare Workers + R2

For some maps, we may need both global overviews and regional details. In these case, we may want to put the .mbtiles archive in a remote bucket and have an edge function perform the "tile server" role, pulling out individual tiles on demand.

This isn't free in an unlimited sense the way committing tiles to version control is (at least for Cloudflare Pages), but it is potentially hundreds of tiles cheaper than a SaaS offering like Mapbox or MapTiler. It may also be useful in cases where we have a large vector dataset that we want to serve.

Write the tilespec (tiles.json)

So your map's style.json will use a sources field to point to your new tileset:

{
  # ...
  "sources": {
      "maplibre": {
        "url": "tiles/tiles.json",
        "type": "vector"
      }
    }
}

But tippecanoe doesn't produce tiles.json, so you'll have to write it according to the Mapbox tilespec. (I've linked to version 2.0.0 here, as I'm unsure if MapLibre supports newer versions.)

(You can actually inline the tilespec in style.json!)

Also consider

Mapbox's node-mbtiles has some utilities for working with .mbtiles files.

Mapbox's tippecanoe looks like it can also handle create .mbtiles files from GeoJSON features as .json files, .geobufs and potentially other input sources, so it could possible also replace the first step. It also looks quite configurable, allowing you to specify max zoom, detail, attributes, etc. It's the basis of Mapbox's Vector Tile Service, so it ought to be quite powerful.

Here're some somebody made earlier

maplibre/demotiles has free vector tiles made with MapTiler Desktop that they've released under BSD and which are already laid out in folder format, ready to be served keyless. These only go to zoom 6, so they aren't detailed enough for regional maps, but for global ones they're great and can be restyled quickly.

openmaptiles/fonts has pre-encoded versions of common open-source fonts, ready to drop into map styles.

Vector data: Flatgeobuf

These aren't really vector tiles, but you can do a similar thing with them: request a byte range that corresponds to a bounding box. The library's MapLibre example does this (demo here). The library handles parsing a bounding box and URL into a ranged fetch request.

Vector data: PMTiles

PMTiles also looks like a tiled vector format designed for serverless, ranged requests. They have two MapLibre examples:

The code on one of their demos converts a shapefile to PMTiles via .json and .mbtiles formats:

ogr2ogr -t_srs EPSG:4326 cb_2018_us_zcta510_500k.json cb_2018_us_zcta510_500k.shp

tippecanoe -zg --projection=EPSG:4326 --no-tile-compression --no-feature-limit --no-tile-size-limit -o cb_2018_us_zcta510_500k_nolimit.mbtiles -l zcta cb_2018_us_zcta510_500k.json

pmtiles-convert cb_2018_us_zcta510_500k_nolimit.mbtiles cb_2018_us_zcta510_500k_nolimit.pmtiles

Apparently PMTiles can hold arbitrary raster or vector data.

Fonts

There're a few mapbox tools floating around to convert TrueType/OpenType fonts into .pbf set sby glyph (which style.json requires), but Protomaps' signed SDF tool is by far the easiest. Give it a font, it'll convert it client-side and give you a ZIP file with the .pbf set.

File sizes

Tilesets get bigger exponentially with zoom. Here're the file sizes I ended up with when I converted the ABS postcodes:

Zoom level Size
0 84K
1 116K
2 164K
3 236K
4 332K
5 516K
6 816K
7 1.4M
8 3.0M
9 8.2M
10 27M
11 104M
12 406M
13 1.6G
14 6.2G

It might be worth finding a suitable max zoom for tiles like those around 10 or 11 and just overzooming!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment