nyurik

OpenStreetMap data is heavily normalized, making it very hard to process. Modeled on a relational database, it seems to have missed the second part of the "Normalize until it hurts; denormalize until it works" proverb.

Each node has an ID, and every way and relation uses an ID to reference that node. This means that every data consumer must keep an enrmous cache of 8 billion node IDs and corresponding lat,lng pairs while processing input data. In most cases, node ID gets discarded right after parsing.

I would like to propose a new easy to process data strucutre, for both bulk downloads and streaming update use cases.

Target audience

• YES -- Data consumers who transform OSM data into something else, i.e. tiles, shapes, analytical reports, etc.

nyurik

#!/usr/bin/env python3
# A script to detect when Terraform projects or depended modules are part of a GIT pull request change
#
# Usage:  python3 pr_tf_changes.py <branch> <dir>...
#
#  <branch>  GIT branch to compare using  git diff branch...  shell call
#  <dir>     One or more directories to monitor, including all sub-dirs, relative to repo's root
#
# Set DEBUG env var to see additional debugging information
# If match is found, exitcode is 0, otherwise 1

nyurik

#
#   This script converts margins, padding, and borders to logical values,
#   allowing RTL and vertical languages to show correctly.
#   Supports both *.css and *.scss files.
#
#   Some renames are not yet implemented widely, and may require CSS plugin
#     https://github.com/csstools/postcss-logical
#   They have been commented out for now, but feel free to try them out.
#
#   Full spec:   https://drafts.csswg.org/css-logical/

nyurik

------- Testing:
-- git clone https://github.com/openmaptiles/openmaptiles
-- git checkout upgrade-v5-pg12
-- place this file in the openmaptiles/ dir as "test-func.sql"
-- use  make start-db   to create a new database (in docker)   
-- use  make bash       to start tools (another docker)
-- test with this command. The test call is taken from the  openmaptiles-tools/tests/sql/LabelGrid.sql  test. Note the "volatile" keyword - without it the query planner will optimize away multiple calls with the same value.
--    profile-pg-func --file test-func.sql "LabelGrid_pgsql(ST_GeomFromText('POINT(100 -100)',900913), 64*9.5546285343)" "LabelGrid_sql(ST_GeomFromText('POINT(100 -100)',900913), 64*9.5546285343)"

-- The results are not that great:

nyurik

CREATE OR REPLACE FUNCTION getmvt(zoom integer, x integer, y integer)
RETURNS bytea AS $$
SELECT STRING_AGG(mvtl, '') AS mvt FROM (
SELECT IsEmpty, count(*) OVER () AS LayerCount, mvtl FROM (
  SELECT CASE zoom <= 8 WHEN TRUE THEN FALSE ELSE ST_WITHIN(ST_GeomFromText('POLYGON(
(0 4096,0 0,4096 0,4096 4096,0 4096))', 3857), ST_COLLECT(mvtgeometry)) END AS IsEmpty, ST_AsMVT(tile, 'water', 4096, 'mvtgeometry') as mvtl FROM (SELECT ST_AsMVTGeom(geometry, TileBBox(zoom, x, y), 4096, 4, true) AS mvtgeometry, class, intermittent FROM layer_water(TileBBox(zoom, x, y), zoom) WHERE ST_AsMVTGeom(geometry, TileBBox(zoom, x, y), 4096, 4, true) IS NOT NULL) AS tile HAVING COUNT(*) > 0
    UNION ALL
  SELECT FALSE AS IsEmpty, ST_AsMVT(tile, 'waterway', 4096, 'mvtgeometry') as mvtl FROM (SELECT ST_AsMVTGeom(geometry, TileBBox(zoom, x, y), 4096, 4, true) AS mvtgeometry, name, name_en, name_de, NULLIF(tags->'name:ar', '') AS "name:ar", NULLIF(tags->'name:az', '') AS "name:az", NULLIF(tags->'name:be', '') AS "name:be", NULLIF(t

nyurik

{
    "extra": {
        "merge-plugin": {
            "include": [
                "extensions/Wikibase/composer.json",
                "../settings.d/composer/*.json"
            ]
        }
    }
}

nyurik

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

fixes['osmlinks'] = {
    'regex': True,
    'nocase': True,
    'msg': {
        # '_default': 'Param cleanup, remove obsolete lang parameters - template detects it automatically',
        '_default': 'Tag template cleanup - format combinations, use proper {{Tag}} template with params, remove kl= and vl= (handled automatically)',
    },

nyurik

How to submit Kibana Vega question

It is usually very difficult to debug Vega questions without having your data. To make it easier, please follow these steps to include data with your graph when posting:

• Reduce your data query to the smallest possible dataset, e.g. set the time range to 15 minutes. It will work as long as it is not empty and represents your data well enough.
• Open Browser Debugger (for Chrome, right click and click Inspect)
• Switch to the Console tab
• Copy the right command, paste it in the console at the > symbol and hit enter (check the schema in your graph to see if you use Vega or Vega-Lite)

nyurik

Keybase proof

I hereby claim:

• I am nyurik on github.
• I am nyurik (https://keybase.io/nyurik) on keybase.
• I have a public key ASA_DNz0UNVjjeDMAIOHOLYL2y2ybFNWIMgB9dkX5A5j3Qo

To claim this, I am signing this object:

nyurik

{
  $schema: https://vega.github.io/schema/vega/v3.json
  padding: 10
  config: {
    kibana: {tooltips: false}
  }
  signals: [
    {
      name: rotation
      value: 65