OpenStreetMap data is heavily normalized, making it very hard to process. Modeled on a relational database, it seems to have missed the second part of the "Normalize until it hurts; denormalize until it works" proverb.
Each node has an ID, and every way and relation uses an ID to reference that node. This means that every data consumer must keep an enrmous cache of 8 billion node IDs and corresponding lat,lng
pairs while processing input data. In most cases, node ID
gets discarded right after parsing.
I would like to propose a new easy to process data strucutre, for both bulk downloads and streaming update use cases.
- YES -- Data consumers who transform OSM data into something else, i.e. tiles, shapes, analytical reports, etc.
- NO -- Apps that submit changes back to OSM, unless they also download individual objects in the original format with all IDs intact.
- Split nodes into two types --
position node
andcontent node
:Position node
is a node that has no tags -- just alat,lng
coordinate pair. Position node's coordinates are inlined into ways and relations. Their ID is essentially deleted from the output.Content node
is a regular node object with an ID, geo coordinate pair, and a list of tags (same as we have now).
- A
way
has a list of geo points instead of a list of node IDs.- TBD: A
way
may have an optional list ofcontent node
IDs.
- TBD: A
- A
relation
has a list of values, where each value can be:- a
lat,lng
pair with an optionalcontent node
ID. - a
way
ID - a
relation
ID
- a
- If a single OSM node is moved, the change stream will include all objects that include that node
- Each PBF block should have uncompressed meta information (useful to skip unrelated info):
- Number of each feature type contained in the block: counts of nodes, ways, relations, changesets (?).
- Bounding box of all features in the block.
Thank you @mmd-osm !! It looks like @joto is leading the effort (?), but also seems that the effort is far more involved than what I proposed -- I think he is trying to change the internal API and data storage model. If so, this would be a far more complicated and take longer. I wonder if it would make sense to solve the "99%" problem, and just provide an alternative data dump/streaming format, and once its in place, work on changing the internals and/or API independently?
One issue @joto does mention is topology. But just as Jochen writes, the vast majority of the duplicate nodes at the same positions are errors. Plus it introduces ambiguity with updates -- if at first we strip node IDs from a way, and later a new feature is added with an identical node's location, it would not be possible to determine if the new one is the same node or a different one. One solution here would be to force OSM model to refuse duplicate nodes at the same location - forcing users to put separate nodes nearby instead. But this would create a slew of other issues - all editor tools would need to be aware of how a geo coords pair gets normalized into two 32 bit values, and ensure that they are different. Tricky, and the "obvious" solution is to treat this ultra-rare problem as non-existent... Not ideal, but solves the problem for the other 99.999% use cases.
P.S. A hacky workaround to handle dups just in the API - if multiple nodes share identical location, treat them as separate, and nudge them all apart by the minimum distance in any direction