Map Data Conversion

map-data-conversion.md

This document is a proposal for how the map data for gen 3 decomps could be converted to a data representation that isn't raw assembler and C. The goal of this data conversion would be to make it simpler for external tools to work with the data. Working directly with assembler and C code takes more effort, is more error-prone, and is less extensible. JSON files are used in this document as the chosen data representation because JSON is widely supported and simple enough for a human to read. The chosen format could potentially be something else, but it should be widely supported (i.e. no custom formats). The work for this proposal is split into several parts:

1. Convert maps data into individual JSON files
2. Convert layouts data into a single JSON file
3. Convert map groups into single JSON file
4. Create new compile-time tool that transforms these JSON files into their old assembler representations (i.e. codegen).

1. Convert maps data into individual JSON files

Each map will move its data into a single JSON file that lives at data/maps/<name>/map_data.json. This file will contain the header, connections, and events data. Additionally, it will contain the map's name, id (map constant), and layout. Here is an example that contains all possible fields:

{
  "id": "MAP_LITTLEROOT_TOWN",
  "name": "LittlerootTown",
  "layout": "LAYOUT_LITTLEROOT_TOWN",
  "music": "MUS_MISHIRO",
  "region_map_section": "MAPSEC_LITTLEROOT_TOWN",
  "requires_flash": false,
  "weather": "WEATHER_SUNNY",
  "map_type": "MAP_TYPE_TOWN",
  "allow_bike": true,
  "allow_escape_rope": false,
  "allow_running": true,
  "show_map_name": true,
  "battle_scene": "MAP_BATTLE_SCENE_NORMAL",
  "connections": [
    {
      "direction": "north",
      "offset": 0,
      "map": "MAP_ROUTE101"
    }
  ],
  "object_events": [
    {
      "graphics_id": "EVENT_OBJ_GFX_TWIN",
      "x": 16,
      "y": 10,
      "elevation": 3,
      "movement_type": "MOVEMENT_TYPE_WANDER_AROUND",
      "movement_range_x": 1,
      "movement_range_y": 2,
      "trainer_type": 0,
      "trainer_sight_or_berry_tree_id": 0,
      "script": "LittlerootTown_EventScript_1E8034",
      "flag": 0
    },
    ...
  ],
  "warp_events": [
    {
      "x": 14,
      "y": 8,
      "elevation": 0,
      "dest_map": "MAP_LITTLEROOT_TOWN_MAYS_HOUSE_1F",
      "dest_warp_id": 1
    },
    ...
  ],
  "coord_events": [
    {
      "type": "trigger",
      "x": 10,
      "y": 1,
      "elevation": 3,
      "var": "VAR_0x4050",
      "var_value": 0,
      "script": "LittlerootTown_EventScript_1E8091"
    },
    {
      "type": "weather",
      "x": 10,
      "y": 15,
      "elevation": "3",
      "weather": "COORD_EVENT_WEATHER_SANDSTORM"
    }
    ...
  ],
  "bg_events": [
    {
      "type": "sign",
      "x": 15,
      "y": 13,
      "elevation": 0,
      "player_facing_dir": "BG_EVENT_PLAYER_FACING_ANY",
      "script": "LittlerootTown_EventScript_1E8151"
    },
    {
      "type": "hidden_item",
      "x": 15,
      "y": 12,
      "elevation": 3,
      "item": "ITEM_POTION",
      "flag": "FLAG_HIDDEN_ITEM_ROUTE_119_CALCIUM"
    },
    {
      "type": "secret_base",
      "x": 15,
      "y": 13,
      "elevation": 3,
      "secret_base_id": "SECRET_BASE_RED_CAVE2_1"
    }
    ...
  ]
}

One uncertainty that can be called out for the object_events items is the trainer_type and trainer_sight_or_berry_tree_id fields. A reasonable change here could be to add a type field similar to the coord_events and bg_events items. For example, "type": "berry_tree", "type": "trainer", and "type": "normal".

It's possible for maps to share both MapEvents and MapScripts. In the case where events are shared with another map, the above schema must omit object_events, warp_events, coord_events, and bg_events. It must additionally include shared_events_map. In the case where map scripts are shared with another map, the above schema must additionally include 'shared_scripts_map`. Example below where both events and scripts are shared with another map:

{
  "id": "MAP_LINK_CONTEST_ROOM2",
  "name": "LinkContestRoom2",
  "layout": "LAYOUT_LINK_CONTEST_ROOM2",
  "music": "MUS_CONTEST0",
  "region_map_section": "MAPSEC_DYNAMIC",
  "requires_flash": false,
  "weather": "WEATHER_NONE",
  "map_type": "MAP_TYPE_INDOOR",
  "allow_bike": false,
  "allow_escape_rope": false,
  "allow_running": false,
  "show_map_name": false,
  "battle_scene": "MAP_BATTLE_SCENE_NORMAL",
  "connections": null,
  "shared_events_map": "MAP_LINK_CONTEST_ROOM1",
  "shared_scripts_map": "MAP_LINK_CONTEST_ROOM1"
}

If a map has no connections, the connections value should be null instead of an empty array.

2. Convert layouts data into a single JSON file

Layouts are lightweight, and they are defined in a very specific order. The order is very important, as the layout indices are used explicitly in code when dynamically changing map layouts. Currently, pokeemerald is not using constants for the layout indices, but it should be, so this document adds them. For the above reasons, it's convenient to encode both the layouts' data and the layout order in the same JSON file. This file will live at data/layouts/layouts.json. Note that data/layouts/<name>/border.bin and data/layouts/<name>/map.bin will remain unchanged. Example below:

{
  "layouts_table_label": "gMapLayouts",
  "layouts": [
    {
      "id": "LAYOUT_LITTLEROOT_TOWN",
      "name": "LittlerootTown_Layout",
      "width": 20,
      "height": 20,
      "primary_tileset": "gTileset_General",
      "secondary_tileset": "gTileset_Petalburg",
      "border_filepath": "data/layouts/LittlerootTown/border.bin",
      "blockdata_filepath": "data/layouts/LittlerootTown/map.bin"
    },
    {
      "id": "LAYOUT_SLATEPORT_CITY",
      "name": "SlateportCity_Layout",
      "width": 40,
      "height": 60,
      "primary_tileset": "gTileset_General",
      "secondary_tileset": "gTileset_Slateport",
      "border_filepath": "data/layouts/SlateportCity/border.bin",
      "blockdata_filepath": "data/layouts/SlateportCity/map.bin"
    },
    {
      "id": "LAYOUT_MAUVILLE_CITY",
      "name": "MauvilleCity_Layout",
      "width": 40,
      "height": 20,
      "primary_tileset": "gTileset_General",
      "secondary_tileset": "gTileset_Mauville",
      "border_filepath": "data/layouts/MauvilleCity/border.bin",
      "blockdata_filepath": "data/layouts/MauvilleCity/map.bin"
    },
    ...
  ]
}

3. Convert map groups into single JSON file

This file contains the grouping and ordering of maps. It's essentially literal translation of data/maps/groups.inc to JSON. An alternative to this would be to include the map group information in the individual maps' JSON files described in part 1 above. However, it seems more useful to include this information in a centralized location. Another alternative would be to combine all of the individual map JSON files together, similar to the layouts described in part 2 above. This seems too unwieldy, and makes manually inspecting map files more difficult.

Example map_groups.json below:

{
  "group_order": [
    "MapGroup0",
    "MapGroup1",
    "MapGroup2",
    ...
  ],
  "MapGroup0": [
    "PetalburgCity",
    "SlateportCity",
    "MauvilleCity",
    "RustboroCity",
    "FortreeCity",
    "LilycoveCity",
    "MossdeepCity",
    "SootopolisCity",
    "EverGrandeCity",
    "LittlerootTown",
    "OldaleTown",
    "DewfordTown",
    "LavaridgeTown",
    "FallarborTown",
    "VerdanturfTown",
    "PacifidlogTown",
    "Route101",
    "Route102",
    "Route103",
    "Route104",
    "Route105",
    "Route106",
    ...
  ],
  "MapGroup1": [
    "LittlerootTown_BrendansHouse_1F",
    "LittlerootTown_BrendansHouse_2F",
    "LittlerootTown_MaysHouse_1F",
    "LittlerootTown_MaysHouse_2F",
    "LittlerootTown_ProfessorBirchsLab"
  ],
  "MapGroup2": [
    "OldaleTown_House1",
    "OldaleTown_House2",
    "OldaleTown_PokemonCenter_1F",
    "OldaleTown_PokemonCenter_2F",
    "OldaleTown_Mart"
  ],
  ...
}

The original game includes map connections in an unpredictable manner, and so it is necessary to encode this order somewhere to produce a matching ROM. This array is completely optional, but should be included in the map_groups.json file described above. Example array shown below:

{
  "group_order": [
    ....
    ....
  "connections_include_order": [
    "LittlerootTown",
    "OldaleTown",
    "DewfordTown",
    "LavaridgeTown",
    "FallarborTown",
    "VerdanturfTown",
    "PacifidlogTown",
    ...
  ]
}

4. Create new compile-time tool that transforms these JSON files into their old assembler representations (i.e. codegen).

This tool would process the JSON files described above and output the assembler code files very similar (perhaps identical) to what exists today. Ideally this tool would be written in C to match the existing toolset and not introduce any new dependencies. (This seems like a decent portable JSON library: https://zserge.com/jsmn.html) One open question is where the output files should go? Should they go in the build/ directory, or should they go in the source tree, like how graphics processing currently works?

This tool will operate in different modes depending on the input file.

1. Map data
2. Map ordering
3. Layouts

Another question is how dependency scanning will work for the layouts files, since they have a one-to-many mapping. One .json file results in 441 individual layout header files, as well as the layout table file. I think this should be a simple Makefile rule, but I just wanted to call it out here.

Seems good to me, as you said that would be easier to process and edit than directly dealing with C code.

I think, back in the day, Touched also proposed a JSON solution.

Yeah this proposal seems like a good idea. All I need is some converter from the old map format so I don't have to redo everything I already did.