Northland/Nordland or 8th Wonder of the World/Weltwunder map.dat file format

gistfile1.txt

<!-- mediawiki -->

<!-- Note: the decoding is not completely right because on bigger maps my parser crashes. For smaller maps it works though. -->

The map.dat has no index table at the beginning of the file. It rather uses entry-headers (called "hoix"es) that state what type of content the next block contains and how long it is. The byte-order is  Little Endian.

== Hoix == 
A hoix denotes a header of a datablock in the map.dat. The file starts with this.

{| class="wikitable"
! Datatype
! Name
! Notes
|-
| char[8]
| name
| It is the name of the entry. Additionally, it determines the type of data it contains.
|-
| uint32
| unknown
| 
|-
| uint32
| blocklength
| The size of the data block in bytes that starts directly after this hoix block. A either a new hoix comes directly after or the end of file is reached.
|-
| uint32
| unknown
| 0 every time?
|-
| ubyte[3]
| unknown
| 0x3FA017 every time?
|-
| ubyte
| unknown
| 
|-
| ubyte[6]
| unknown
| 
|-
| ubyte[2]
| unknown
| Sometimes the last hoix is two bytes too short so end of file might be already reached here!
|}


The "name" field can be one of the following:
{| class="wikitable"
! Datatype
! Notes
|-
| <tt>[[#hoixigol|hoixigol]]</tt>
| 
|-
| <tt>[[#hoixmmgl|hoixmmgl]]</tt>
| 
|-
| <tt>[[#hoixzisl|hoixzisl]]</tt>
| Mapsize
|-
| <tt>[[#hoixehml|hoixehml]]</tt>
| Heights
|-
| <tt>[[#hoixapml|hoixapml]]</tt>
| 
|-
| <tt>[[#hoixbpml|hoixbpml]]</tt>
| 
|-
| <tt>[[#hoixtlml|hoixtlml]]</tt>
| Minimapcolor - normally linked with LSC (e.g. Trees are green on minimap)
|-
| <tt>[[#hoixvlml|hoixvlml]]</tt>
| Additional data for LSC - number of items on the ground; content of chests; ...
|-
| <tt>[[#hoixplml|hoixplml]]</tt>
| 
|-
| <tt>[[#hoixocml|hoixocml]]</tt>
| 
|-
| <tt>[[#hoixwtml|hoixwtml]]</tt>
| 
|-
| <tt>[[#hoixsmml|hoixsmml]]</tt>
| 
|-
| <tt>[[#hoixrpml|hoixrpml]]</tt>
| 
|-
| <tt>[[#hoixbwml|hoixbwml]]</tt>
| LSC; unknown; On trees: if harvestable=1, else=0
|-
| <tt>[[#hoixbbml|hoixbbml]]</tt>
| LSC
|-
| <tt>[[#hoixorml|hoixorml]]</tt>
| 
|-
| <tt>[[#hoixbsml|hoixbsml]]</tt>
| 
|-
| <tt>[[#hoixoaml|hoixoaml]]</tt>
| 
|-
| <tt>[[#hoixfhml|hoixfhml]]</tt>
| 
|-
| <tt>[[#hoixocal|hoixocal]]</tt>
| 
|-
| <tt>[[#hoixwsal|hoixwsal]]</tt>
| 
|-
| <tt>[[#hoixmfal|hoixmfal]]</tt>
| 
|-
| <tt>[[#hoixdnex|hoixdnex]]</tt>
| 
|-
| <tt>[[#hoixmmme|hoixmmme]]</tt>
| 
|-
| <tt>[[#hoixrbme|hoixrbme]]</tt>
| Lights
|-
| <tt>[[#hoixcvme|hoixcvme]]</tt>
| Vertexcolors
|-
| <tt>[[#hoix1mme|hoix1mme]]</tt>
| 
|-
| <tt>[[#hoiximme|hoiximme]]</tt>
| 
|-
| <tt>[[#hoixdpae|hoixdpae]]</tt>
| Ground type dict
|-
| <tt>[[#hoixapme|hoixapme]]</tt>
| Ground texture triangles
|-
| <tt>[[#hoixbpme|hoixbpme]]</tt>
| Ground texture triangles
|-
| <tt>[[#hoixdtae|hoixdtae]]</tt>
| 
|-
| <tt>[[#hoix1tme|hoix1tme]]</tt>
| 
|-
| <tt>[[#hoix2tme|hoix2tme]]</tt>
| 
|-
| <tt>[[#hoix3tme|hoix3tme]]</tt>
| 
|-
| <tt>[[#hoix4tme|hoix4tme]]</tt>
| 
|-
| <tt>[[#hoixdlae|hoixdlae]]</tt>
| 
|-
| <tt>[[#hoixalme|hoixalme]]</tt>
| LSC - Types
|-
| <tt>[[#hoixdnet|hoixdnet]]</tt>
| EOF
|}
They are in order of natural occurrences.

=== hoixzisl ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| uint32
| width
| 
|-
| uint32
| height
| 
|}
These values are equivalent to the ones seen in the [[internal Editor]] and half of the values specified in the [[external Editor]].

=== hoixehml ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Every decoded "data" field inside the [[#Common Data|COMMON_DATA]] represents the height value from 0 to 255 of one field. common_decoded holds the height data as one big array for the whole map. From this array the game builds the map from left to right, from top to bottom.
|-
|}

=== hoixtlml ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Decoded like [[#hoixehml|hoixehml]].
|-
|}

=== hoixbwml ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Decoded like [[#hoixehml|hoixehml]].
|-
|}

=== hoixbbml ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Decoded like [[#hoixehml|hoixehml]].
|-
|}

=== hoixrbme ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Exactly like [[#hoixehml|hoixehml]] except that it contains the light information instead of height.
|-
|}

=== hoixalme ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Decoding|COMMON_DECODING]]</tt>
| common_decoded
| Decoded similar to [[#hoixehml|hoixehml]] EXCEPT that it uses two bytes instead of one for the data (the "number" is not affected; still one byte).
|-
|}

== Common ==
=== Common Decoding ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| <tt>[[#Common Header|COMMON_HEADER]]</tt>
| header
| 
|-
| <tt>[[#Common Data|COMMON_DATA]]</tt>
| data
| 
|}

=== Common Header ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| ubyte
| unknown
| 
|-
| uint32
| blockLength
| represents the number of bytes that are between the end of this uint32 until the next hoix
|-
| ubyte[12]
| unknown
| 
|-
| uint32
| someLength
| ? same as blocklength before
|}

=== Common Data ===
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| ubyte
| number
| Either the number of fields for which the next byte is the data OR the number of bytes following where every byte is the data for one field.
|}

If number is > 128, the number stands for the number of fields the data is valid for,
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| ubyte
| data
| The data that is encoded here is valid for the following "number" fields. (-> run-length encoding) The data is specific to the hoix.
|}

else, "number" is the number of bytes that are following where every byte is the data for one field.
{| class="wikitable"
! Datatype
! Name
! Notes
|-
| ubyte
| data[number]
| Every data-byte is the data for exactly one field. The data is specific to the hoix. 
|}

Hi @Nithanim, thanks for the quick reply! (I didn't actually expect a response anytime soon, if ever 😀) I found this gist when I was desperately searching for some data regarding map.dat files, so eventually I googled hoixigol and this page was the only result. 😀 Too bad about your wiki, I was actually hoping that you can point me to the original wiki where I can the links to other pages. I copied this gist into my wikipedia sandbox where I'm adding a little bit of more info as I get deeper.

Also thanks for recommending Siguza, I think I stumbled upon a web page about Cultures that had his Github avatar!

My ultimate goal is to make this game actually run on web with all its details, although I will probably never reach that far! 😆 So far I've been able to decode the game's main data file (DataX\data0001.lib), decode all of the CIF files, load pcx palettes, render BMD graphics on a webpage, and - thanks to the info on this page - decode sections of a map.dat files. I can now render each hoix section's data to get a better sense of the kind of data it holds.

I was also able to make sense out of hoixehml section for elevation data, so it's solved now. Thanks! I understand that your time is limited, but if you were ever interested again in picking up where you left on reversing this game, just ping me somewhere I'll be more than glad work together. Oh and by the way, feel free to check out my cultures2-engine project! I push my latest developments in there almost everyday. (Although it's not a polished tool or anything, just a bunch of scripts and a HTML canvas that I render things onto! 😁)

Thanks again, Nithanim!

Hello @martianboy,

this is very interesting how you found this gist and that it at least helped you a little bit. It is really genious of you that you just put the gist in the wikipedia sandbox to view it :D
In terms of my "wiki", do not think that it is a lost goldmine because I was the creator and only contributer and can tell you that the only page of interest was the one above. Others were just basic articles or deeplinks to Molt's file format specs.

Wanting to recreate this game is a bold move. I have started a "remake" two or three times now but it seems that I am not the kind of person for such an endeavour. I tried to hold it close to the original so I could load all the game data and have the original feeling.

You actually seem to have more decoder for the formats because you also managed to decode BMD. I still have Molt's php script and testfiles for it but never managed to decrypt that and get something working, sadly. I am glad that you got you heightmap working now! I actually built a hightmap viewer once after I found out the data should be read. Just be advised that the 128 number was guessed and it probably is rather >= 128 because it most likely checks the first bit. Take everything with a grain of salt :D

I should have put all my decoders (and encoders) into one project like you did with [https://github.com/martianboy/cultures2-engine]. It's good to have all working code and therefore knowledge in one place. However it is JS so not really my thing :P My stuff is also pretty bad and also non-sensical to use so don't worry about that.

I have never "given up" reversing cultures2 but since I have a job I just put the remaining free time in my to-be startup company.

In the attachments you can find a couple of files I wanted to send you. One is where I am fooling around with the textures. The next two are about debug map which I actively modify to try things out. Then there is (probably) the file with the most recent findings in it. Tough, there are a lot of conditionals in there which are most likely wrong because they do not make sense to be there but make things work. The triangle parts also have the problem, that parsing only works on small maps and crashes on bigger. Maybe we find the root cause someday! And the last file is a screenshot of the "latest" version of the remake.

If you have any specific question I will gladly try my best to answer. Or if you want to have a look at some parts together would be great too. My heart is still burning for this game and I am extremely happy that you are going down the rabbit hole too to discover its secrets!

Thank you very much, martianboy!

EDIT: The fileupload is nice and broken so here you go: https://drive.google.com/open?id=1I6EI2mHyy1sO2kBVzkE4epEDK2VE_7WV
Also I made this https://www.moddb.com/games/8th-wonder-of-the-world/downloads/cultures-map-development-kit-0-0-2-windows if you haven't seen it yet (because I had to delete it from the forums).

YES YES YES! FINALLY I AM ABLE TO DECODE BMD FILES! Thanks to your C code, @martianboy! The documentation I have from Molt was wrong at two points. I ignored the docs more or less this time and just straight up ported your C to java. Finally!

@Nithanim oh wow that's awesome, I didn't know you still wanted to experiment with those! 😃 My latest code for reading BMD files are in my Rust Wasm package, here's a link: https://github.com/martianboy/cultures2-wasm/blob/master/src/bmd.rs

I'm new to Rust, though, and the code quality is still mediocre. But I figured a bunch of other details like frame type 4, and the metadata fields on frame info headers.

I'm already able to render the whole ground of the map, and in just a few days I'll be able to render landscape items (trees, meadow, goods, etc) over it as well. There are still a few bugs here and there. I can prepare a demo for you if you wanna see the progress!

By the way @Nithanim, if you like, I'll be more than glad to chat and exchange ideas regarding this over Telegram or something.

I never got BMDs working and left them because it was just a mess, until you suddenly showed up by surprise and found your reader. I will have a look at your rust code. I do like rust but I have never gotten into it but looks like a good opportunity for reading some!

You are loooong ways ahead of me in terms of reversing the game and I am pretty sure that you are the one with the most knowledge in that regard. Though, there is still the trickery of Remik's Editor where I don't know what he did.

After you showed up I wanted to get back to the c2m file but quickly ran into the issue that there is no good viewer/(un)packer available. So I am currently developing a multitool like one that exist for other games (CascView, Gw2Browser, FinalBig, ...) and then ran into the dreaded bmd format again.

Feel free to add me on Telegram! You should be able to find me with the same name/handle.