For: BATCH #1-3 -- LMP Structs Exported from Neo as CSV.7z

Explanation.txt

_NEO_EXPORTS
============

From HHD Hex Editor Neo to *.CSV files.

While 010 Editor also offers exporting of structures, Neo's
format and look is overall better. This is mostly an effort to
liberate this data from Neo -- using WebGL to read directly
from BSP files is a 9-year old pursuit with many examples:

	Epic BSP Python library with all 64 Lumps I never
	found until right now -- but it's pretty epic

	https://pysourcesdk.github.io/ValveBSP/api/lumps/lump_0.html

		IN THIS *.TXT ALL 64 LUMPS ARE MENTIONED,
		EVEN THOUGH NEITHER OF US UNDERSTAND ALL 64.

	Live Working Demos

	https://hlbsp.netlify.app/
	https://devanbuggay.com/bspview/
	http://media.tojicode.com/q3bsp
	https://metapyziks.github.io/SourceUtils/maps/de_nuke/
	https://noclip.website/

	Githubs, respectively (in order of above)

	https://github.com/rein4ce/hlbsp
	https://github.com/sbuggay/bspview
	https://github.com/toji/webgl-quake3
	https://github.com/Metapyziks/SourceUtils
	https://github.com/magcius/noclip.website

	First one above is the best, here's structs/maplist

	https://github.com/rein4ce/hlbsp/blob/master/js/hlbsp.js
	https://github.com/rein4ce/hlbsp/blob/master/js/maplist.js

	And I like that this one is entirely contained in Github

	https://github.com/Metapyziks/SourceUtils

	Tutorials

	https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Tutorial/Creating_3D_objects_using_WebGL
	https://www.tutorialspoint.com/webgl/webgl_cube_rotation.htm

	More info

	https://news.ycombinator.com/item?id=22686452
	https://twitter.com/rewindcs?lang=en
	https://blog.tojicode.com/2011/09/source-engine-in-webgl-tech-talk.html
	https://discourse.elm-lang.org/t/render-quake-3-arena-maps-with-elm-and-webgl/3820

Apparently BSP viewers are the go-to thing for game coders
to hone their skills on before going off on their own stuff,
considering these countless examples that are just WebGL!

Biggest thing to improve Arcweave is an in-editor / website
for collaborating / visualizing ideas, like this game's dev:

	https://en.wikipedia.org/wiki/Hydrophobia_(video_game)

BATCHES
=======

Done in literal order of importance, and only after I've fully
understood each of them. This is about obsoleting use of Neo,
and even PowerShell could make use of the *.CSV's "Addresses"
to directly replace the data.

While a WebGL viewer should (rationally) parse the BSP and not
need these *.CSV's, these *.CSV's support an effort to create
a fully human-readable document, that keeps track of all the
hex edits to *.LMP files, and applies all edits automatically.

When it comes to Lump 0 Entities, those have _commentary.txt
files -- supplementary *.CSV files could "edit" plaintext
exports like the ones being made here (hastened by AutoHotkey).

##############################################################
##############################################################
##############################################################
##############################################################

####  BATCH #1
##############

Batch contains all the lowest-level lumps: the ones critical
for visualizing the map independent of BSP files.

Liberating structured BSP data out of binary form to CSV is
intended to facilitate in-game DebugDraw, browser WebGL, or
other simple 3D render systems to effortlessly parse this
human-readable plaintext data and do anything with it. Data
represented like this does produce much larger filesizes,
but plaintext compresses to only 3-5% their original size.

Lump 0 Entities is plaintext... now the best lumps are, too.

##############################################################

========================
Geometry -- Input
Geometry -- Output
========================

These lumps:

	03_LUMP_VERTEXES
	12_LUMP_EDGES
	13_LUMP_SURFEDGES

Describe the lowest level of world geometry available.
Their only downside is they don't seem to contain data
for baked-in clips (visible with r_drawclipbrushes 2).

Visualizing these in-game would facilitate VScript clip
creation (with env_physics_blocker) and moving vertices
to change how things appear -- collision is unaffected!

Once extracted and parsed, it'd be possible to view any
map's worldspawn, in a "dark empty" BSP file, to then
move/adjust any vertex, and dump them for hexedits later.

Following structures from:

	https://github.com/Tsuey/L4D2-BSP-Lump-Editing/blob/main/Releases/bsp_l4d2.h

Lump 3 (max 65536) refers to the X/Y/Z of each vertex:

	struct dvertex_t
	{
		Vector	point;
	};

Lump 12 (max 256000) refers to a pair of vertex indices
and is a straight line between them:

	struct dedge_t
	{
		unsigned short	vertex[ 2 ];	// vertex indices
	};

Lump 13 (max 512000) which reference Lump 12 where if
positive the edge goes from 1st to 2nd vertex, if negative
from 2nd to 1st:

	struct dsurfedge_t
	{
		signed int	edge;		// references LUMP 12 edge with + or - sign
	};

Lump 3 is the most important to visualize in-game, but
surf/edges may be informative (but the vertices will always
be what is edited, the edges will just "play along").

========================
Textures -- Input
Textures -- Output
========================

TO_DO_-_MAY_BE_USEFUL_WITH_ABOVE_BUT_IGNORE_FOR_NOW

These lumps:

	02_LUMP_TEXDATA				SEE BATCH #2
	06_LUMP_TEXINFO				SEE BATCH #2
	07_LUMP_FACES -- non-HDR fallback	SEE BATCH #2
	43_LUMP_TEXDATA_STRING_DATA		SEE BATCH #2
	44_LUMP_TEXDATA_STRING_TABLE		SEE BATCH #2
	58_LUMP_FACES_HDR -- HDR default	BATCH #1

Command "mat_surfaceid 2" reveals the 58_LUMP_FACES_HDR
indices. Currently, beyond using this data to more precisely
replicate the material/texture data that VMF parsing offers,
there's no use for it since this command allows built-in
identification of Face ID's.

Note that 07_LUMP_FACES can be zeroed entirely since it is
only used if 58_LUMP_FACES_HDR is empty -- or if HDR is
disabled, an option that doesn't even seem to be available
in the vanilla Settings menu. Since Lump 58 takes priority,
it is there where texture changes visually take effect.

For now, only 58_LUMP_FACES_HDR will be exported to CSV,
because it has tons of valuable data in it, even though it
will require interfacing with ALL Lumps 2/6/43/44 to merely
figure out what texture info is on the face... its data is
still largely made worthless since "mat_surfaceid 2" really
does trivialize getting in-game Face ID's, though.

The following structure is for 07_LUMP_FACES, but Lump 58
is 100% identical (just a small amount of data differs),
and there can be a max 65536 ID's:

	struct dface_t
	{
		unsigned short	planeNum;			// the plane number
		char		side;				// faces opposite to the node's plane direction
		char		onNode;				// 1 of on node, 0 if in leaf
		int		firstEdge;			// index into surfedges
		short		numEdges;			// number of surfedges
		short		texInfo;			// texture info
		short		dispInfo;			// displacement info
		short		surfaceFogVolumeID;		// ????
		char		styles[ 4 ];			// switchable lighting info
		int		lightOfs;			// offset into lightmap lump
		float		area;				// face area in units^2
		int		lightmapTexMinsInLuxels[ 2 ];	// texture lighting info
		int		lightmapTexSizeInLuxels[ 2 ];	// texture lighting info
		int		origFace;			// original face this was split from
		unsigned short	numPrims;			// primitives
		unsigned short	firstPrimID;			// ????
		unsigned int	smoothingGroups;		// lightmap smoothing group
	};

Regarding 58_LUMP_FACES_HDR, note that the following maps:

	c5m1_waterfront_sndscape
	curling_stadium

Don't have Lump 58 ("credits" does, it was actually compiled
with HDR and has a nice light in it, interestingly).

========================
Solidity -- Input
Solidity -- Output
========================

TO_DO_-_UNLIKELY_TO_BE_USEFUL_SO_USE_HAMMER_INSTEAD

These lumps:

	01_LUMP_PLANES				SEE BATCH #2
	18_LUMP_BRUSHES				SEE BATCH #2
	19_LUMP_BRUSHSIDES			SEE BATCH #2

Influence worldspawn/brush solidity. LUMP 1 should never
be modified since it also affects visibility, and LUMP 19
only affects bullet/decal application -- making LUMP 18
brushes non-solid already makes them non-solid to bullets.

Specially-decompiled Hammer VMF's eliminate the need for
these since its "Show Selected Brush Number" is workable,
once its criminal inconsistencies are over-come (i.e. by
avoiding func_detail decompile dupes, and refraining from
toggling of Visgroups since those surprisingly change their
numbers, it's extremely volatile but works if careful).

Visualizing these Brush ID's in-game would probably be
very convoluted and require above-mentioned 58_LUMP_FACES_HDR
data to even have a chance.

##############################################################
##############################################################
##############################################################
##############################################################

####  BATCH #2
##############

Hereafter, see this for all structure definitions:

	https://github.com/Tsuey/L4D2-BSP-Lump-Editing/blob/main/Releases/bsp_l4d2.h

This batch is all the other lumps I'm comfortable with so
far. Lump 43 plaintext dump is definitely incomplete (c2m4
end has a Midnight Riders name cut off) and is very messy,
but is only for passing reference -- Lump 0 Entities and
Lump 43 warrant a different non-CSV export approach later.

Note that Lump 6 has 2 "Vecs[ 8 ]" floats, which have been
simplified from their actual "Vecs[ 2 ][ 4 ]" since Neo
gives an error for double-arrays -- the structures used
are otherwise 100% accurate/reliable.

As always, Lump 7's struct is identical for 27 and 58.

##############################################################

========================
Textures -- Input
Textures -- Output
========================

Last batch, 58_LUMP_FACES_HDR was handled -- now it's time
for all the others I skipped over last time.

The intent here is to expose the "hierarchy" of texture
lumps and how they interact -- while Lump 58's data is
very cryptic, exact texture names used are recursively
found through these. Lump 27 is the "pre-split" ID's and
exclusively useful for BSPSource / decompilation, but
are included this time for completion.

These lumps:

	02_LUMP_TEXDATA
	06_LUMP_TEXINFO
	07_LUMP_FACES -- non-HDR fallback
	27_LUMP_ORIGINALFACES
	43_LUMP_TEXDATA_STRING_DATA
	44_LUMP_TEXDATA_STRING_TABLE
	45_LUMP_OVERLAYS
	47_LUMP_FACE_MACRO_TEXTURE_INFO

While Lump 45 Overlays is a bit of a stretch, they really
are textures / materials and behave similarly. One exception
is I am NOT including Lump 40 PakFile and Lump 42 Cubemaps
here, because while those do technically contain textures,
the PakFile is a glorified ZIP file and Cubemaps are built-in
coordinates that are best modified through recompile, since
if replacing them all (as with the "c1m3_mall" fixes) the
generated VTF filenames also need to be updated.

So, the following lumps are 100% ignored. Additionally,
Lump 0 Entities is plaintext and needs better than CSV
exports (Lump 35 GameLump strings are null-terminated
so those work well with CSV exports); Lump 59 MapFlags
is always exactly 4 (non-header) bytes for all maps;
and while I have no facts / awareness on Lump 60 and 61,
Lump 45 for Overlays is already surely all the data I'll
need regarding those -- and Overlays are niche as is.

	00_LUMP_ENTITIES			IGNORED FOREVER
	40_LUMP_PAKFILE				IGNORED FOREVER
	42_LUMP_CUBEMAPS			IGNORED FOREVER
	59_LUMP_MAP_FLAGS			IGNORED FOREVER
	60_LUMP_OVERLAY_FADES			IGNORED FOREVER
	61_LUMP_OVERLAY_SYSTEM_LEVELS		IGNORED FOREVER

Note that Lumps 0 and 59 I'm throwing here just to get
them covered -- all the other "IGNORED" lumps are texture
related... even Lump 0, honestly, because in order to
have a web-based OpenGL viewer, the most significant thing
almost all of them lack is showing *.MDL files because
they require a lot of additional assets (materials) -- so,
in that sense it fits; and Lump 59:

	https://pysourcesdk.github.io/ValveBSP/api/lumps/lump_59.html

Is apparently "Lighting" related, covered in BATCH #3,
but is ignored for CSV given its extreme simplicity.

========================
Solidity -- Input
Solidity -- Output
========================

Skipped last batch -- but now it's time.

Lump 1 is very dangerous to modify but is important to
work with given its vital role in "segmenting visibility",
aka "cordoning" off the map into boxes that are made
inside of and in between every brush (if not func_detail).

But Lump 1 does affect solidity / collision, as do the
map's "prop_static" entities (which can be moved out of
the map and you can walk through where they once were).
Lump 14 "brush model" (opposite of "studiomodel" / *.mdl)
vertices curiously aren't in Lump 3, and while VScript
can clone any of these already, there's still investigating
to do, so export those. Lump 18 can negate player collision
and 19 bullet collision / decals.

These lumps:

	01_LUMP_PLANES
	14_LUMP_MODELS
	18_LUMP_BRUSHES
	19_LUMP_BRUSHSIDES
	35_LUMP_GAME_LUMP

Note that Lump 29 PhysCollide still needs work so BATCH #3
will address that. When it comes to the "prop_static", we
are lucky Lump 35 simplifies them so much -- the following
lumps are ALWAYS empty / deprecated from L4D2, and all 4
of them are older / more archaic "prop_static" implementation:

	22_LUMP_PROPCOLLISION -- convex lists	IGNORED FOREVER
	23_LUMP_PROPHULLS -- convex hulls	IGNORED FOREVER
	24_LUMP_PROPHULLVERTS -- collis verts	IGNORED FOREVER
	25_LUMP_PROPTRIS -- tri idx start/end	IGNORED FOREVER

While Lump 29 PhysCollide is basically a glorified VPHYSICS
studiomodel file, it's not the be-all-end-all since Lump 18
can modify brush solidity all the same.

I define "Solidity" as different from "Collision":

	Solidity	Handled in this BATCH #2
	Collision	Handled later in BATCH #3 (i.e. Lump 29)

Where "Solidity" is easier / more accessible to change, and
Collision deals with (assumed) math-based pre-calculations.

##############################################################
##############################################################
##############################################################
##############################################################

####  BATCH #3
##############

Collision, Displacement, Lighting, Visibility, and various
lumps that will be ignored forever are covered here.

Compared to everything above: everything here is very advanced
and offers mitigating returns on what can be changed -- so
there's not much benefit to do much with these, and exports
to CSV may not even be useful as use is yet to be undetermined.

Truthfully: I'm not sure about these categories and I'm
not sure which are to be ignored -- don't take anything below
as serious or final until a LOT more tests are done.

##############################################################

========================
Collision -- Input
Collision -- Output
========================

These lumps:

	11_LUMP_FACEIDS
	29_LUMP_PHYSCOLLIDE
	30_LUMP_VERTNORMALS
	31_LUMP_VERTNORMALINDICES
	37_LUMP_PRIMITIVES
	39_LUMP_PRIMINDICES

Lump 11 FaceIDs (can't possibly be?) that useful when
the other 3 Face lumps contain TONS of useful data.
The others are large files and may be useful, but I've
not yet tested any structures for. They also don't
seem crucial to learn in terms of their enabling of
desirable map modifications that can be done with other
lumps -- the hope here, is possibly making convoluted
approaches (like finding Lump 18 Brush ID's) easier,
which requires a LOT more testing.

Lump 29 is a glorified VPHYSICS "studiomdl" and might
supplement Lump 3 Vertexes inability to modify collision.
Also I still have no idea how to modify geometry of
Lump 14 Models since those vertices aren't even in Lump 3,
so one of these may support exactly that.

Regarding:

	38_LUMP_PRIMVERTS

Next to no maps (or literally none) use that lump, so
again: more investigation required... it's merely thrown
here along with the other "prim" and "verts" names.

========================
Displacement -- Input
Displacement -- Output
========================

These lumps:

	26_LUMP_DISPINFO
	28_LUMP_PHYSDISP
	33_LUMP_DISP_VERTS
	34_LUMP_DISP_LIGHTMAP_SAMPLE_POSITIONS
	48_LUMP_DISP_TRIS

Modifying displacements comes long after everything else,
if at all -- just recompile maps at this point of crazy.

Also these:

	32_LUMP_DISP_LIGHTMAP_ALPHAS
	63_LUMP_DISP_MULTIBLEND

May be used on few (or literally no) maps, they're far
more exclusive than the ones above -- this is simply a
matter of gathering all lumps with "disp" in them.

========================
Lighting -- Input
Lighting -- Output
========================

These lumps:

	51_LUMP_LEAF_AMBIENT_INDEX_HDR
	52_LUMP_LEAF_AMBIENT_INDEX
	53_LUMP_LIGHTING_HDR
	54_LUMP_WORLDLIGHTS_HDR
	55_LUMP_LEAF_AMBIENT_LIGHTING_HDR
	56_LUMP_LEAF_AMBIENT_LIGHTING

Are by far the most important ones when it comes to lights.
Lump 53's roughly equal the size of Lump 40 PakFiles and
are bitmaps for the lighting -- they glitch out if their
file sizes change, so changing maps Nighttime to Daytime
without recompiling IS possible BUT requires the full
file size (and increasing resolution would increase size).

The fact "Ambient" is in their name is the dead give-away,
but they're also closely related to leafs / visibility...
how, though, I've not yet observed.

Also:

	08_LUMP_LIGHTING
	15_LUMP_WORLDLIGHTS

Respectively, those correspond to Lump 53 and 54. It's very
interesting that these Lumps 8 and 15 do not exist for the
overwhelming majority of Valve maps -- only 3 maps escape
the mould of expectation.

There's a quick distinction to be made here, since while
not enough maps have these lumps to bother with CSV exports,
it's interesting to observe which do and why:

	"credits"

		Same common lumps as all other maps,
		because Valve compiled this in HDR.

	"curling_stadium"

		The only Valve map that has a Lump 8.
		Lump 8 is for non-HDR, so all HDR-named
		lumps are absent for this map -- all
		because this is the only map Valve
		deliberately compiled with HDR un-checked.

	"c5m1_waterfront_sndscape"

		Doesn't have Lump 8 non-HDR, and doesn't
		have Lump 53 HDR, or Lump 8 -- for this
		map, Valve didn't compile with ANY lighting.

HDR is optional, and Lump 7 etc. exist as REQUIRED fall-backs.
So, it's very unexpected that Lump 8 and 15 don't exist for
Valve's major campaigns -- which tells me that I should be
able to fully ignore Lump 7 Faces in favor of Lump 58 Faces HDR,
since non-HDR lighting doesn't exist in the lumps anyway...
and it's one less lump to modify when making brushes invisible.

========================
Visibility -- Input
Visibility -- Output
========================

These lumps:

	04_LUMP_VISIBILITY
	05_LUMP_NODES
	10_LUMP_LEAFS
	16_LUMP_LEAFFACES
	17_LUMP_LEAFBRUSHES
	41_LUMP_CLIPPORTALVERTS

Are the big ones to keep in mind. In terms of WebGL
drawing the maps, Visibility can be ignored completely
if there's no performance drawbacks in rendering entire
maps at once -- they're made for 2009 and it's 2021.

Not all of these effect change (like modifying which
nodes are connected to each other), and Lump 4 is not
fully explored -- and is also compressed and tricky.

Also, these are usually very small and I'd be surprised
if they end up being useful:

	09_LUMP_OCCLUSION
	20_LUMP_AREAS
	21_LUMP_AREAPORTALS
	46_LUMP_LEAFMINDISTTOWATER

The *.PTL (Portal) file is assumed to need regeneration
from (one of these?) lumps in order to re-run VVIS
on already-compiled BSP's, to edit their visibility.

Lastly this:

	36_LUMP_LEAFWATERDATA

I just have no idea -- and mitigating returns for exports
to CSV is absolutely certain at this point.

========================
OTHER / IDK -- Input
OTHER / IDK -- Output
========================

These lumps:

	49_LUMP_PROP_BLOB
	50_LUMP_WATEROVERLAYS
	57_LUMP_XZIPPAKFILE
	62_LUMP_PHYSLEVEL

Really all of BATCH #3 belongs under "IDK", but the above are
possibly full-on deprecated, few (or no) L4D2 maps make use
of them, and Lump 50 is more Overlay stuff -- for something as
niche as Overlays, they sure as hell do have tons of lumps.

##############################################################
##############################################################
##############################################################
##############################################################