Sparkxxx/How_to_NodeMCU_lua.txt

## How_to_NodeMCU_lua.txt

Errata (Start Reading Here):
	Flash = micro-controller storage facility that can be written ~100.000 times (hardware related, ESP-12e has 4Mb)
	Flash is used for 3 things:
		- Firmware = the main program interpreter, in our case Lua
		- SPIFFS = think at this as a partition of the hard-drive in your computer that stores all sort of files
			SPIFFS is used for 3 things:
				- to store/load program files.lua directly, including init.lua !!! these files will be loaded in RAM and interpreted/compiled to be executed !!!
				- to store ANY type of files you need (Eg: configs, logs, images, certificates, etc.)
				- to store a special .bin file called LFS that contains PRE-COMPILED .lua files (your program) - https://nodemcu.readthedocs.io/en/dev/lfs/
					The LFS file is uploaded like any other file in SPIFFS (Eg: use ESPlorer or at build time) AND NEEDS TO BE absorbed/initialized by the firmware with:
						node.LFS.reload("lfs.img")
				????????????Q.0 Is there any option/define that will allow us to set the SPIFFS size to allow for LFS partition + lfs.img + some extra ????????????
		- LFS (Lua-File-Storage) upon initialization above will transfer the contents of the "lfs.bin" file in a specially designated flash/SPIFFS sub-partition (Eg: static precompiled folder)
			After initialization the SPIFFS lfs.bin is no longer needed and can/should be removed/deleted
				Check the existence of the fls contents with:
					print("\nFiles in LFS")
					for k,v in pairs(node.LFS.list()) do print(k,v) end
				????????????Q.1 IS THERE a ls/dir function for LFS so that we see also the folder structure/namespaces - Usefull for loading modules ????????????
				????????????Q.2 How are the modules from LFS loaded, who's registering them - Is there a framework/convention we can work with like
					- https://github.com/HHHartmann/nodemcu-LFS-base
					- https://github.com/freyssin/elmis - has an example (X.lua) of how to develop your own module
				maybe this approch will help mitigate changes occuring between LFS/NodeMCU versions ????????????

	init.lua = default file that is called by the firmware at startup (see it as your program entrance)
		- this behavior can be modified in app/include/user_config.h
			#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)" <<<--- on boot look for _init.lua in LFS not for default init.lua from SPIFFS
				NOT advisable if you want to mitigate problems with OTA when the new lfs.img can be corrupted.
		- init.lua has to:
			- check for the existence of LFS files and consistency
				- (for k,v in pairs(node.LFS.list()) do print(k,v) end)
				!!!!! as far as I understand the files are present in LFS but not yet LOADED, I mean not yet available to be called with 'require "LuaOTA.check"'
					https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/luaOTA
					and I don't understand how is it enough to call require if modules are not registered (or are they?)
				I couldn't make the example work, could not get to run LuaOTA.check (calling it many names) from init.lua !!!!!
			- allow some 5sec for the program to be interrupted in case things got messed up (infinite reboot)
			- transfer control/call to _init.lua
	_init.lua = the SPIFFS and/or LFS "twin brother / big brother" of init.lua
		????????????Q.3 Here things start to become fuzzy, Where should _init.lua reside in SPIFFS or in LFS root or in LFS/namespace/ ????????????
		????????????Q.4 How should _init.lua be called node.flashindex("_init")(), dofile ("LFS._init.lua"), dofile ("_init.lua") or require "_init" ????????????
		????????????Q.5 Should node.flashindex("_init")() be called to register the modules in proper namespaces before doing 'require "LuaOTA.check"' like in luaOTA example ????????????
		- _init.lua has to:
			- register the modules and make them available in their respective namespaces ????????????
			- establish wifi (sta or/and ap) (optional)
			- transfer control/call to NAMESPACE.yoursoftware.lua ????????????

	namespaces = When building lfs.img (see below) files can/should be put in FOLDERS and these FOLDER_NAMES BECOME NAMESPACES for lua.
		!!!!! In my opinion this would help in structuring the code if you have a very complex software !!!!!

	/nodemcu-firmware/local/lua = Contents of this folder is not directly installed into LFS, instead, the files therein are built into /nodemcu-firmware/local/fs/LFS.img.
		See the LFSimage target in tools/Makefile.
	/nodemcu-firmware/local/fs = Is used to generate the SPIFFS images (see the spiffsscript target in tools/Makefile), which land in files like bin/0xc0000-32mb.img which are ${flash_position}-${flash_size}.bin.
		In general, it's fine to use smaller SPIFFS images on larger flash devices, though I do not think SPIFFS will expand to fill free space without formatting from scratch.
		- https://github.com/nodemcu/nodemcu-firmware/issues/3299


	Flash available on the board
	0x00000000 flash strat address																						0xZZZZZZZZZ flash end address
	|------------------------|-----------------------------------------------------------------------------------------|

	Firmware space
	0x00000000 strat address  0xSOMETHING end address
	|------------------------| #define SPIFFS_SIZE_1M_BOUNDARY <<<--- Might be a good idea so that when flashing new firmware you don't overwrite/overflow to the SPIFFS start address ??????

	SPIFFS space
							   Start SPIFFS address 0xSOMETHING + headroom for a larger firmware	   End SPIFFS address 0xSPIFFS
	|-----Firmware space-----|--blank--|---------------------------SPIFFS-----------------------------|------blank-----| 0xZZZZZZZZZ flash end address

	LFS space
										Start LFS address 0xSOMETHING + headroom   					   End SPIFFS address 0xSPIFFS
	|-----Firmware space-----|--blank--||------LFS------|----------SPIFFS-----------------------------|------blank-----| 0xZZZZZZZZZ flash end address
														 End SPIFFS address 0xSPIFFS
	The address/location of LFS is not a concern for us since it's taken care of by NodeMCU. It is shown here for reference.
	We only have to allocate enough space to it based on the size of our developed software with #define LUA_FLASH_STORE 0x40000 <<<--- size of LFS // 0x40000 = 256kb


What we want:
	We want to load our .lua files in LFS so that they are not stored in memory and we have ram available 43KB to use them. (there is no other way for a serious program !!!)
ESP-x family has the firmware partition and the SPIFFS partition. LFS is part of the SPIFFS
	0x00000.bin will contain just the firmware.
	0x10000.bin will contain the SPIFFS.
	nodemcu_integer_dev_20200929-0843.bin has both

.bin files are in nodemcu-build/bin
.img LFS files can be found in {PathToLuaSourceFolder}

!!! Same file can exist in both LFS (not using ram) and SPIFFS (this uses ram)
!!! If above happens take care which one is executed when developing
!!! Change the order (Use LFS or SPIFFS file version) like described in https://nodemcu.readthedocs.io/en/dev/lfs/#choosing-your-development-life-cycle

Success looks like:
	We want to be able to:
		- build firmware with chosen modules on whichever git branch we choose,
		- build wih Lua 5.1.x or Lua 5.3.x,
		- build LFS containing our software,
		- build SPIFFS with preloaded files (Eg: init.lua, lfs.img),
		- flash them to hardware,
		- call funtions from LFS.

	Following the https://nodemcu.readthedocs.io/en/dev/getting-started/#minimal-lfs-example
		1. we must get to check our files are in LFS:
			for k,v in pairs(node.LFS.list())  do print(k,v) end
				1	HTTP_OTA
				2	_init
				3	dummy_strings
				4	lfs_fragments
		2. we must be able to load and use the functions from lfs

0. Create the folder that will serve as our base
	mkdir nodemcu; cd nodemcu

1. Get Docker NodeMcu builder image - https://github.com/marcelstoer/docker-nodemcu-build
	docker pull marcelstoer/nodemcu-build

2. Clone Repo - https://nodemcu.readthedocs.io/en/latest/build/#git
	master branch -> git clone --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git
	release branch -> git clone --recurse-submodules -b release https://github.com/nodemcu/nodemcu-firmware.git
	dev branch -> git clone --recurse-submodules -b dev https://github.com/nodemcu/nodemcu-firmware.git

3. Build Firmware - https://nodemcu.readthedocs.io/en/latest/build/

	Edit app/include/user_modules.h and comment-out the #define statement for modules you don't need.
	Edit app/include/user_config.h
		#define LUA_NUMBER_INTEGRAL <<<--- if not defined a float image will be built
		#define LUA_FLASH_STORE 0x40000 <<<--- size of LFS // 0x40000 = 256kb
		#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)" <<<--- on boot look for _init.lua in LFS not for default init.lua from SPIFFS
		#define SPIFFS_SIZE_1M_BOUNDARY <<<--- Might be a good idea so that when flashing new firmware you don't overwrite/overflow to the SPIFFS start address

	LUA 5.1 and 5.3 differ so we have to take care to build the correct firmware with docker option -e LUA=53
		LUA 5.1
		docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware marcelstoer/nodemcu-build build
			https://github.com/marcelstoer/docker-nodemcu-build#build-the-firmware
			nodemcu_${BUILD_TYPE}_${IMAGE_NAME}.bin is the combined firmware image you can flash. BUILD_TYPE is integer or float. For IMAGE_NAME, see the Options chapter below.
				Almost same but with .map ending, a mapfile will be saved that contains the relative offsets of functions.
			0x00000.bin will contain just the firmware.
			0x10000.bin will contain the SPIFFS.
		LUA 5.3
		docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -e LUA=53 marcelstoer/nodemcu-build build

4. Build LFS - .img files can be found in {PathToLuaSourceFolder}
	- https://github.com/marcelstoer/docker-nodemcu-build#create-an-lfs-image-for-esp8266
	- https://nodemcu.readthedocs.io/en/dev/lfs/#compiling-and-loading-lfs-images

	docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware -v {PathToLuaSourceFolder}:/opt/lua marcelstoer/nodemcu-build lfs-image
		This will compile and store all Lua files in the given {PathToLuaSourceFolder} folder including all SUBFOLDERS which WILL BECOME NAMESPACES for lua.
	docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware -v {PathToLuaSourceFolder}:/opt/lua marcelstoer/nodemcu-build lfs-image final/files.lst
		To only add specific files you can prepare a file containing the files to add and give them as paramater.

	LUA 5.1 and 5.3 differ so we have to take care to build LFS for the correct one with docker option -e LUA=53
		LUA 5.1
			docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua marcelstoer/nodemcu-build lfs-image
		LUA 5.3
			docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua -e LUA=53 marcelstoer/nodemcu-build lfs-image

		Example: Build LFS from lua_examples/lfs/
			docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua marcelstoer/nodemcu-build lfs-image
			Adding files: ./lfs_fragments.lua ./HTTP_OTA.lua ./_init.lua ./dummy_strings.lua
			creating LFS_float_20200928-0840.img
			creating LFS_integer_20200928-0840.img

3&4 Build NodeMCU firmware with init.lua and our firmware compiled in LFS.img file that is pre-loaded in SPIFFS image all at once
	- https://github.com/nodemcu/nodemcu-firmware/issues/3299

		- /nodemcu-firmware/local/lua = You put your firmware here to be compiled and stored in LFS.img.
			Contents of this folder is not directly installed into LFS, instead, the files therein are built into /nodemcu-firmware/local/fs/LFS.img. See the LFSimage target in tools/Makefile.

		- /nodemcu-firmware/local/fs = You put your init.lua here to be stored in SPIFFS and called at boot.
			Contents of this folder is used to generate the SPIFFS images (see the spiffsscript target in tools/Makefile), which land in files like bin/0xc0000-32mb.img
			which are ${flash_position}-${flash_size}.bin. In general, it's fine to use smaller SPIFFS images on larger flash devices,
			though I do not think SPIFFS will expand to fill free space without formatting from scratch.

		- You can land everything you want on the board in a single esptool.py flash command, but you will need to node.LFS.flash("LFS.img") at first boot.
			Probably the correct way to do this is to also land an init.lua containing the correct incantation in SPIFFS.
			(To avoid a reboot loop if something goes wrong with loading LFS, it may be useful to first test for LFS.img and then file.rename it to something else before node.LFS.flash-ing.
			At least in this case, you'll still have a Lua interpreter on the UART.)

5. Use NodeMCU PyFlasher to format from scratch - https://github.com/marcelstoer/nodemcu-pyflasher
	This will flash the firmware and upon boot will reformat SPIFFS (SPIFFS will expand to fill free space)
	This scenario can be used to manually upload init.lua and lfs.bin AFTER the formatting is complete
	NodeMCU firmware - choose the one you built (Eg: nodemcu_integer_dev_20200929-2150.bin)
	!!!!! TOTEST if files pre-built in SPIFFS dissapear upon formatting !!!!!
	Baud rate - 115200
	Flash mode Dual I/O (DIO) - This is dependent on your device - https://www.esp32.com/viewtopic.php?p=5523&sid=08ef44e13610ecf2a2a33bb173b0fd5c#p5523
	Erase flash - yes,wipes all data <<<--- SPIFFS will expand to fill free space formatting from scratch

5'. Reflash using NodeMCU PyFlasher to flash the image on ESP-12e - https://github.com/marcelstoer/nodemcu-pyflasher
	NodeMCU firmware - choose the one you built (Eg: nodemcu_integer_dev_20200929-2150.bin)
	Baud rate - 115200
	Flash mode Dual I/O (DIO) - This is dependent on your device - https://www.esp32.com/viewtopic.php?p=5523&sid=08ef44e13610ecf2a2a33bb173b0fd5c#p5523
	Erase flash - no <<<--- SPIFFS is already expanded to fill maximum space by step 5
		!!!!! TOTEST I think doing this will also resize the SPIFFS partition !!!!!
		!!!!! TOTEST We do this with PyFlasher or use esptool directly and only flash the SPIFFS? which brings me to Q.0 !!!!!

	Command: esptool.py --port COM4 --baud 115200 --after no_reset write_flash --flash_mode dio 0x00000 D:\_NodeMCU LUA\dumps\nodemcu_integer_release_20200928-0829.bin --erase-all
	esptool.py v2.6
	Serial port COM4
	Connecting....
	Detecting chip type... ESP8266
	Chip is ESP8266EX
	Features: WiFi
	MAC: 18:fe:34:f4:ed:44
	Uploading stub...
	Running stub...
	Stub running...
	Configuring flash size...
	Auto-detected Flash size: 4MB
	Erasing flash (this may take a while)...
	Chip erase completed successfully in 9.0s
	Flash params set to 0x0240
	Compressed 659456 bytes to 443592...
	Wrote 659456 bytes (443592 compressed) at 0x00000000 in 39.2 seconds (effective 134.6 kbit/s)...
	Hash of data verified.
	Leaving...
	Staying in bootloader.
	Firmware successfully flashed. Unplug/replug or reset device
	to switch back to normal boot mode.

6. Using ESPlorer v0.2.0 - https://esp8266.ru/esplorer/
	HowTo ESPlorer - https://www.best-microcontroller-projects.com/installing-and-using-esplorer.html
	Remove the flash pin, open the serial COM with the board and power it

	Formatting file system. Please wait...
	..................................................
	NodeMCU 3.0.0.0 built with Docker provided by frightanic.com
		branch: release
		commit: 64bbf006898109b936fcc09478cbae9d099885a8
		release: 3.0-master_20200910
		release DTS: 202009090323
		SSL: true
		build type: integer
		LFS: 0x40000 bytes total capacity
		modules: crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm2,rtctime,sjson,sntp,softuart,tmr,uart,wifi
	 build 2020-09-28 08:29 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

	----------------------------
	No files found.
	----------------------------
	>
	Total : 3073746 bytes
	Used  : 0 bytes
	Remain: 3073746 bytes

7. Uploading to ESP file LFS_integer_20200928-0840.img...Success
	>
	----------------------------
	LFS_integer_20200928-0840.img : 5670 bytes
	----------------------------
	Total file(s)   : 1
	Total size      : 5670 bytes

	Total : 3073746 bytes
	Used  : 6024 bytes
	Remain: 3067722 bytes
	file.rename("LFS_integer_20200928-0840.img","lfs.img")
	>
	----------------------------
	lfs.img         : 5670 bytes
	----------------------------
	Total file(s)   : 1
	Total size      : 5670 bytes

	Total : 3073746 bytes
	Used  : 6024 bytes
	Remain: 3067722 bytes

8.	Initialize LFS with node.LFS.reload("lfs.img")
		Board will reboot after successfull self-flashing

9. 	Check that your files are in LFS and ready to be LOADED
		for k,v in pairs(node.LFS.list()) do print(k,v) end
		1	HTTP_OTA
		2	_init
		3	dummy_strings
		4	lfs_fragments

10. How to use files from LFS ??????????????


=========================================== TEST THE EXAMPLE ======================================================================================
init.lua
	print("The LFS addr is " .. node.getpartitiontable().lfs_addr)
	print("The LFS size is " .. node.getpartitiontable().lfs_size)
	print("The SPIFFS addr is " .. node.getpartitiontable().spiffs_addr)
	print("The SPIFFS size is " .. node.getpartitiontable().spiffs_size)

	do
	  local s,p={},node.getpartitiontable()
	  for _,k in ipairs{'lfs_addr','lfs_size','spiffs_addr','spiffs_size'} do
		s[#s+1] ='%s = 0x%06x' % {k, p[k]}
	  end
	  print ('{ %s }' % table.concat(s,', '))
	end

> dofile("init.lua");
The LFS addr is 688128
The LFS size is 262144
The SPIFFS addr is 950272
The SPIFFS size is 3231744
{ lfs_addr = 0x0a8000, lfs_size = 0x040000, spiffs_addr = 0x0e8000, spiffs_size = 0x315000 }
>
----------------------------
init.lua        : 473 bytes
----------------------------
Total file(s)   : 1
Total size      : 473 bytes

Total : 3050403 bytes
Used  : 753 bytes
Remain: 3049650 bytes

> =node.heap()
40808
> file.rename("LFS_integer_20200929-0851.img","lfs.img")
>
----------------------------
init.lua        : 473 bytes
lfs.img         : 5671 bytes
----------------------------
Total file(s)   : 2
Total size      : 6144 bytes

Total : 3050403 bytes
Used  : 6777 bytes
Remain: 3043626 bytes
>
node.LFS.reload("lfs.img")
LFS region updated.  Restarting.
>
node.flashindex("_init")()
>
print(LFS)
table: 0x3fff04f0
>
print(node.LFS.list())
table: 0x3fff05a0
>
for k,v in pairs(node.LFS.list()) do print(k,v) end
1	HTTP_OTA
2	_init
3	dummy_strings
4	lfs_fragments

> =node.heap()
42392

	Errata (Start Reading Here):
	Flash = micro-controller storage facility that can be written ~100.000 times (hardware related, ESP-12e has 4Mb)
	Flash is used for 3 things:
	- Firmware = the main program interpreter, in our case Lua
	- SPIFFS = think at this as a partition of the hard-drive in your computer that stores all sort of files
	SPIFFS is used for 3 things:
	- to store/load program files.lua directly, including init.lua !!! these files will be loaded in RAM and interpreted/compiled to be executed !!!
	- to store ANY type of files you need (Eg: configs, logs, images, certificates, etc.)
	- to store a special .bin file called LFS that contains PRE-COMPILED .lua files (your program) - https://nodemcu.readthedocs.io/en/dev/lfs/
	The LFS file is uploaded like any other file in SPIFFS (Eg: use ESPlorer or at build time) AND NEEDS TO BE absorbed/initialized by the firmware with:
	node.LFS.reload("lfs.img")
	????????????Q.0 Is there any option/define that will allow us to set the SPIFFS size to allow for LFS partition + lfs.img + some extra ????????????
	- LFS (Lua-File-Storage) upon initialization above will transfer the contents of the "lfs.bin" file in a specially designated flash/SPIFFS sub-partition (Eg: static precompiled folder)
	After initialization the SPIFFS lfs.bin is no longer needed and can/should be removed/deleted
	Check the existence of the fls contents with:
	print("\nFiles in LFS")
	for k,v in pairs(node.LFS.list()) do print(k,v) end
	????????????Q.1 IS THERE a ls/dir function for LFS so that we see also the folder structure/namespaces - Usefull for loading modules ????????????
	????????????Q.2 How are the modules from LFS loaded, who's registering them - Is there a framework/convention we can work with like
	- https://github.com/HHHartmann/nodemcu-LFS-base
	- https://github.com/freyssin/elmis - has an example (X.lua) of how to develop your own module
	maybe this approch will help mitigate changes occuring between LFS/NodeMCU versions ????????????

	init.lua = default file that is called by the firmware at startup (see it as your program entrance)
	- this behavior can be modified in app/include/user_config.h
	#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)" <<<--- on boot look for _init.lua in LFS not for default init.lua from SPIFFS
	NOT advisable if you want to mitigate problems with OTA when the new lfs.img can be corrupted.
	- init.lua has to:
	- check for the existence of LFS files and consistency
	- (for k,v in pairs(node.LFS.list()) do print(k,v) end)
	!!!!! as far as I understand the files are present in LFS but not yet LOADED, I mean not yet available to be called with 'require "LuaOTA.check"'
	https://github.com/nodemcu/nodemcu-firmware/tree/dev/lua_examples/luaOTA
	and I don't understand how is it enough to call require if modules are not registered (or are they?)
	I couldn't make the example work, could not get to run LuaOTA.check (calling it many names) from init.lua !!!!!
	- allow some 5sec for the program to be interrupted in case things got messed up (infinite reboot)
	- transfer control/call to _init.lua
	_init.lua = the SPIFFS and/or LFS "twin brother / big brother" of init.lua
	????????????Q.3 Here things start to become fuzzy, Where should _init.lua reside in SPIFFS or in LFS root or in LFS/namespace/ ????????????
	????????????Q.4 How should _init.lua be called node.flashindex("_init")(), dofile ("LFS._init.lua"), dofile ("_init.lua") or require "_init" ????????????
	????????????Q.5 Should node.flashindex("_init")() be called to register the modules in proper namespaces before doing 'require "LuaOTA.check"' like in luaOTA example ????????????
	- _init.lua has to:
	- register the modules and make them available in their respective namespaces ????????????
	- establish wifi (sta or/and ap) (optional)
	- transfer control/call to NAMESPACE.yoursoftware.lua ????????????

	namespaces = When building lfs.img (see below) files can/should be put in FOLDERS and these FOLDER_NAMES BECOME NAMESPACES for lua.
	!!!!! In my opinion this would help in structuring the code if you have a very complex software !!!!!

	/nodemcu-firmware/local/lua = Contents of this folder is not directly installed into LFS, instead, the files therein are built into /nodemcu-firmware/local/fs/LFS.img.
	See the LFSimage target in tools/Makefile.
	/nodemcu-firmware/local/fs = Is used to generate the SPIFFS images (see the spiffsscript target in tools/Makefile), which land in files like bin/0xc0000-32mb.img which are ${flash_position}-${flash_size}.bin.
	In general, it's fine to use smaller SPIFFS images on larger flash devices, though I do not think SPIFFS will expand to fill free space without formatting from scratch.
	- https://github.com/nodemcu/nodemcu-firmware/issues/3299


	Flash available on the board
	0x00000000 flash strat address 0xZZZZZZZZZ flash end address
	\|------------------------\|-----------------------------------------------------------------------------------------\|

	Firmware space
	0x00000000 strat address 0xSOMETHING end address
	\|------------------------\| #define SPIFFS_SIZE_1M_BOUNDARY <<<--- Might be a good idea so that when flashing new firmware you don't overwrite/overflow to the SPIFFS start address ??????

	SPIFFS space
	Start SPIFFS address 0xSOMETHING + headroom for a larger firmware End SPIFFS address 0xSPIFFS
	\|-----Firmware space-----\|--blank--\|---------------------------SPIFFS-----------------------------\|------blank-----\| 0xZZZZZZZZZ flash end address

	LFS space
	Start LFS address 0xSOMETHING + headroom End SPIFFS address 0xSPIFFS
	\|-----Firmware space-----\|--blank--\|\|------LFS------\|----------SPIFFS-----------------------------\|------blank-----\| 0xZZZZZZZZZ flash end address
	End SPIFFS address 0xSPIFFS
	The address/location of LFS is not a concern for us since it's taken care of by NodeMCU. It is shown here for reference.
	We only have to allocate enough space to it based on the size of our developed software with #define LUA_FLASH_STORE 0x40000 <<<--- size of LFS // 0x40000 = 256kb


	What we want:
	We want to load our .lua files in LFS so that they are not stored in memory and we have ram available 43KB to use them. (there is no other way for a serious program !!!)
	ESP-x family has the firmware partition and the SPIFFS partition. LFS is part of the SPIFFS
	0x00000.bin will contain just the firmware.
	0x10000.bin will contain the SPIFFS.
	nodemcu_integer_dev_20200929-0843.bin has both

	.bin files are in nodemcu-build/bin
	.img LFS files can be found in {PathToLuaSourceFolder}

	!!! Same file can exist in both LFS (not using ram) and SPIFFS (this uses ram)
	!!! If above happens take care which one is executed when developing
	!!! Change the order (Use LFS or SPIFFS file version) like described in https://nodemcu.readthedocs.io/en/dev/lfs/#choosing-your-development-life-cycle

	Success looks like:
	We want to be able to:
	- build firmware with chosen modules on whichever git branch we choose,
	- build wih Lua 5.1.x or Lua 5.3.x,
	- build LFS containing our software,
	- build SPIFFS with preloaded files (Eg: init.lua, lfs.img),
	- flash them to hardware,
	- call funtions from LFS.

	Following the https://nodemcu.readthedocs.io/en/dev/getting-started/#minimal-lfs-example
	1. we must get to check our files are in LFS:
	for k,v in pairs(node.LFS.list()) do print(k,v) end
	1 HTTP_OTA
	2 _init
	3 dummy_strings
	4 lfs_fragments
	2. we must be able to load and use the functions from lfs

	0. Create the folder that will serve as our base
	mkdir nodemcu; cd nodemcu

	1. Get Docker NodeMcu builder image - https://github.com/marcelstoer/docker-nodemcu-build
	docker pull marcelstoer/nodemcu-build

	2. Clone Repo - https://nodemcu.readthedocs.io/en/latest/build/#git
	master branch -> git clone --recurse-submodules https://github.com/nodemcu/nodemcu-firmware.git
	release branch -> git clone --recurse-submodules -b release https://github.com/nodemcu/nodemcu-firmware.git
	dev branch -> git clone --recurse-submodules -b dev https://github.com/nodemcu/nodemcu-firmware.git

	3. Build Firmware - https://nodemcu.readthedocs.io/en/latest/build/

	Edit app/include/user_modules.h and comment-out the #define statement for modules you don't need.
	Edit app/include/user_config.h
	#define LUA_NUMBER_INTEGRAL <<<--- if not defined a float image will be built
	#define LUA_FLASH_STORE 0x40000 <<<--- size of LFS // 0x40000 = 256kb
	#define LUA_INIT_STRING "pcall(function() node.flashindex'_init'() end)" <<<--- on boot look for _init.lua in LFS not for default init.lua from SPIFFS
	#define SPIFFS_SIZE_1M_BOUNDARY <<<--- Might be a good idea so that when flashing new firmware you don't overwrite/overflow to the SPIFFS start address

	LUA 5.1 and 5.3 differ so we have to take care to build the correct firmware with docker option -e LUA=53
	LUA 5.1
	docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware marcelstoer/nodemcu-build build
	https://github.com/marcelstoer/docker-nodemcu-build#build-the-firmware
	nodemcu_${BUILD_TYPE}_${IMAGE_NAME}.bin is the combined firmware image you can flash. BUILD_TYPE is integer or float. For IMAGE_NAME, see the Options chapter below.
	Almost same but with .map ending, a mapfile will be saved that contains the relative offsets of functions.
	0x00000.bin will contain just the firmware.
	0x10000.bin will contain the SPIFFS.
	LUA 5.3
	docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -e LUA=53 marcelstoer/nodemcu-build build

	4. Build LFS - .img files can be found in {PathToLuaSourceFolder}
	- https://github.com/marcelstoer/docker-nodemcu-build#create-an-lfs-image-for-esp8266
	- https://nodemcu.readthedocs.io/en/dev/lfs/#compiling-and-loading-lfs-images

	docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware -v {PathToLuaSourceFolder}:/opt/lua marcelstoer/nodemcu-build lfs-image
	This will compile and store all Lua files in the given {PathToLuaSourceFolder} folder including all SUBFOLDERS which WILL BECOME NAMESPACES for lua.
	docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware -v {PathToLuaSourceFolder}:/opt/lua marcelstoer/nodemcu-build lfs-image final/files.lst
	To only add specific files you can prepare a file containing the files to add and give them as paramater.

	LUA 5.1 and 5.3 differ so we have to take care to build LFS for the correct one with docker option -e LUA=53
	LUA 5.1
	docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua marcelstoer/nodemcu-build lfs-image
	LUA 5.3
	docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua -e LUA=53 marcelstoer/nodemcu-build lfs-image

	Example: Build LFS from lua_examples/lfs/
	docker run --rm -ti -v `pwd`/nodemcu-firmware:/opt/nodemcu-firmware -v `pwd`/nodemcu-firmware/lua_examples/lfs/:/opt/lua marcelstoer/nodemcu-build lfs-image
	Adding files: ./lfs_fragments.lua ./HTTP_OTA.lua ./_init.lua ./dummy_strings.lua
	creating LFS_float_20200928-0840.img
	creating LFS_integer_20200928-0840.img

	3&4 Build NodeMCU firmware with init.lua and our firmware compiled in LFS.img file that is pre-loaded in SPIFFS image all at once
	- https://github.com/nodemcu/nodemcu-firmware/issues/3299

	- /nodemcu-firmware/local/lua = You put your firmware here to be compiled and stored in LFS.img.
	Contents of this folder is not directly installed into LFS, instead, the files therein are built into /nodemcu-firmware/local/fs/LFS.img. See the LFSimage target in tools/Makefile.

	- /nodemcu-firmware/local/fs = You put your init.lua here to be stored in SPIFFS and called at boot.
	Contents of this folder is used to generate the SPIFFS images (see the spiffsscript target in tools/Makefile), which land in files like bin/0xc0000-32mb.img
	which are ${flash_position}-${flash_size}.bin. In general, it's fine to use smaller SPIFFS images on larger flash devices,
	though I do not think SPIFFS will expand to fill free space without formatting from scratch.

	- You can land everything you want on the board in a single esptool.py flash command, but you will need to node.LFS.flash("LFS.img") at first boot.
	Probably the correct way to do this is to also land an init.lua containing the correct incantation in SPIFFS.
	(To avoid a reboot loop if something goes wrong with loading LFS, it may be useful to first test for LFS.img and then file.rename it to something else before node.LFS.flash-ing.
	At least in this case, you'll still have a Lua interpreter on the UART.)

	5. Use NodeMCU PyFlasher to format from scratch - https://github.com/marcelstoer/nodemcu-pyflasher
	This will flash the firmware and upon boot will reformat SPIFFS (SPIFFS will expand to fill free space)
	This scenario can be used to manually upload init.lua and lfs.bin AFTER the formatting is complete
	NodeMCU firmware - choose the one you built (Eg: nodemcu_integer_dev_20200929-2150.bin)
	!!!!! TOTEST if files pre-built in SPIFFS dissapear upon formatting !!!!!
	Baud rate - 115200
	Flash mode Dual I/O (DIO) - This is dependent on your device - https://www.esp32.com/viewtopic.php?p=5523&sid=08ef44e13610ecf2a2a33bb173b0fd5c#p5523
	Erase flash - yes,wipes all data <<<--- SPIFFS will expand to fill free space formatting from scratch

	5'. Reflash using NodeMCU PyFlasher to flash the image on ESP-12e - https://github.com/marcelstoer/nodemcu-pyflasher
	NodeMCU firmware - choose the one you built (Eg: nodemcu_integer_dev_20200929-2150.bin)
	Baud rate - 115200
	Flash mode Dual I/O (DIO) - This is dependent on your device - https://www.esp32.com/viewtopic.php?p=5523&sid=08ef44e13610ecf2a2a33bb173b0fd5c#p5523
	Erase flash - no <<<--- SPIFFS is already expanded to fill maximum space by step 5
	!!!!! TOTEST I think doing this will also resize the SPIFFS partition !!!!!
	!!!!! TOTEST We do this with PyFlasher or use esptool directly and only flash the SPIFFS? which brings me to Q.0 !!!!!

	Command: esptool.py --port COM4 --baud 115200 --after no_reset write_flash --flash_mode dio 0x00000 D:\_NodeMCU LUA\dumps\nodemcu_integer_release_20200928-0829.bin --erase-all
	esptool.py v2.6
	Serial port COM4
	Connecting....
	Detecting chip type... ESP8266
	Chip is ESP8266EX
	Features: WiFi
	MAC: 18:fe:34:f4:ed:44
	Uploading stub...
	Running stub...
	Stub running...
	Configuring flash size...
	Auto-detected Flash size: 4MB
	Erasing flash (this may take a while)...
	Chip erase completed successfully in 9.0s
	Flash params set to 0x0240
	Compressed 659456 bytes to 443592...
	Wrote 659456 bytes (443592 compressed) at 0x00000000 in 39.2 seconds (effective 134.6 kbit/s)...
	Hash of data verified.
	Leaving...
	Staying in bootloader.
	Firmware successfully flashed. Unplug/replug or reset device
	to switch back to normal boot mode.

	6. Using ESPlorer v0.2.0 - https://esp8266.ru/esplorer/
	HowTo ESPlorer - https://www.best-microcontroller-projects.com/installing-and-using-esplorer.html
	Remove the flash pin, open the serial COM with the board and power it

	Formatting file system. Please wait...
	..................................................
	NodeMCU 3.0.0.0 built with Docker provided by frightanic.com
	branch: release
	commit: 64bbf006898109b936fcc09478cbae9d099885a8
	release: 3.0-master_20200910
	release DTS: 202009090323
	SSL: true
	build type: integer
	LFS: 0x40000 bytes total capacity
	modules: crypto,dht,file,gpio,http,mdns,mqtt,net,node,pwm2,rtctime,sjson,sntp,softuart,tmr,uart,wifi
	build 2020-09-28 08:29 powered by Lua 5.1.4 on SDK 3.0.1-dev(fce080e)

	----------------------------
	No files found.
	----------------------------
	>
	Total : 3073746 bytes
	Used : 0 bytes
	Remain: 3073746 bytes

	7. Uploading to ESP file LFS_integer_20200928-0840.img...Success
	>
	----------------------------
	LFS_integer_20200928-0840.img : 5670 bytes
	----------------------------
	Total file(s) : 1
	Total size : 5670 bytes

	Total : 3073746 bytes
	Used : 6024 bytes
	Remain: 3067722 bytes
	file.rename("LFS_integer_20200928-0840.img","lfs.img")
	>
	----------------------------
	lfs.img : 5670 bytes
	----------------------------
	Total file(s) : 1
	Total size : 5670 bytes

	Total : 3073746 bytes
	Used : 6024 bytes
	Remain: 3067722 bytes

	8. Initialize LFS with node.LFS.reload("lfs.img")
	Board will reboot after successfull self-flashing

	9. Check that your files are in LFS and ready to be LOADED
	for k,v in pairs(node.LFS.list()) do print(k,v) end
	1 HTTP_OTA
	2 _init
	3 dummy_strings
	4 lfs_fragments

	10. How to use files from LFS ??????????????



	=========================================== TEST THE EXAMPLE ======================================================================================
	init.lua
	print("The LFS addr is " .. node.getpartitiontable().lfs_addr)
	print("The LFS size is " .. node.getpartitiontable().lfs_size)
	print("The SPIFFS addr is " .. node.getpartitiontable().spiffs_addr)
	print("The SPIFFS size is " .. node.getpartitiontable().spiffs_size)

	do
	local s,p={},node.getpartitiontable()
	for _,k in ipairs{'lfs_addr','lfs_size','spiffs_addr','spiffs_size'} do
	s[#s+1] ='%s = 0x%06x' % {k, p[k]}
	end
	print ('{ %s }' % table.concat(s,', '))
	end

	> dofile("init.lua");
	The LFS addr is 688128
	The LFS size is 262144
	The SPIFFS addr is 950272
	The SPIFFS size is 3231744
	{ lfs_addr = 0x0a8000, lfs_size = 0x040000, spiffs_addr = 0x0e8000, spiffs_size = 0x315000 }
	>
	----------------------------
	init.lua : 473 bytes
	----------------------------
	Total file(s) : 1
	Total size : 473 bytes

	Total : 3050403 bytes
	Used : 753 bytes
	Remain: 3049650 bytes

	> =node.heap()
	40808
	> file.rename("LFS_integer_20200929-0851.img","lfs.img")
	>
	----------------------------
	init.lua : 473 bytes
	lfs.img : 5671 bytes
	----------------------------
	Total file(s) : 2
	Total size : 6144 bytes

	Total : 3050403 bytes
	Used : 6777 bytes
	Remain: 3043626 bytes
	>
	node.LFS.reload("lfs.img")
	LFS region updated. Restarting.
	>
	node.flashindex("_init")()
	>
	print(LFS)
	table: 0x3fff04f0
	>
	print(node.LFS.list())
	table: 0x3fff05a0
	>
	for k,v in pairs(node.LFS.list()) do print(k,v) end
	1 HTTP_OTA
	2 _init
	3 dummy_strings
	4 lfs_fragments

	> =node.heap()
	42392