Gjum/#mcdevs.log Secret

## #mcdevs.log
# irc.freenode.net/#mcdevs from Tue Mar 24 2015
20:58 < Grum>  <_Gjum> rom1504: seems to be a bit more up-to-date, but does not have stack size http://minecraft-ids.grahamedgecombe.com/items.json <-- also going to be old and broken for 1.9 :)
21:02 < rom1504> yeah but currently items.json has been update manually by Kupferhirn (https://github.com/andrewrk/mineflayer/pull/227)
21:02 < rom1504> so until we have an automatic extractor for items.json , it's better than nothing
21:12 < _Gjum> Grum: Any hints on the new internal format so we can prepare?
21:15 < Grum> internal format of?
21:16 < _Gjum> items and blocks, as how they are handled, because there are rumors of not using the same item/block IDs as before
21:17 < Grum> they are going to be really different
21:19 < Grum> humz
21:19 < Grum> many ideas atm
21:20 < _Gjum> completely different ideas or is there some consensus?
21:20 < Grum> we'll drop the block->id completely
21:21 < Grum> purge metadata completely
21:21 < _Gjum> so everything will be indexed via name? or how?
21:22 < _Gjum> like transferring a chunk without IDs seems pretty complicated
21:22 < Grum> within the code there is no way to obtain an id
21:22 < Grum> the currency is going to be BlockStates
21:22 < Grum> storage will handle BlockState -> id
21:23 < _Gjum> so the IDs in storage and transfer are temporary and only for the stored/transferred chunk?
21:23 < Grum> chunk storage will handle it
21:24 <+Amaranth> Different int ids per chunk? Seems wasteful
21:25 < Grum> not really
21:25 < Grum> comes with one huge benefit
21:25 < Grum> memory is going to be the only limit for the amounts of blockstates you can handle
21:25 < Grum> aka infinite blocks
21:25 < _Gjum> hm, if there are no ores in a chunk, it can use all the numbers for e.g. mod-added blocks
21:26 < _Gjum> so there will be another completely new world format?
21:26 <+Amaranth> A string->id mapping with up to 4096 unique entries and strings are probably about 20 characters each (10 just for "minecraft:")
21:26 < Grum> but yes, there is a tradeof point where just storing raw BlockState's is going to be cheaper than dual-indexing
21:26 < Grum> Amaranth: also compresses *really* well
21:27 <+Amaranth> Grum: I'm talking about in memory
21:27 < Grum> and obviously after it is loaded it will have all those things resolved to BlockStates
21:27 < Grum> so we'll have: id->BlockState (BlockState[]) and BlockState->id which is going to be something more memory efficient than Map<BlockState, Integer>
21:27 <+Amaranth> Ok so your in memory mapping is going to be 4096 pointer->int mappings and 4096 int->pointer mappings
21:28 < Grum> at very worst case, and as i said, math will tell when its cheaper to just store 'BlockState[]' instead of: char[] + Bidirectional mappings
21:29 < Grum> cheapest however, nearly all generated chunks can be stored using 16 entry pallete
21:29 < Grum> which means: BlockState[16] (which we can just traverse, who the hell cares) and Nibble[4k] for size
21:29 < Grum> which is barely anything sizewise
21:30 <+Amaranth> Assuming you're willing to rely on compressed pointers it's going to almost always be cheaper to store BlockState[] assuming your BlockState objects are singletons
21:30 < Grum> they are singletons and it will have a tradeoffpoint
21:31 < Grum> eg: with <16 pallete entries; you just store 16*8+2k
21:31 < Grum> sorry, 16*4
21:31 <+Amaranth> I'd have to think on it a bit for the palette approach but for the bidirectional map I don't think it'd ever win
21:31 < Grum> and 'worst case' it will be BlockState[] which is 4x4k = 16k
21:31 < Grum> it will?
21:31 < Grum> ok lets assume we just do this stupidly
21:32 < Grum> and we search through the BlockState[] to find the index ok ?
21:32 < Grum> and then use that index in the char[4k] to write
21:32 < Grum> so; char[4k] we'll always have, which is 8k of data
21:33 < Grum> so we can host another 8k of data until 'just storing BlockState[]' is better
21:33 < Grum> which is 2k pallet entries
21:33 <+Amaranth> But of course that'd be too slow to use...
21:33 < Grum> so, when <2048 blockstates and super naive searching it is cheaper to do that over BlockState[]
21:34 < Grum> so lets assume trove is cool
21:34 < Grum> and its ObjectCharIdentityMap is 'smart'
21:34 < Grum> so it would just store: BlockState and char for each 'entry'
21:34 <+Amaranth> I don't remember when the cutoff point is where having a sorted array and binary search is faster than linear though
21:34 < Grum> so that is 6 bytes each
21:35 < Grum> and another 4 bytes for the other index
21:35 < Grum> so when it can spend 8k, we can do ~800 entries until BlockState[] is cheaper
21:35 < Grum> in the end its just math, and cramming 800 blockstates into a single chunk is actually quite hard to do
21:36 < Grum> and we need a migation path anyhow, so just mathing all the options and then having it selecting stuff based on the pallet-size and growing just makes sense
21:36 <+Amaranth> btw, this would be a good time to change to 32x32x32 chunk size :D
21:36 <+Amaranth> Double your draw distance
21:37 < Grum> that is not how it works Amaranth :p
21:38 <+Amaranth> Well, unless you're already bottlenecking somewhere else that's close enough to how it works
21:38 < Grum> not really
21:38 <+Amaranth> Although you'll also slow down mesh generation on updates
21:38 < Grum> yeah and this means you have to generate 8 times the amount of mesh
21:39 <+Amaranth> But the overhead for setting up state and drawing each chunk as a separate draw call is non-trivial and having bigger chunks means you get to cut that down or draw more with the same effort
21:40 <+Amaranth> Of course you also have to worry about memory and blah blah blah
21:40 <+Amaranth> But once you optimize the rest the chunk size will end up being a limiter
21:43 < Grum> yeah gpu memory is also a pain
21:44 < Grum> we can get quite some gain from jsut rendering a sphere
21:45 < Grum> saves 33% area assuming i remember right =)
21:47 <+Amaranth> Yeah, something like that
21:50 < Grum> anyhow, until we fix generation that is just all pointless
21:50 < Grum> threading that is going to be needed to do anything >16
	# irc.freenode.net/#mcdevs from Tue Mar 24 2015
	20:58 < Grum> <_Gjum> rom1504: seems to be a bit more up-to-date, but does not have stack size http://minecraft-ids.grahamedgecombe.com/items.json <-- also going to be old and broken for 1.9 :)
	21:02 < rom1504> yeah but currently items.json has been update manually by Kupferhirn (https://github.com/andrewrk/mineflayer/pull/227)
	21:02 < rom1504> so until we have an automatic extractor for items.json , it's better than nothing
	21:12 < _Gjum> Grum: Any hints on the new internal format so we can prepare?
	21:15 < Grum> internal format of?
	21:16 < _Gjum> items and blocks, as how they are handled, because there are rumors of not using the same item/block IDs as before
	21:17 < Grum> they are going to be really different
	21:19 < Grum> humz
	21:19 < Grum> many ideas atm
	21:20 < _Gjum> completely different ideas or is there some consensus?
	21:20 < Grum> we'll drop the block->id completely
	21:21 < Grum> purge metadata completely
	21:21 < _Gjum> so everything will be indexed via name? or how?
	21:22 < _Gjum> like transferring a chunk without IDs seems pretty complicated
	21:22 < Grum> within the code there is no way to obtain an id
	21:22 < Grum> the currency is going to be BlockStates
	21:22 < Grum> storage will handle BlockState -> id
	21:23 < _Gjum> so the IDs in storage and transfer are temporary and only for the stored/transferred chunk?
	21:23 < Grum> chunk storage will handle it
	21:24 <+Amaranth> Different int ids per chunk? Seems wasteful
	21:25 < Grum> not really
	21:25 < Grum> comes with one huge benefit
	21:25 < Grum> memory is going to be the only limit for the amounts of blockstates you can handle
	21:25 < Grum> aka infinite blocks
	21:25 < _Gjum> hm, if there are no ores in a chunk, it can use all the numbers for e.g. mod-added blocks
	21:26 < _Gjum> so there will be another completely new world format?
	21:26 <+Amaranth> A string->id mapping with up to 4096 unique entries and strings are probably about 20 characters each (10 just for "minecraft:")
	21:26 < Grum> but yes, there is a tradeof point where just storing raw BlockState's is going to be cheaper than dual-indexing
	21:26 < Grum> Amaranth: also compresses really well
	21:27 <+Amaranth> Grum: I'm talking about in memory
	21:27 < Grum> and obviously after it is loaded it will have all those things resolved to BlockStates
	21:27 < Grum> so we'll have: id->BlockState (BlockState[]) and BlockState->id which is going to be something more memory efficient than Map<BlockState, Integer>
	21:27 <+Amaranth> Ok so your in memory mapping is going to be 4096 pointer->int mappings and 4096 int->pointer mappings
	21:28 < Grum> at very worst case, and as i said, math will tell when its cheaper to just store 'BlockState[]' instead of: char[] + Bidirectional mappings
	21:29 < Grum> cheapest however, nearly all generated chunks can be stored using 16 entry pallete
	21:29 < Grum> which means: BlockState[16] (which we can just traverse, who the hell cares) and Nibble[4k] for size
	21:29 < Grum> which is barely anything sizewise
	21:30 <+Amaranth> Assuming you're willing to rely on compressed pointers it's going to almost always be cheaper to store BlockState[] assuming your BlockState objects are singletons
	21:30 < Grum> they are singletons and it will have a tradeoffpoint
	21:31 < Grum> eg: with <16 pallete entries; you just store 16*8+2k
	21:31 < Grum> sorry, 16*4
	21:31 <+Amaranth> I'd have to think on it a bit for the palette approach but for the bidirectional map I don't think it'd ever win
	21:31 < Grum> and 'worst case' it will be BlockState[] which is 4x4k = 16k
	21:31 < Grum> it will?
	21:31 < Grum> ok lets assume we just do this stupidly
	21:32 < Grum> and we search through the BlockState[] to find the index ok ?
	21:32 < Grum> and then use that index in the char[4k] to write
	21:32 < Grum> so; char[4k] we'll always have, which is 8k of data
	21:33 < Grum> so we can host another 8k of data until 'just storing BlockState[]' is better
	21:33 < Grum> which is 2k pallet entries
	21:33 <+Amaranth> But of course that'd be too slow to use...
	21:33 < Grum> so, when <2048 blockstates and super naive searching it is cheaper to do that over BlockState[]
	21:34 < Grum> so lets assume trove is cool
	21:34 < Grum> and its ObjectCharIdentityMap is 'smart'
	21:34 < Grum> so it would just store: BlockState and char for each 'entry'
	21:34 <+Amaranth> I don't remember when the cutoff point is where having a sorted array and binary search is faster than linear though
	21:34 < Grum> so that is 6 bytes each
	21:35 < Grum> and another 4 bytes for the other index
	21:35 < Grum> so when it can spend 8k, we can do ~800 entries until BlockState[] is cheaper
	21:35 < Grum> in the end its just math, and cramming 800 blockstates into a single chunk is actually quite hard to do
	21:36 < Grum> and we need a migation path anyhow, so just mathing all the options and then having it selecting stuff based on the pallet-size and growing just makes sense
	21:36 <+Amaranth> btw, this would be a good time to change to 32x32x32 chunk size :D
	21:36 <+Amaranth> Double your draw distance
	21:37 < Grum> that is not how it works Amaranth :p
	21:38 <+Amaranth> Well, unless you're already bottlenecking somewhere else that's close enough to how it works
	21:38 < Grum> not really
	21:38 <+Amaranth> Although you'll also slow down mesh generation on updates
	21:38 < Grum> yeah and this means you have to generate 8 times the amount of mesh
	21:39 <+Amaranth> But the overhead for setting up state and drawing each chunk as a separate draw call is non-trivial and having bigger chunks means you get to cut that down or draw more with the same effort
	21:40 <+Amaranth> Of course you also have to worry about memory and blah blah blah
	21:40 <+Amaranth> But once you optimize the rest the chunk size will end up being a limiter
	21:43 < Grum> yeah gpu memory is also a pain
	21:44 < Grum> we can get quite some gain from jsut rendering a sphere
	21:45 < Grum> saves 33% area assuming i remember right =)
	21:47 <+Amaranth> Yeah, something like that
	21:50 < Grum> anyhow, until we fix generation that is just all pointless
	21:50 < Grum> threading that is going to be needed to do anything >16