gridaphobe/ghc.log Secret

## ghc.log
12:15 PM <thomie> gridaphobe: binary sizes are 3% up. Just checking if that is expected, see: https://perf.haskell.org/ghc/#revision/6740d70d95cb81cea3859ff847afc61ec439db4f
12:16 PM  → LnL, tempname11, mathk, osa1, ivan\, moei and nkartash_ joined  ← Welkin left  ⇐ gentleben, mgsloan, Yuras, ryantrinkle and bb010g quit  ↔ dcoutts_, edsko, hamishmack and Gal3rielol popped in  ↔ tibbe, Luke (was Luke_) and matrixa1 nipped out  •  waern_ → waern
2:05 PM <gridaphobe> thomie: i wouldn't expect an increase in the binary size unless the programs are using implicit callstacks (e.g. by using error, undefined, or assert)
2:06 PM  → quchen joined  ⇐ matrixa1 and ivan\ quit
2:13 PM <slyfox> what does ghc generate for nonexhaustive pattern matches?
2:14 PM  ↔ ivan\ popped in
2:16 PM <gridaphobe> i'm afraid i don't know anything about the nofib suite, but i suspect the programs are compiled with optimizations :)
2:16 PM → matrixa1 joined (~matrixa1@h-88-35.a230.priv.bahnhof.se)
2:16 PM <ghc-github> [ghc] ghc-mirror pushed 1 new commit to master: http://git.io/vG7aN
2:16 PM <ghc-github> ghc/master 34b106f Edward Z. Yang: Accept underscores in the module parser. (Thanks spinda for the fix.)...
2:17 PM <gridaphobe> oh but this is talking about the produced binary, not ghc's allocations, right?
2:17 PM <gridaphobe> so it wouldn't be connected to the allocation increase for the two tests
2:18 PM  → nomeata and ryantrinkle joined
2:18 PM <slyfox> i believe it's binary size, yes
2:19 PM <slyfox> the very end of page has 2 +17% allocations failures: tests/alloc/T5321FD and tests/alloc/T5321Fun
2:20 PM → ivan\ joined (~ivan@unaffiliated/ivan/x-000001)
2:20 PM <gridaphobe> right, those are expected increases because those tests are compiled without optimizations, so ghc doesn't see that undefined diverges
2:21 PM <gridaphobe> and it ends up doing unnecessary work
2:22 PM  → hamishmack joined  ⇐ gustavold and tibbe quit  ↔ matrixa1 and ivan\ nipped out
2:40 PM <gridaphobe> thomie: what's the best way to determine why the binary sizes have increased? is there a particular dump flag that would be useful?
2:41 PM ⇐ hamishmack quit (~hamishmac@72.2.55.21) Quit: hamishmack
2:45 PM <gridaphobe> thomie: actually, looking at the core for one of the benchmarks (the atom benchmark), `error` is used via `read`, so a small increase would be expected there
2:45 PM <gridaphobe> i'll look at the others
2:46 PM  → dcoutts_ and tibbe joined  ⇐ Luke quit  ↔ osa1 nipped out
3:13 PM <rwbarton> "binary size" is the size of the whole stripped executable including the needed parts of the rts/ghc-prim/base such as, presumably, most of the IO manager
3:14 PM ⇐ tibbe quit (~tibbe@209-82-80-116.dedicated.allstream.net) Quit: tibbe
3:14 PM <rwbarton> which probably uses error some I guess?
3:14 PM <rwbarton> Still 3% sounds like a lot
3:14 PM <rwbarton> It might be better to start by looking at "module sizes" (which are the sizes of the .o files)
3:16 PM → Luke joined (~luke@unaffiliated/luke)
3:17 PM <gridaphobe> well the module itself uses error too (many of the benchmarks do since they read some data from stdin)
3:17 PM <gridaphobe> the call to `read` is inlined
3:17 PM <rwbarton> sure, but that fact will not make a visible change to binary size
3:18 PM <rwbarton> or not a 30K change anyways
3:18 PM <gridaphobe> true..
3:18 PM  ⇐ raichoo and LnL quit
3:22 PM <rwbarton> you can download the logs and run nofib-analyse on them to see which individual nofib modules changed sizes
3:22 PM <rwbarton> oddly, some got smaller too but on average they are apparently 1.0% larger
3:24 PM <rwbarton> (though that number is potentially bogus because of how it is computed)
3:24 PM  → Javran joined  ⇐ dcoutts_ and nomeata quit  ↔ osa1 nipped out
3:27 PM <gridaphobe> rwbarton: are the logs you're referring to just the build logs? (i get an illegal byte sequence error)
3:27 PM <thomie> wait a minute, you updated the Cabal module. I've been making changes to the way stripping is handled in Cabal...
3:27 PM <rwbarton> gridaphobe, yes
3:28 PM → nomeata joined (~jojo@nomeata.de)
3:28 PM <gridaphobe> yes i updated Cabal because i had to patch it to use the new pattern synonym for ErrorCall
3:28 PM <rwbarton> they have non-ASCII (UTF-8) characters though, make sure you have a UTF-8 locale?
3:29 PM <rwbarton> well nofib programs aren't built with cabal
3:29 PM <rwbarton> I guess base is built with cabal though?
3:29 PM <thomie> so the change I made is that .a files are only stripped when you install them, not before
3:29 PM <gridaphobe> i have LANG="en_US.UTF-8" set
3:30 PM <gridaphobe> ah, so if the benchmarks are linked against the inplace libraries, the libraries won't have been stripped yet?
3:30 PM <rwbarton> gridaphobe: well then I don't know what is wrong
3:30 PM — rwbarton is still confused about stripping .a files, btw
3:31 PM <rwbarton> gridaphobe: FWIW I used wget to download the log files and that worked for me, if you used a browser maybe it did something "smart"
3:31 PM <gridaphobe> i used wget too :)
3:31 PM <geekosaur> depends on how strip is being used
3:31 PM <rwbarton> hm
3:32 PM <thomie> gridaphobe: yes, could be
3:32 PM <geekosaur> also OS X strip has variant semantics and is safe on .a files (it *adjusts* relocation entries)
3:32 PM <rwbarton> adjusts how?
3:33 PM ⇐ Javran quit (~javran@c-73-200-214-152.hsd1.md.comcast.net) Quit: Leaving.
3:33 PM <geekosaur> I think OS X relocations contain symbol table references, which need to be patched up if entries are removed
3:33 PM <rwbarton> I see (also reading the OS X man page)
3:34 PM <geekosaur> I note it still has -x and -s which are the options I was thinking of (remove local / debugging symbols, respectively)
3:35 PM ⇐ dolio quit (~dolio@haskell/developer/dolio) Quit: WeeChat 1.2
3:36 PM <gridaphobe> interesting.. so while the `atom` binary increases in size, `Main.o` actually *decreases* by about 250 bytes
3:37 PM <geekosaur> oh, no, it replaced traditional -s with -S and made -s mean something else
3:37 PM ⇐ matrixa1 quit (~matrixa1@h-88-35.a230.priv.bahnhof.se) Quit: moo
3:37 PM <gridaphobe> https://gist.github.com/gridaphobe/dab13eb56d4691c71a31
3:38 PM <gridaphobe> argh, github reordered the files, the top is my commit, bottom is the parent commit
3:39 PM <rwbarton> well sure, that's not too surprising. since most of the exectuable is not Main.o
3:39 PM <gridaphobe> but Main.o now includes a couple CallStacks
3:39 PM <rwbarton> yes, the part where it's smaller now is weird
3:40 PM <rwbarton> possibly ghc did less inlining, or something like that
3:40 PM <rwbarton> or who knows, more inlining
3:40 PM → matrixa1 joined (~matrixa1@h-88-35.a230.priv.bahnhof.se)
3:41 PM <rwbarton> thomie: my inplace ghc has the base package registered with its libraries in /home/rwbarton/ghc/libraries/base/dist-install/build. so I'm guessing that is pre-install?
3:43 PM <thomie> rwbarton: yes. If you're using a build.mk that originated from build.mk.sample, with the `STRIP_CMD = :` setting, no stripping will be performed at all (at least I think that's the idea)
3:44 PM → tibbe joined (~tibbe@72.2.55.21)
3:44 PM <thomie> afk
3:45 PM  ⇐ Luke quit  ↔ osa1 nipped out
3:50 PM <rwbarton> the ghcspeed builds have "WARNING: You don't have a mk/build.mk file", so it looks like they do potentially run strip?
3:53 PM  → dolio joined  ⇐ tibbe quit
3:55 PM <rwbarton> I guess it's plausible that stripping local symbols would decrease executable size around 3%
3:56 PM <rwbarton> Wait, but nofib already runs strip I think...
3:57 PM <rwbarton> so I don't think the Cabal change should be relevant
3:58 PM <rwbarton> or at least don't understand why it would be relevant
3:59 PM  → MP2E and tibbe joined  ⇐ mathk, nomeata and matrixa1 quit  ↔ osa1 nipped out
4:33 PM <ezyang> rwbarton: Cabal controls symbol sizes
4:34 PM <ezyang> through the choice of what is passed into -this-package-key
4:34 PM <ezyang> I didn't follow the whole discussion, but it could matter.
4:34 PM  → predator117 joined  ⇐ predator217, quchen and tempname11 quit  ↔ qnikst and MP2E nipped out
4:47 PM <rwbarton> ezyang: the Cabal change in question was when it runs strip. But there could have been other relevant Cabal changes in gridaphobe's Cabal submodule update as far as I know
4:49 PM <rwbarton> but actually, nofib's strip should remove all symbols anyways
4:50 PM  → Luke joined  ↔ hexagoxel nipped out
4:58 PM <gridaphobe> the previous commit for Cabal was f47732a50d4bd103c5660c2fbcd77cbce8c521b5
4:58 PM <gridaphobe> looking at the log, thomie's commit (7ef72b3184d7ab1b27f43756a0ad8a828431fac3) is the only one that looks relevant
4:59 PM <rwbarton> the simplest and most likely explanation is that the code got 3% bigger
4:59 PM <rwbarton> but if you wanted to test that you could try building the parent commit of yours with the cabal submodule update applied
5:00 PM <gridaphobe> some code that was imported
5:00 PM ⇐ yeji quit (~yeji@216.165.95.78) Remote host closed the connection
5:02 PM <gridaphobe> how concerning is the 3% increase?
5:06 PM  ⇐ osa1 and ryantrinkle quit
5:10 PM <rwbarton> I dunno. Mostly it seems like potentially more than expected? It would be good to account for the size changes and investigate whether they might be caused by some unexpected side effect. The modules that got smaller in nofib would be a natural place to start.
	12:15 PM <thomie> gridaphobe: binary sizes are 3% up. Just checking if that is expected, see: https://perf.haskell.org/ghc/#revision/6740d70d95cb81cea3859ff847afc61ec439db4f
	12:16 PM → LnL, tempname11, mathk, osa1, ivan\, moei and nkartash_ joined ← Welkin left ⇐ gentleben, mgsloan, Yuras, ryantrinkle and bb010g quit ↔ dcoutts_, edsko, hamishmack and Gal3rielol popped in ↔ tibbe, Luke (was Luke_) and matrixa1 nipped out • waern_ → waern
	2:05 PM <gridaphobe> thomie: i wouldn't expect an increase in the binary size unless the programs are using implicit callstacks (e.g. by using error, undefined, or assert)
	2:06 PM → quchen joined ⇐ matrixa1 and ivan\ quit
	2:13 PM <slyfox> what does ghc generate for nonexhaustive pattern matches?
	2:14 PM ↔ ivan\ popped in
	2:16 PM <gridaphobe> i'm afraid i don't know anything about the nofib suite, but i suspect the programs are compiled with optimizations :)
	2:16 PM → matrixa1 joined (~matrixa1@h-88-35.a230.priv.bahnhof.se)
	2:16 PM <ghc-github> [ghc] ghc-mirror pushed 1 new commit to master: http://git.io/vG7aN
	2:16 PM <ghc-github> ghc/master 34b106f Edward Z. Yang: Accept underscores in the module parser. (Thanks spinda for the fix.)...
	2:17 PM <gridaphobe> oh but this is talking about the produced binary, not ghc's allocations, right?
	2:17 PM <gridaphobe> so it wouldn't be connected to the allocation increase for the two tests
	2:18 PM → nomeata and ryantrinkle joined
	2:18 PM <slyfox> i believe it's binary size, yes
	2:19 PM <slyfox> the very end of page has 2 +17% allocations failures: tests/alloc/T5321FD and tests/alloc/T5321Fun
	2:20 PM → ivan\ joined (~ivan@unaffiliated/ivan/x-000001)
	2:20 PM <gridaphobe> right, those are expected increases because those tests are compiled without optimizations, so ghc doesn't see that undefined diverges
	2:21 PM <gridaphobe> and it ends up doing unnecessary work
	2:22 PM → hamishmack joined ⇐ gustavold and tibbe quit ↔ matrixa1 and ivan\ nipped out
	2:40 PM <gridaphobe> thomie: what's the best way to determine why the binary sizes have increased? is there a particular dump flag that would be useful?
	2:41 PM ⇐ hamishmack quit (~hamishmac@72.2.55.21) Quit: hamishmack
	2:45 PM <gridaphobe> thomie: actually, looking at the core for one of the benchmarks (the atom benchmark), `error` is used via `read`, so a small increase would be expected there
	2:45 PM <gridaphobe> i'll look at the others
	2:46 PM → dcoutts_ and tibbe joined ⇐ Luke quit ↔ osa1 nipped out
	3:13 PM <rwbarton> "binary size" is the size of the whole stripped executable including the needed parts of the rts/ghc-prim/base such as, presumably, most of the IO manager
	3:14 PM ⇐ tibbe quit (~tibbe@209-82-80-116.dedicated.allstream.net) Quit: tibbe
	3:14 PM <rwbarton> which probably uses error some I guess?
	3:14 PM <rwbarton> Still 3% sounds like a lot
	3:14 PM <rwbarton> It might be better to start by looking at "module sizes" (which are the sizes of the .o files)
	3:16 PM → Luke joined (~luke@unaffiliated/luke)
	3:17 PM <gridaphobe> well the module itself uses error too (many of the benchmarks do since they read some data from stdin)
	3:17 PM <gridaphobe> the call to `read` is inlined
	3:17 PM <rwbarton> sure, but that fact will not make a visible change to binary size
	3:18 PM <rwbarton> or not a 30K change anyways
	3:18 PM <gridaphobe> true..
	3:18 PM ⇐ raichoo and LnL quit
	3:22 PM <rwbarton> you can download the logs and run nofib-analyse on them to see which individual nofib modules changed sizes
	3:22 PM <rwbarton> oddly, some got smaller too but on average they are apparently 1.0% larger
	3:24 PM <rwbarton> (though that number is potentially bogus because of how it is computed)
	3:24 PM → Javran joined ⇐ dcoutts_ and nomeata quit ↔ osa1 nipped out
	3:27 PM <gridaphobe> rwbarton: are the logs you're referring to just the build logs? (i get an illegal byte sequence error)
	3:27 PM <thomie> wait a minute, you updated the Cabal module. I've been making changes to the way stripping is handled in Cabal...
	3:27 PM <rwbarton> gridaphobe, yes
	3:28 PM → nomeata joined (~jojo@nomeata.de)
	3:28 PM <gridaphobe> yes i updated Cabal because i had to patch it to use the new pattern synonym for ErrorCall
	3:28 PM <rwbarton> they have non-ASCII (UTF-8) characters though, make sure you have a UTF-8 locale?
	3:29 PM <rwbarton> well nofib programs aren't built with cabal
	3:29 PM <rwbarton> I guess base is built with cabal though?
	3:29 PM <thomie> so the change I made is that .a files are only stripped when you install them, not before
	3:29 PM <gridaphobe> i have LANG="en_US.UTF-8" set
	3:30 PM <gridaphobe> ah, so if the benchmarks are linked against the inplace libraries, the libraries won't have been stripped yet?
	3:30 PM <rwbarton> gridaphobe: well then I don't know what is wrong
	3:30 PM — rwbarton is still confused about stripping .a files, btw
	3:31 PM <rwbarton> gridaphobe: FWIW I used wget to download the log files and that worked for me, if you used a browser maybe it did something "smart"
	3:31 PM <gridaphobe> i used wget too :)
	3:31 PM <geekosaur> depends on how strip is being used
	3:31 PM <rwbarton> hm
	3:32 PM <thomie> gridaphobe: yes, could be
	3:32 PM <geekosaur> also OS X strip has variant semantics and is safe on .a files (it adjusts relocation entries)
	3:32 PM <rwbarton> adjusts how?
	3:33 PM ⇐ Javran quit (~javran@c-73-200-214-152.hsd1.md.comcast.net) Quit: Leaving.
	3:33 PM <geekosaur> I think OS X relocations contain symbol table references, which need to be patched up if entries are removed
	3:33 PM <rwbarton> I see (also reading the OS X man page)
	3:34 PM <geekosaur> I note it still has -x and -s which are the options I was thinking of (remove local / debugging symbols, respectively)
	3:35 PM ⇐ dolio quit (~dolio@haskell/developer/dolio) Quit: WeeChat 1.2
	3:36 PM <gridaphobe> interesting.. so while the `atom` binary increases in size, `Main.o` actually decreases by about 250 bytes
	3:37 PM <geekosaur> oh, no, it replaced traditional -s with -S and made -s mean something else
	3:37 PM ⇐ matrixa1 quit (~matrixa1@h-88-35.a230.priv.bahnhof.se) Quit: moo
	3:37 PM <gridaphobe> https://gist.github.com/gridaphobe/dab13eb56d4691c71a31
	3:38 PM <gridaphobe> argh, github reordered the files, the top is my commit, bottom is the parent commit
	3:39 PM <rwbarton> well sure, that's not too surprising. since most of the exectuable is not Main.o
	3:39 PM <gridaphobe> but Main.o now includes a couple CallStacks
	3:39 PM <rwbarton> yes, the part where it's smaller now is weird
	3:40 PM <rwbarton> possibly ghc did less inlining, or something like that
	3:40 PM <rwbarton> or who knows, more inlining
	3:40 PM → matrixa1 joined (~matrixa1@h-88-35.a230.priv.bahnhof.se)
	3:41 PM <rwbarton> thomie: my inplace ghc has the base package registered with its libraries in /home/rwbarton/ghc/libraries/base/dist-install/build. so I'm guessing that is pre-install?
	3:43 PM <thomie> rwbarton: yes. If you're using a build.mk that originated from build.mk.sample, with the `STRIP_CMD = :` setting, no stripping will be performed at all (at least I think that's the idea)
	3:44 PM → tibbe joined (~tibbe@72.2.55.21)
	3:44 PM <thomie> afk
	3:45 PM ⇐ Luke quit ↔ osa1 nipped out
	3:50 PM <rwbarton> the ghcspeed builds have "WARNING: You don't have a mk/build.mk file", so it looks like they do potentially run strip?
	3:53 PM → dolio joined ⇐ tibbe quit
	3:55 PM <rwbarton> I guess it's plausible that stripping local symbols would decrease executable size around 3%
	3:56 PM <rwbarton> Wait, but nofib already runs strip I think...
	3:57 PM <rwbarton> so I don't think the Cabal change should be relevant
	3:58 PM <rwbarton> or at least don't understand why it would be relevant
	3:59 PM → MP2E and tibbe joined ⇐ mathk, nomeata and matrixa1 quit ↔ osa1 nipped out
	4:33 PM <ezyang> rwbarton: Cabal controls symbol sizes
	4:34 PM <ezyang> through the choice of what is passed into -this-package-key
	4:34 PM <ezyang> I didn't follow the whole discussion, but it could matter.
	4:34 PM → predator117 joined ⇐ predator217, quchen and tempname11 quit ↔ qnikst and MP2E nipped out
	4:47 PM <rwbarton> ezyang: the Cabal change in question was when it runs strip. But there could have been other relevant Cabal changes in gridaphobe's Cabal submodule update as far as I know
	4:49 PM <rwbarton> but actually, nofib's strip should remove all symbols anyways
	4:50 PM → Luke joined ↔ hexagoxel nipped out
	4:58 PM <gridaphobe> the previous commit for Cabal was f47732a50d4bd103c5660c2fbcd77cbce8c521b5
	4:58 PM <gridaphobe> looking at the log, thomie's commit (7ef72b3184d7ab1b27f43756a0ad8a828431fac3) is the only one that looks relevant
	4:59 PM <rwbarton> the simplest and most likely explanation is that the code got 3% bigger
	4:59 PM <rwbarton> but if you wanted to test that you could try building the parent commit of yours with the cabal submodule update applied
	5:00 PM <gridaphobe> some code that was imported
	5:00 PM ⇐ yeji quit (~yeji@216.165.95.78) Remote host closed the connection
	5:02 PM <gridaphobe> how concerning is the 3% increase?
	5:06 PM ⇐ osa1 and ryantrinkle quit
	5:10 PM <rwbarton> I dunno. Mostly it seems like potentially more than expected? It would be good to account for the size changes and investigate whether they might be caused by some unexpected side effect. The modules that got smaller in nofib would be a natural place to start.