Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

First of all, sorry for the messy code (it was in a bad condition, and I've been commenting/uncommenting bunch of things to understand what's going on and where, only got time to fix it but not clearnup)

At first, I've started getting

Stack space overflow: current size 8388608 bytes.

Whenever running trying to iterate over 1M records (that's a small kind of database). The "suspect" was of course a function that was recuring into itself in order to iterate over the whole database: https://github.com/ifesdjeen/continuum/blob/master/src/Continuum/Storage.hs#L129-L144

I've started adding seqs everywhere, nothing helped. Tried to examine a function a level above (one that's passing L.Fold, and first thought that Folds themselves are causing trouble). Checked it - no success.

After long time of inserting trace on each level, I've finally found a spot, right before which everything was cool and great, and i've been getting output, but at that point (where fmap sits in "fixed" version), I've been getting 0 traces. So that meant that function wasn't evaluated, therefore needed to be forced.

I've added "strict fmap", and it worked: "Fixed version": https://github.com/ifesdjeen/continuum/blob/stackoverflow/src/Continuum/Storage.hs#L133-L134

In general, I indeed made many more changes. And most likely "just" inserting fmap wouldn't even be possible in the initial code. Although the largest leak (at least, speaking semantically from the code) is when I'm folding that 1M records into something smaller. The more I aggregate at that point, the worse. And, most likely, because of fmap (which is just fmap f (Right y) = Right (f y) in that case). I've been getting a Right + non-evaluated new function.

Of course, now I have to change semantics from "generic" strict fmap to particular one. But in general that still seems to explain the problem :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment