Skip to content

Instantly share code, notes, and snippets.

@applicative
Forked from jartur/mem.hs
Created July 20, 2012 16:15
Show Gist options
  • Save applicative/3151613 to your computer and use it in GitHub Desktop.
Save applicative/3151613 to your computer and use it in GitHub Desktop.
Memory hog + Stack overflow
import qualified Data.Map as M -- cabal install unordered-containers
import qualified Data.ByteString.Char8 as B
import Data.List (foldl')
main = B.interact $ exhibit . foldl' updateCount M.empty . B.words
where exhibit counts = B.concat $ zipWith pretty keys_ counts_ where
counts_ = M.elems counts ; keys_ = M.keys counts
pretty w count = w +++ " :: " +++ show count +++ "\n"
updateCount m w = M.insertWith (+) w (1::Int) m
(+++) x y = B.append x (B.pack y)
@applicative
Copy link
Author

Here's how I would have done this if I were writing it from scratch, I couldn't tell if you needed more complicated IO. If the file has unicode you will need Text. If it is huge you might need Text.Lazy or ByteString.Char8.Lazy . If compiled with ghc --make -O2 hsprogram.hs then cat 40mb.txt | ./hsprogram has no difficulty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment