Skip to content

Instantly share code, notes, and snippets.

@jartur
Created July 20, 2012 15:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jartur/3151176 to your computer and use it in GitHub Desktop.
Save jartur/3151176 to your computer and use it in GitHub Desktop.
Memory hog + Stack overflow
module Main where
import System.IO
import qualified Data.Map as M
import qualified Data.Text.Lazy as T
import qualified Data.Text.Lazy.IO as T
import Data.List (foldl')
-- maybe Data.Text.Lazy maybe ByteString
(+++) = T.append
main = do
inh <- openFile "doubles.txt" ReadMode
document <- T.hGetContents inh
outh <- openFile "dict.txt" WriteMode
let counts = mainLoop'' document M.empty
let excsPairs = M.assocs counts
let excsStr = T.concat $ map (\(w, count) -> w +++ T.pack " :: " +++ T.pack (show count) +++T.pack "\n") excsPairs
T.hPutStr outh excsStr
hClose inh
hClose outh
-- mainLoop' :: T.Text -> M.Map T.Text Int -> M.Map T.Text Int
mainLoop' m line = updateMapForWords m (T.words line)
mainLoop'' document m = foldl' mainLoop' m (T.lines document)
-- mainLoop :: Handle -> M.Map String Int -> IO (M.Map String Int)
mainLoop inh counts = do
inEof <- hIsEOF inh
if inEof
then return counts
else do
ws <- (T.hGetLine inh >>= \inStr -> return $ T.words inStr)
mainLoop inh $ updateMapForWords counts ws
-- - updateMapForWords :: [String] -> M.Map String Int -> M.Map String Int
updateMapForWords counts ws =
foldr (\w m -> M.alter updateCount w m) counts ws
where
updateCount (Just !c) = Just $ c + 1
updateCount Nothing = Just 1
@applicative
Copy link

I put up another gist, but all this forking is confusing https://gist.github.com/3151613

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment