Skip to content

Instantly share code, notes, and snippets.

@swarley

swarley/lexer_output.ex

Created March 1, 2013 01:20
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save swarley/5061726 to your computer and use it in GitHub Desktop.
Save swarley/5061726 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
lexer_output.ex
iex(1)> Lexer.PIR.tokenize("<<\"TEST\" <<\"SECOND_LEVEL\" <<'THIRD'\nthis probably won't work the first time\nTEST\nthis should go into SECOND_LEVEL\nSECOND_LEVEL\nthis almost certainly will not work\nTHIRD")
[
{
:str,
1,
'this probably won\'t work the first time\n'
},
{
:str,
1,
'this should go into SECOND_LEVEL\n'
},
{
:str,
1,
'this almost certainly will not work\n'
},
{
:newline,
1,
nil
}
]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment