Skip to content

Instantly share code, notes, and snippets.

@yukitos
Last active August 29, 2015 14:19
Show Gist options
  • Save yukitos/8e4a3558487a0db94a28 to your computer and use it in GitHub Desktop.
Save yukitos/8e4a3558487a0db94a28 to your computer and use it in GitHub Desktop.
Trying to find the root cause of https://github.com/Microsoft/visualfsharp/issues/380
#if INTERACTIVE
#r "..\\packages\\FSharp.Compiler.Service.0.0.89\\lib\\net45\\FSharp.Compiler.Service.dll"
#endif
open System
open Microsoft.FSharp.Compiler.SourceCodeServices
let sourceTok = FSharpSourceTokenizer([], "C:\\test.fsx")
let lines = """
type Foo = class end
type Bar = interface
abstract Baz: Foo: string -> int
end
"""
let rec tokenizeLine (tokenizer:FSharpLineTokenizer) state =
match tokenizer.ScanToken(state) with
| Some tok, state ->
printf "%s " tok.TokenName
tokenizeLine tokenizer state
| None, satte -> state
let rec tokenizeLines state count lines =
match lines with
| line::lines ->
printfn "\nLine %d" count
let tokenizer = sourceTok.CreateLineTokenizer(line)
let state = tokenizeLine tokenizer state
tokenizeLines state (count + 1) lines
| [] -> ()
lines.Split('\r', '\n')
|> List.ofSeq
|> tokenizeLines 0L 1
Line 1
Line 2
TYPE WHITESPACE IDENT WHITESPACE EQUALS WHITESPACE CLASS WHITESPACE END
Line 3
Line 4
TYPE WHITESPACE IDENT WHITESPACE EQUALS WHITESPACE INTERFACE
Line 5
WHITESPACE ABSTRACT WHITESPACE IDENT COLON WHITESPACE IDENT COLON WHITESPACE IDENT WHITESPACE RARROW WHITESPACE IDENT
Line 6
END
Line 7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment