Environment: I think we will try the coding on Linux. I use Mac but I can just work from a vagrant machine or from an EC2 instance.
robots-txt-scanner takes in a robots.txt file and outputs a list of (token, lexeme) tuple.
print scanner.scan("User-agent: Google\nDisallow: *")
(('\\USER_AGENT_VALUE/', 'User-agent: Google'), ('\\DISALLOW_VALUE/', 'Disallow: *'))
Features: