Reds lexer (drives separator + grammar)

reds-lexer.r

REBOL [
	Title:		"Red/System lexical analysis"
	Date:		1-Jul-2011
	Name:		"Reds lexer"

	Type:		none
	Version:	1.0.0
	File:		%/G/Projects/Common/RED/red-system/sources/reds-lexer/reds-lexer.r
	Home:		http://users.telenet.be/rwmeijer

	Author:		"Rudolf W. Meijer"
	Rights:		"Copyright (C) 2011 Rudolf W. Meijer. All Rights Reserved"

	History:	[
				0.0.0 [19-Jun-2011 {Start of project} "RM"]
				0.5.0 [24-Jun-2011 {First working version} "RM"]
				0.7.0 [27-Jun-2011 {Added file! and tuple! literals} "RM"]
				0.8.0 [27-Jun-2011 {Simplified the separator} "RM"]
				0.9.0 [29-Jun-2011 {
					Separator reduced to stripping comments only,
					Grammar takes care of whitespace
				} "RM" ]
				1.0.0 [1-Jul-2011 {Grammar takes care of comments also} "RM"]
	]

]
;---|----1----|----2----|----3----|----4----|----5----|----6----|----7----|-

do %reds-lex-grammar.r

reds-lexer: func [
	inp [file! url! binary! string!]
][
	unless string? inp [
		unless binary? inp [inp: read/binary inp]
		inp: to-string inp
	]
	either empty? inp
	[
		print "nothing to analyse: empty input"
	][
		print "start"
; diagnostic
		print
		["parse" dt [
		parse/all inp lex-grammar/program
		]]
		; parsed source is in lex-grammar/source
	]
]

test-text: 
 %../../tests/source/units/exit-test.reds

print "call"
reds-lexer 
;copy
test-text


; diagnostic
print mold/all head lex-grammar/source



ask ""

halt

This grammar does not consider the shift operators that were added just today.