Skip to content

Instantly share code, notes, and snippets.

@jamis
Created Feb 24, 2017
Embed
What would you like to do?
Source files for Jamis' "designing for tests" presentation at AtomicJolt

Let's talk about testing.

002-calc

  • Simple recursive-descent parser
  • Abstract Syntax Tree (AST) with three node types (Binary, Unary, Atom)
  • #evaluate is the only public API on Calc
    • parses the input
    • evaluates the AST
  • It really works!

003-test

  • take 1: FEELS like a decent test, but we're not testing parsing in isolation, and we CAN'T test parsing in isolation via the #evaluate method, because it's parsing AND evaluating the tree.
  • take 2: this feels wrong, because it is. you should never test the private interface of an object. if you find yourself needing to call a private interface in a test, then you've designed something wrong (either your test, or your object).

004-calc

Refactored!

  • Added Parser class
    • ! is for required rules (fail if nothing matches), ? is for optional rules (return nil if nothing matches)
    • made methods all public.
    • semantics!
      • If you say "Parser parses expressions", then anything but #expr! becomes private interface
      • If instead you just say "Parser parses", then everything is fair game for the public API, because you're exposing an API for parsing stuff.
    • DOCUMENT the semantics!
  • Gave responsibility to the AST nodes
  • Calc does almost nothing now

005-test

  • We can now test the parser directly to make sure factors, terms, and expressions are being parsed correctly.
  • We can now test the evaluator directly, to make sure evaluation is happening properly.

006-test

One other gotcha I want to point out, which I've seen happening:

never ever ever use instance_variable_get in tests! if you need to access internal state, either:

007-calc

expose the internal state as part of the public interface

  • Makes a contract -- 'this interface is supported'
  • Careful not to expose too much, and only where appropriate
    • if something truly is private, exposing it for testing is probably a bad idea

OR

008-calc

refactor into separate classes so the internal state becomes testable as part of another object

  • tokenizer remains as private, internal state of the parser, but accepts it as initialization data
  • parser can instantiate tokenizer and reference it without violating encapsulation

Conclusion

  • Smaller objects test better
  • Smaller objects are easier to reuse
  • Focused responsibilities are easier to describe and think about
    • is the calculator calculating, or is it parsing too?
    • is the parser parsing, or tokenizing too?
require './002-calc'
calc = Calc.new
puts calc.evaluate('3')
puts calc.evaluate('3 + 5')
puts calc.evaluate('3 + 5 * 3 - 1')
puts calc.evaluate('(3 + 5) * (3 - 1)')
puts calc.evaluate('(3 + 5) * -(3 - 1)')
puts calc.evaluate('((3 + 5) / 4) * -(3 - 1)')
require 'strscan'
# expr = term expr-op ;
# expr-op = '+' expr
# | '-' expr
# | () ;
#
# term = factor term-op ;
# term-op = '*' term
# | '/' term
# | () ;
#
# factor = number
# | '(' expr ')'
# | '-' factor ;
class Calc
class BinaryExpr < Struct.new(:left, :op, :right)
end
class UnaryExpr < Struct.new(:op, :node)
end
class Atom < Struct.new(:value)
end
#
def evaluate(string)
tree = _parse(string)
_evaluate_tree(tree)
end
#
def _parse(string)
scanner = StringScanner.new(string)
_parse_expr(scanner)
end
def _parse_expr(scanner)
left = _parse_term(scanner)
op, right = _parse_expr_op(scanner)
if op
BinaryExpr.new(left, op, right)
else
left
end
end
def _parse_expr_op(scanner)
_eat_space(scanner)
op = scanner.scan(/[-+]/) || return
[op, _parse_expr(scanner)]
end
def _parse_term(scanner)
left = _parse_factor(scanner)
op, right = _parse_term_op(scanner)
if op
BinaryExpr.new(left, op, right)
else
left
end
end
def _parse_term_op(scanner)
_eat_space(scanner)
op = scanner.scan(%r{[\*/]}) || return
[op, _parse_term(scanner)]
end
def _parse_factor(scanner)
_eat_space(scanner)
_parse_number(scanner) ||
_parse_parens(scanner) ||
_parse_negation(scanner) ||
raise("can't parse #{scanner.rest}")
end
def _parse_number(scanner)
token = scanner.scan(/[0-9]+(\.[0-9]+)?/) || return
Atom.new(token.to_f)
end
def _parse_parens(scanner)
scanner.scan(/\(/) || return
_parse_expr(scanner).tap do
scanner.scan(/\)/) || raise('unclosed parens')
end
end
def _parse_negation(scanner)
scanner.scan(/-/) || return
UnaryExpr.new('-@', _parse_factor(scanner))
end
def _eat_space(scanner)
scanner.scan(/\s+/)
end
def _evaluate_tree(node)
case node
when UnaryExpr then
_evaluate_tree(node.node).send(node.op)
when BinaryExpr then
_evaluate_tree(node.left).send(node.op, _evaluate_tree(node.right))
when Atom then
node.value
else
raise "can't evaluate #{node.inspect}"
end
end
end
require 'minitest/autorun'
require './calc-1'
class CalcTest < Minitest::Test
def setup
@calc = Calc.new
end
def test_factor_is_parsed_correctly___take_1
assert_equal 3, @calc.evaluate('3')
end
def test_factor_is_parsed_correctly___take_2
scanner = StringScanner.new('3')
node = @calc._parse_factor(scanner)
assert_kind_of Calc::Atom, node
assert_equal 3, node.value
end
# etc...
end
require 'strscan'
# expr = term expr-op ;
# expr-op = '+' expr
# | '-' expr
# | () ;
#
# term = factor term-op ;
# term-op = '*' term
# | '/' term
# | () ;
#
# factor = number
# | '(' expr ')'
# | '-' factor ;
class BinaryExpr < Struct.new(:left, :op, :right)
def evaluate
left.evaluate.send(op, right.evaluate)
end
end
class UnaryExpr < Struct.new(:op, :node)
def evaluate
node.evaluate.send(op)
end
end
class Atom < Struct.new(:value)
def evaluate
value
end
end
# An interface for parsing strings
class Parser
attr_reader :scanner
def initialize(string)
@scanner = StringScanner.new(string)
end
def expr!
left = term!
op, right = expr_op?
if op
BinaryExpr.new(left, op, right)
else
left
end
end
def expr_op?
eat_space!
op = scanner.scan(/[-+]/) || return
[op, expr!]
end
def term!
left = factor!
op, right = term_op?
if op
BinaryExpr.new(left, op, right)
else
left
end
end
def term_op?
eat_space!
op = scanner.scan(%r{[\*/]}) || return
[op, term!]
end
def factor!
eat_space!
number? ||
parens? ||
negation? ||
raise("can't parse #{scanner.rest}")
end
def number?
token = scanner.scan(/[0-9]+(\.[0-9]+)?/) || return
Atom.new(token.to_f)
end
def parens?
scanner.scan(/\(/) || return
expr!.tap do
scanner.scan(/\)/) || raise('unclosed parens')
end
end
def negation?
scanner.scan(/-/) || return
UnaryExpr.new('-@', factor!)
end
def eat_space!
scanner.scan(/\s+/)
end
end
class Calc
def evaluate(string)
parser = Parser.new(string)
tree = parser.expr!
tree.evaluate
end
end
require 'minitest/autorun'
require './calc-2'
class ParserTest < Minitest::Test
def test_atom_is_parsed_correctly
atom = _parse('3').factor!
assert_kind_of Atom, atom
assert_equal 3, atom.value
end
# etc...
def _parse(string)
Parser.new(string)
end
end
class ASTTest < Minitest::Test
def test_evaluate_atom_returns_value
atom = Atom.new(3)
assert_equal 3, atom.evaluate
end
def test_evaluate_unary_evaluates_child_and_applies_operator
unary = UnaryExpr.new('-@', Atom.new(3))
assert_equal(-3, unary.evaluate)
end
# etc...
end
class CalcTest < Minitest::Test
def setup
@calc = Calc.new
end
def test_evaluate_with_single_atom
assert_equal 3, @calc.evaluate('3')
end
def test_evaluate_with_complex_expression
expect = -20 + 8.0 / 6.0
assert_equal expect, @calc.evaluate('-((3 + 1) * 5) + 8 / (2 * 3)')
end
end
require 'minitest/autorun'
require './calc-2'
class ParserTest < Minitest::Test
def test_expr_should_consume_all_input
parser = _parse('(3 + 5) / 4 - 1')
parser.expr!
# RED ALERT!
assert parser.instance_variable_get(:@scanner).eos?
end
# etc...
def _parse(string)
Parser.new(string)
end
end
class Parser
attr_reader :scanner # <-- expose private state for testing
# ...
end
class ParserTest < Minitest::Test
def test_expr_should_consume_all_input
parser = _parse('(3 + 5) / 4 - 1')
parser.expr!
assert parser.scanner.eos?
end
# etc...
end
class Tokenizer
def initialize(string)
# ...
end
def empty?
# ...
end
# ...
end
class Parser
def initialize(tokenizer)
#...
end
# ...
end
def test_expr_should_consume_all_input
tokenizer = Tokenizer.new(string)
parser = Parser.new(tokenizer)
# ...
assert tokenizer.empty?
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment