Skip to content

Instantly share code, notes, and snippets.

@non

non/seeds.md

Last active July 10, 2024 20:34
Show Gist options
  • Save non/aeef5824b3f681b9cfc141437b16b014 to your computer and use it in GitHub Desktop.
Save non/aeef5824b3f681b9cfc141437b16b014 to your computer and use it in GitHub Desktop.
Download ZIP
Simple example of using seeds with ScalaCheck for deterministic property-based testing.
Raw
seeds.md

introduction

ScalaCheck 1.14.0 was just released with support for deterministic testing using seeds. Some folks have asked for examples, so I wanted to produce a Gist to help people use this feature.

simple example

These examples will assume the following imports:

import org.scalacheck._
import org.scalacheck.Prop.forAll
import org.scalacheck.rng.Seed

Here's a basic property that tests we can compose map calls when working with List:

class DemoProps extends Properties("Demo") {
  property("map fusion") =
    forAll { (xs: List[Int], f: Int => Int, g: Int => Int) =>
      xs.map(f).map(g) == xs.map(f andThen g)
    }
}

introucing a bug

Let's introduce a bug! There is no guarantee that filtering a list will reduce its size to below 60, but since ScalaCheck is more likely to generate short lists than long lists, there's a chance this property will pass.

class BugProps1 extends Properties("Bug1") {
  property("filtered lists are short") =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      xs.filter(p).size < 60
    }
}

Running this initially I got:

[info] + Bug1.filtered lists are short: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1

However, after several runs I saw a failure:

[info] ! Bug1.filtered lists are short: Falsified after 88 passed tests.
[info] > ARG_0: List("73603", "-21", "-2147483648", "-1", "-1", "-115", "1", "1", "-1", "30", "187880611", "53968", "783", "-1584256195", "1937207237", "4", "-1", "-2113554838", "-568121085", "0", "0", "-1", "906382812", "255", "-2147483648", "0", "-1", "-8388608", "971737", "-1689466537", "-1", "-1", "-823800529", "-2147483648", "0", "-1", "-1143061184", "-1", "510634863", "1", "-1063758949", "2141521118", "311931058", "-266177369", "19835993", "1018393343", "-1992022794", "2147483647", "1669721679", "578926712", "1", "0", "-1", "-1", "2147483647", "495764033", "0", "2147483647", "-916147596", "-267865817", "1788828777", "-2147483648", "-1", "1", "0", "1", "1646969489", "2147483647", "-551877913", "-2147483648", "-2147483648")
[info] > ARG_0_ORIGINAL: List("150738984", "728116065", "-2147483648", "-1", "-1", "-971913915", "1", "1", "-1", "1039244423", "-1503044895", "-884215070", "-821511363", "-1584256195", "1937207237", "-2147483648", "-1", "-2113554838", "-568121085", "0", "0", "1736041799", "-2147483648", "-2147483648", "2147483647", "-1375641703", "-1", "-1", "906382812", "2147483647", "-2147483648", "0", "-1876898002", "-2147483648", "1990117421", "-1689466537", "-1", "-1", "-823800529", "-2147483648", "0", "-1", "-1143061184", "-1", "510634863", "1", "-1063758949", "1079824459", "-504900761", "431944360", "-2147483648", "2141521118", "311931058", "-266177369", "19835993", "1018393343", "-1992022794", "2147483647", "1669721679", "-2062075049", "578926712", "1", "0", "-1", "-1", "2147483647", "495764033", "0", "2147483647", "-916147596", "-267865817", "1788828777", "-2147483648", "-1", "-690937068", "-1", "1465506269", "1", "0", "1", "1646969489", "2147483647", "-551877913", "-2147483648", "-2147483648", "-1345475936", "-2147483648")
[info] > ARG_1: org.scalacheck.GenArities$$Lambda$16504/1944917297@415d0821
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0

It's good that ScalaCheck found an error, but it will be hard for us to recreate this specific failure case. Our first argument (ARG_0) is a very large literal list , and we don't have any visibility into our second argument (ARG_1), which is a function.

extracting a seed

Using the new propertyWithSeed method, we can rewrite our property to get a specific seed if it fails:

class BugProps2 extends Properties("Bug") {
  propertyWithSeed("filtered lists are short", None) =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      xs.filter(p).size < 60
    }
}

Running this, we'll continue to see spurious passing cases as before:

[info] + Bug2.filtered lists are short: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1

After many more passing runs, we observe another failing case:

failing seed for Bug.filtered lists are short is H84AeXXB037k6uRDc8xUqygjoDnJ8XMdAD9TpRY4iqP=
[info] ! Bug.filtered lists are short: Falsified after 94 passed tests.
[info] > ARG_0: List("0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0")
[info] > ARG_0_ORIGINAL: List("-1756030137", "-1", "1", "-1577978365", "1996323280", "1235987858", "-1", "-119289092", "1796794817", "-2147483648", "0", "-185835109", "2147483647", "-573437505", "-1", "-2147483648", "-2147483648", "1117642543", "-1002215582", "0", "1532444616", "-1171363902", "-786379840", "-2147483648", "-1", "2147483647", "451199318", "324096022", "-107161746", "-1", "-1445092754", "2147483647", "1", "-1688700755", "0", "-1719981158", "1", "2147483647", "1", "-1", "-2147483648", "0", "2147483647", "0", "2147483647", "-2147483648", "-1", "-2147483648", "1", "1632255924", "-1", "1", "-988803452", "0", "102049049", "-842347084", "-1", "996100904", "0", "2147483647", "2147483647", "-1363532431", "2147483647", "-1086043995", "-2147483648", "-1", "-1385447539", "2147483647", "0", "2147483647", "1", "-1", "-927675505", "-2147483648", "-2147483648", "0", "327082319", "-493095153", "-2147483648", "-149872041", "945020429", "1577344162", "-2147483648", "-1694300309", "-2147483648", "-2147483648", "834915392", "-1661078214", "2147483647", "1313409436")
[info] > ARG_1: org.scalacheck.GenArities$$Lambda$14768/327031953@4c7afdff
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0

We see the same kind of output as before. But this time, there's an additional line present:

failing seed for Bug.filtered lists are short is H84AeXXB037k6uRDc8xUqygjoDnJ8XMdAD9TpRY4iqP=

This is an instance of org.scalacheck.rng.Seed encoded in Base-64 format.

We can take this seed and pass it to propertyWithSeed to reproduce this exact failing case:

class BugProps3 extends Properties("Bug3") {
  propertyWithSeed("filtered lists are short", Some("H84AeXXB037k6uRDc8xUqygjoDnJ8XMdAD9TpRY4iqP=")) =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      xs.filter(p).size < 60
    }
}

(Aside: There's a good argument ScalaCheck should be printing failing seeds by default. When this feature was first added we chose to be conservative and preserve the existing behavior for property.)

syntactic sugar

Using the propertyWithSeed constructor works, but is not necessarily ergonomic. Also, when passing an explicit seed (Some(...)) the property will only run that one seed. In many cases users might want to test against that seed, but also test against other randomized seeds as normal.

Here's an extension to Properties that supports the usage we want with a relatively ergonomic syntax:

abstract class SeededProperties(name: String) extends Properties(name) { self =>

  // this behaves similarly to scalacheck's `property(...) = ...` DSL
  // but includes better support for seeds by default.
  object Property {

    // if no existing seeds are given we still want to display seeds
    // for failing properties.
    def update(pname: String, p: => Prop): Unit =
      self.propertyWithSeed(pname, None) = p

    // run the property as normal, but *also* run it for the given
    // seed as extra test cases.
    def update(pname: String, seed: String, p: => Prop): Unit = {
      self.propertyWithSeed(pname, None) = p
      val s = seed.substring(0, 4) + "..."
      self.propertyWithSeed(s"$pname[$s]", Some(seed)) = p
    }

    // run the property as normal, but *also* run it for the given
    // seeds as extra test cases.
    def update(pname: String, seeds: List[String], p: => Prop): Unit = {
      self.propertyWithSeed(pname, None) = p
      seeds.foreach { seed =>
        val s = seed.substring(0, 4) + "..."
        self.propertyWithSeed(s"$pname[$s]", Some(seed)) = p
      }
    }
  }
}

Using the Property method will ensure that every failing property prints out a seed. It also allows users to optionally pass one or more seeds to the property -- these seeds will be evaluated in addition to the standard random testing.

Here's an example of using Property without explicit seeds.

class BugProps6 extends SeededProperties("Bug6") {
  Property("filtered lists are short") =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      xs.filter(p).size < 60
    }
}

Running this test will often produce the following successful output:

[info] + Bug6.filtered lists are short: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1

It will also occasionally produce the following failure output (complete with failing seed):

failing seed for Bug6.filtered lists are short is I3BuwrInBlfIZADkQGOCGfR5tvoa5uivIBJKJ6-js4D=
[info] ! Bug6.filtered lists are short: Falsified after 97 passed tests.
[info] > ARG_0: List("0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0")
[info] > ARG_0_ORIGINAL: List("1", "-1801906625", "2147483647", "-2147483648", "1400389728", "-796837488", "-1722253487", "-1461587117", "2147483647", "1052502412", "-2147483648", "1", "2147483647", "1441161313", "1375759661", "25241546", "-1001848571", "940941472", "0", "-440091543", "0", "-2147483648", "-1039525937", "-426807587", "-541360694", "209241219", "2147483647", "-546679553", "0", "32478226", "258082633", "-1626249214", "-2147483648", "-1006840807", "1", "-1", "-2147483648", "2147483647", "-1", "-592708690", "-1047944561", "-1", "-1962346612", "-1", "-1871438852", "-1021750730", "0", "1", "-415171090", "-2147483648", "-2147483648", "1289255483", "2147483647", "-950287220", "0", "2147483647", "-1", "2147483647", "-2147483648", "-2147483648", "-1419096609", "2147483647", "-1285569950", "2147483647", "1", "443128714", "-1089892649", "-337356189", "-1538427422", "-1663816990", "1568216407", "2147483647", "-2147483648", "-1", "1330255428", "2057696823", "-2080449637", "0")
[info] > ARG_1: org.scalacheck.GenArities$$Lambda$22530/804884405@3bc0438a
[info] Failed: Total 1, Failed 1, Errors 0, Passed 0

Here's an example of passing multiple seeds to Property. This test will run all the given seeds, but will also run the property in the usual way.

class BugProps7 extends SeededProperties("Bug7") {
  Property(
    "filtered lists are short",
    List(
      "cSsfwP-m_gNLTzJcsDJiaHmpYCLYGL0N0PyMbDEwlfI=", // pass
      "rktKv44iRzr8-F8OAbwkZScIEv9t8nZyX4Y9aFmn7gN=", // pass
      "H84AeXXB037k6uRDc8xUqygjoDnJ8XMdAD9TpRY4iqP="  // fail
    )
  ) =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      xs.filter(p).size < 60
    }
}

The above Property actual runs four distinct properties:

  • The usual property, run with randomized cases
  • Three fixed-seed properties, whose names are suffixed with [wyxz...] to indicate which fixed seed they're using.

Here's what that output looks like:

[info] + Bug7.filtered lists are short: OK, passed 100 tests.
[info] + Bug7.filtered lists are short[cSsf...]: OK, passed 100 tests.
[info] + Bug7.filtered lists are short[rktK...]: OK, passed 100 tests.
[info] ! Bug7.filtered lists are short[H84A...]: Falsified after 81 passed tests.
[info] > ARG_0: List("0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0")
[info] > ARG_0_ORIGINAL: List("-1756030137", "-1", "1", "-1577978365", "1996323280", "1235987858", "-1", "-119289092", "1796794817", "-2147483648", "0", "-185835109", "2147483647", "-573437505", "-1", "-2147483648", "-2147483648", "1117642543", "-1002215582", "0", "1532444616", "-1171363902", "-786379840", "-2147483648", "-1", "2147483647", "451199318", "324096022", "-107161746", "-1", "-1445092754", "2147483647", "1", "-1688700755", "0", "-1719981158", "1", "2147483647", "1", "-1", "-2147483648", "0", "2147483647", "0", "2147483647", "-2147483648", "-1", "-2147483648", "1", "1632255924", "-1", "1", "-988803452", "0", "102049049", "-842347084", "-1", "996100904", "0", "2147483647", "2147483647", "-1363532431", "2147483647", "-1086043995", "-2147483648", "-1", "-1385447539", "2147483647", "0", "2147483647", "1", "-1")
[info] > ARG_1: org.scalacheck.GenArities$$Lambda$21499/38903525@622d6b18
[info] Failed: Total 4, Failed 1, Errors 0, Passed 3

fully-deterministic runs

The previous examples have all showed how to reproduce specific failing cases using a failing seed. There's another possible use for deterministic testing: to ensure a given test will pass.

In some cases users may choose to hardcode a particular seed that is known to pass, particularly if a test is flaky and is tripping up other users without revealing "real" issues. This can be dangerous, since the flaky test may be masking real failures, but in some cases it may be necessary.

ScalaCheck doesn't make it easy to do this, but it is possible. Here's another extension to Properties that exposes this functionality:

abstract class DeterministicProperties(name: String, seed0: String) extends Properties(name) { self =>

  val seed1: Seed = Seed.fromBase64(seed0)
    .getOrElse(sys.error(s"invalid seed: $seed0"))

  object Property {
    def update(pname: String, p: => Prop): Unit = {
      val seed2: Seed = seed1.reseed(pname.hashCode)
      self.property(pname) = p.contramap { params =>
        params.withInitialSeed(seed2)
      }
    }
  }
}

Using this property, we can write a test that will always pass:

class BugProps8 extends DeterministicProperties("Bug8", "xSsfwP-m_gNLTzJcsDJiaHmpYCLYGL0N0PyMbDEwlfI=") {
  Property("filtered lists are short") =
    forAll { (xs: List[Int], p: Int => Boolean) =>
      println(xs.hashCode)
      xs.filter(p).size < 60
    }
}

This example differs from the above cases that use explicit seeds. In this case, the seed is an "initial seed" -- we are not limiting ScalaCheck to only verifying this seed, but we're ensuring that every run will start with this seed, and proceed in a deterministic order. This means that if the test passes once, it will continue passing (at least until the ScalaCheck version or generators are changed).

(Implementation detail: since properties can be evaluated in parallel, this approach combines the single initial seed with the hashed name of the property, to produce per-property initial seeds.)

conclusion

I hope this document helps you get started using fixed seeds to reproduce ScalaCheck failures! There's still work to be done to create the best high-level API that takes advantage of seeds (and work sorting out ScalaCheck internals to make doing this work easier). However, I think even as it stands now seeds can be very useful to ScalaCheck users.

@SethTisue
Copy link

There's a good argument ScalaCheck should be printing failing seeds by default

opened typelevel/scalacheck#400 on it

@non
Copy link
Author

non commented Apr 24, 2018

Cool, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment