soroban-fuzzing-rfc-supp.md

Changes that needed to be made to support fuzzing

• arbitrary features
• Making soroban contracts rlibs
• Additions of IntoVal impls

My concerns

• I have tried to come up with the simplest set of traits that will make it work, but it's still not a simple story to document:

  You can always name a contract type's Arbitrary type with <ContractType as SorobanArbitrary>::Arbitrary, and you can always get the final contract type with let val: ContractType = arbitrary_val.into_val(Env). For some types though you don't need to bother with that though and can just get an arbitrary e.g. Symbol. Here's a convenient table for you to reference...

• Reusing IntoVal. Of the existing traits it seems right to use, but this trait has few constraints, and adding impls potentially breaks previously-working type inference.

Additional fuzz testing ergonomic obstacles

• 1 binary per test
• fuzz crates live outside the workspace
• fuzzing and CI don't work well together
  • can't say e.g. "fuzz every test for 10s"
• threading arbitrary features everywhere

Intended next steps

I would like to try to fuzz some real world code next, probably starting with the timelock example in soroban-examples. I would also like to fuzz the native token contract, though I don't understand how to access it yet.

General Soroban ergonomic problems

• IntoVal is underconstrained
• importing contracts from wasm breaks the cargo build system

todo

• types that don't implement arbitrary yet: signature, contracts
• type confusion between XDR and RawVal encoding
• type confusion between RawVal and Rust types
• should certain types produce "invalid" arbitrary values
• fuzzing the sdk and runtime itself
• proptest
• where should fuzzing examples ultimately live
• where should fuzzing be ultimately documented
• fuzzing the native token program
• arbitrary implementations of contracts
• having arbitrary produce the same object multiple times
• ergonomically converting failed fuzz tests into unit tests