Skip to content

Instantly share code, notes, and snippets.

@cameel

cameel/solc-seqbench-intro-and-initial-results.md

Last active February 21, 2024 14:16
Show Gist options
  • Save cameel/d3c80993e7dd93ed1c1dd0f37f7ab990 to your computer and use it in GitHub Desktop.
Save cameel/d3c80993e7dd93ed1c1dd0f37f7ab990 to your computer and use it in GitHub Desktop.
Download ZIP
solc-seqbench intro and initial results
Raw
solc-seqbench-intro-and-initial-results.md

solc-seqbench

https://github.com/cameel/solc-seqbench

Analysis pipeline

Workflow

  • Define inputs
    • optimizer sequences
    • contracts
    • sequences of external calls with parameters
  • Download sources
  • solc: Compile each contract into a single .yul file
  • solc: Compile each contract for each sequence, optimizing with every prefix
  • foundry: Deploy contract and execute test calls for each contract for each sequence for each prefix
  • pandas: Combine partial results into tables and plots

The final output is a markdown document for each sequence/contract/calls triple

Patched solc

Sequences with repetitions

  • Number of repetitions depends on a metric that is very inconvenient to calculate from outside of the compiler.
    • Initially I used a different condition in the pipeline: diffing yul output.
  • Each subassembly can get a different number of repetitions for the same sequence segment.
  • Compilation of an intermediate sequence sometimes fails with StackTooDeep.
    • When this happens at the end of a repeated segment, I can't check the stop condition.
    • In other cases: results in missing data.

Timing

  • Timing the whole solc process on each prefix is unreliable for small contracts. Non-monotonic results.

Patched solc

  • Can dump timing info directly after each step
  • Can get the list of actually executed steps
    • Subassemblies are still a problem. For now I only take the sequence used for top-level object.
  • Dumping intermediate .yul did not work out because it's just one subassembly, not a complete contract.
  • Based on #14854, which fixes unnecessary repetitions of the sequence.
  • StackTooDeep never happens inside the optimizer so I get data for all steps.

Input

Sequences

  • default: default sequence from v0.8.24.
  • single-pass: default sequence but with only one pass of the main loop.
  • always-ssa-min: default sequence with the Vcul [j] (SSA reversing) part removed.

Contracts

From semanticTests/externalContracts:

  • deposit_contract
  • FixedFeeRegistrar
  • prbmath_unsigned
  • ramanujan_pi (also used PRBMath, might be redundant)
  • strings
  • base64 (dropped)
    • cast send fails when executing test calls: execution reverted, data: Some(String("0x"))
  • `snark (dropped)
    • cast send fails when executing test calls: EVM error InvalidFEOpcode
    • Probably due to the use of precompiles not supported by anvil

General remarks:

  • No special selection for now. I used whatever worked.
  • Next I need to get more input and aim for some variety.
  • I want to try some contracts from OpenZeppelin next.

Calls

  • Based on expectations I found in the external tests. Not sure how representative that is.
  • Using contracts and calls that external projects use for benchmarking might be a good idea.

Results

deposit_contract

ramanujan_pi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment