Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
GSoC final report -- Automated regression tests against real-world projects

Automated regression tests against real-world projects

GSoC 2018 final report

Organisation: PMD

Mentor: Andreas Dangel andreas.dangel@adangel.org, Juan Sotuyo juan.sotuyo@mercadolibre.com

Student: Binguo Bao djydewang@gmail.com

Project Output

Pmdtester

Pmdtester is a regression testing tool that ensures no new problems and unexpected behaviors will be introduced to PMD after fixing an issue. It can also be used to verify, that new rules work as expected. The tool supports the following modes:

  • single: This mode is mainly used to observe the behavior of new pmd rules or to generate the baseline for a pmd branch/tag.
  • online: The tool download the baseline of base branch rather than generating it locally in this mode. We run this mode to compare the PR baseline against the baseline for the last published snapshot when we integrate the tool into travis build job of pmd.
  • local: The baselines for base and patch branches/tags are generated locally, then compare their differences.

The tool also supports generating dynamic PMD java ruleset based on base and patch branchs differences, then compare the differences in the behavior of the two branches based on the ruleset.

Integrating the tool into the travis build job of pmd

Impact

Pmdtester already had an impact and prevented bugs/false positives:

Future work

  • We execute PMD just on the source code of standard projects now, but specifying the classpath for libraries used by the source code is also crucial for the type resolution.
  • We have exclude patter attribute in project list XSD file, but it has not been implemented yet in fact.
  • Support other languages. Current focus is Java.

PR summary

PRs merged into pmdtester - https://github.com/pmd/pmd-regression-tester

  • #1: Initialize project
  • #2: Add projects parser & design format of projectlist
  • #3: Add pmd report builder
  • #4: Test PmdReportBuilder
  • #5: Add DiffBuilder for PmdTester
  • #6: Change the package command for building PMD
  • #7: Add test cases for DiffBuilder
  • #8: Add HtmlReportBuilder to PmdTester
  • #9: Add test cases for HtmlReportBuilder
  • #10: Add bundler to manage dependency
  • #11: Using rubocop to check code style of the project
  • #12: Fix Metrics/BlockLength offenses
  • #13: Separate integration test cases
  • #14: Add Runner to PmdTester
  • #15: Fix rubocop Style/Documentation offenses
  • #16: Add single mode, add mocha library for unit test
  • #17: Add more details about pmd branchs and pmd reports
  • #18: Add SummaryReportBuilder to PmdTester
  • #19: Add online mode for PmdTester
  • #20: Change the way of parsing xml file from DOM to SAX
  • #21: Add auto-gen-config option for PmdTester
  • #22: Add 'introduce new errors' table head for html summary report
  • #23: Preparing for the release of PmdTester
  • #24: Adding a logging framework for PmdTester
  • #25: Remove working directory substring from filename of pmd violation
  • #26: Release pmdtester 1.0.0.beta1
  • #27: Release pmdtester 1.0.0.beta2
  • #28: Refactor require statements
  • #29: Add 'verify' rake task to verify code quality before committing changes
  • #30: Fix diff_cmd in RuleSetBuilder
  • #31: Fix color scheme for diff report, add default values for various options
  • #32: Update Readme.rdoc
  • #33: Ignore changes to test code of PMD when generating dynamic rule sets

PRs merged into pmd/pmd - https://github.com/pmd/pmd

  • #1250: [ci] [GSoC] Upload baseline of pmdtester automatically
  • #1254: [ci] [GSoC] Integrating the danger and pmdtester to travis CI
  • #1278: [ci] [GSoC] Use pmdtester 1.0.0.pre.beta3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment