GSoC 2018 final report
Organisation: PMD
Mentor: Andreas Dangel andreas.dangel@adangel.org, Juan Sotuyo juan.sotuyo@mercadolibre.com
Student: Binguo Bao djydewang@gmail.com
Pmdtester is a regression testing tool that ensures no new problems and unexpected behaviors will be introduced to PMD after fixing an issue. It can also be used to verify, that new rules work as expected. The tool supports the following modes:
- single: This mode is mainly used to observe the behavior of new pmd rules or to generate the baseline for a pmd branch/tag.
- online: The tool download the baseline of base branch rather than generating it locally in this mode. We run this mode to compare the PR baseline against the baseline for the last published snapshot when we integrate the tool into travis build job of pmd.
- local: The baselines for base and patch branches/tags are generated locally, then compare their differences.
The tool also supports generating dynamic PMD java ruleset based on base and patch branchs differences, then compare the differences in the behavior of the two branches based on the ruleset.
- Once the PR is merged, a new baseline for last published snapshot is upload to https://sourceforge.net/projects/pmd/files/pmd-regression-tester/.
- The regression analysis is automated via the Dangerfile, that is automatically executed for each pull request.
- Analysis diffs are commented back to the Pull Request for the reviewer's information e.g. pmd/pmd#1265 (comment)
Pmdtester already had an impact and prevented bugs/false positives:
- detected false positives: pmd/pmd#1252 (comment)
- detected the PR may intruduce PMD errors: pmd/pmd#1265 (comment)
- We execute PMD just on the source code of standard projects now, but specifying the classpath for libraries used by the source code is also crucial for the type resolution.
- We have exclude patter attribute in project list XSD file, but it has not been implemented yet in fact.
- Support other languages. Current focus is Java.
PRs merged into pmdtester - https://github.com/pmd/pmd-regression-tester
- #1: Initialize project
- #2: Add projects parser & design format of projectlist
- #3: Add pmd report builder
- #4: Test PmdReportBuilder
- #5: Add DiffBuilder for PmdTester
- #6: Change the package command for building PMD
- #7: Add test cases for DiffBuilder
- #8: Add HtmlReportBuilder to PmdTester
- #9: Add test cases for HtmlReportBuilder
- #10: Add bundler to manage dependency
- #11: Using rubocop to check code style of the project
- #12: Fix Metrics/BlockLength offenses
- #13: Separate integration test cases
- #14: Add Runner to PmdTester
- #15: Fix rubocop Style/Documentation offenses
- #16: Add single mode, add mocha library for unit test
- #17: Add more details about pmd branchs and pmd reports
- #18: Add SummaryReportBuilder to PmdTester
- #19: Add online mode for PmdTester
- #20: Change the way of parsing xml file from DOM to SAX
- #21: Add auto-gen-config option for PmdTester
- #22: Add 'introduce new errors' table head for html summary report
- #23: Preparing for the release of PmdTester
- #24: Adding a logging framework for PmdTester
- #25: Remove working directory substring from filename of pmd violation
- #26: Release pmdtester 1.0.0.beta1
- #27: Release pmdtester 1.0.0.beta2
- #28: Refactor require statements
- #29: Add 'verify' rake task to verify code quality before committing changes
- #30: Fix diff_cmd in RuleSetBuilder
- #31: Fix color scheme for diff report, add default values for various options
- #32: Update Readme.rdoc
- #33: Ignore changes to test code of PMD when generating dynamic rule sets