https://summerofcode.withgoogle.com/projects/#6100626637848576
Tags Devops, Developer Happiness,Static Analysis
-
Analyzing Civi-core with Psalm and CI/CD tool(Jenkins) on merge
-
Analyzing Civi-core with Psalm and CI/CD tool(Jenkins) on Pull Request
-
Utilizing Psalm Baseline Feature for Civi-core Legacy code error handling
This project aims at integrating Vimeo’s Psalm into the CiviCRM Jenkins toolchain for static analysis purposes. This project will also work on improving Code coverage report generation for CiviCRM. Both static analysis and improved unit code test coverage reports will make for better quality assurance benchmarks for CiviCRM.
The static analysis tool selected for integration with Civi-core is Psalm. For integrating an open source static analysis tool I considered Phpstan, Phan and Psalm. I choose Vimeo’s Psalm because of its ease of installation, configuration , descriptive error messages and also its extensive documentation.
During the Community bonding phase the CiviCRM team provided a remote server for use by me and my mentors for this project implementation. Though a remote server was provided, my mentors made me understand that Static analysis would have an easier workflow if I tried Psalm out Locally on my Computer. To achieve local Static analysis I did the following;
- I created my own fork of the Civi-core repository
- I cloned the fork to my Local System
- Tried Running Psalm on the local repository
- Opened Issues in the Psalm Repository to fix errors encountered while running psalm locally
From the beginning it was clear there was no way I would implement my static analysis project directly on the CiviCRM Jenkins Tool chain, as the infrastructure was used on a daily basis for active development, hence I was provided with a server having similar configurations.
The developers maintaining Psalm were very quick to fix errors encountered while running Psalm on Civi-core,after the errors were fixed and Psalm was confirmed to work locally I tried Static Analysis on the remote server by doing the following;
- I installed PHP on the server
- I installed Composer on the server
- I Installed git on the server
- I cloned the Civi-core repository to the remote server
- I installed Psalm via Composer as one of Civi-core Composer Dependencies
With all the requirements in place, I was able to run Psalm static analysis on Civi-core without errors. One important observation from this was that Psalm returned varying errors(locally and remotely). Mentors from the CiviCRM community mentioned that this was normal and that was the reason CiviCRM had test suites for different environments and different PHP versions.
The CiviCRM CI/CD Tool Chain ran on Jenkins, so I was required to try implementing Civi-core Psalm Static Analysis with Jenkins. To achieve that I did the following
- Setup Jenkins on remote server provided by the CiviCRM team with help from mentors
- Configure Jenkins to work with Github
- Configure Github webhooks to work with Jenkins server url endpoint
- Attempt code merge to test integration
Having figured that Psalm worked with github merge, the next step was working to run Psalm whenever a pull request is made to the Civi-core mirror repository.
- Install Jenkins Git plugin
- Install Jenkins Github plugin
- Install Jenkins GitHub Pull Request Builder
- Install Jenkins Rebuilder plugin
- Configure Github webhooks to work with Jenkins server url endpoint
- Attempt pull request to test integration
Here is the output from the above pull request, notice that even though Psalm static analysis completes it returns with many errors. This is expected as there are lots of legacy code within the codebase.
Even though the error output from Psalm Static analysis was appreciated, there was a need to handle legacy code errors with an approach that keeps them in check, while allowing for focus on errors resulting from new Pull requests. This was necessary because it was not feasible for the CiviCRM development team to fix all the errors already existing within the codebase(Thousands of errors) before utilizing the benefits of Psalm static analysis for new pull requests. Most of these errors did not cause a breakdown in functionality and the development team had both feature requests and bugs breaking CiviCRM functionality to handle.
Psalm has a baseline feature which allows it to grandfather existing errors while picking up new errors from new pull requests. This was exactly what was needed. With the Baseline feature implemented, Psalm was able to analyze Civi-core without errors. It is worthy of note that CiviCRM Psalm legacy errors were marked for incremental fixing.
Here is a passing Pull request and its output with the baseline feature Implemented.
With Civi-core Psalm static analysis integrated on pull requests, it was time to take a closer look at the error output, my mentors were able to detect that most of the errors reported by Psalm were false negatives. This was the case because The Classes/Methods not found errors from Psalm pointed to Classes and functions that were found within the Civi-core codebase.
The CiviCRM team faced a similar issue when integrating PHPUnit as part of their CI/CD Jenkins tool chain so the created a Bootstrap file to pull in dependencies(classes and functions) required by Civicore. Psalm has an Autoloader directive for cases like this and the directive was utilized in solving this issue.
Adding the Bootstrap file through the autoloader directive eliminated the false Classes/Functions not found errors. See more information on the implementation in the documentation Section.
An Important part of this project was determining if errors detected by Psalm static analysis were fixable.To ascertain this , we analysed a section of the Civi-core codebase and I submitted a pull request to fix the errors detected. Merging the pull request to the codebase eliminated the errors on second static analysis.
- Continue working with the CiviCRM team to Integrate Psalm as part of CI/CD Tool chain Infrastructure
- Fix more Legacy Codebase errors detected by Psalm
- Install PHP on server, check out this link
- Install composer on server, check out this link
- Install Jenkins on Server, check out this link
- Here is a concise guide on setting up Jenkins to Listen for pull requests on a Github Repo
The steps from the above help you set up a Jenkins Job, And also Configure the Job, Now you can Add A build Step so Jenkins can install required Dependencies for Psalm, as well as run psalm static analysis on Civi-core
Add the following to build Step
composer updatecomposer require --dev vimeo/psalm:3.4.9./vendor/bin/psalm --show-info=false
Configuring Psalm to Work with Civi-core
- Add a psalm.xml file to Civi-core root folder
- This can be achieved by running
./vendor/bin/psalm --init
- This can be achieved by running
- Add psalmbase.xml file to Civi-core root folder, this is a baseline file
- This can be achieved by running
./vendor/bin/psalm --set-baseline=psalmbaseline.xml
- This can be achieved by running
Psalm baseline feature allows us to grandfather errors already existing in Civi-core, while checking for new errors arising from subsequent PR's after its installation.
-
Github repo Mirroring Civi-core, notice the psalm.xml file and the psalmbaseline.xml file
-
For consistent results, It is advisable to create psalm configuration files i.e psalm.xml & psalmbaseline.xml on the actual server where Jenkins is installed, as Psalm might have different error count when run on the same repository, with same configurations but on different servers. This is the case because different systems have different configurations. This is the reason why Civi-core exists for different versions of PHP for example.
-
running
./vendor/bin/psalm --initcreates a psalm.xml file which lists the different folders in the current repository, for Civi-core folders ext,packages are always included but non existent, do well to remove the folders from psalm.xml before running./vendor/bin/psalmon the folder, to avoid errors
False Negatives refer to error pointers that are actually incorrect, this was the case with Psalm reporting that many dependencies were missing when in fact they were present example e.g. Undefined Function/Class errors. To make up for this the Autoloader(Bootstrap) file for Civi-core Unit tests was used for Psalm by adding an autoloader in the psalm.xml file
<psalm autoloader="tests/phpunit/CiviTest/bootstrap.php" >
and the observations are as follows
-
For CiviTest/bootstrap.php to work as our Psalm autoloader, it requires a working local installation of CiviCRM and associated civicrm.settings.php file. The bootstrap.php file also makes calls to the CiviCRM CLI utility
cvwhich needs to be installed using the CiviCRM Buildkit. This can be achieved locally using a Vagrant installation of CiviCRM. -
Running analysis with Psalm and an Autoloader usually returns different errors and error count as compared to running without an autoloader
-
Running analysis with Psalm and an Autoloader takes lots of hours when run on the entire Civi-core repository, without the autoloader this analysis works in minutes
-
--threads=10
This increases analysis speed as it utilizes 10 threads instead of one
-
--show-info=false
Ensures that only error messages are returned by Psalm and not info warnings, this keeps the output clean
-
--debug-by-line
Very useful to trace line causing Psalm to fail, this makes fixing easy
-
--config=psalm.xml
Set path to Psalm.xml in case you intend to put the config file outside the root directory