In order to display the output of several source code analysis / build tools it is necessary to have the ability to store and display their output.
QA Tools can output in two different ways:
- To STDOUT
- To a physical disk as file(s)
The latter output type can be split into two sub-types:
- Raw files, or logs
- Human-readable output
The first part of this RFC will focus on Raw files where the second section proposes a means to contain human-readable output within the current architecture of Travis.
Raw files are files containing machine readable output that can be transformed by Travis into human-readable output. It is assumed that it does not matter whether the output is via STDOUT or to a physical file, as long as there is just one entity to record.
During the processing stage of Travis it should be possible to execute an arbitrary tool, as defined in the project build file, and have its output written to an artifact record in the Travis database. The artifact record should also have a type of artifact filled in matching the output type of the given tool.
For example:
PHPUnit outputs JUnit XML, this output can be stored with a type JUnit.
PHP_CodeSniffer outputs Checkstyle formatted XML, this output can be stored with a type Checkstyle.
Based on the type of these raw files Travis can transform these into human-readable output using a transformation script, or using (for example) XSLT. This human-readable output can be stored in the database in a second artifact record, or it could not be a by-product of the build process but have the transformation applied ad-hoc when the user attempts to open the page.
The first approach (generating during a post-processing phase of building) has the benefit of faster viewing (some large projects cannot have their human-readable artifact generated ad-hoc due to time constraints) and self-contained builds. Whereas the downside is the increased disk space usage for the database and longer build times.
The second approach reduces the build time and database storage but runs the risk of having performance issues when you want to view the results.
Based on the arguments above I recommend the first approach.
In addition to the human-readable output mentioned in the previous chapter some tools will only allow human-readable artifacts to be created consisting of multiple files.
In the next section I will propose a solution as to storing these kind of artifacts in the current architecture of Travis. By no means is this a best practice, but it could be used as a simple starting point from where to expand in the future.
PHP has an executable archive format called Phar. By using a micro-framework inside a phar archive to route all URLs to the static files contained inside this Phar file is it possible to have compressed output of a QA tool in a single file. This in turn makes it possible to store it in an artifact record and have an action in Travis route traffic to this Phar file. The Phar will in turn retrieve all static content.
With probably relative minor adjustments it might be possible to add support for several types of artifacts produced by source code analysis tools. Focus here should be to store single entity output of specific QA tools in the artifact model of Travis and display them with human readable contents.
In this appendix I hope to document as much analysis tools with their output format per language. _Author's note: since my area of expertise is PHP, I hope that other will append to this list.
Please note that documentation generators are explicitly not included as this does not match the primary scope of this RFC.
- PHPUnit: xUnit
- PHP_CodeCoverage: Clover
- PHP_CodeSniffer: checkstyle
- phpmd: PMD
- pdepend: JDepend
- phploc: custom format
- phpcpd: custom format
- JUnit: xUnit
- PMD: PMD
- JDepend: JDepend
I prefer aiming for something way more simple and just upload files from a predefined location to either s3 directly or the app. then expose these through the api and put links on the build details page. we might want to display images at some point.