Documenting Composer scripts

For open source projects I'm involved with, I developed the habit to define, and document the steady growing amount of repository and build utilities via Composer scripts. Having Composer scripts available makes it trivial to define aliases or shortcuts for complex and hard to remember CLI calls. It also lowers the barrier for contributors to start using these tools while helping out with fixing bugs or providing new features. Finally they're also simplifying build scripts by stashing away complexity.

Defining Composer scripts

If you've already defined or worked with Composer scripts or even their npm equivalents you can skip this section, otherwise the next code snippet allows you to study how to define these. The here defined Composer scripts range from simple CLI commands with set options (e.g. the test-with-coverage script) to more complex build utility tools (i.e. the application-version-guard script) which are extracted into specific CLI commands to avoid cluttering up the composer.json or even the .travis.yml.

composer.json

{
  "scripts": {
    "test": "phpunit",
    "test-with-coverage": "phpunit --coverage-html coverage-reports",
    "cs-fix": "php-cs-fixer fix . -vv || true",
    "cs-lint": "php-cs-fixer fix --diff --stop-on-violation --verbose --dry-run",
    "configure-commit-template": "git config --add commit.template .gitmessage",
    "application-version-guard": "php bin/application-version --verify-tag-match"
  }
}

Describing Composer scripts

Since Composer 1.6.0 it's possible to set custom script descriptions via the scripts-descriptions element like shown next. It's to point out here that the name of a description has to match the name of a defined custom Composer script to be recognised at runtime. On another note it's to mention that the description should be worded in simple present to align with the other Composer command descriptions.

composer.json

{
  "scripts-descriptions": {
    "test": "Runs all tests.",
    "test-with-coverage": "Runs all tests and measures code coverage.",
    "cs-fix": "Fixes coding standard violations.",
    "cs-lint": "Checks for coding standard violations.",
    "configure-commit-template": "Configures a local commit message template.",
    "application-version-guard": "Checks that the application version matches the given Git tag."
  },
  "scripts": {
    "test": "phpunit",
    "test-with-coverage": "phpunit --coverage-html coverage-reports",
    "cs-fix": "php-cs-fixer fix . -vv || true",
    "cs-lint": "php-cs-fixer fix --diff --stop-on-violation --verbose --dry-run",
    "configure-commit-template": "git config --add commit.template .gitmessage",
    "application-version-guard": "php bin/application-version --verify-tag-match"
  }
}

Now when running $ composer via the terminal the descriptions of defined custom scripts will show up sorted in into the list of available commands, which makes it very hard to spot the Composer scripts of the package at hand. Luckily Composer scripts can also be namespaced.

Namespacing Composer scripts

To namespace (i.e. some-namespace) the custom Composer scripts for any given package define the script names with a namespace prefix as shown next. As the chances are very high that you will be using the one or other Composer script several times, while working on the package, it's recommended to use a short namespace like in the range from two to four characters.

composer.json

{
  "scripts": {
    "some-namespace:test": "phpunit",
    "some-namespace:test-with-coverage": "phpunit --coverage-html coverage-reports",
    "some-namespace:cs-fix": "php-cs-fixer fix . -vv || true",
    "some-namespace:cs-lint": "php-cs-fixer fix --diff --stop-on-violation --verbose --dry-run",
    "some-namespace:configure-commit-template": "git config --add commit.template .gitmessage",
    "some-namespace:application-version-guard": "php bin/application-version --verify-tag-match"
  }
}

Now this time when running $ composer via the terminal the defined custom scripts will show up in the list of available commands in a namespaced manner giving an immediate overview of the available Composer script of the package at hand.

$ composer
 ... ommitted content
Available commands:
  ... ommitted content
 some-namespace
  some-namespace:application-version-guard  Checks that the application version matches the given Git tag.
  some-namespace:configure-commit-template  Configures a local commit message template.
  some-namespace:cs-fix                     Fixes coding standard violations.
  some-namespace:cs-lint                    Checks for coding standard violations.
  some-namespace:test                       Runs all tests.
  some-namespace:test-with-coverage         Runs all tests and measures code coverage.

To use any namespaced Composer script, e.g. to fix coding standard violations after a substantial refactoring, it has to be called with its namespace e.g.$ composer some-namespace:cs-fix, which is the one disadavantage of Composer script namespacing.

Eight knobs to adjust and improve your Travis CI builds

After having refactored several Travis CI configuration files over the last weeks, this post will provide eight adjustments or patterns immediately applicable for faster, changeable, and economic builds.

1. Reduce git clone depth

The first one is a simple configuration addition with a positive impact on the time and disk space consumption, which should be quite noticeable on larger code bases. Having this configured will enable shallow clones of the Git repository and reduce the clone depth from 50 to 2.

.travis.yml

git:
  depth: 2

2. Enable caching

The second one is also a simple configuration addition for caching the Composer dependencies of the system under build (SUB) or its result of static code analysis. Generally have a look if your used tools allow caching and if so cache away. This one deserves a shout out to @localheinz for teaching me about this one.

The next shown configuration excerpt assumes that you lint coding standard compliance with the PHP Coding Standards Fixer in version 2.0.0-alpha and have enable caching in its .php_cs configuration.

.travis.yml

cache:
  directories:
    - $HOME/.composer/cache
    - $HOME/.php-cs-fixer

3. Enforce contribution standards

This one might be a tad controversial, but after having had the joys of merging GitHub pull requests from a master branch I started to fail builds not coming from feature or topic branch with the next shown bash script. It's residing in an external bash script to avoid the risk of terminating the build process.

./bin/travis/fail-non-feature-topic-branch-pull-request

#!/bin/bash
set -e
if [[ $TRAVIS_PULL_REQUEST_BRANCH = master ]]; then
  echo "Please open pull request from a feature / topic branch.";
  exit 1;
fi

.travis.yml

script:
  - ./bin/travis/fail-non-feature-topic-branch-pull-request

The bash script could be extended to fail pull requests not following a branch naming scheme, e.g. feature- for feature additions or fix- for bug fixes, by evaluating the branch name. If this is a requirement for your builds you should also look into the blocklisting branches feature of Travis CI.

4. Configure PHP versions in an include

With configuring the PHP versions to build against in a matrix include it's much easier to inject enviroment variables and therewith configure the version specific build steps. You can even set multiple enviroment variables like done on the 7.0 version.

.travis.yml

env:
  global:
    - OPCODE_CACHE=apc

matrix:
  include:
    - php: hhvm
    - php: nightly
    - php: 7.1
    - php: 7.0
      env: DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: 
      - DISABLE_XDEBUG=true

before_script:
  - if [[ $DISABLE_XDEBUG = true ]]; then
      phpenv config-rm xdebug.ini;
    fi

I don't know if enviroment variable injection is also possible with the minimalistic way to define the PHP versions list, so you should take that adjustment with a grain of salt.

It also seems like I stumbled upon a Travis CI bug where the global enviroment variable OPCODE_CACHE is lost, so add another grain of salt. To work around that possible bug the relevant configuration has to look like this, which sadly adds some duplication and might be unsuitable when dealing with a large amount of environment variables.

.travis.yml

matrix:
  include:
    - php: hhvm
      env: 
      - OPCODE_CACHE=apc
    - php: nightly
      env: 
      - OPCODE_CACHE=apc
    - php: 7.1
      env: 
      - OPCODE_CACHE=apc
    - php: 7.0
      env: OPCODE_CACHE=apc DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: OPCODE_CACHE=apc DISABLE_XDEBUG=true

before_script:
  - if [[ $DISABLE_XDEBUG = true ]]; then
      phpenv config-rm xdebug.ini;
    fi

5. Only do static code analysis or code coverage measurement once

This one is for reducing the build duration and load by avoiding unnecessary build step repetition. It's achived by linting against coding standard violations or generating the code coverage for just a single PHP version per build, in most cases it will be the same for 5.6 or 7.0.

.travis.yml

matrix:
  include:
    - php: hhvm
    - php: nightly
    - php: 7.1
    - php: 7.0
      env: DISABLE_XDEBUG=true LINT=true
    - php: 5.6
      env: 
      - DISABLE_XDEBUG=true

script:
  - if [[ $LINT=true ]]; then
      composer cs-lint;
      composer test-test-with-coverage;
    fi

6. Only do release releated analysis and checks on tagged builds

This one is also for reducing the build duration and load by targeting the analysis and checks on tagged release builds. For example if you want to ensure that your CLI binary version, the one produced via the --version option, matches the Git repository version tag run this check only on tagged builds.

.travis.yml

script:
  - if [[ ! -z "$TRAVIS_TAG" ]]; then
      composer application-version-guard;
    fi

7. Run integration tests on very xth build

For catching breaking changes in interfaces or API's beyond your control it makes sense do run integration tests against them once in a while, but not on every single build, like shown in the next Travis CI configuration excerpt which runs the integration tests on every 50th build.

.travis.yml

script:
  - if [[ $(( $TRAVIS_BUILD_NUMBER % 50 )) = 0 ]]; then
      composer test-all;
    else
      composer test;
    fi

8. Utilise Composer scripts

The last one is all about improving the readability of the Travis CI configuration by extracting command configurations i.e. options into dedicated Composer scripts. This way the commands are also available during your development activitives and not hidden away in the .travis.yml file.

composer.json

{
    "__comment": "omitted other configuration",
    "scripts": {
        "test": "phpunit",
        "test-with-coverage": "phpunit --coverage-html coverage-reports",
        "cs-fix": "php-cs-fixer fix . -vv || true",
        "cs-lint": "php-cs-fixer fix --diff --verbose --dry-run"
    }
}

To ensure you don't end up with an invalid Travis CI configuration, which might be accidently committed, you can use composer-travis-lint a simple Composer script linting the .travis.yml with the help of the Travis CI API.

Happy refactoring.

Hooking a Growl publisher plugin into Xinc

This week I finally had the time to setup Xinc, PHP's shiny new Continuous Integration(CI) server, on my MacBook and started to fiddle with it's publisher plugins and plugin architecture. While still down with 'tool envy' influenza I recently came across some nice and inspiring blog posts from the Ruby/Rails camp, showing how to employ Growl as an useful and fun feedback radiator for test/spec results. Since then the idea of building a Growl publisher plugin for Xinc was travelling my mind repeatedly, so the following post will break this circle and show a possible approach to build such a plugin, which can be used to notify the build result for continuously integrated projects and thereby provide an on-point/immediate feedback.

Building the Growl publisher plugin

Xinc comes with an elegant plugin system, allowing you to easily role your own one. The following code shows the plugin class and is the place where the actual Growl notifications are executed/raised by calling the growlnotify CLI. The images in the public growl() method, used to accent the two main build statuses(failure and success) in the Growl notification, are borrowed from this blog post and should be located in the $HOME/Pictures directory.

<?php
/**
 * A Growl publisher plugin for Xinc
 *
 * @package Xinc.Plugin
 * @author Raphael Stolt
 * @version 2.0
 * @copyright 2008 Raphael Stolt, Constance
 * @license  http://www.gnu.org/copyleft/lgpl.html GNU/LGPL, see license.php
 */
require_once 'Xinc/Plugin/Base.php';
require_once 'Xinc/Plugin/Repos/Publisher/Growl/Task.php';

class Xinc_Plugin_Repos_Publisher_Growl extends Xinc_Plugin_Base
{
    public function validate()
    {
        return true;
    }
    public function getTaskDefinitions()
    {
        return array(new Xinc_Plugin_Repos_Publisher_Growl_Task($this));
    }
    private function _sendGrowlNotification($message, $image, $name)
    {
        $command = "growlnotify -w -m '{$message}' "
                 . "-n '{$name}' "
                 . "-p 2 --image {$image}";

        Xinc_Logger::getInstance()->info('Executed growlnotify command: ' . $command);

        exec($command, $response, $return);

        if ($return === 0) {
            return true;
        }
        return false;
    }
    public function growl(Xinc_Project &$project, $message, $buildstatus, $name = 'Xinc')
    {
        if ($buildstatus === Xinc_Build_Interface::PASSED) {
            $image = '$HOME/Pictures/pass.png';
            $buildstatus = 'PASSED';
        } elseif ($buildstatus === Xinc_Build_Interface::FAILED) {
            $image = '$HOME/Pictures/fail.png';
            $buildstatus = 'FAILED';
        }

        $project->info('Executing Growl publisher with content '
                      ."\nMessage: " . $message
                      ."\nBuildstatus: " . $buildstatus
                      ."\nImage: " . $image);

        return $this->_sendGrowlNotification($message, $image, $name);
    }
}

The next big code chunk shows the task definition of the plugin and is the place where the publisher plugin name and is acceptable and required attributes are programmaticly defined, as you can see the Growl task currently only takes a required message attribute which can be set for the Growl publisher within the Xinc project configuration file.

<?php
/**
 * A Growl publisher plugin task for Xinc
 *
 * @package Xinc.Plugin
 * @author Raphael Stolt
 * @version 2.0
 * @copyright 2008 Raphael Stolt, Constance
 * @license  http://www.gnu.org/copyleft/lgpl.html GNU/LGPL, see license.php
 */
require_once 'Xinc/Plugin/Repos/Publisher/AbstractTask.php';

class Xinc_Plugin_Repos_Publisher_Growl_Task extends Xinc_Plugin_Repos_Publisher_AbstractTask
{
    private $_message;

    public function setMessage($message)
    {
        $this->_message = $message;
    }
    public function getName()
    {
        return 'growl';
    }
    private function _isGrowlnotifyAvailable()
    {
        exec('growlnotify -v', $reponse, $return);

        if ($return === 0) {
            return true;
        }
        return false;
    }
    public function validateTask()
    {
        if (!$this->_isGrowlnotifyAvailable()) {
            $message = 'The growlnotify command seems to be not available '
                     . 'on this system';
            Xinc_Logger::getInstance()->error($message);
            throw new RuntimeException($message);
        }

        if (!isset($this->_message)) {
              $message = 'Element publisher/growl - required attribute '
                       . '\'message\' is not set';
              throw new Xinc_Exception_MalformedConfig($message);
        }
        return true;
    }
    public function publish(Xinc_Build_Interface &$build)
    {
        $statusBefore = $build->getStatus();
        $res = $this->_plugin->growl($build->getProject(), $this->_message, $build->getStatus());
        if (!$res && $statusBefore == Xinc_Build_Interface::PASSED ) {
            /**
             * Status was PASSED, but now the publish process made it fail
             */
            $build->setStatus(Xinc_Build_Interface::FAILED);
        }
    }
}

Hooking the Growl publisher plugin into the build system

To make the just crafted Growl publisher plugin available to the Xinc build system you have to add the plugin file and class name/path to the /etc/xinc/system.xml file. After restarting the Xinc server the Growl publisher should be available to use and also be listed in the section of the /var/log/xinc.log file stating all registered plugins.

The next listing shows the Growl publisher plugin added to the aforementioned XML file.

<?xml version="1.0" encoding="UTF-8"?>
<xinc>
  <configuration>
    <setting name="loglevel" value="2"/>
    <setting name="timezone" value="Europe/Berlin"/>
  </configuration>
  <plugins>
    <plugin filename="Xinc/Plugin/Repos/ModificationSet.php" classname="Xinc_Plugin_Repos_ModificationSet"/>
    ...
    <plugin filename="Xinc/Plugin/Repos/Publisher/Growl.php" classname="Xinc_Plugin_Repos_Publisher_Growl"/>
    ...
    <plugin filename="Xinc/Contrib/Warko/Plugin/ModificationSet/SvnTag.php"
            classname="Xinc_Contrib_Warko_Plugin_ModificationSet_SvnTag"/>
  </plugins>

  <engines>
    <engine classname="Xinc_Engine_Sunrise" filename="Xinc/Engine/Sunrise.php" default="default"/>
  </engines>
</xinc>

Putting the Xinc Growl publisher plugin to action

Now that the Growl publisher plugin is available it can be used in the build cycle. The following Xinc project configuration file shows the utilization of the Growl publisher plugin in both of the main publisher realms onsuccess and onfailure.

<?xml version="1.0" encoding="UTF-8"?>
<xinc engine="Sunrise">
  <project name="recordshelf">
    <configuration>
      <setting name="loglevel" value="2"/>
    </configuration>
    <property name="dir" value="${projectdir}/${project.name}" />
    <property name="build.failure.message" value="Build for project ${project.name} failed!"/>
    <schedule interval="180" />
    <modificationset>
      <svn directory="${dir}" update="true" />
    </modificationset>
    <builders>
      <phingbuilder buildfile="${dir}/build.xml" target="main"/>
    </builders>
    <publishers>
      <onfailure>
        <email to="[email protected]" subject="[${project.name}] Build failure"
               message="${build.failure.message}" />
        <growl message="${build.failure.message}" />
      </onfailure>
      <onsuccess>
        <email to="[email protected]" subject="[${project.name}] Build success"
               message="Build for project was successful" />
        <growl message="Build for project ${project.name} was successful." />
      </onsuccess>
    </publishers>
  </project>
</xinc>

The next two images are finally showing the Growl notifications for a successful and a failed build. To minimize the notification noise it's a good and common practice to only 'ring the alarm' for failed builds, which can be achieved by using the Growl publisher only in the onfailure publisher realm of the Xinc project configuration file. Nuff talk, happy Xincing!

PHP_CodeSniffer task is in the current Phing branch

A few days ago I was evaluating the need for writing a Phing task for the awesome PHP_CodeSniffer Pear package and noticed that Dirk Thomas blessedly already added one to the current Phing Svn branch. That's again extending the range of available tools that PHP's Continuous Integration(CI) toolchain has to offer and I quess there are more to follow, as Manuel Pichler just announced the development start of PHP_Depend.

To get the 2.3 Phing branch including the mentioned PHP_CodeSniffer task and it's documentation simply run a svn checkout http://svn.phing.info/branches/2.3 /path/to/phing-checkout-dir and you're ready to add continuous monitoring of coding standard adherence to your builds.

For simplifying the integration of the just checked out Phing 'branch' release into an existing Pear environment you can build a new Pear package by calling the buildfile in /path/to/phing-checkout-dir/pear. The built Pear package will be located in /path/to/phing-checkout-dir/pear/build and can than be used to switch the installed Phing Pear release.

Also make sure you have installed the underlying PHP_CodeSniffer Pear package via the Pear Installer.

Although the task is very well documented the next code snippet shows the shiny PHP_CodeSniffer task in an example build file.

<?xml version="1.0"?>
<project name="example" default="build" basedir=".">

   <target name="build" depends="clean, svn-checkout, code-inspection, test">

   ...

   </target>

   ...

   <target name="code-inspection" description="runs code inspection on the stated directory">
     <phpcodesniffer standard="PEAR"
                     format="summary"
                     tabWidth="4"
                     file="/path/to/source-files"
                     allowedFileExtensions="php"/>
     </phpcodesniffer>
   </target>

</project>