Posted by m.jackson on 3 November 2016 - 11:12am

Warning: please be aware that Travis CI, which is mentioned in this article, has a security issue with its Free Tier service. By design, “secret” data such as access credentials are exposed within historical clear-text logs which are accessible by anyone via the Travis CI API. Please see this article for more information.

By Mike Jackson, Software Sustainability Institute

Continuous integration frameworks build and test our software, so we don't have too (well, we do, but they do it too!) As part of my work on automated testing to boost recipy's confidence, I've had my first experience of AppVeyor, a continuous integration service for Windows, and it was good!

Growing up in Scotland, I endured an oft-repeated peak and trough of emotion when watching BBC television. A new drama or comedy would be trailed by an enthusiastic announcer, anticipation would rise at the forthcoming delights, only to be dashed upon the rocks of the announcer's concluding words "...except for viewers in Scotland". I've noticed a similar trait in the world of research software where phrases such as "...except for Windows" or "...except for Internet Explorer" occur frequently enough to be noticeable, along with their fellow "works on Linux/UNIX", leaving "Windows" unsaid. So, having used Travis CI, a deservedly-popular Linux-based open source continuous integration framework for projects hosted on GitHub, it was refreshing to see that there's a similar service for Windows, AppVeyor.

I'd first used Travis CI when developing an interoperability test harness for Provenance Tool Suite. I was impressed by Travis CI, both by how easy it was to get started, and also how much they allow to be done within the scope of a test job (e.g. installation of Linux packages, cloning of Git repositories etc), and all for free! Buoyed by this good experience, Travis CI was my go-to service as part of my collaboration with Robin Wilson on Automated testing to boost recipy's confidence, developing a test suite for Robin's recipy Python provenance tracking package.

To ensure recipy has as wide a usage as possible, Robin wanted recipy to be tested under Windows as well as Linux and wondered if AppVeyor was a possibility. I'd never heard of AppVeyor and I confess that my own Windows biases came to the surface at this point as I foresaw a battle to get Python deployed onto AppVeyor and guddling with problems due to backslashes in paths. Fortunately, my fears didn't come true and AppVeyor proved as straightforward to use as Travis CI: as for Travis CI, Python 2 and 3 flavours are supported (along with Java, Ruby, C#, F#, C++ and Perl, amongst others). Indeed, both are very similar in how they are used: you link each to your GitHub account (AppVeyor also supports BitBucket), decide which Git repositories you want to build-and-test, and write a configuration file. Every commit you push to your Git repository triggers a new build-and-test run.

Compare my Travis CI page for recipy with my AppVeyor page. These show the results of testing recipy's configuration files, command-line parameters and its support for logging popular Python packages used in research. AppVeyor, like Travis CI, also provides status badges which embedded in files (for example a repository's README) on GitHub.

Like Travis CI, AppVeyor uses YAML for its configuration files, which, for recipy, led to two very similar configuration files: .travis.yml, and appveyor.yml. The key difference is how the environment is initially configured. As an aside, both Travis CI and AppVeyor have the same issue when it comes to using numpy and scipy in that these packages can take far too long to build from scratch. But, this can be solved the same way for each too, by using Miniconda (see, for example Five steps to add the 'bling' factor your Python package and How to continuously test your Python code on Windows using AppVeyor by Tjelvar Olsson).

So, if you're tempted by, or are already using, Travis CI, then give AppVeyor a go, and replace "...except Windows" with "...and Windows"!

For more on continuous integration, please see our guides on Using continuous integration to build and test your software and Hosted continuous integration.