By Steve Crouch, Research Software Group Lead

Why and how to test your code?

Software development doesn't end when the software is written. How can you, and any developers you work with, be sure that your software meets its requirements? Does your software work as expected and will it continue to work over its lifetime?

The important requirement here is showing your software functioning in a demonstrable way so that your code can be seen to meet its test criteria. This is where software testing becomes invaluable. It is very likely that you do some form of software testing already: build the code, run the code in a certain way, and ensure the outputs are as expected. By formalising this process into a set of tests that can be run independently and automatically, you provide a much greater degree of confidence that the software behaves correctly and increase the likelihood that defects are found.

There are some straightforward principles, methods and tools that can greatly assist in the testing process.

Why write this guide?

We wrote this guide to give an overview of a subject that we think is important to software sustainability.

Make the code easy to compile: an automated build process

Once a developer has changed your code, they'll need to compile it and rebuild your software. To help them, you should, at the very least, provide documentation about what is needed to build the software and detail any compilers and third-party software that is required.

Providing an automated build mechanism allows developers to quickly and easily validate their changes. By making this a simple and rapid process, developers are more likely to want to develop your software and will be able to efficiently contribute higher quality code.

A good approach is to use existing automated build tools such as Make, Ant, NAnt (Ant for .NET) and Maven. A comprehensive list of these tools can be found on Wikipedia. Typically, build tools are easily installable. They can work with a variety of programming languages, but there are commonly accepted conventions, such as Make for C/C++ and Ant for Java. Build tools allow you to capture the technical details required to build your software in a high-level build script. This can take into account, and indeed can be run on, different operating systems. Build scripts enable a single point of entry to building the software, regardless of platform, and can enable automated tests to be executed from the script.

Providing different levels of granularity within the build process is also a good idea. By enabling separate components to be built independently, as opposed to building everything, developers working on a single component, or a subset, can more quickly verify the changes on only those areas of the code they have changed. This can be especially beneficial for very large software packages. For example, if you have a client, service and automated documentation or JavaDoc generation, you will frequently find that a developer is unconcerned with the latter every time they build the software.

Some build tools, such as Maven, support dependency management which automatically downloads third-party dependencies prior to building the software. This can make the software more maintainable by removing the required libraries statically within the software project, and allowing easier inclusion of more up-to-update versions of those dependencies.

Make it easy to validate changes: provide automated (or unit) tests

After changing your code and rebuilding it, a developer will want to check that their changes or fixes have not broken anything. Providing developers with a fail-fast environment allows the rapid identification of failures introduced by changes to the code.

Automated (or unit) tests are modules or classes that invoke operations on your code. A test might verify an individual function or method, a class or module, related modules or components or the software as a whole. Tests can ensure that the correct results are returned from a function, that an operation changes the state of a system as expected, and that the code behaves as expected when things go wrong (for example, if incorrect inputs are given by a user or a database connection is cut off prematurely).

Unit tests can be viewed as executable requirements: they are a representation in code of the requirements for the software. They are represented in code, so they lack the ambiguity that might be present if described in a document. Executable requirements allow a developer to check that their changes have not broken anything and, if anything has stopped working, to identify why it is broken and how to fix it. Executable requirements can also help during initial development, by assuring you that your code works as planned. Test like these can be integrated into the build process, providing an efficient means of building and (optionally) running the tests.

Tests can be used as documentation - providing the code is readable - because they contain examples of how your code is used and how it works. Unlike other documentation, e.g. tutorials, they also have the advantage of always being consistent with your code, If your tests aren't consistent with your code, they won't compile! Tests, then, can assist developers in understanding your code.

There are many tools available for the development of automated unit tests in a range of languages, e.g. JUnit for Java, CPPUnit for C++ or fUnit for Fortran. A good list of the automated tests for each language is available on Wikipedia.

It is important to provide documentation that describes how to run the tests. This should cover subjects such as whether a database or a web server is needed and, if so, how should it be configured? Ideally, you would provide scripts to set up and configure any resources that the developer needed. This way, you could provide a test code or script that, for example, configures and populates a database needed for the test. Automation further minimises the work a developer must do before being able to run your tests and allows them to check their changes quickly and easily.

One way of ensuring tests are not neglected in a project, is to adopt test-driven development. This is an approach in which the tests for a function, class, module or component are written before the code. Once the tests are written, the code is developed so that it passes all the associated tests. Testing the code from the outset ensures that your code is always in a releasable state (as long as it passes the tests!).

Make it easier to build and test: towards continuous integration

An automated build and test infrastructure can be brought together to perform an independent automated build and test function. One extremely useful benefit of this system is that it can be used to update your developers. If the build and test is run overnight, it is relatively simple to send out an automatically generated report to your developers that describes the state of your software.

In Linux, an automated build and test system can be done by scheduling a Cron job on a non-critical machine to be executed automatically during the night. This job should build and test scripts in sequence and then disseminate them in any way you choose, for example by posting them to a locally hosted web server or sending them to a mailing list. If a developer team requires large-scale testing, then sophisticated systems such as Inca can harness a number of machines through a central server to coordinate, distribute and execute many testing tasks in a parallel and scalable way. More information on Inca is available on their website.

Taking this one step further, a continuous integration server can detect when a new version of code is checked into a source-code repository, and then automatically run the software tests. This saves you from having to run more complex or time-consuming tests on your own computer. Continuous integration also helps to ensure that you immediately get feedback on the impact of your changes. Examples of continuous integration tools include CruiseControl, Hudson and Bamboo. A good list of these tools is available on Wikipedia.