Posted by m.jackson on 18 February 2015 - 11:11am

By Mike Jackson, Software Architect.

The UK open source service, OSS Watch, have recently published their Openness Rating tool. This tool allows projects to assess their openness and can be applied to both open and closed source software. In this blog post, I'll provide a summary of the Openness Rating tool and how it complements our own online Sustainability Evaluation Service.

Openness Rating Tool

OSS Watch provide independent consultancy on all aspects concerning both use and development of free open source projects. They developed the Openness Rating tool to support this consultancy and have used it in-house for years. The tool asks users to answer 51 questions, grouped into 5 categories:

• Legal Issues: software licensing and whether the code can be read, modified and redistributed.
• Data Formats and Standards: data and communication formats, use of closed/proprietary or open standards, use of project management processes (e.g. PRINCE 2, XP).
• Knowledge: availability and location of information and communications about the project and its software and any financial, technical or other barriers.
• Governance: decision making and project policies, leadership and membership.
• Market: extent to which the software is, or could be, exploited commercially, possible revenue models, and any financial, technical or other barriers.

The questions are based on the paper Foundations of Openness - Evaluating aspects of openness in software projects by Pia Waugh and Randy Metcalf (October 2007). Each question has supporting commentary and is multiple choice with certain answers highlighted in red if they undermine openness (for example selecting "Proprietary" for "Is the licence recognised as a common Free and Open Source licences?" or "Yes" for "Does the project rely on any closed proprietary standards?"). Answers are weighted and are used to calculate an "openness rating" percentage for each of the categories. The ratings are presented as a web page. As an example, the openness rating for OGSA-DAI, an open source project I worked on before joining the Institute, would have been: Legal 69%, Standards 88%, Knowledge 56%, Governance 76%, and Market 55%. Once users have their rating, they can then follow-up with OSS Watch on dimensions within which their openness could be improved.

Sustainability Evaluation Service

We developed our online Sustainability Evaluation Service (SES) as a way of capturing the criteria we'd use when measuring the sustainability of a project or software. This automated what had previously been a manual process, and allows us to focus on consultancy and advice that cannot be readily automated (yet).

The SES consists of 93 yes/no questions (for example "Does your software provide tests to demonstrate compliance with open standards?" or "Is dependency management used to download dependencies (e.g. ANT, Ivy, Maven)? "). These are grouped into 16 categories including Source code structure, Support, Portability, Community, Testing, Copyright, Licences. The questions are based on our guide Software Evaluation: Criteria-based Assessment (November 2011).

Answers are analysed, and, for any question to which the user answered "no", commentary is presented as to why the user should address this issue (for example "Anyone can claim compliance with an open standard, but tests allow this claim to be proven. By inspecting your tests, potential users can assess whether your software does indeed comply to the standard"). This advice is presented as a web page. As an example, see the report for OGSA-DAI. As for the Openness Rating tool, users can contact us for discussions on resolving specific areas of concern raised by the SES.

Comparing the services

Our SES looks at all aspects of sustainability. Open source and open development are a means by which software can escape the shackles of a single project, institution or funding stream and, so, can contribute to improving the sustainability of software. As a result, the SES overlaps with the Openness Rating Tool as both include questions relating to licensing, open standards, contributions and communications. The SES also addresses other aspects relating to software sustainability including software development practices, source code management maintainability and testing. These are out of scope for the Openness Rating Tool, which, in contrast, investigates more deeply issues around project openness, looking at, for example, the openness of non-code resources, sub-licencing, auditing licences of third-party dependencies, commercial exploitation, concerns that the SES does not currently explore.

Conclusion

Both the Openness Rating Tool and SES embody and expose expertise via automated services, making this expertise available to a wider audience than would otherwise be the case. They both allow users to identify concerns which they can then discuss with OSS Watch or ourselves, respectively. The services are complementary - the SES provides guidance on the breadth of issues relating to software sustainability, one of which is open source, whereas the Openness Rating Tool provides depth on issues specific to open source and open development. We recommend both to any researchers wishing to improve the sustainability or openness of the projects or software.

Postscript

I could not find the source code for the Openness Rating Tool. Our SES is implemented as a set of non-public Drupal extensions. Observations on this state of affairs is left as an exercise for the reader.