Posted by s.hettrick on 12 December 2012 - 10:40am

By Paul Madden, Solutions Architect, Professional Services, SeaChange International.

Several weeks ago Simon Hettrick posted a reference on LinkedIn’s Scientific Software Development and Management group to a blog post by James Hetherington entitled The Craftsperson and the Scholar. I responded to the post by parroting the conventional wisdom that it's more expensive to create sustainable software, and that it typically only makes sense to pay that additional cost for commercial software development, so Simon asked me if I’d write about my views on the economic viability of writing good code in academia.

I first thought I might not be qualified. I do have some background in scientific programming. Long ago I earned a PhD in experimental particle physics then worked as a staff computer scientist at Lawrence Berkeley Labs developing data acquisition and analysis software for a Free Quark Search experiment at the SLAC National Accelerator Laboratory. But all that was over thirty years ago before I joined the Laboratory Data Products group at Digital Equipment Corporation - then moved from that into a career in commercial software engineering. Upon reflection though, I think the issues that commercial and research software development have in common outweigh those that are different and I have some things to say about these common issues.

There are three reasons why sustainable software typically costs more to create. First, more upfront design work is needed to create sustainable software; second, sustainable software requires the production of additional artifacts - design, support and user level documentation; and third, more effort must be expended in the creation of test harnesses so that any ill effects from changes can be efficiently detected and repaired.

So, at first blush, it would appear that the creation of sustainable software will add cost to every research effort.

There is, however, a way around this increased cost. In fact, it could be that an emphasis on the creation of sustainable software might actually reduce the software development cost associated with every research effort. My experience has been that the single most important factor in determining software development cost is team composition - having the right people with the right skills in the right roles, and that by far the most important team member is the software architect.

A requirement for sustainable software creation might increase the development cost estimate by a factor of two or three. But a good software architect can decrease the cost by a factor of three or more. I offer this quote from Fred Brooks, author of The Mythical Man Month.

"We can get good designs by following good practices instead of poor ones. Good design practices can be taught. Programmers are among the most intelligent part of the population, so they can learn good practice. New curricula, new literature, new organizations such as the Software Engineering Institute, all have come into being in order to raise the level of our practice from poor to good. This is entirely proper.

Nevertheless, I do not believe we can make the next step upward in the same way. Whereas the difference between poor conceptual designs and good ones may lie in the soundness of the design method, the difference between good designs and great ones surely does not. Great designs come from great designers. Software construction is a creative process. Sound methodology can empower and liberate the creative mind; it cannot enflame or inspire the drudge.

The differences are not minor - it is rather like Salieri and Mozart. Study after study shows that the very best designers produce structures that are faster, smaller, simpler, cleaner, and produced with less effort. The differences between the great and the average approach is an order of magnitude."

Brooks is describing the role of a software architect. If you want a first class building, you hire a first class architect to design it. The architect will oversee the construction - directing the construction engineers as they implement the design. Brooks is suggesting that software systems development ought to follow the same model.

A building architect’s job does not end with the production of the design. The architect is expected to be conversant with building codes, the state of the art in plumbing and wiring, concrete preparation, the quality of steel and all the related construction techniques. So the building architect is ultimately responsible for both the design and the implementation and is involved with every aspect of the building’s construction.

Brooks’ observation has been around for twenty five years now but the academic, software engineering, and the commercial software development communities still struggle with defining the software architect’s role. Degrees in computer science are common. Degrees in software architecture are rare. Software architect titles are common in industry but there remains much murkiness with respect to the definition of the role - and most of that murkiness is related to the part about overseeing the implementation.

I am always struck by the number of programmers that like to write code, but don’t like to read it, and by the number of software architects who think their job is done once the spec’s are published. Over the years I’ve found that all the really good software architects and engineers I’ve worked with not only can read other people’s code but enjoy doing it. Computer code, in any language, is codified thought. Reading code is the only sure way to know which programmers are paying attention to the design and which programmers are not. And reading code is the only way to really understand the state of any software development activity. After all, the code is the software.

Two examples of how great software architects operate...

Dave Cutler designed the RSX-11M and Vax/VMS operating systems at Digital Equipment Corporation, then went on to design the NT operating system for Microsoft. Many programmers who worked with him tell stories of leaving a code printout on their desk at the end of the day only to come in the next morning to discover that Dave had read the code and annotated it with suggestions, or directives, about how to improve it.

I had the good fortune to work with Bernie Stumpf at Apollo Computer in the mid-1980s. Bernie had been the architect for PrimeOS - the operating system for Prime Computer Systems - before moving to Apollo to design the Aegis operating system. One of the programmers there tells a story about walking into Bernie’s office one night to find him intently reading code on his monitor with his hand covering the right side of the screen.

“Bernie, why are you hiding the comments?”, he asked.

“Well”, Bernie said, “The comments are sometimes wrong.”

There is ample evidence from the commercial software development world that identifying and empowering an exceptionally talented software architect can dramatically reduce the cost of a software development effort- and dramatically increase its chance of success. Yet the importance of the software architect’s role is still not widely understood.

Digital Equipment Corporation addressed this problem in the 1980s by creating a dual technical ladder. For each level in the technical management hierarchy, they created a corresponding level to be occupied by a purely technical contributor. This made explicit the fact that in a technology company there is management to be done and there is design to be done - and that the company’s success depended on both these things being done well. But the norm in most organizations today is to live with a substantial fuzziness with respect to the boundaries between systems design and technical management. That fuzziness can be very expensive.

I suspect much of the success of today’s open-source community stems from the fact that it affords real opportunities for exceptionally talented software architects.

Returning to the subject at hand - how can we best help the scientific community create sustainable research software - there are a variety of things we might try. We can augment the development team with trained scientific software engineers. We can try to impose accepted coding standards and practices. We can insist on the use of some modern development methodology. We can argue for the use of the right programming language. We can help with Make vs. Buy. But in the end, none of these things can have anything like the positive impact to be had from just ensuring that the right software architect is in place and is empowered to guide the project.

I’m not sure there can be a career path in academia for research software engineers and I’m not sanguine about the success to be had from advocating for it. But I do think there should be a career path in academia and a distinct discipline for research software architects. Most major universities already have separate schools of engineering and of architecture. The role of the building architect is widely recognised and accepted as distinct from that of the civil engineer. It seems entirely reasonable to make the same argument by analogy for the role of the software architect as distinct from that of the software engineer.

There are plenty of research topics in software architecture. How does one best interact with stakeholders to produce a design that meets their needs? How does one make a design visible so the aesthetics can be judged? How does one communicate a design to the stakeholders so they are comfortable with funding the implementation? How does one efficiently communicate a design to the implementors - is the UML really the end of it? How does one efficiently monitor an implementation for design conformance? Can test cases be distilled from the design in parallel with the implementation? How does one compare current designs to those that have gone before - evolution of design? How does one quantify the effects of good vs. poor upfront design on both development costs and long term support costs?

I hope we can eventually - perhaps a decade from now - get to a place where every research proposal that requires substantial software development will include the name of a software architect as one of the principals. That principal will be a faculty member from a university’s school of software architecture. Then when the research project commences that faculty member will direct one of his graduate level software architecture students to design and oversee the project’s software development effort as part of the student’s Master’s level or PhD requirement. The things learned from the project can be fed back into the software architecture school’s research activities - and the result of that research can eventually flow to the commercial software development community.

Now is the time though to take Brooks’ now twenty-five-year-old observations seriously and start nurturing some research software Mozarts that we can inject into the research community to continually reduce our software development costs, improve our software quality, and show us how to do great software design.