Is Agile still agile?
By SSI Fellow Peter Schmidt.
Cross-posted from Medium.com.
Oh no, not another article on Agile! Is that what you think? You may be right! After all, we have quite a lot of material on agile practices out there already. So, what good will these few lines do?
And yet, after having worked in the IT sector for a long while, I thought I’d throw in my 2 pence on the subject. It all came about when I interviewed Raj Heda on my podcast Code for Thought. Raj is a consultant who helps organisations roll out agile practices. And I realised how much has changed since I started my career in IT many, many years ago.
For today, daily stand-ups, pair-programming, sprints, backlogs, retrospectives are all part of a common vernacular in IT offices. So common are these practices, often grouped together under the umbrella term of “agile”, that it’s sometimes easy to forget where they come from. Or, indeed, what kind of problems they are meant to solve. Which I thought might be worth writing a few lines about - so here it is.
Let there be … a manifest
Agile software methodologies have been around more than 20 years. In 2001 a group of engineers got together to write a manifesto on how they believe software engineers should work and help them deliver better results.
The output, called the agile manifesto, consists of 4 key statements (and 12 subsequent principles). Here they are:
- individuals and interactions over processes and tools
- working software over comprehensive documentation
- customer collaboration over contract negotiation
- responding to change over following a plan
The problem the authors tried to address was the fact that up to that date, too many software projects ran over budget, disappointed customers and stakeholders and seem to require a lot of maintenance once delivered (read: full of bugs). Organisations tended to design and plan everything upfront, then implemented and tested it before showing it to the end user and customer. Indeed, in some software architecture books from that period authors compared this process to building a house: first the detailed blueprint, then the actual construction before the customer signs off and moves in. The trouble with this process, often described as “waterfall”, is that it takes too long before the end user sees the product. Time, during which requirements and expectations often change (and costs).
In fact, I have been victim of this myself. In 1997 I worked for a UK hospital to develop software for a clinical study. Part of the work was outsourced to external contractors, who were tasked with writing a driver for a bespoke medical device. We met with the contractors in planning sessions. They then produced a very long and beautifully printed document containing all features and all requirements in minute details.
And (you guessed it, dear reader) … they failed to deliver. After severe delays, crisis talks and meetings, we had to do the work ourselves. At which point, needless to say, we ran hopelessly over budget and disappointed funders and clinicians alike.
It was a valuable lesson. And following my move into the private sector at the beginning of the 2000s I was pleased to see that there have been initiatives to fix this. These initiatives aligned themselves closely to the agile manifesto and aimed to make software deliveries more predictable and more robust. Many developers, myself included, were excited about this and wanted to adopt it in their work environments. Why not give customers regular snapshots of the product right from the start and get their feedback early on? Why not check and test the product while developing it — rather than having a long test session at the end?
Funnily enough, as engineers we faced a lot of resistance from senior management at the time. But not any more! The ‘State of Agile’ annual survey, which has been running for the last 15 years, states in its latest report that only 4% of the participants said they didn’t know anything about agile methodologies. Only less than 2% of the respondents said they don’t practice it at all. Agile has gone mainstream.
Veni, Vidi … Scrum
The survey also shows, which of the many agile methodologies has come out on top. And the clear winner is Scrum (including some of its variants).
So what is it about Scrum that makes it so widely adopted? Maybe, it’s because the ingredients of Scrum are few and easy? Mainly:
- work is organised in sprints (e.g. 2 weeks length)
- only few team roles, which are clearly defined: the team includes developers and the customer or product manager — called the product owner
- regular sessions to get feedback, correct course
- few artefacts (sprint/product backlogs) to keep track of what should be and what is being worked on
(For more go to scrum.org or read the K Schwaber/J Sutherland book)
Scrum can be viewed as a practical implementation of the agile manifesto in the following way:
- having regular reviews and demos allows software teams to respond to change
- having a customer rep in the team (product owner) aims to collaborate with customers directly (or at least through established channels)
- having demonstrable software at the end of a sprint focuses on delivering working software
- the regular review sessions of the team, including product representatives, aims to encourage individuals and interactions
Scrum and other animals
Since Ken Schwaber and Jeff Sutherland wrote the book on Scrum, the methodology has gone through many hands and several changes.
For one, some realised that Scrum is all very well on a team level. But what about large enterprises and organisations with many teams, often distributed?
How well do methodologies like Scrum scale there? An attempt to solve this has been made with SAFe - the scaled agile framework (think of a daisy chain of scrums). Even on a team level, other initiatives like Kanban (and then the combination between Scrum and Kanban — yes, you guessed it — Scrumban) were introduced to improve the overall process further.
Finally, the agile manifesto itself has gone through some makeover, with the proposal of a “modern agile manifesto”.
And, as I mentioned at the beginning, we have an ever expanding cottage industry (or perhaps not so cottage anymore) of agile consultancies, training courses, “Agile” certifications, workshops, books — you name it. In short, agile methodologies have become a bit of a zoo.
Is Agile still agile?
20 years after the manifesto, some (many?) of us in the industry have become rather jaded and disillusioned with “Agile”. In fact, in a recent conversation one of my colleagues opined:
“Seems to me Agile doesn’t mean very much nowadays, maybe it never did.” A more elaborate statement comes from Dave Thomas, one of the original manifesto’s authors, who stated as far back as 2015 that “agile is dead.”
How has it come to this? Back to my interview with Raj Heda. Right at the beginning, Raj points out that agile methodologies require a change in mindset when implementing it in organisations. How can we deliver maximum value to our users and clients quickly? How can we empower engineering teams to deliver the best they can? After all, practices such as Scrum were specifically designed to bring this about.
But 20 years on, I have not seen widespread evidence in terms of changing mindsets and empowering teams. A few times I have seen “Agile” roll-outs being little more than checkbox ticking exercises:
- doing stand-ups? Tick!
- Having JIRA boards as backlogs? Tick!
- Meeting every 2 weeks for a review? Tick!
And what’s more, in some cases I experienced almost an inversion of the agile manifesto, where tools and processes have indeed become, once again, more important than individuals and interactions. Who hasn’t seen engineers and teams wasting away hours, days even, on getting workflows in JIRA right?
It seems that Agile has become less agile and we have again a system that engineers have to work around rather than work with.
Maybe it shouldn’t surprise us? After all, working down a to-do and check list is a lot easier than affecting real change in the way we work. But if we look at implementing agile practices as a way of changing the mindset inside organisations, this effort to lead change is necessary. Luckily, there are ways to approach change systematically. In my conversation with Raj, we specifically discuss the 8-steps program by John Kotter.
From the 8 steps, there are 2 in particular, which I think are crucially important: The first is to get buy-in from staff early on. John Cotter e.g. mentions a number of ways in which this can be attempted and achieved. Whatever the ways, it requires time and effort. It won’t happen overnight or as a result of a 30min PowerPoint presentation.
The second is to get the active support and sponsoring from the senior leadership. In fact, in survey after survey participants reported that lack of management support was one of the reasons why agile roll-outs failed. Like their staff members, senior leaders need to invest time and effort, in order to turn it into a success. Paying lip service to iterative development, using the right lingo or signing up to expensive tool subscriptions isn’t getting your software out of the door any cheaper, faster or better.
But even before then, I often wished there had been some clarity of why an organisation wants to implement certain agile practices in the first place: What problems are we trying to solve? What benefits do we expect the organisations to draw from this? And how do we expect this to improve outcomes and deliveries?
Back to the title: is Agile still agile? The answer is: it really depends on us. The tools we use, the books we read and the courses we got certified in only help us so far. But they don’t deliver the goods by themselves. Agile is not an assembly line you put in place that, once switched on, will deliver what you want.
And just before I close, I’d like to say this. It can really work! I had the privilege to see excellent and truly agile teams in practice. Where it helped us to focus on what really needs to be done and consistently deliver high quality products. It’s not easy to get to that point. Raj wasn’t kidding when he said it requires a change in mindset. But it is also immensely rewarding when it works. Because, who doesn’t like their hard work being put to use and their software out there? And — at the end of the day — isn’t that what really matters?
Listen to the Code for Thought episode To Agile or not to Agile.