The agile manifesto: where do things go wrong?
This is published under the terms
of the licence summarized in the footnote.
All free-to-read materials on the Avancier web site are paid for
out of income from Avancier’s training courses and methods licences.
If you find them helpful, please spread the word and link to the
site in whichever social media you use.
Abstract
This paper follows the Agile Manifesto paper that listed “a top ten agile imperatives”.
If you want to promote agile principles, you must understand why some people find those ten imperatives challenging.
This paper discusses why and where you may find our top ten imperatives for agility are challenging – reasons why many resist agile principles – reasons why people stray from the agile path.
Contents
The manifesto for agile
software development
Why the ten agile imperatives
are difficult and threatening
Other contexts where you may
find agility is difficult
So what’s wrong with the
Agile Manifesto?
This manifesto <http://www.agilealliance.org>
presents the consensus view of several agilists about
the values and principles they look for in a successful project.
We are uncovering better ways of developing software by doing it and helping others do it.
Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.
This ten agile imperatives below elaborate on these four values.
Many agile methods place considerable emphasis on preconditions for a successful agile project. DSDM proposes four keys to agility
· Flexible and prioritised requirements (80/20 rule, Value stack, MoSCoW)
· Time/cash-boxed (Rather than requirements-boxed, We deliver what we deliver
· Incremental development (Vertical slices added, evolutionary prototype)
· Close user involvement (Joint application development (JAD) workshops)
I have found many projects on which agile methods simply can’t be applied, at least, not as far as agilists would like them to be.
Analysis of scores of projects led me to produce this scoring system.
|
What kind of project |
||
|
Time/cost-driven |
0, 1, 2, 3 |
Mandatory requirements-driven |
|
Users available for JAD |
0, 1, 2, 3 |
Users not available |
|
Developers empowered |
0, 1, 2, 3 |
Developers not empowered |
|
What kind of system and work |
||
|
Incremental releases possible |
0, 1, 2, 3 |
No partial solution
can be released |
|
Client/user interface dominated |
0, 1, 2, 3 |
Server/database dominated |
|
Output/enquiry dominated |
0, 1, 2, 3 |
Input/update dominated |
|
Simple business rules (CRUD) |
0, 1, 2, 3 |
Complex business rules (non-CRUD) |
|
On-line |
0, 1, 2, 3 |
Batch |
|
Stand-alone |
0, 1, 2, 3 |
Highly integrated w other systems |
If your project score more than 10, then you will find it difficult meet our top ten imperatives, listed below.
That is not to say you shouldn’t try to be agile, rather to explain why you’ll find it difficult and may have to accept some compromises.
For example, you may find your customer forces you to
capture detailed requirements in some kind of requirements catalogue and
develop a pyramid of documentation below that.
The trick is to be as agile as you can be.
But I’ve studied why people resist agile methods
And come to conclude there is more to be understood about the challenges to agility than the conditions above.
People can find
agile to be a harsh discipline. And if the whole team is not committed, it can
decay.
1) If we are going to fail, we want to fail
faster
People used to wallowing in
a waterfall process find this a harsh discipline.
The supplier working on a
time and materials contract needs nerves of steel to hold to the view that
early failure on this project will be perceived as a success.
It also risks the premature
cancellation of projects that simply need more time for analysis, for
architecture definition, for technology familiarisation etc.
2)
We develop iteratively, driven by priorities
Incremental development
suits web sites and some other software products.
It is much harder to apply
the principle a batch program suite.
3)
We use prototypes to tackle risks early
Prototypes consume time and resources.
Sometimes they reveal little.
Often, the final product is
so different that the lessons drawn from the prototype are of limited value.
Sometimes it is well-nigh
impossible to engage the users with what a prototype does.
Perhaps you will be able to
allocate one of two of the team members to the role of users.
Sometimes the system is
principally batch processing, where even the keenest of users find it difficult
to engage with and validate a prototype.
4)
We empower a small team (including users) to get on with it
Many project managers find
this threatening.
5)
We maintain continuous and close stakeholder involvement
Persuading customers that
ongoing customer acceptance is a key to development, and all stakeholders must
be involved, can be very difficult.
Customers may simply be
unable to do acceptance testing on a half-finished batch job.
Worse, acceptance tests may
prove useless because they fail to guarantee the operational system will meet
non-functional requirements to do with performance, availability, recovery and
security.
Michael Z: I haven't met
many folks who wouldn't love to maintain continuous and close stakeholder
involvement.
The problem of course is
trying to get business management to let you have a good person as part of the
team on a near full-time basis for an extended duration.
Chris B: I have heard some
nominally agile projects fail for the interesting reason that one key
stakeholder was not allowed to be involved. In one instance the key stakeholder
was a user of the system.
In another it was an IT guy
in the central office who was tasked to enforce a company-wide policy of some
kind.
6)
We focus on program code rather than on specifications
What if the customer insists
on a detailed functional specification?
What if your analysts know
no other way? And that way has worked for them in the past?
7)
We focus on tests rather than on specifications
What if your analysts and
developers perceive test data preparation as a low status activity, or boring
compared with specification?
8)
We iteratively refactor code written so far
Refactoring does not seem to
apply to data structures as well as to process structures.
Specific advice on how to
speed up database structure changes would be interesting and valuable (more so than
classifications of changes that can be made).
9)
The customer must be willing to change and sacrifice requirements
This is a huge challenge.
Flexing requirements is not
tenable in many commercial customer-supplier relationships.
Welcoming late changes is
impossible given an immovable deadline and in some commercial customer-supplier
relationships.
Some requirements are fixed
by legislation, or contract, or legacy interfaces, or legacy databases, or the
requirement to mimic a legacy system.
10)
We get customers on side before we start
Often, the customer and or
users won’t play ball with agile principles.
E.g. The
customer insists every specification is approved by a committee that meets once
a month.
The customer will never
delete a requirement.
The project manager is quite likely to think of several of these points, and conclude than most of our day to day problems lie in projects where agile principles cannot be readily applied.
This section lists some
lessons that projects have reported in connection with the use of the ten
imperatives above.
It is intended to help
you steer others away from making the same mistakes. It is not an explanation
of any agile method or a definitive list of agile principles.
An
agile method is more than pick 'n mix principles:
It is a good idea to apply agile principles where you can, but that is not the same as applying an agile method.
An agile method implies specific constraints and consequences.
The best-known agile method today is probably Scrum
There are others: DSDM, XP, Feature-Driven Development and Adaptive Development).
Each is a method with specific disciplines, suited to specific circumstances.
Lesson: Do not pretend you can use these
methods if your team is not trained and the circumstances are not right.
An
agile method implies a customer with contractual obligations:
Lesson: Never adopt an agile approach that you have not first presented to and agreed with your customer.
You will hear: "the customer has a short timescale, can we use RAD?" "can we do extreme programming to speed things up?”. "can we do it faster and cheaper using an agile method?"
Perhaps yes, but if your customer is not signed up, ideally in the contract, to play their part and meet their obligations then say no!
An agile method means "what gives is what you get".
If time and resources are fixed, then the
requirements must be flexible.
Is your customer signed up to revising requirements, and postponing some to a later iteration?
If not, then don’t pretend you can follow an agile method, though some agile principles might be useful.
Prioritisation means willingness to
sacrifice:
Lesson: listing 90% of the requirements as "must have" is not prioritisation.
Will your customers/users accept a system that meets significantly less than the originally requested functional or non-functional requirements?
Empowerment means decisions are immediate and
final:
Lesson: a customer/user committee is not an empowered team.
Agile methods require close knit communication and co-operation, and usually a degree of physical proximity that is incompatible with offshore development.
Can you work daily with a small number of customer/user representatives who have the authority to revise and reprioritise requirements?
Flexible requirements imply light-weight or
slick change management:
An empowered team is daily revising and reprioritising requirements.
This is incompatible with heavy-weight change management.
Incremental development yields software releases:
Incremental development (which covers both refinements and increments) is about delivering working code, ideally to live use, or to acceptance testing.
But if you can’t get customers involved in
the testing, then you aren’t really doing incremental development.
Iterative development really does yield
software releases:
Redrafting specifications is not iterative development.
Prototyping is very often not iterative development.
Most projects find uses for prototypes of various kinds.
The kind of prototype important to an agile method
involves demonstration of the evolving system itself, and requires empowered
users to make accept/reject decisions during the demonstration.
Agility comes from training, experience and
skill:
It takes years of training for a boxer to be light on his feet when it matters.
Agile methods assume staff know what they are doing.
That includes knowledge not only of the chosen technologies and agile method, but also of structured methods and techniques, which will still be useful here and there.
And managing an agile project is more taxing
than managing a structured project!
Schedule in technical architecture and
testing:
In commercial SI projects, elements of the regular project life cycle are usually needed.
You still need to attend to your solution development process.
Software specialists still need sufficient specification of requirements.
Customers are not satisfied with a prototype rather than a tested, scalable system.
We have yet to find a customer willing to accept a system that does not scale up to support the real number of users and volume of transactions.
We have yet to find a customer prepared to
forgo acceptance testing (which implies some degree of system testing).
Above all, some customers and end users are simply unable or
unwilling to engage in agile and iterative styles of development, especially,
unwilling to accept flexing requirements
Some projects are not well suited to an agile approach.
Characteristics to watch out for include:
The typical project lacks many of the agilist's preconditions for success. You will often find:
Consider a troublesome project I reviewed.
There existed a legacy system to apportion the money from many ticket sales between a few business owners.
The system had to replace the legacy system, run daily instead of weekly, in a 4 hour batch window each night.
The business rules, previously hard-coded, must be converted into data attributes modifiable by business users
Our new system could easily have printed the blank cheques for the business owners, or cheques with wrong numbers on.
The trouble is, it took us several years
· to work out how to process the input data in the narrow time window available for batch processing,
· to print the right numbers on the cheques, and
· to prove to the customer’s satisfaction that the numbers were right.
How would you approach incremental development in a case like that?
What partial system could the customer accept as an increment - before the system is fully finished?
When pressed, one agilist mumbled unconvincingly about doing “earned value analysis” along the way.
(Writing a paper dissing that technique would be to shoot fish in a barrel.)
What we need are good indicators for when agility works well and when it doesn't.
People need to understand the trade-offs and compromises they have to make.
Several challenges are discussed below, starting with the challenge of a large-scale systems integration programme.
Some enterprises, looking to control the costs and ensure the value of projects, impose a simple sequential process on all projects carried out by their IT department or IT services provider.
Here is an example, drawn from a major international
company: Appraise
- Select - Define - Build – Operate.
The enterprise management team insists that all requirements are defined in detail during the Define stage.
The idea is to control the costs of the Build phase and ensure the value of the Operate phase.
So even though the IT services provider may prefer agile development, but they have to work to the terms the enterprise sets.
Enterprise managers believe that imposing stop/go reviews at the phase boundaries is the best way to spot and stop a runaway train.
Some IT services providers (OK, most of them), insist on requirements capture and tight change control.
Because that’s the only way they have of stopping the customer from endlessly changing their mind about what they want.
And then refusing to pay the bill for the work the provider
has done on earlier (now redundant) requirements.
An IT service provider may have to spend months working on a bid for a large project.
They simply cannot start coding the solution.
They have to work with more abstract specifications.
The agile manifesto is perfectly reasonable, but I think it is also fair to make these observations.
It is hard to direct or assess a project using the values.
Our top ten imperatives offer a stronger challenge to common practice, and give us some directions to follow.
It promotes conditions that occur naturally in easy projects.
We need help with the projects where we find the opposite conditions.
·
There are many mandatory requirements (and
perhaps heavy-weight change management processes
·
Users are not available
·
Developers are not empowered
· No partial solution can be released
·
The solution is server/database dominated
·
The solution is input/update dominated
·
There are complex business rules (non-CRUD)
·
There is batch processing, and highly
performance-constrained
· There are many or complex interfaces w other systems
· There are data management or migration challenges.
It can encourage untutored, undisciplined, trial and error.
Surely no manifesto author intended this.
However, left to their own devices, many software developers will disregard any contract, maintain only slight documentation of requirements, and develop iteratively, learning as they go along.
And the agile manifesto enables them (albeit improperly) to aggrandize this by calling it agile development.
Footnote:
Creative Commons Attribution-No Derivative Works Licence 2.0 12/03/2015 20:22
Attribution: You may copy,
distribute and display this copyrighted work only if you clearly credit
“Avancier Limited: http://avancier.website”
before the start and include this footnote at the end.
No Derivative Works:
You may copy, distribute, display only complete and verbatim copies of this
work, not derivative works based upon it.
For more information about the licence, see http://creativecommons.org