Copyright
Avancier Limited. All rights
reserved.
Abstract
Agile methods form a broad church.
There are management-centric approaches like
DSDM and SCRUM.
There are also development-centric approaches
like XP.
There are
many and various disciplines to be found in these and other methods.
Agile
development sounds like a soft option, yet it involves accepting some tough
disciplines in order to increase the efficiency of software development.
This paper promotes some agile development principles
over others, and suggests agility should not mean all things to all people.
Contents
Our
quality plan for agility: a summary list1
Principles
of Rapid Application Development (RAD)2
Principles
from the Manifesto for Agile Software Development
<http://www.agilealliance.org>2
A
principle from XP: Test-driven development (TDD)6
An
11th principle from Martin Fowler: Continuous (daily) integration. 7
Four
agile principles that didn’t make our top ten8
What’s
wrong with being all things to all people?9
Conclusion:
our quality plan for agility10
This paper distils what I think matters most from
several attempts to formalize what agile development means.
Briefly,
before we discuss agile principles at length, a project team who can say yes to
all ten questions below would have to be regarded as highly agile.
1)
If we are going to fail, we want
to fail faster
2)
We develop iteratively, driven by
priorities
3)
We use prototypes to tackle risks
early
4)
We empower a small team (including
users) to get on with it
5)
We maintain continuous and close
stakeholder involvement
6)
We focus on program code rather
than on specifications
7)
We focus on tests rather than on
specifications
8)
We iteratively refactor code
written so far
9)
The customer must be willing to
change and sacrifice requirements
10) We get customers on side before we start
Chris B: I like your points and they seem to me to be a fair reflection of agile practices.
Michael Z: This list gets down to brass tacks, and can be reflected in concrete project practices, and in the quality plan up front.
Even if these aren’t your top ten, you will surely agree that a project that can say yes to all ten principles in this table would have to be regarded by as highly agile.
The value of these imperatives to a manager lies in their simplicity.
Starting with these as clear statement of intent, we can look at the situation in front us, then adapt the agile approach accordingly.
The following sections discuss the sources from which I have abstracted the ten imperatives.
Agile principles were first described in the early 1990s.
One of the earliest agile methods, DSDM, promoted 9 principles, mostly similar to those discussed below.
DSDM was originally presented as a RAD method.
Nowadays, agilists deprecate the term RAD, because it appears to promise that you can get the same result quicker.
RAD methods were indeed about producing a system quicker, but not necessarily the system the customer first envisaged (which may prove undeliverable).
Rather, the aim is to deliver a system that does what matters most, and is deliverable.
The key to DSDM, and agility in general, is flexibility about requirements.
This and other DSDM principles can largely be subsumed under the umbrella of the agile alliance.
Before introducing their manifesto for Agile Software Development, there is just one scary principle from the earlier RAD school worth mentioning.
A failed agile project is much cheaper than a failed sequential/waterfall project – because it fails faster!
Agile methods weed out ill-conceived projects, through iteration and short-term delivery milestones.
Nobody wants to fail, or be seen to fail.
It is human nature to put off the day when failure is exposed.
That’s one of the reasons why being agile can be tough – it isn’t the easy option some people assume it to be.
This manifesto presents the consensus view of several agilists about the values and principles they look for in a successful project.
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
That is, while there is value in the items on the right, we value the items on the left more.
This guide elaborates on these four values, but is not meant to be the definitive description of them and everything proposed as an agile development principle.
The aim is to produce a list of ten point that are remarkable, and perhaps controversial, about agile projects, not to repeat motherhood and apple pie that agile methods share with non-agile methods.
Naturally! Though communication about technical things is assisted by references to lists, tables, structures and diagrams that have been documented using tools.
Naturally! Though the normal context for IT service providers is work to a detailed contract.
This is less obviously a natural principle.
Waterfallists disapprove: “Agile is, in my mind, as much an excuse to cover
for not knowing what a problem is or how to go about understanding it as it is
a methodology.
From what they say, we wouldn't be very agile if we had a plan.”
Agilists welcome change requests, because they show the customers and users are clarifying their business needs.
Customers realize what they really want when they see the solution take shape.
And change requests describe their requirements more accurately than what they said in interviews before development started.
Actually, agilists do make plans.
But since they assume the plans will change, they don't detail later time/cash boxes as much as the first.
In Scrum for example, the next 30 day sprint is planned in more detail than what may happen after it.
Agilists emphasise the need for slick change management.
They deprecate bureaucratic change control that has the effect of restricting the changes (that is, improvements).
Because to resist change means resisting building the solution the customer really needs.
And you may end up building the perfect solution to the wrong problem.
Agilists continually juggle priorities.
If a change is substantial and important, then you must ask the customer to de-prioritize something else to make way for it.
See our paper “Configuration management challenges”.
The more detailed principle is:
Agile developers sit down with users and start coding almost immediately, stopping frequently to evaluate progress and make necessary changes based on user input without the need for a substantial requirements specification pyramid.
Waterfallists disapprove: they say fixing a problem in requirements analysis is N times cheaper than a fixing a problem in development.
Their argument goes: “The cost of code defects is only about 10% of the cost of failure.
About 35% of the
cost of failure is due to design defects.
About 55% is due
to requirements defects.
So, don't spend as much on code testing as on reviewing design and requirements first.”
Our “Software is not hardware” paper explains why agilists conclude exactly the reverse from the same statistics about test-revealed defects. They say:
Reservations: There is sometimes a presumption that the quality of a software product can be assessed by visual inspection of the user interface, which is true of some but not all applications.
Also, agilists like to assume that all team members are high-skilled and multi-skilled, and reading the code is natural.
In practice, stakeholders often need to discuss reader-friendly descriptions of the business rules that are to be coded.
And it is reasonable to maintain some architecture-level models in complete detail - most obviously, a data model.
Agilists say that completing detailed requirements specifications makes for a rigid process that puts walls between users and developers.
And, given the ever-changing nature of a business and its software, it is doomed to fail.
Agile methods excel where requirements are unclear, or where the instruction is simply, make it “faster, cheaper, better", no matter how.
But even where requirements can be detailed, agilists are against weighty requirements documentation.
Look at what maintenance teams do.
They must maintain the solution configurations used in development and testing.
Few of them maintain the original requirements configuration.
They find the requirements specification pyramid that helped during elaboration becomes less helpful when
Maintenance teams find at least some of the early specification and design artifacts to be an overhead during change management.
Agilists say to minimize the weight of documentation.
Their working assumption is that you should not detail every high-level requirement through lower-level requirements specification and design documents.
Look at what happens during development.
Agilists observe that stakeholders and business users often:
And on the IS/IT side, analysts, architects and coders don’t do their job well.
They
A requirements configuration is both incomplete and flawed.
A requirements configuration is never as fully detailed as a solution configuration.
Also, the requirements probably contain more flaws than the solution, because requirements are not executable and testable.
It is better to minimise the volume of requirements documentation, minimize the weight of specification and design artifacts between the top and bottom of the pyramid.
Even when a weighty hierarchical requirements specification pyramid is called for
(and it may be essential in a large project, during elaboration, during the journey from high level to low level)
the higher levels of the requirements specification pyramid get overtaken by lower levels, and the requirements are overtaken by the solution.
Once the software is coded, built and tested, the value of mid-level specification and design artifacts diminishes.
It isn’t just that some prototypes, design sketches and test cases are redundant once the built system is running.
Or that some detailed specification and design notes are transferred into comments in code or in test cases.
It is that the situation changes dramatically as soon as the customer can see the inputs and outputs of the running solution.
The customer loses interest in the requirements catalogue and presents requirements as change requests to the built solution.
Developers and testers turn these change requests directly into changes to code and tests.
But agile development shouldn’t mean no logical models are maintained. See the footnote.
Agilists talk a lot about the importance of a flexible approach to requirements.
But in the main, they talk about functional requirements.
They don’t talk so much about the non-functional and legal requirements.
These requirements are, by their very nature, more rigid.
And they are the requirements the architect must focus on.
Even in the most agile project, it is a good idea to reverse-engineer a requirements catalogue from the on-going work –
because it gives you a vehicle for communicating with customers and users, for example, to help you discuss priorities, cost and effort projections.
The larger the project, the more people need specifications that are more abstract than code and tests.
There is a balance to be struck between working on a model or design to get it right and incremental development.
The footnote illustrates the point with a short extract from long and wide-ranging discussions with a leading agilist.
There are different ways to improve requirements capture.
You can work harder at requirements analysis, complete a deeper hierarchical requirements specification pyramid and put more effort into quality reviews of documentation.
Or you can adopt a more agile approach.
Each approach can work.
Each has sometimes failed.
Neither approach guarantees a project will end in success.
The right answer depends on your circumstance.
In practice, you usually have to do a bit of both and find the right balance between them.
This section discusses one increasingly popular agile approach to requirements capture, which is to design and code your tests before your software.
TDD means more than defining test cases and test data before development. It means coding the tests:
The resulting tests are sometimes called executable specifications.
TDD is an iterative method – code the first test – code the first bit of program - code the second test - code a bit more program etc.
So the project moves forward in very short test-code cycles. A bank of regression test data is built up.
Q) Do you have any more of a chance to write an accurate test than you do to write accurate requirements? Yes.
Text requirements statements are abstract, imprecise and ambiguous.
Tests cases and test data are concrete, precise, unambiguous and 'executable'.
TDD is highly iterative: test case - code - test case - code.
So the majority of the test and the code are accurately aligned at any point in time.
Requirements are uncovered through incremental solution development, and software designs are refactored to minimize the code written.
Q) Is TDD about unit testing or system testing? TDD doesn’t map well to either unit or system testing in a waterfall method.
It is a different approach, with something of both unit testing and system testing about it:
Is
TDD about unit testing?
TDD does involve development tools and skills.
And some developers use TDD for unit and link testing alone.
But users and requirements analysts can provide test cases for TDD.
And some agilists speak of using the TDD test cases in regression tests.
Is
TDD about system testing?
Tests cases are written before the code - to express (and incrementally develop) the requirements.
The test cases are called "executable requirements".
But TDD doesn’t map well to system testing in a waterfall method, because of its need for tests to be coded.
Reservations: Defining test cases and test data as early as possible is surely right.
And customers should help suppliers to do it.
However, there are some questions about TDD that need to be addressed:
· How far does it reduce the need for weighty requirements specification?
· How does it integrate with higher levels of system testing, and regression testing?
· How does it work on a large project with several banks of test data?
· How does it square with swift change management?
· How far does TDD increase the volume of test cases and test data in the configuration that needs to be managed?
See our paper on test-driven design.
We have chosen ten principles of a fairly general nature.
There more at the level of software engineering disciplines.
Continuous integration is promoted, especially by Martin Fowler.
“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day.
Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.
Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”
“Continuous integration might be seen as merely an elaboration of the principles in the previous three sections.
It is however a big and powerful step change from waterfall methods.
Properly done, it forces teams (customers, managers and developers) to face up to the real state of their evolving product (the software) and reinforces the rapid feedback loops at the heart of agile efficiency claims.”
“The main continuous integration practices:
• Maintain a Single Source Repository
• Automate the Build
• Make Your Build Self-Testing
• Everyone Commits Every Day
• Every Commit Should Build the Mainline on an Integration Machine
• Keep the Build Fast
• Test in a Clone of the Production Environment
• Make it Easy for Anyone to Get the Latest Executable
• Everyone can see what's happening
• Automate Deployment”
“A common mistake is not to include everything in the automated build.
The build should include getting the database schema out of the repository, and firing it up in the execution environment.
I'll elaborate my earlier rule of thumb: anyone should be able to bring in a virgin machine, check the sources out of the repository, issue a single command, and have a running system on their machine.” <http://www.martinfowler.com/articles/continuousIntegration.html>
How empty is Fowler’s virgin machine? Does it have an operating system on it? What else?
The table below lists four more principles that are reasonable - so reasonable it is hard to imagine many wanting to promoting the alternatives.
That’s why I don’t regard them as distinguishing an agile project from non-agile projects, and they didn’t make our top ten.
|
Four agile principles that
didn’t make our top ten |
Is alternative conceivable? |
|
Agile processes promote sustainable development. |
Promote burn out and
non-sustainable development? |
|
The sponsors, developers, and users should be able to
maintain a constant pace indefinitely. |
Drive sponsors,
developers, and users until they are too tired or disheartened to carry on
working? (Though of course hard deadlines must be
met, and people do work better under a bit of pressure now and then.) |
|
Continuous attention to technical excellence and good
design enhances agility. |
Attention to
technical excellence and good design is bad? gets in the way? |
|
Simplicity--the art of maximising the amount of work not
done--is essential. |
Needless complexity
is a good idea? |
Three more agile principles that didn’t make our top ten
There is also a school of agilism which is opposed to central command and control as a matter of principle. Scott Ambler says:
“Traditional development governance frameworks have had limited success because developers are intellectual workers who don’t work well under command-and-control approaches.
To be effective, a governance program must reflect the actual environment in which it is being applied, and take into consideration the people involved.
“Successful development governance focuses on enabling the right behaviors and the best practices through collaborative and supportive techniques.
It proves to be far more effective to motivate people to do the right thing instead of forcing them to do so.”
This school promotes the three principles below.
You can see where they are coming from, but they don’t feature our top ten.
Not
only designs but also practices are flexible
A view expressed in agile modelling discussions is that agility is not about any set of practices; it means being able to change your practices whenever you feel the need.
That is certainly “agile” in a sense.
But should managers measure the agility of a project by how frequently it changes its practices? The danger here is that agility descends into trial and error.
The
focus of agility is on promoting high-communication environments
This is a valuable focus. Communication is certainly a worthy topic to address.
Communication of requirements and designs is a challenge, and essential, on every software development project, be it agile or waterfall.
Poor communications cause difficulties in many projects.
The environment should not prevent communication.
Note that high communication per se can be a symptom of problems.
Above the critical mass of the agile team, we run into the “mythical man month” problem discussed in our paper on change management challenges.
Since agile methods involve prioritizing and narrowing of scope; they explicitly remove some topics of discussion.
Note that agilists deprecate some communication paths.
Some complain that adding an extra layer of communication with design authorities and enterprise architects prevents agility.
This brings us to one more agile principle.
The
best architectures, requirements, and designs emerge from self-organizing teams
This is true only to an extent and in some contexts. It wraps up three or four ideas in one sentence.
It appears to suggest a project team should resist the advice of design authorities, technical risk reviewers – and enterprise architects.
I’ll leave that for discussion in other papers.
The last three principles are not wrong.
However, they can encourage the anything-goes school in which everybody discusses everything and developers do whatever they choose.
If agile becomes everything and nothing, then it becomes recipe for methodology mayhem.
What is it saying to customers?
· We aren’t trained in recognised practices?
· We don’t have or follow consistent practices?
· We aren’t responsible for past mistakes?
Would you let your mother into a hospital run this way? Would you put your car into a repair shop run this way?
An enterprise wants agility to be objective in its theory, and useful in practice.
Anybody promoting any methodology ought to be pressed by senior managers to provide objective answers two questions:
How can I tell whether a project is applying this methodology or not?
How will I know a year from now whether my projects are more or less cost effective as a result of following this methodology?
What is an agile project? If we can’t give our managers some objective ways to distinguish agile projects from non-agile ones, then we do not deserve to be taken seriously.
If agile is important, then our managers should be able to recognise a highly agile project.
In order to place a project on the spectrum from low to high agility, managers need specific criteria they can readily assess if not measure.
“I typically start with a known set of Fairly
Good Practices, then when one of them fails, find out why and fix the problem.
The funny thing is that because I choose XP
[extreme programming] as my starting point, and because I recommend others use
XP as their starting point, I am often branded a zealot.
This makes me sad, then angry.
Angry because I don't think I ought to be
labelled a zealot just because I use the same starting set of practices for
each project – something I have seen work." J.B.Rainsberger,
If you want agile methods to have a positive and
significant impact on a large organisation, then I propose you:
·
Define
a platform of agile disciplines (like our ten imperatives) as a common starting
point.
·
Exclude
any principle that is bland, middle-of-the-road, all-things-to-all-people,
motherhood and apple pie.
·
Extend
the agile platform with advice on specific challenges and compromises (see
later discussions) that lead projects away from being highly agile.
·
Get managers' agreement that
your agile platform, challenges and compromises are taught to all, from the top
down.
You will surely agree that a project that can say yes
to all ten principles in this table is highly agile.
|
Ten
imperatives for agility |
Discussion |
|
1) If
we are going to fail, we want to fail faster |
Knowing that software projects
are difficult, and a high percentage end up failing, fail faster is good. Agile methods help us
more swiftly spotlight daft requirements, poor process and ignorance of
technologies. |
|
2) We develop iteratively, driven by priorities |
The agile manifesto
principle is: “Our highest priority is to satisfy the customer through early
and continuous delivery of valuable software. Deliver working
software frequently, from a couple of weeks to a couple of months, with a
preference to the shorter time scale.” An agile project proceeds
in increments. SCRUM speaks of 30 day increments. Some projects (probably
well into construction if not maintenance) get to work on the basis of daily
increments. |
|
3) We use prototypes to tackle risks early |
RUP and other
methods say use prototypes to tackle any tricky design risk early. |
|
4) We empower a small team (including users) to get on
with it |
The agile manifesto
principle is: “Build projects around motivated individuals. Give them the environment
and support they need, and trust them to get the job done.” This does of
course imply the team has the necessary skills. SCRUM speaks of
self-managing teams. DSDM speaks of empowered teams. Users and developers
should work cooperatively in a team empowered to make decisions. The people typically
appointed to steering groups and project boards rarely have the knowledge or
the ability to direct the development team on a day-to-day business. They should be (and are
in PRINCE2) a body to handle exceptions elevated to them. |
|
5) We maintain continuous and close stakeholder
involvement |
The agile manifesto
principle is: “Business people and developers must work together daily
throughout the project.” |
|
6) We focus on program code rather than on specifications |
The agile manifesto
principle is: “Working software is the primary measure of progress. The most efficient
and effective method of conveying information to and within a development
team is face-to-face conversation.” In XP,
analysts record requirements on cards, which are allocated to developers. Analysts
talk to the developers about what the requirements mean. Further
documentation effort is focused on what is executable, or testable. Agile
methods are light on documentation; there is little by way of functional
specification or UML. Where
code has to be documented, reverse engineer some UML and erase details until
you have something readable. |
|
7) We focus on tests rather than on specifications |
The XP practice is for
developers to write unit test code before solution code. Also, analysts specify
system test data rather than system/functional specifications. |
|
8) We iteratively refactor code written so far |
This is an implication
of iterative development promoted by Martin Fowler, Scott Ambler and Bob
Martin. Refactoring means
restructuring a process to make it more economical, more optimal. Refactoring is
usually and preferably done during a change made for requirements reasons. (If done afterwards,
there is an additional regression testing burden.) |
|
9) The customer must be willing to change and sacrifice
requirements |
The agile manifesto
puts it thus: “Welcome changing requirements, even late in development. Agile process
harness change for the customer's competitive advantage.” A DSDM guru would
put it. “What gives is what the customer gets” (Jennifer Stapleton) Given we time and/or
cash box our delivery, we (customer and supplier) must be flexible about
requirements. |
|
10) We get customers on side before we start |
The agile manifesto
doesn’t say this, but agilists ignore it at their peril, and it is a feature
of DSDM at least. You must first ensure
sponsors and customers are committed to all of the above principles. |
Remember these are imperatives for agility, not for every project.
You can set out with these imperatives, then adapt according to circumstances.
Other papers discuss difficulties that tend to undermine these imperatives, and look at applying them to a large-scale systems integration programme.
Footnote:
Does agile development preclude model-driven development?
“Graham: Suppose you agilists gathered up all user stories and notes about business rules - iron out the duplications and the conflicts - and collect them into a coherent glossary.
Won’t you'll find yourself building a domain/data model.
Are agilists against this form of documentation?
Agilist: Agilists are against spending so much time writing and refining text that software doesn't come out of the project in a timely way.
That really is the genesis of the agile movement.
Agilists got fed up with watching projects run for 2 years, collecting business rules, and then getting canceled because no one delivered any software.
Graham: Sure. My sympathy is with you there. Though I have seen sequential projects succeed.
Over a few months of requirements analysis I completed a domain/data model then handed it over to development.
The model sailed through development with no significant change over many more months.
The up-front effort was much appreciated by the development team, since it eliminated the need for iterative restructuring.
The team pored over notes I left on business rules - found those a valuable knowledge source.
Agilist: The agile approach would be to gather as many business rules as needed to make the first delivery –
opinion differs on whether after 1 week, 1 month or 1 quarter, but in that time range; and then move on to the next set of business rules.
It's called "incremental development" and it's been documented as a best practice since the early 1980s. Agilists just make a big fuss about it.
Graham: The fuss started over here about 1994, when DSDM was getting off the ground. I am convinced incremental development is the most cost effective approach.
Agilist: At the end, you may have the same domain model, and even documented in exactly the same way, but it was constructed in parallel with delivering software.
Graham: Yes often. Probably most times. And sometimes you end up with lash up.
The first implemented data model is simply wrong - some relationships are missing for example.
Changing the database schema (or rather changing the test data and the programs) is tedious and costly.
So people work around the bad structure. And live with the pain for years. Sometimes forever. Done properly, models reduce such pain.”