Agile change management

This page is published under the terms of the licence summarized in the footnote.

Integrity or Agilitiy? You can’t maximise both.

To ensure integrity you need careful change management, but that slows you down, and makes some changes infeasible.

My interest is the tension or balance between integrity (requiring thorough system description and strict configuration management) and agility (requiring speed of system adaptation/evolution).

I propose isolation of silos is antagonistic to integrity but necessary to agility.

Change management processes are necessary.

You cannot produce a useful target architecture description unless you have processes for changing requirements and solution visions during the architecture development process.

You will never realize that target architecture without processes to manage changes to your existing systems.

But you need more sophisticated views of change management than standard processes give you.

Change management time and costs

Traditional change management standards

Proposals for more agile change management

Ways to minimize the cost of change management

Ways to judge where change management can be light

Ways to manage a large scale and complex configuration

Proposals for agile scope management

Proposals for agile configuration management and version control

Proposals for agile requirements traceability

Proposals for agile change control

Change management time and costs

Change management processes are designed to ensure quality is maintained and to reduce risk.

Managers (in both enterprises and service providers) are keen on change management processes, which are seen as the way to limit "scope creep".

But the effect of heavy-weight change management practices on time and cost are often underestimated.

Change management processes can add hugely to the time and cost of any creative activity, because they tend to:

Increase the organization size by adding management and administration overheads
Slow progress by introducing bureaucratic processes and delays
Disable a project team from accepting requirements and improving their products
Hinder the delivery of solutions customers really want.

The table below contains some formula that (though invented and unproven) help to explain the cost of change management:

This property	Is derived thus
Item change cost	(Number of dependencies on the item * configuration complexity) + process overheads
Configuration complexity	Total dependencies between items divided by total items
Process overheads	Item failure worry * number of change approval authorities.

Of course change management processes are needed, but you must understand their implications for time and cost.

You must know when/where/how to be agile.

This paper uses the formulae above to propose ways make change management more agile.

Agile change management might seem an oxymoron.

This paper offers proposals for agile scope management, agile configuration management, agile requirements traceability and agile change control.

If you are any enterprise, or any supplier working to specify, build and test solutions for an enterprise, you must explicitly and repeatedly decide where you want heavy change management processes, and where lightweight processes will suffice.

You must balance the costs of applying heavyweight processes against the risks of applying lightweight processes.

And to do this, you must understand the various times and places that change management should be relaxed - where you must deliberately allow related configurations to get out of step.

Change management domains and styles

Change management is about the monitoring and control of changes to configurations.

But not all configurations are the same.

Software configurations differ from hardware configurations.

System description configurations differ from operational system configurations.

The management of large and complex configurations differs from the management of small and simple configurations.

In the world of IT services management, change management addresses the analysis and implementation of changes to the configuration of an operational system.

There is a network of machines on which software packages are deployed.

Change management processes tend to assume that configuration items can be seen and identified, every configuration item can be documented, every dependency between those items can be documented.

Strict configuration management is expected.

In the world of software architecture, change management addresses the analysis and implementation of changes to software, test cases and descriptions of those.

The size and scope of the configuration items is relatively arbitrary.

A software package can be divided into hundreds or thousands of extremely fine-grained elements.

The software elements are connected in an abstract message passing network – a mix of local and remote communication.

Above the executable software are many views and layers of human-readable description.

At the highest level, the most abstract enterprise architecture descriptions can include human roles and processes.

These descriptions are of various kinds and varying levels of quality.

Strict configuration management is not expected.

What assumptions underlying your change management processes?

Heavyweight	Lightweight
A single change management board willing and able to process every change request.	Lower level units are empowered to accept changes.
There is a single indivisible configuration of items.	There are several loosely-coupled configurations, or several levels of configuration description.
All inter-item dependencies are equally important.	Some dependencies can be ignored, at least temporarily.
Complete integrity between all items is paramount.	Dis-integrity is acceptable.
Changes to requirement specifications need the same heavy-weight control as changes to live systems.	Changes to requirements can be accepted without painstaking impact analysis.
It is both desirable and practical to maintain perfectly accurate cross-references between specifications at every level of abstraction.	Higher level specifications of implemented systems can be out of date.
You need to record dependencies between configuration items twice – both within the items and in external traceability records.	Dependencies between configuration items are visible in those items (not recorded separately)
Processes that work for small a configuration (say 500 items) work equally for the configuration that is a complete enterprise architecture.	Different kinds of change management process are followed in different situations.

The Agile Traceability paper suggests that while you might perfect maintain the internal consistency a small and simple configuration.

You cannot do this perfectly in a large and complex system composed of many inter-related configurations.

Ways to minimize the cost of change management

The formulae above suggest ways to minimise change management costs.

You can:

Reduce both impact and risk of each change - by keeping changes small and reversible.
Reduce the scope of the overall change - by keeping projects short.
Reduce the scope/size of a configuration - by modularization and keeping projects and teams small.
Reduce effort to maintain descriptions that are not-critical to the working system (see the Agile Traceability paper)
Reduce the obscurity of the configuration structure - by making it visible and sharing knowledge of it.
Reduce the remoteness of change approval authorities - by empowering architects to make decisions.

This list unintentionally paraphrases some principles of agile development.

Some reappear in proposals below.

Ways to judge where change management can be light

Change management involves elements of risk management.

All the systems you develop or maintain are imperfect.

No change will remove all flaws.

Every change carries a risk of new flaws, or exposing old flaws.

There are different kinds of items and interdependencies, with different degrees of significance for risk:

Run-time system dependencies - item B cannot work without item A.
Design-time or architecture dependencies - item B may require changing if item A changes.
Traceability relationships between structures at different levels of abstraction – item B is a realization of a more abstract item A.

The formulae above suggest conditions that imply heavier or lighter change management, listed in this table.

Question	Lighter change management	Heavier change management
What risks does a change bring?	New flaws are tolerable, can be lived with and corrected later.	New flaws are intolerable, the potential damage is large.
How dependent is the system on the items’ correctness?	Ambiguities and flaws are inevitable in descriptions of systems. Humans never follow specifications slavishly. Humans can detect flaws and can correct or work them.	Flaws are relatively intolerable in working systems. Computers follow programs slavishly. Computers cannot correct or work around flaws in the code.
How mature is the system?	At the start there is uncertainty, requirements and designs are volatile Room must be allowed for creativity and radical change. The configuration is small.	As a system nears completion there is certainty and stability. The room for creativity and change is restricted. The configuration is large.
How obscure is the configuration structure?	The structure and items in it are highly visible, You can see what the system contains and how are items are related.	The structure and items in it are hard to see. No individual knows much of what is in the system.
How remote are the change approval authorities?	You are empowered to make decisions about changes,	Decisions about changes must be referred for approval to higher authorities or external agencies.

Don’t think lighter change management means that inconsistencies, for examples in requirements specifications, don’t matter.

They do matter.

We should always strive to ensure the integrity of a requirements catalogue, a use case definition, a data model, whatever.

But we can’t justify applying the same weight of change control to a data model as to a live system.

The quality and integrity of specifications should be maintained through education, professionalism, personal responsibility and peer review.

Ways to manage a large scale and complex configuration

Some configuration management standards assume there are only two levels – the configuration and the items it contains.

The obvious strategy for managing a large configuration is to create several levels of configuration and/or loosely-coupled configurations.

You can use hierarchical decomposition to create a more manageable structure.

You may divide a large configuration in several ways, for example between different

subsystems or lines of business

domains - business, applications, technology.
levels of abstraction or management hierarchy.

You can allow some degree of inconsistency between items within related configurations, at least on a temporary basis, and perhaps over the long term.

Your aim will be to:

Maintain perfect consistency within each small configuration – at least when anybody uses it.
Allow each small configuration to advance without being perfectly aligned at all times to all others

You can keep related configurations loosely-coupled by limiting the points of correspondence between them.

You can do this by a combination of:

Skillful separation of concerns, so there are relatively few dependencies between items in two configurations
Raising the level of traceability, as described in the Agile Traceability paper.

Raising the level of traceability does not remove dependencies between lower level items – it removes only the records of those dependencies.

It leaves you to discover the lower level dependencies when you do impact analysis – and to ignore them if you wish.

You can delegate the internal integrity of coarse-grained items to individuals.

You can maximise the use of brain power by registering items that are as large as one person can manage.

You allocate responsibility for the integrity and testing of one item to a willing and able individual (while striving to avoid a blame culture).

You can avoid deep configuration hierarchies by dividing one configuration into levels: a top-level configuration of large items and lower level configurations of small items.

(E. g. see the interplay between IT and IS architecture discussed in the previous paper. )

A large enterprise usually needs a deep management hierarchy, and this hierarchy will be reflected in levels of architecture description.

You may maintain descriptions of an enterprise, each division, each department.

You may maintain descriptions of a programme, a project, a system release.

You need your change management organisation and processes to be correspondingly hierarchical.

The highest level of change management organization and processes should be concerned with only the highest level and most generic architectural visions.

Most of the changes requested and processed at the bottom level of change management are irrelevant and should be invisible to higher levels of management.

A challenge is to make sure lower levels recognize the issues that need to be reported upwards, and for higher levels to respond appropriately.

Keep the management hierarchies as shallow as possible.

Make sure they work bottom-up as well as top-down.

Top-down-only control is rarely effective (outside some operations of armed forces?).

Higher levels must gather feedback from lower levels.

See also the paper on very large-scale enterprise architecture.

Proposals for agile scope management

The traditional waterfall project view is.

"Scope management requires in depth study of complete requirements of the project, including the processes needed to ensure that the project will be accomplished successfully.

Project scope management is primarily focused on defining and controlling all the requisite work areas of the project.

The main processes included are Scope Definition, Scope Planning, Scope Verification, Work Breakdown Structure and Scope Control. "

The agile project view is somewhat different.

This table outlines two extreme approaches.

Extreme waterfall approach - working assumptions	Extreme agile approach - working assumptions
The project scope starts as the first-given requirements scope	The project scope starts as the first-given time box and/or cash box
The supplier resists changes	The supplier welcomes changes
The customer gets what they asked for	The customer gets what they most need, and what supplier can deliver to the time/money
Customer and supplier agree changes by formal change approval authorities and processes	Customer and supplier agree changes by continually negotiation, trimming requirements to fit the given time/money.
A hierarchical requirements specification pyramid should be defined and maintained	A hierarchical requirements specification pyramid should not be defined and maintained
Change management and traceability must be heavy	Change management and traceability processes must be slick

The extreme waterfall approach is to build the requirement configuration as hierarchical specification pyramid and maintain it under change control with the customer.

Each requirement is elaborated at several levels of abstraction.

From high-level business visions and objectives though a requirement catalogue, process models and data models, to screen designs and program flow charts, class diagrams and interaction diagrams.

Every item is cross-referred to all related items.

This adds a considerable overhead to projects that are large and/or receive many change requests.

Ironically, introducing iterative development into a waterfall approach adds even more cost and delay, since it encourages customers to raise more change requests, without relaxing the original scope.

Iterative development is not necessarily agile.

The extreme agile approach is to maintain under change control with the customer only a high-level vision (the top of the specification pyramid).

E. g. Agile method DSDM says to maintain only high-level requirements.

Scott Ambler has a good summary at http://www.agilemodeling.com/essays/changeManagement.htm.

There is no single formula for success.

What works on a small agile project does not always work on a large waterfall project, and vice-versa.

Proposals for agile configuration management and version control

Configuration management is as much art as science.

The effectiveness and efficiency of configuration management processes depend on human judgements.

Judgements about configurations and items, about how big they are, and about how far (or how long) discrepancies between them are acceptable.

Don’t expect to complete/perfect the requirements specification

The lighter the specification, the easier it is to change.

“Early on – agility trumps completeness. ” Joshua Bloch.

In iterative development, you should start the first solution without attempting to complete the specification.

You should deliberately freeze or throw away some specification and design products rather than carry on maintaining them past their sell-by date – once the product is coded and tested.

If your customer demands a complete and perfect specification, then do this when the solution configuration is reaching stability – and budget for doing that.

Minimise effort to maintain external traceability records

Who uses them? When? How? Why? Often, configuration items are published in a form that anybody can read, can find and follow the dependencies between items.

It is better, if you can, to

· Trace dependencies by following internal cross-references using a “search” or “find next” tool

· Sort and list the items in a configuration any attribute of interest

· Automatically generate external traceability mappings (when needed) from internal cross-references.

Divide to conquer

Do you really need to maintain everything in one large and complex configuration? Can you divide it into smaller loosely-coupled configurations in the ways described above?

Do you really need to store everything in one configuration management database? Using one database does not make its content internally consistent.

In practice, you may choose to manage different configuration baselines in different repositories.

It is normal practice to manage requirements statements in one database and software components in another database.

Proposals for agile requirements traceability

Most discussion of traceability centers on mapping between items in a requirements configuration and items in a solution configuration.

This is done to:

Help a customer check the delivered solution meets declared requirements.
Help a supplier prove compliance – to show a requirement has tests and has been tested.
Validate the coherence of abstract requirements configuration – which is not itself executable or testable.

This section challenges the conventional advice that you should record all dependencies between requirements items and solution items.

Making a contractual commitment to “full traceability” up and down a hierarchical requirements specification pyramid can be a hostage to fortune.

Initial requirements are rarely accurate or precise.
In any case, the requirements will probably change.
In the end, the solution will be assessed against the real need, documented or not.
The happy customer who has been participating in the design and development won't look at traceability records at all.
The unhappy customer will use traceability records to find faults and inform arguments about what went wrong.

Ways to decouple requirements and solutions configurations

Record traces in external traceability records rather than in internal cross references.
You don’t want to have to read the code of a software component to find the requirements it meets.
Record traces indirectly, via a third configuration.
E. g. via test cases.
Temporarily freeze one configuration (requirements or solution) during evolution of the other, and align them later.
Freeze a configuration and archive it: If developers have stopped using detailed requirements and design documents, why are you maintaining them?

DSDM suggests: “keep only high-level requirements under formal change control”.

This limits formal traceability by keeping middle and low-level specification items out of traceability mappings.

There are at two contrasting ways to document requirements traceability:

Trace requirements to solution components to test cases (the traditionally-recommended waterfall development sequence)
Trace requirements to test cases to solution components (a modern agile development sequence).

I suggest: map high-level requirements to test cases directly (rather than through intermediate specification and design products of questionable long-term value) and map test cases to software components.

This proposal comes with an implication.

Define test cases early.

Do not wait to the end of the project life cycle to define test cases.

Proposals for agile change control

Minimise the time for change control

Run the process for change approval concurrently with the process for the project manager providing the executive sponsor with the project impact.

Establish SLAs, e. g.

24 hours for the change board to approve the cost/time estimate.
If the process extends beyond 24 hours the request goes in the parking lot.
48 hours for the project team to provide a cost/time estimate for the request and impact to project baselines.
If the approval is received from the business unit and you are in a lock down period or there is significant project impact to milestone dates, the request goes in the parking lot.
If the request does not get side tracked into the parking lot, it is incorporated.

Minimise the cost of change control

Agilists argue that in practice, the most efficient projects are ones where customers accept requirements flexibility.

Agilists consider heavy change control to be harmful, a hindrance to progress.

They say to welcome change within reason.

The agile manifesto says: "We value responding to change over following a plan".

This section makes some more specific proposals:

Set up three of four different change authorities for different levels of the requirements specification pyramid.
Classify change requests according to the level of approval they need.
Use the requirements catalogue as focus for customer-level change control.
In fact, limit it to including only those requirements that require formal change control, that cannot be changed without customer and supplier agreement.
Allow that lower level of the requirements specification pyramid can be extended and improved without the overheads of formal change control.
Exclude informal documentation items from formal configuration management and change control.

Use the change control process to manage customer expectations

Use the change control process to your advantage.

Define what triggers change control.

This is an important step towards management of customer expectations.

Triggers should include not only events that create new additional requirements, be they:

changes to the business processing requirement or business rules
new “must have” requirements that reduce what else can be done
new acceptance criteria that change the baseline
but also events that change the prioritisation of requirements
delay allocated resources working on already known requirements, and thus reduce the requirements that can be met in the given time and budget:
users not attending a scheduled session or not replying to request for information in a timely fashion,
change to the deadline for the project, or current time box
change of key member of customer or supplier staff.

Use the change control process to re-prioritise

Identify the risk level of requirement.

If there is a need to prototype part of the requirement to reduce risk then do it.

But beware: expect conflicts (even from one domain expert at different times).

Users are rarely the best people to abstract requirements from solutions.

Interpret customer/user needs, what’s the real requirement? Do not detail requirements at a level of detail that becomes a hostage to the change control process.

Analyse the requirements.

Map requirements to higher-level business vision, strategic or objective statements.

Extend the requirements catalogue if need be, raising a formal change request where appropriate.

Ask the sponsor for permission to remove or lower the prioritisation of requirements which meet no higher-level objective.

Footnote: Creative Commons Attribution-No Derivative Works Licence 2.0

Attribution: You may copy, distribute and display this copyrighted work only if you clearly credit “Avancier Limited: http://avancier.co.uk” before the start and include this footnote at the end.

No Derivative Works: You may copy, distribute, display only complete and verbatim copies of this page, not derivative works based upon it.

For more information about the licence, see http://creativecommons.org