Agile documentation
(minimising the overhead)
This page is published under
the terms of the licence summarized in the footnote.
Integrity
or Agilitiy? You can’t maximise both. To ensure integrity you need careful change
management, but that slows you down, and makes some changes infeasible. My
interest is the tension or balance between integrity (requiring thorough system
description and strict configuration management) and agility (requiring speed
of system adaptation/evolution). It seems isolation of silos is death to
integrity but essential to agility.
There are practical reasons
why most human and computer activity systems will never be documented so
completely that change management is the wholly mechanical and reliable process
that naïve managers and customers might hope. How to optimize the change
management effort? This paper explores the following documentation principles.
Divide
a configuration between stewards
Minimise
relationships between configurations
Balance
proactive and reactive effort
Remember
the operational system is paramount
Raise
the level of traceability
Automate
documentation where possible
A configuration steward is
the individual who looks after the integrity of a configuration. When a change
request is made, impact analysis always depends at some level on the knowledge
that a human being holds about the items within a configuration and their
inter-dependencies.
Proposal: Given a
configuration where H is larger than M, divide it into substructures, each
maintained by one configuration steward whose role is to ‘know’ that
substructure well enough to do impact analysis on it.
If a configuration steward
cannot maintain their own configuration without continually corresponding with
other configuration stewards, then the benefit of dividing the configuration is
defeated. The trick of course is to do what software engineers are taught to
do.
Proposal: maximise the
internal cohesion of a configuration (so closely related things are looked
after by the same configuration steward) and minimise the coupling between
separated configurations (so configuration stewards don’t have to worry about
each other).
Strike a
balance between investing in:
·
Proactive
documentation-time effort, documenting every item and inter-item dependency in
anticipation of a change request.
·
Reactive
change-time effort, applying human knowledge and intelligence at change time to
spot dependencies.
Those
involved in change time effort do want enough documentation to point them in
the right direction. But documentation-time effort is dull work, and the effort
is unfocused, so records will be flawed, so don’t go overboard with it.
In the real
world, operational systems are more important that system descriptions.
Business managers and users don’t care if operational systems are out of step
with higher level descriptions. But they care a great deal if the operational
system stops working.
To business
managers, change management of the operational systems (in use by many) is far
more critical than change management of more abstract descriptions (used only
by architects and designers).
The ultimate
description of an operational systems is the
bottom-level one, composed of statements that actors (machine or human) read in
order to carry out activities in an operational system. If this description is
not right, then the system will not work
Proposal:
Documentation of higher level system descriptions after the fact - by some kind
of reverse engineering - is a good thing, and sometimes the best we can hope
for.
The Single Point of Definition (SPOD) principle tells
us to document an item or relationship once and once only. That minimises the
effort to change the documentation, and minimises the risk of discrepancies in
documentation.
Consider for example how record
inter-item dependencies within a configuration?
·
In the structure
of items: by grouping inter-dependent items together, perhaps under one node in
a hierarchical structure.
·
In the items
themselves: as attributes (foreign keys, pointers, invocation statements) that
refer to related items.
·
In external
traceability records: in a configuration register or traceability table aside
from the place where items appear or are catalogued.
Some standards advise us to
record each dependency in more than one of these ways
Consider for example a web site whose pages connected
by hyperlinks. These links can be seen in the bottom level description – in the
documents themselves. To aid change impact analysis, you might complete a
higher level description, a table showing web page names in rows and columns,
links between them in the cells.
|
Home page |
Web page B |
Web page C |
Web page D |
|
|
Home page |
x |
Links to |
Links to |
Links to |
|
Web page B |
Links to |
x |
|
Links to |
|
Web page C |
Links to |
|
x |
Links to |
|
Web page D |
Links to |
|
|
x |
The trouble is that this
higher level description duplicates information in the bottom-level description
(the web pages). This runs counter to the SPOD principle. Each higher level of description
increases the documentation workload, and the risk that records get out of
step.
The Zachman Framework
suggests maintaining parallel descriptions at five levels of abstraction from
the operational system. Has anybody maintained the five levels of abstract
specification that the Zachman Framework implies? And maintained traceability
records between each level and its adjacent level(s)? That would be a huge
task.
That documentation burden may
be excessive, but to specify systems, we do describe them from several
viewpoints and at several levels of abstraction. Is it practical to ensure all views
and levels of system descriptions are perfectly aligned? Our knowledge of how
best to maintain abstract descriptions in step with implemented systems remains
weak.
Proposal: If you have a lower
level description, and you can manage items and their relationships in it, then
that may be enough. Don’t maintain higher-level descriptions unless they prove
helpful. Allow old ones to be archived rather than maintained.
It is normal to produce
higher level system descriptions (say, requirements catalogues, logical data
and process models) before lower level descriptions. The higher level
descriptions help in analysis and design. They are milestones on the way to
fully describing the items and inter-item relationships in the operational
system.
In my experience, many
higher-level systems descriptions are side-lined after they have been refined
into lower level and more concrete descriptions. SPOD is one reason, but there
are others.
·
The lower level
description is such a straightforward elaboration of the higher level
description, there is no value in retaining the higher level one.
·
The lower level
description is such a radical transformation or elaboration of the higher level
one that the relationship between them is obscure, and it is impractical to
produce and maintain traceability relationships.
·
The higher level
description was misguided or supplanted.
The trick is to maintain a
higher level description that is more abstract by an order of magnitude than
lower level description, but can be traced to it.
Proposal: edit higher-level
descriptions so that they fit to abstract from lower ones. Once you have
completed a lower level description, return to a higher one, simplify it,
restructure it to reflect the lower one, and insert references to the lower
level for detail.
A system description is
usually divided into parallel views. The table below is intended to illustrate
how a small information system might be described in three distinct views. Each
view is a configuration that should be internally consistent.
|
Configuration |
Business process models |
User interface design |
Database schema |
|
Substructure |
10 flowcharts |
10 web pages |
10 tables |
|
Item |
100 process steps |
100 fields and controls |
100 fields |
In the operational system
these views are tightly interrelated. The processes capture data items at the
user interface and store them in database fields. In theory, we can relate the system
description views in the same way. We can document which database fields appear
in which web page fields – and which of those are used in which process steps.
The potential number of inter-item relationships (100 * 100 * 100) is huge.
Where configurations are
highly interdependent, change management becomes a serious obstacle to change.
You could bring in a 3rd party to build and maintain traceability records.
|
Configuration B |
Substructure B1 |
Substructure B2 |
||
|
Configuration A |
Item B11 |
Item B12 |
Item B21 |
Item B22 |
|
Substructure A1 |
|
|
|
|
|
Item A11 |
Depends on |
Depends on |
|
Depends on |
|
Item A12 |
|
Depends on |
Depends on |
|
|
Substructure A2 |
|
|
|
|
|
Item A21 |
|
|
|
|
|
Item A22 |
|
|
Depends on |
|
The trouble is that a 3rd
party has no ownership of and perhaps little interest in the configuration.
They will make mistakes and fail to keep the traceability records up to date.
The change management process will become not only immensely bureaucratic, but
error prone.
How can the configuration
stewards reduce the bureaucratic overhead of maintaining traceability records
between their configurations? They can raise the level of traceability to a
higher level of composition, confident that each configuration steward will be
able to do impact analysis well enough in their head when a change happens.
You can raise the level of
traceability records in a lop-sided way. For example, you can record an item in
configuration A depends on the whole of configuration B. This means you won’t
have to update the traceability record every time items in B are reorganised.
More likely you’ll raise the level of traceability records in a balanced way by
mapping between substructures rather than bottom level items.
|
|
Configuration substructure
B1 |
Configuration substructure
B2 |
|
Configuration substructure
A1 |
Depends on |
Depends on |
|
Configuration substructure
A2 |
|
Depends on |
Of course, an implication of this
approach is that the architect should group closely-related items into
sub-configurations or substructures. You can refactor (reorganise) existing
configurations by clustering related items into middle-level substructures that
are (in the classic manner) internally cohesive but loosely-coupled one with
another. This is the principal way in which we make large configurations
manageable.
Proposal: Raise the level of
traceability. If mappings between configurations too complex to make
maintenance possible, then group items into substructures, so you can raise the
level of traceability.
Enterprise Architecture is
the most abstract level of system description. Software/application architects
manage the configuration of fine-grained components within an application.
Enterprise architects manage the applications portfolio in which the items are
whole applications. In theory, there is a continuous top-to-bottom
decomposition structure. In practice, the two levels of configuration are
rarely if ever recorded in the same repository. The application maintenance
teams regard EA as an ivory-tower activity of no relevance to them.
We need a way for application
teams to document application structure behaviour that automatically populate the EA repository. We need to record screen flows
and map user interface functions to data in a way that is independent of user
interface appearance and implementation detail. The resulting models will be
useful as a description of legacy system behaviour and/or target system
behaviour. And in completing these models, application designers will
automatically document the bottom-level functions and data types. So enterprise
architects can connect to that documentation in EA level descriptions.
Proposal: Automate
documentation. Customers and employees can maintain their own data records.
Deployment of machines and applications can be automatically recorded. Automate
import/export between configuration management are related tools. Reduce the labour
of mapping between levels of specification by investing in automation of
forward and reverse engineering between the two levels of abstraction.
Ref 1: Papers on Scale,
Complexity and Change in the Library at http://avancier.co.uk
Footnote: Creative Commons
Attribution-No Derivative Works Licence 2.0
Attribution: You may copy,
distribute and display this copyrighted work only if you clearly credit
“Avancier Limited: http://avancier.co.uk”
before the start and include this footnote at the end.
No Derivative Works: You may
copy, distribute, display only complete and verbatim copies of this page, not
derivative works based upon it.
For more information about
the licence, see http://creativecommons.org