tldrThe U.S. defense industry is the inventor of numerous software
processes and even a programming language. This series discusses the
challenges to using agile in the U.S. defense industry and some ways
agile has had an impact./tldr
I was having a conversation with some of DZone’s Zone Leaders and
the topic turned to how agile is used by defense contractors, especially
in the United States. Allen Coin, with his eye out for
interesting articles, was nice enough to suggest I try writing out some
of those experiences and my thoughts.
First, the necessary disclaimer. I work for Lockheed Martin, but everything
I write is my own opinion, on my own time, and does not represent the views
of Lockheed Martin or anyone else who works with me.
If we played a word association game, most people would probably not say
“agile” in response to either “big company” or “defense industry”. The U.S. defense
industry gets high marks for being on the cutting edge of technology, but
there is also a reputation for multi-year contracts, lots of approvals required
for decisions, and long lead times before new systems get all the way
And yet, the U.S. Department of Defense is taking agile very seriously.
In the linked presentation is a quote that says, “[s]oftware development agility
is a key contributor to Program success.”
In this first article, I want to discuss some of the reasons large programs
in the defense industry are set up the way they are, and then introduce why
agile is embraced in that environment. In future posts, I will go into more
detail on some individual challenges.
Traditional Large Programs
For years, the defense industry relied on DOD-STD-2167A, the guidebook
for software development for military applications. (Even though 2167A was
superceded by MIL-STD-498 and eventually IEEE 12207, many of us in the
industry still think in terms of “2167A development”.) That standard
has informed software development in many different ways, including the idea of
software requirements analysis (identifying “what” the software must do), a
separation between architectural and detailed design (“how” the software will
meet the requirements), implementation, and various levels of test starting
with unit test and proceeding out to a full system qualification test before
heading to the field for operational testing.
When described this way, it immediately seems like a waterfall process, and of
course waterfall is considered stereotypical. But it is interesting to note
that I have never been on a pure waterfall program; every program I have ever
been on has had some form of incremental development, with each increment
progressing at least to integration and test, and many even deployed to end
Also, within agile, we perform many of the same activities, and in the
same order. We figure out what the software must do before making it happen
(how could it be the other way around?) and we run our unit tests more frequently
and over smaller changes than our integration tests. Some activities in
engineering are logically prior to others.
So the chief point of difference between agile and “traditional” methods of
software development is the size of the increments. If we are identifying
requirements (user stories) a few at a time, it is sufficient to write them
informally, because the developer can ask the product owner what is meant,
the product owner will remember, and once they agree that the software does
what it must, the user story is complete. On the other hand, if we write many
requirements at once, we must record them formally, drive out ambiguity, and
keep them around in a document so we can be sure we implement them all and then
test them all. It is likely that the project team will change composition
during that time, so we cannot rely on anyone’s memory as to what a requirement
“meant”; everything must be documented enough to reduce risk. This is how we
get into the “contract negotiation” mode mentioned in the agile manifesto.
Two Challenges To Agile
Of course, every time we create a proposal, we include hours for all of this
documentation effort, so we know how much it costs to develop systems this way.
There are two important reasons why big documentation is considered a necessary
evil anyway. First, in the defense industry it really is very expensive to fix
issues once they are in operational testing or production. Flying someone like
me to a customer location or integration facility is expensive in time and
living expenses. Even fixing problems in an integration lab is expensive, as
lab resources are all too scarce (often multiple shifts). It is less expensive
to change a document.
The second reason is related. One of the worst things that can befall a big
program is to produce something that passes tests but does not fulfill the
mission. To prevent this, we carefully trace the software and hardware
implementation back to design elements, trace design elements back to
requirements, and trace requirements back to a concept of operations. Then we
feel confident that the system, when complete, will meet its initial concept, driven
by the missions it needs to perform. This also drives the decision to have
formal document delivery, formal reviews, and change control. In a big and
complex system, a change in one part of the system can have broad-reaching
impacts; at worst it can lead to a failure of the system in operational
Two Reasons For Agile
So if there are strong reasons to have extensive formal testing and extensive
documentation with reviews, why are so many in the defense industry embracing agile?
For me there are two main reasons. First, I have worked with many exceptional
engineers, in systems, aircraft, hardware, software, and test disciplines. On
all those programs, I have yet to see a system where it was possible get it
perfect on paper before building it. It is impossible to completely
characterize the behavior of a system from the paper description, or from a
model. The reason is simple and is rooted in information theory: to be a
completely faithful representation of the system, the model would have to be at
least as complex as the system it is trying to model. In which case, you just
built the system.
Even formal proofs of “correctness” using a model prove correctness on a
limited number of axes, subject to assumptions inherent in the model (for
example, the assumption that separate meteor strikes will not hit both the
primary and the backup computer simultaneously).
So there is always discovery later in the engineering lifecycle than we would like.
Second, there always are requirements changes as the program moves along. Sometimes
they are customer driven, as the customer refines their planned use of the
system, or responds to new opportunities or threats. Sometimes they are driven
by obsolescence, or by the need to upgrade security. If change from within
(discovery) and change from without (requirements changes) are inevitable, it
seems logical to me to use a methodology with change built in. Why not have a
set of practices that matches what we actually do, rather than treating change as
an exceptional condition?
Additionally, the security we feel we get from formal reviews and testing is
not as complete as we hope. It has value; I have seen many times where an
experienced reviewer has caught a major architectural issue early enough to fix it,
and seen many times where an experienced tester has broken a system with ease in the
lab, saving more expensive debug later. But I have also seen many times where
hundreds of pages of documents are skimmed because no one has the time to
review them in detail. There is value in taking things in small pieces just in
the improvement to our ability to concentrate on one thing at a time.
In future articles, I intend to discuss a variety of ways agile has affected the U.S.
defense industry, including team composition, planning, continuous integration, and
software change control.