Should It Always Be Agile?

Agile methodology was largely organized as a reaction to the perceived disadvantages of the Waterfall model of project management, and the various kinds of impact (personal, professional) it had on software developers. But the landscape at the time was complicated, and the reaction has been an overreaction. Agile software development is really great, and it has become a necessary component for the industry, and we write about it a great deal here on this blog. But there is more to be said about the systems that came before, and the ways they might be blended with agile, the New Kid on the Block, to achieve even greater things.

In engineering complex systems, the best designs can usually be achieved by considering and manipulating the properties of the system as a whole, rather than focusing only on smaller components and their interfaces. John Carmack gave a 2017 tech talk at UMKC with some excellent and detailed practical examples. The field studying this level of design is known as systems engineering, and like agile methodology, it has been credited with creating better designs, often for systems which involve software components, or indeed are complex software systems. But how can systems engineering, with its emphasis on holistic requirements management and its target output of a complete design specification, work with agile methodology, with its emphasis on iterating in short sprints and its target output of a (perhaps, for now, partially) working product?

Systems engineering emerged in the middle of the 20th century in response to the increasing complexity of engineering projects. The inadequacy of existing tools led to the development of models for transforming high-level customer requirements into complete design specifications for components from diverse technical disciplines in a principled, quantitative way. Using approaches such as Waterfall and the V-Model, engineering teams can proceed into the design of specific components within their disciplines, confident that the effects on the overall system design are being accounted for in a rigorous framework.

As software systems in their own right grew increasingly complex, it was natural to apply these models to their design as well. By the late 1990s, however, the limitations of such approaches in the context of software development were becoming apparent. Agile was born out of the opportunity to create better software—that is to say, to create software that better addresses the customer’s needs, that is more robust, and in less time—by way of an iterative approach incorporating continuous verification and validation.

A major reason that this worked (and that software has, ever since, been the home turf of agile methodology) is that software is uniquely malleable. In the mechanical and electrical domains, continuous integration would involve a rather prohibitive amount of manufacturing retooling, and the analogue to an over-the-air update is, in the best case, a recall. Agile relies fundamentally on relative ease of refactoring to adapt to changing system specifications, and software is easy to refactor. As a result, agile methodology has been spectacularly successful as a project management strategy for software development.

However, the agile approach does exist somewhat in opposition to the principles of systems engineering. This is perhaps unsurprising, given that agile arose in reaction to traditional systems engineering methods. Agile tends to shift attention from the system as a whole to smaller issues, and there is relatively low visibility over the overall project outside of sprints. Tight iterations and strong customer involvement more than compensate for the weaker system-level awareness in terms of the design outcome, in domains where the “easy to refactor” assumption holds. However, this still creates significant drawbacks in the context of a larger engineering project which is (necessarily) being managed under a traditional systems engineering model, notably in requirements verification and in time and cost estimation.

Erik Puik and Darek Ceglarek illustrate the situation nicely in a 2018 conference paper using what they call the Axiomatic Maturity Diagram (AMD). This model places the two axioms of axiomatic design, the Information Axiom and the Independence Axiom, on two axes of a graph, and the development process proceeds following some path from the bottom left corner to the top right corner. The axiomatic design methodology itself insists on satisfying first the Independence Axiom, then the Information Axiom, i.e. moving across the bottom and then up the right side of the AMD. A typical Scrum project, by contrast, drives development up the Information axis (that is, to a working implementation that can be tested as soon as possible), with potentially many revisions back down and up that axis to reach a mature design along the Independence axis. This process of zigzagging across the graph represents inefficiency, in the form of conceptual revision and refactoring.

The solution suggested by Puik and Ceglarek is a strategy in which both design methodologies are applied, with the agile rules attenuated at the early stages, and the systems engineering rules attenuated in the later stages. Their guidelines are rather vague, but the general idea is to establish some level of maturity and decoupling in the requirements before proceeding toward a MVP implementation and iterating on the design sprint by sprint. In the context of integrating a software component, being managed as an agile project, into a larger systems engineering project, that level of maturity would be determined by sufficient decoupling of those requirements which interact with other parts of the system.

In general, blending a systems engineering mindset into the initial design phase of an agile project is not only possible, but potentially a great early boost on the path to a good design. This is especially true when the requirements are being driven down by a larger complex system design into which the software is to be integrated. Although the success of agile in the software domain is often cast as a contrast to traditional systems engineering, we should take care not to throw the baby out with the bathwater.