Welcome to a series on Agile Systems Engineering exploring the practical aspects of this emerging approach. If you didn’t see it already, check out Part 1: What is Agile, Anyway?
A faster horse
“If I had asked people what they wanted, they would have said faster horses.”Apocryphally attributed to Henry Ford1
When people trot out that quote they’re often trying to make the point that seeking user feedback will only constrain the design because our small-minded <sneer>users</sneer> cannot possibly think outside the box. I disagree with that approach. User feedback is valuable information. It should not constrain the design, but it is essential to be able to understand and empathize with your users. They say “faster horse”? It’s your job to generalize and innovate on that desire to come up with a car. The problem with the “singular visionary” approach is that for every wildly successful visionary there are a dozen more with equally innovative ideas that didn’t find a market.
Sometimes, your research will even lead you to discover something totally unexpected which changes your whole perspective on the problem.
Here’s a great, real-world example from a Stanford Hacking for Defense class:
Customer ≠ user
Team Aqualink was tasked by their customer (the chief medical officer of the Navy SEALS) to build a biometric monitoring kit for Navy divers. These divers face both acute and long-term health impacts due to the duration and severe conditions inherent in their dives. A wearable sensor system would allow divers to monitor their own health during a dive and allow Navy doctors to analyze the data afterwords.
Team Aqualink put themselves in the flippers of a SEAL dive team (literally) and discovered something interesting: many of the dives were longer than necessary because the divers lacked a good navigation system. The medical concerns were, at least partially, really a symptom. What the divers truly wanted was GPS or another navigational system that worked at depth. Solving that root cause would alleviate many of the health concerns and improve mission performance, a much broader impact.
The customer was trying to solve the problem they saw without a deeper understanding of the user’s needs. That’s not a criticism of the customer. Truly understanding user needs is hard and requires substantial effort by engineers well-versed in user requirements discovery.
In the US DoD, the Joint Capabilities Integration and Development System (JCIDS) process is intended to identify mission capability gaps and potential solutions. The Initial Capabilities Document (ICD), Capability Development Document (CDD), and Key Performance Parameters (KPPs) are the basis for every materiel acquisition. This process suffers from the same shortcoming as the biometric project: it’s based on data that is often removed from the everyday experiences of the user. But once requirements are written, it’s very hard to change them even if the development team uncovers groundbreaking new insights.
The Bradley Fighting Vehicle
The Bradley Fighting Vehicle was lampooned in the 1998 movie The Pentagon Wars2. By contrast, the program to replace the Bradley is being held up as an example of a new way of doing business.
Instead of determining the requirements from the outset, the Army is funding five companies for an 18-month digital prototyping effort. The teams were given a set of nine desired characteristics for the vehicle and will have the freedom to explore varying designs in a low-cost digital environment. The Army realizes that the companies may have tools, experiences, and concepts to innovate in ways the Army hasn’t considered. The Army is defining the problem space and stepping back to allow the contractors to explore the solution space.
System engineering for the DoD is built around requirements. The aforementioned JCIDS process defines the need. Based on that need, the acquisition command defines the requirements. The contractor bids and develops to those requirements. The test commands evaluate the system against those requirements. In theory, since those requirements tie back to warfighter needs, if we met the requirements we must have met the need.
But, there’s a gap. In the proposal process, contractors evaluate the scope of work and estimate how much effort will be required to complete the work. Sometimes this is based on concrete data from similar efforts in the past. Other times, it’s practically a guess. If requirements are incompletely specified, there could be significant latitude for interpretation. Even really good requirement sets cannot adequately capture the actual, boots-on-the-ground mission and user needs.
So, the contractor has bid a certain cost to complete the work based on their understanding of the requirements provided. If they learn more information about the user need but meeting that need would drive up the cost, they have three options:
- Ask the customer for a contractual change and more money to develop the desired functionality
- Absorb the additional costs
- Build to the requirement even if it isn’t the best way to meet the need (or doesn’t really meet it at all)
Obviously none of these solutions are ideal. Shelling out much more than originally budgeted reflects poorly on the government program office, who has to answer to Congress for significant overruns. Contractors will absorb some additional development cost from a “management reserve” fund built into their bid, but that amount is pretty limited. In many cases, we end up with option 3.
This is heavily driven by incentive structures. Contractors are evaluated and compensated based on meeting the requirements. Therefore, the contractor’s success metrics and leadership bonuses are built around requirements. Leaders put pressure on engineers to meet requirement metrics and so engineers are incentivized to prioritize the metrics over system performance. DoD acquisition reforms such as Human Systems Integration (HSI) have attempted to force programs to do better, but have primarily resulted in more requirements-focused bureaucracy and rarely the desired outcome.
I call this “requirements myopia”: a focus on meeting the requirements rather than delivering value.
Refocusing on value
It doesn’t make sense to get rid of requirements entirely, but we can adapt our approach based on the needs of each acquisition. I touched on this briefly in an earlier article, Agile Government Contracts.
One major issue: if we don’t have requirements, how will we know when the development is “done”? Ponder that until next time, because in the next post in this series we’ll dive into some of the potential approaches.
What are your experiences with requirements, good or bad? Thoughts on the “faster horse”, Team Aqualink’s pivot, or the Optionally Manned Fighting Vehicle (OMFV) prototyping effort? Sound off below!
- Also, Henry Ford was an outspoken anti-Semite. That’s not relevant to the discussion, but any mention of a hateful person in my writings will be so noted.
- If you haven’t see it, you absolutely should. It is a hilarious satire of acquisition bureaucracy.