Move Fast and Validate Things
Complex, critical systems programs are under real pressure to deliver faster. The fastest teams I’ve worked with get their speed from first-time quality; going fast in the wrong direction is ultimately more costly in dollars, schedule, and—most impactfully—mission effectiveness.
“Move fast and break things” was Facebook’s internal motto for years. When the thing that breaks is the ability to repost a Minions meme and the potential upside is a new strategy for keeping billions of eyeballs engaged for longer, it’s a worthwhile tradeoff.
Facebook outgrew the motto in 2014, changing to “move fast with stable infrastructure”1. Defense and other critical systems programs never had the luxury of such experimentation; on the other hand, they developed a reputation for being over budget, behind schedule, and difficult to use. That reputation is unnecessary and programs that move slowly do a disservice to the users. The pressure to deliver at the speed of relevance is real and it isn’t going away.
Instead, we should move fast and validate things2.
Speed on a development program isn’t achieved by coding faster, it’s achieved with first-time quality. “Do it right or do it twice”, as they say. Continuous validation is a core value of agile, user-centered engineering; that doesn’t mean developing a capability and then validating, it means validating the concept, the workflow, and the wireframe before any code is written so that you can be confident that the solution being built will meet the mission needs.
Rework is a schedule problem before it’s a quality problem
Every engineer knows Boehm’s cost-to-fix-defects curve: the cost of correcting a defect rises by an order of magnitude at each successive phase of development. Late requirements discovery is not a defect in the usual sense. The code works and does what the story said. The problem is that the story was wrong, and you find out at the acceptance demo when the operator says some version of ‘that’s not how we do it’. Now you’re having to rework, doubling—or more—the cost. Not to mention the frustration from the user, the dip in morale from the development team, and the loss of credibility with the customer.
The discovery is always going to happen. The only variable under your control is when. Late requirement discovery is incredibly expensive. Rework to meet those requirements takes time and money, but the biggest impact might be that options have narrowed as design decisions have solidified around the wrong assumption. Failures discovered during operational test can only be addressed through band-aids instead of intentional design decisions. This is similar to the previously-discussed requirements myopia.
Continuous validation is not expensive
I have a team on a major, software-intensive program. The system is deployed in operations and we continue to add new capabilities and adapt to the evolving mission needs and TTPs 3. Our lead human factors engineer is on site at the operations center every Wednesday, sitting shift alongside the operators or getting a few minutes with them in a small group to run through new concepts.
Three things happen on those Wednesdays:
- Workflow observation. Watching what operators actually do, especially anything which differs from the CONOPS or reflects evolving SOPs.
- Honest feedback. Users provide unfiltered engagement because they trust the person who is present and responsive week after week and they see the impact of their inputs.
- Prototype validation. Putting the next increment’s designs in front of the people who will use them, at a fidelity that’s still cheap to throw away.
None of this is exotic. It’s ordinary user-centered design practice. What makes it work is the cadence: it’s on the shift schedule, not the milestone review calendar. The team never accumulates more than a week of unvalidated assumptions. The mutual trust permeates the project: users keep an open mind when exploring innovative concepts, program leadership sees the value of the engagement and the user feedback, technical leadership trusts the inputs and discovered requirements brought back by the HFE.
It’s also not expensive. The only marginal cost is a half day of labor each week, which is nothing compared to the hundreds of full-time engineers on the project. Everything else, like UX design, has to happen anyways—except now it is more efficient because it is well-informed.
The result is that delivered software gets accepted the first time, nearly every time.
That claim, by itself, doesn’t prove anything. Every contractor says their software gets accepted. Maybe the team is unusually good, maybe the customer is unusually easy, maybe the increments are small. Fortunately, we had a…
Natural experiment
For about six months, bureaucratic issues4 beyond the project’s control cut off our access to users. The work couldn’t stop, so we built several features against our best understanding of user needs, using the accumulated knowledge of a team that had been sitting shift with these operators for years.
That’s a far better position than most programs are in, but it wasn’t quite enough given the major new capabilities being developed.
When we finally got those features in front of users, they came back with a pile of change requests, all of it traceable to not-quite-accurate assumptions and requirements we discovered too late. The most useful outcome wasn’t the lesson for our team, it was that the stakeholders could clearly see the impact; HFE and UX aren’t just for interface polish, they’re for risk reduction, quality assurance, and schedule compliance.
When the risk is realized
If you want to see what this looks like when validation is deferred not by a few months but by an entire development program, the FIM-92 Stinger is the case I keep coming back to. The system met its 60% hit requirement in developmental testing, but only half that in the hands of actual soldiers in operational test. The technology was ready, the system wasn’t. The Army’s retrospective found incomplete task analyses, poor usability of key functions, and no serious consideration of the physical and cognitive capabilities of the people carrying it.
Stinger got fixed. It cost money to rework and added extensive delays to fielding, all attributable to discovery that could have happened earlier and cheaper. Late requirements discovery on a two-week sprint costs you a sprint, it doesn’t put the whole program at risk.
What this means for practice
Put user contact on a recurring cadence, not a review calendar. A quarterly design review is not validation, it’s an audit. The cadence matters, not the hours: one afternoon a week beats a two-day site visit per quarter, because the feedback arrives while the assumptions are still soft.
Validate the cheapest artifact that can carry the question. Tabletop walkthroughs using paper design artifacts are often the sweet-spot. If the answer changes what you build, you want it before the build, and you want the thing you’re showing to be disposable.
Treat user access as a true program risk. Ours lapsed and the associated cost was realized. User access benefits everyone so both contractors and program offices should demand it. Sometimes that access is genuinely impossible; user proxies are better than nothing, but that is still a risk that should be identified and results in assumptions that should be clearly stated, acknowledged, and validated as soon as practicable. Field trips help too, even when they aren’t formal research.
Measure first-time acceptance. Track change requests and sort them between defects and late discovery. The second category is validation debt, in objective units that turn an HFE/UX argument into a schedule, cost, and risk argument.
Summary
Speed and rigor aren’t in tension. Rework is what makes programs slow, and validation is what prevents rework. The teams that move fastest are the ones that are never set back more than a week when they find out they were wrong.
What’s the most expensive thing your program discovered too late? What actually blocks you from getting in front of your users? How do you track and categorize defects? Share your experiences in the comments below.