• Home
  • Category: View All Articles

Agile SE Part 3: Agile Contracts and the Downfall of Requirements

Welcome to a series on Agile Systems Engineering, exploring the practical aspects of this emerging approach. If you didn’t see it already, check out Part 1: What is Agile, Anyway? and Part 2: What’s Your Problem?

The antithesis of agile

Requirements are a poor way to acquire a system. They’re great in theory, but frequently fail in practice. Writing good requirements is hard, much harder than you’d think if you’ve never had the opportunity. Ivy Hooks gives several examples of good and bad requirements in the paper “Writing Good Requirements“. Poor requirements can unnecessarily constrain the design, be interpreted incorrectly, and pose challenges for verification. Over-specification results in spending on capabilities that aren’t really needed while under-specification can result in a final product that doesn’t provide all of the required functions.

If writing one requirement is hard, try scaling it up to an entire complex system. Requirements-based acquisition rests on the assumption that specification and statement of work are complete, consistent, and effective. That requires a great deal of up-front work with limited opportunity to correct issues found later. A 2015 GAO report found that “DoD often does not perform sufficient up-front requirements analysis”, leading to “cost, schedule, and performance problems”.

And that’s just the practical issue. The systematic issue with requirements is that the process of analyzing and specifying requirements is time consuming. One of the more recent DoD acquisition buzzphrases is “speed of relevance”. Up-front requirements are antithetical to this goal. If it takes months or even years just to develop those requirements, the battlefield will have evolved before a contract can be issued. Add years of development and testing, and then we’re deploying last-generation technology geared to meeting a past need. That’s the speed of irrelevance.

Agile promises a better approach to deliver capabilities faster. But we have to move away from large up-front requirements efforts.

Still from Back to the Future Part 2 with subtitles changed to: Requirements? Where we're going, we don't need requirements!

Agile contracting

Traditional requirements-based acquisition represents a fixed scope, with up-front planning to estimate the time and cost required to accomplish that scope. Pivoting during the development effort (for example, as we learn more about what is required to accomplish the mission) requires re-planning with significant cost and schedule impacts. The Government Accountability Office (GAO) conducts annual reviews of Major Defense Acquisition Programs (MDAPs). The most recent report analyzing 85 MDAPs found that they have experienced over 54 percent total cost growth and 29 percent schedule growth, resulting in an average delay of more than 2 years.

Defense acquisition leaders talk about delivering essential capabilities faster and then continuing to add value with incremental deliveries, which is a foundational Agile and Dev*Ops concept. But you can’t do that effectively under a fixed-scope contract where the emphasis is on driving to that “complete” solution.

The opposite of a fixed-scope contract is a value stream or capacity of work model. Give the development teams broad objectives and let them get to work. Orient the process around incremental deliveries, prioritize the work that will provide the most value soonest, and start getting those capabilities to the field.

triangle with vertices labeled "SCOPE", "COST", and "TIME", the center is the word "QUALITY"
Project Management Triangle

“But wait,” you say, “doesn’t the project have to end at some point?” That’s the best part of this model. The developer’s ‘fixed’ cost and schedule keeps getting renewed as long as they’re providing value. The contractor is incentivized to delivery quality products and to work with the customer to prioritize the backlog, or the customer may choose not to renew the contract. The customer has flexibility to adjust funding profiles over time, ramping up or down based on need and funding availability. If the work reaches a natural end point—any additional features wouldn’t be worth the cost or there is no longer a need for the product—the effort can be gracefully wrapped up.

You may be familiar with the project management triangle1. Traditional approaches try to fix all of the aspects, and very often fail. Agile approaches provide guardrails to manage all of the aspects but otherwise allow the effort to evolve organically.

Agile requirements

The most important aspect of agile approaches is that they shift requirements development from an intensive up-front effort to an ongoing, collaborative effort. The graphic below illustrates the difference between traditional and agile approaches. With traditional approaches, the contractor is incentivized to meet the contractual requirements, whether or not the system actually delivers value to the using organization or is effective to the end user.

Block diagram showing acquisition models. Traditional acquisition model includes using organization defining need to acquisition organization writing requirements for contractor organization delivering system to using organization deploying system to end users. Agile acquisition model includes using organization defining need to acquisition organization creating an agile contract to contractor, iterative feedback between contractor and end users, collaboration among all groups, the contractor continuous delivery to using organization which then deploys to end users.

In an agile model, the development backlog will be seeded with high-level system objectives. Requirements are developed through collaboration among the stakeholders and the development is shaped by iterative user feedback. The agile contract may have a small set of system requirements or constraints. For example, it may be a requirement for the system to comply with an established architecture or interface, meet particular performance requirements, or adhere to relevant standards. The key is that the provided set of requirements are as minimal as possible.

The requirements discovery, analysis, and development process is collaborative, iterative, and ongoing. It really isn’t extremely different from a traditional requirements decomposition, as requirements still have to be traceable from top-level objectives. A key difference is that the decomposition happens closer to the development, both in time and organization. The rationale and mission context for a requirement won’t get lost because the development team is involved in the process, so they understand the drivers behind the features they’ll be implementing.

I’m getting ahead of myself, though! In the next installment of this series we’ll look at cross-functional development teams, the role of Product Owner, and scaling up to a large project.

What are your experiences with agile contracts and agile requirements? Share your best practices, horror stories, and pitfalls to avoid below.

Agile SE Part Two: What’s Your Problem?

Welcome to a series on Agile Systems Engineering exploring the practical aspects of this emerging approach. If you didn’t see it already, check out Part 1: What is Agile, Anyway?

A faster horse

“If I had asked people what they wanted, they would have said faster horses.”

Apocryphally attributed to Henry Ford1

When people trot out that quote they’re often trying to make the point that seeking user feedback will only constrain the design because our small-minded <sneer>users</sneer> cannot possibly think outside the box. I disagree with that approach. User feedback is valuable information. It should not constrain the design, but it is essential to be able to understand and empathize with your users. They say “faster horse”? It’s your job to generalize and innovate on that desire to come up with a car. The problem with the “singular visionary” approach is that for every wildly successful visionary there are a dozen more with equally innovative ideas that didn’t find a market.

Sometimes, your research will even lead you to discover something totally unexpected which changes your whole perspective on the problem.

Here’s a great, real-world example from a Stanford Hacking for Defense class:

Customer ≠ user

Team Aqualink was tasked by their customer (the chief medical officer of the Navy SEALS) to build a biometric monitoring kit for Navy divers. These divers face both acute and long-term health impacts due to the duration and severe conditions inherent in their dives. A wearable sensor system would allow divers to monitor their own health during a dive and allow Navy doctors to analyze the data afterwords.

Team Aqualink put themselves in the flippers of a SEAL dive team (literally) and discovered something interesting: many of the dives were longer than necessary because the divers lacked a good navigation system. The medical concerns were, at least partially, really a symptom. What the divers truly wanted was GPS or another navigational system that worked at depth. Solving that root cause would alleviate many of the health concerns and improve mission performance, a much broader impact.

The customer was trying to solve the problem they saw without a deeper understanding of the user’s needs. That’s not a criticism of the customer. Truly understanding user needs is hard and requires substantial effort by engineers well-versed in user requirements discovery.

In the US DoD, the Joint Capabilities Integration and Development System (JCIDS) process is intended to identify mission capability gaps and potential solutions. The Initial Capabilities Document (ICD), Capability Development Document (CDD), and Key Performance Parameters (KPPs) are the basis for every materiel acquisition. This process suffers from the same shortcoming as the biometric project: it’s based on data that is often removed from the everyday experiences of the user. But once requirements are written, it’s very hard to change them even if the development team uncovers groundbreaking new insights.

The Bradley Fighting Vehicle

Still capture from The Pentagon Wars (1998)

The Bradley Fighting Vehicle was lampooned in the 1998 movie The Pentagon Wars2. By contrast, the program to replace the Bradley is being held up as an example of a new way of doing business.

Instead of determining the requirements from the outset, the Army is funding five companies for an 18-month digital prototyping effort. The teams were given a set of nine desired characteristics for the vehicle and will have the freedom to explore varying designs in a low-cost digital environment. The Army realizes that the companies may have tools, experiences, and concepts to innovate in ways the Army hasn’t considered. The Army is defining the problem space and stepping back to allow the contractors to explore the solution space.

Requirements myopia

System engineering for the DoD is built around requirements. The aforementioned JCIDS process defines the need. Based on that need, the acquisition command defines the requirements. The contractor bids and develops to those requirements. The test commands evaluate the system against those requirements. In theory, since those requirements tie back to warfighter needs, if we met the requirements we must have met the need.

But, there’s a gap. In the proposal process, contractors evaluate the scope of work and estimate how much effort will be required to complete the work. Sometimes this is based on concrete data from similar efforts in the past. Other times, it’s practically a guess. If requirements are incompletely specified, there could be significant latitude for interpretation. Even really good requirement sets cannot adequately capture the actual, boots-on-the-ground mission and user needs.

So, the contractor has bid a certain cost to complete the work based on their understanding of the requirements provided. If they learn more information about the user need but meeting that need would drive up the cost, they have three options:

  1. Ask the customer for a contractual change and more money to develop the desired functionality
  2. Absorb the additional costs
  3. Build to the requirement even if it isn’t the best way to meet the need (or doesn’t really meet it at all)

Obviously none of these solutions are ideal. Shelling out much more than originally budgeted reflects poorly on the government program office, who has to answer to Congress for significant overruns. Contractors will absorb some additional development cost from a “management reserve” fund built into their bid, but that amount is pretty limited. In many cases, we end up with option 3.

This is heavily driven by incentive structures. Contractors are evaluated and compensated based on meeting the requirements. Therefore, the contractor’s success metrics and leadership bonuses are built around requirements. Leaders put pressure on engineers to meet requirement metrics and so engineers are incentivized to prioritize the metrics over system performance. DoD acquisition reforms such as Human Systems Integration (HSI) have attempted to force programs to do better, but have primarily resulted in more requirements-focused bureaucracy and rarely the desired outcome.

I call this “requirements myopia”: a focus on meeting the requirements rather than delivering value.

Refocusing on value

It doesn’t make sense to get rid of requirements entirely, but we can adapt our approach based on the needs of each acquisition. I touched on this briefly in an earlier article, Agile Government Contracts.

One major issue: if we don’t have requirements, how will we know when the development is “done”? Ponder that until next time, because in the next post in this series we’ll dive into some of the potential approaches.

What are your experiences with requirements, good or bad? Thoughts on the “faster horse”, Team Aqualink’s pivot, or the Optionally Manned Fighting Vehicle (OMFV) prototyping effort? Sound off below!

Agile SE Part One: What is Agile, Anyway?

Welcome to a new series on Agile Systems Engineering exploring the practical aspects of this emerging approach.

What is “Agile”?

Agile is a relatively new approach to software development based on the Agile Manifesto and Agile Principles. These documents are straightforward. I will sum them up as stating that development should be driven by what is most valuable to the customer and that our projects should align around delivering value.

Yes, I’ve obnoxiously italicized the word value as if it were in the glossary of a middle school textbook. That’s because value is the essence of this discussion.

Little-a Agile

With a little-a, “agile” is the ability to adapt to a changing situation. This means collaboration to understand the stakeholder needs and the best way to satisfy those needs. It means changing the plan when the situation (or your understanding of the situation) changes. It means understanding what is valuable to the customer, focusing on delivering that value, and minimizing non-value add effort.

Big-A Agile

With a big-A, “Agile” is a software development process that aims to fulfill the agile principles. There are actually several variants that fall under the Agile umbrella such as Scrum, Kanban, and Extreme Programming. Each of these have techniques, rituals, and processes that supposedly lead to delivery of a quality product by helping teams focus on value-added work.

“Cargo Cult” Agile

“Agile” has become the hot-new-thing, buzzword darling of the U.S. defense industry. Did I mean Big-A or Little-a? It hardly matters. As contractors have rushed to promote their “new” development practices, they have trampled the distinction. The result is Cargo Cult Agile: following the rituals of an Agile process and expecting that the project will magically become more efficient and effective as a result. I wrote about this previously, calling it agile-in-name-only and FrAgile.

This isn’t necessarily the fault of contractors. They want to follow the latest best practices from commercial industry to most effectively meet the needs of their customers. But as anyone who has worked in the defense industry can tell you, the pace of change is glacial due to a combination of shear bureaucratic size and byzantine regulations. Most contracts just don’t support agile principles. For example, the Manifesto prioritizes “working software over comprehensive documentation” and one of the Principles is that “working software is the primary measure of progress”; but, most defense contracts require heaps of documentation that are evaluated as the primary measure of progress.

The upshot is that, to most engineers in the defense industry, “Agile” is an annoying new project management approach. Project management is already the least enjoyable part of our job, an obstacle to deal with so that we can get on with the real work. Now we have to learn a new way of doing things that may not be the most effective way to organize our teams and has no real impact on the success of the program. This has resulted in an undeserved bad taste for many of us.

If this is your experience with Agile, please understand that this is not the true intent and practice. The rest of this series will talk about how we achieve real agility.

Agile Systems Engineering

So far, I’ve only mentioned Agile as a software development approach. Of course, we’re here because Agile is being appropriated to all types of engineering, especially as “Agile Hardware Development” and “Agile Systems Engineering”. Some people balk at this; how can a software process be applied to hardware and systems? Here, the distinction between little-a agile and big-A Agile is essential. Software agile development evangelists have taken the values in the Manifesto and Principles and created Agile processes and tools that realize them.

It’s incumbent upon other engineering disciplines to do the same. We must understand the agile values, envision how they are useful in our context (type of engineering, type of solution, customer, etc.), and then craft or adapt Agile processes and tools that make sense. Where many projects and teams go wrong is trying to shoehorn their needs into an Agile process that is a poor fit, and then blaming the process.

Stay Tuned

In the rest of this series we’ll explore how agile SE can provide customer value, how our contracts can be crafted to enable effective Agile processes, and what those processes might look like for a systems engineering team. Stay tuned!

Have you worked on a project with “Cargo Cult Agile”? Have you adapted agile principles effectively in your organization? What other resources are out there for Agile systems engineering? Share your thoughts in the comments below.

The Operations Concept: Developing and Using an OpsCon

  • An Operations Concept is more detailed than a Concept of Operations
  • It is a systems engineering artifact that describes how system use cases are realized
  • It is versatile and serves many uses across the project
  • There is no set format, though there are some best practices to consider

Concept of Operations (ConOps)

Let start by talking about the OpsCon’s better-known big brother, the ConOps.

Read More

Learn from the mistakes of others

The problem with being too busy to read is that you learn by experience… i.e. the hard way. By reading, you learn through others’ experiences, generally a better way to do business…

General James Mattis

The most successful people in any profession learn from the experiences of others. You can learn from their successes, sure. But don’t focus on doing things exactly they way they did, you’ll stifle your own innovation. Instead, understand their successes, extract relevant lessons, and forge your own path.

More importantly, learn from others’ failures and mistakes.

That’s why I publish a Reading / Listening List. As of the publishing of this article, 5 of the 6 recommendations are about poor engineering and design1. I find these stories fascinating, enlightening, and valuable. By avoiding the pitfalls of the past, we improve the likelihood of success in our own projects.

It’s okay to make mistakes, but strive to at least make original mistakes.

Board man gets paid

For years I’ve been advocating for the effective inclusion of human systems integration (HSI) in the systems engineering (SE) process. I had to address a persistent misunderstanding of what HSI is and how it relates to human factors; while that can be frustrating, I recognized that it wasn’t going to change overnight. Instead, I worked diligently to share my message with anyone who would listen.

Recently, my diligence paid off. I was contacted by a group putting together a proposal for a defense contract. The government’s request outlined their expectations for HSI as part of the systems engineering effort in a way that the proposal team hadn’t seen before. Someone on the team had heard me speak before, knew I had the right expertise they needed, and reached out to request my support.

It will be a while before we find out who won the contract, but I am certain that our proposal is much stronger for the inclusion of HSI. The HSI piece of the work is small but essential, and any competitors without the requisite expertise may not have understood its impact or importance to the customer.

This experience reminded me of basketball star Kawhi Leonard’s most popular catchphrase: “The board man gets paid.” See, Leonard is known for his skill at grabbing his team’s rebounds1. This is a key differentiator on the basketball court. The team has done all that work to get the ball up the court, yet failed to score. Grabbing the rebound before the opponent does gives the team another chance. Most of the time, the defensive team is in a better position to grab the rebound; Kawhi Leonard has made a career of getting to those balls first.

Leonard identified an underexploited opportunity and worked hard to develop the skill to take advantage of it. Throughout high school and college, he called himself “The Board Man”. He shaped his career around this unique skill and has been extraordinarily successful because of it.

That’s not to say you have to find a niche to be successful. Obviously there are superstars in every field. But, it’s a heck of a lot easier if you can identify those opportunities nobody else is taking advantage of2.

Bonus read: The top 5%. Share your own tips, inspiration, and niche in the comments below.

Diversity in engineering careers

I had the privilege to attend the Society of Women Engineers conference WE19 in Anaheim, CA last week. I left inspired and optimistic.

Speakers and panelists relayed their experiences over the previous decades. These women had been denied entrance into engineering schools, marginalized in the workplace, and forced to become ‘one of the guys’ to be accepted among their peers.

We’ve come a long way. It’s never been a better time to enter the workforce as a woman/person of color/LGBTQ/etc. Diversity in the workforce and leadership of engineering companies is on the rise, barriers are falling, and the value of diversity is being recognized. And yet, we still have so far to go.

We recognize that diversity is good for business 1 and companies are actively recruiting more diverse talent. Our organizational cultures are still adapting to this diversity. In many ways, we still expect all employees to conform to the existing culture, rather than proactively shape the inclusive culture we desire.

A great example is the “confidence gap” theory for why men are more successful in the workplace. Writing in The Atlantic  in 2014, Katty Kay and Claire Shipman explain that “compared with men, women don’t consider themselves as ready for promotions, they predict they’ll do worse on tests, and they generally underestimate their abilities. This disparity stems from factors ranging from upbringing to biology.”

Jayshree Seth‘s WE19 closing keynote combated the confidence gap with a catchy “confidence rap”. I was excited to share it with you in a gender-neutral post about combating imposter syndrome. In researching this post, I learned that the “confidence gap” is symptom, not a cause. Telling women to be more confident won’t close the gap because our workplace cultures are often biased against women who display confidence.

Jayshree Seth countered the “confidence gap” with the “confidence rap” in an excellent keynote.

Research demonstrates that an insidious double standard2 is what’s holding women back. Women who talk up their accomplishments the same way men do are perceived as less likeable. Women who are modest are more likeable, but nobody learns of their accomplishments and they appear to lack confidence. Women can be just as confident as men, but the cultural expectations of the workplace do not allow it.

That’s not to totally dismiss the confidence gap theory. This double-standard stems partly (primarily?) from continuing societal expectations. Though gender equality has advanced significantly in recent decades, many parents continue to raise girls and boys differently3. A girl raised to be modest and display less confidence will join the workforce with the same attitude.

That’s not the whole story, of course. Our behaviors and habits continue to be shaped by the workplace culture, especially for younger employees just learning to fit in at the office. Currently most office cultures encourage confidence in men and discourage it in women.

I think this is changing slowly over time along with other aspects of gender equality. I also think that a gradual change is not good enough. We owe it to ourselves, to our female peers, and to the advancement of the profession to consciously bring gender equality in engineering more swiftly.

We should define what a gender-equal workplace looks like, identify where our cultures diverge from this ideal, and create strategies for closing that gap. As a starting point, Harvard Business Review shared some management and organizational strategies. And all of us can contribute by recognizing our own biases and by finding ways to highlight others’ accomplishments.

What does workplace gender equality mean to you? How does the culture of your office support (or not) gender equality? What strategies would you recommend for addressing bias on an individual, team, or organizational level? Post in the comments below.

The Swiss cheese model: Designing to reduce catastrophic losses

Failures and errors happen frequently. A part breaks, an instruction is misunderstood, a rodent chews through a power cord. The issue gets noticed, we respond to correct it, we clean up any impacts, and we’re back in business.

Occasionally, a catastrophic loss occurs. A plane crashes, a patient dies during an operation, an attacker installs ransomware on the network. We often look for a single cause or freak occurrence to explain the incident. Rarely, if ever, are these accurate.

Read More

Thoughts on “A Message to Garcia”

“A Message to Garcia” is a brief essay on the value of initiative and hard work written by Elbert Hubbard in 1898. It is often assigned in leadership courses, particularly in the military. Less often assigned but providing essential context is Col. Andrew Rowan’s first-person account of the mission, “How I Carried the Message to Garcia”.

There are also a number of opinion pieces archived in newspapers and posted on the internet both heralding and decrying the essay. There are a number of interpretations and potential lessons to be extracted from this story. It’s important that developing leaders find the valuable ideas.

Work ethic

Hubbard’s original essay is something of a rant on the perceived scarcity of work ethic and initiative in the ranks of employees. He holds Rowan up as an example of the rare person who is dedicated to achieving his task unquestioningly and no matter the cost.

Of course, this complaint is not unique to Hubbard1 nor is it shared universally. Your view on this theme probably depends on whether you are a manager or worker and your views on the value of work2. Nevertheless, Hubbard’s point is clear: Strong work ethic is valuable and will be rewarded.

No questions asked

If that were the extent of the message, it would be an interesting read but not particularly compelling. One reason the essay gained so much traction is Hubbard’s waxing about how Rowan supposedly carried out his task: with little information, significant ingenuity, and no questions asked. This message appeals to a certain type of ‘leader’ who doesn’t think highly of their subordinates.

It’s also totally bogus.

Lt. Rowan was a well-trained Army intelligence officer and he was sufficiently briefed on the mission. Relying on his intelligence background, he understood the political climate and implications. Additionally, preparations were made for allied forces to transport him to Garcia. He did not have to find his own way and blindly search Cuba to accomplish his objective.

I don’t intend to minimize Rowan’s significant effort and achievement, only to point out Hubbard’s misguided message. Hubbard would have us believe that Rowan succeeded through sheer determination, when the truth is that critical thinking and understanding were his means.

There may be a time and place for blind execution, but the majority of modern work calls for specialized skills and critical thinking. Hubbard seems to conflate any question with a stupid question, which is misguided. We should encourage intelligent questions and clarifications to ensure that people can carry out their tasks effectively. After all, if Rowan didn’t have the resources to reach Garcia he may still be wandering Cuba and Spain may still be an empire.

The commander who dismisses all questions breeds distrust and dissatisfaction. Worse, they send their troops out underprepared.

Leadership

On the topic of work ethic, Hubbard is preaching to the choir. Those with work ethic already have it while those with is won’t be swayed by the message. Of course, managers always desire employees who demonstrate work ethic.

“A Message to Garcia” would be more effectively viewed as a treatise on leadership. After all, Army leadership effectively identified, developed, and utilized Rowan’s potential.

Perhaps the most important lesson, understated in the essay, is choosing the right person for the job. Rowan had the right combination of determination, brains, and knowledge to get the job done. In another situation, he may have been the worst person. How did Col. Wagner know about Rowan and decide he was the right person for the job? How do we optimize personnel allocation in our own organizations?

That’s my two pesetas, now you chime in below. What lessons do you take from Hubbard’s essay? Feel free to link to an interpretation, criticism, or praise which resonates with you.