The SEI conducts independent technical assessments (ITAs) periodically for any programs that request them, looking at both technical and programmatic aspects. Such requests often come from either programs that are experiencing challenges with delivering their systems or from external stakeholders to check on the progress that is being made. In the course of performing such an assessment, the ITA team may interview as many as 50 to 100 people from the program management office (PMO) staff, contractor staff, users, and other external stakeholder organizations, all under assurance of anonymity. Interviewees generally give very open and candid responses, giving the team insight into what is actually happening on a program and the ability to gain a deep understanding of the pressures and incentives under which people are operating.
One notable aspect of such assessments is that similar problems arise across separate and dissimilar programs. The key questions that arise when conducting assessments of many different programs are “Why do some of these adverse behaviors keep happening across entirely different programs?” and “Is there a way to stop them?” In this blog post, I discuss the recurring problem in software acquisition and development of what I call clinging to the old ways. I describe the behavior in the context of a real-world scenario and provide recommendations on recovering from and preventing future occurrences of this problem. Future posts in this series will explore other recurring problems.
About Acquisition Archetypes
The SEI’s work on these types of recurring patterns of behavior is based on our experiences doing assessments of large government programs, and employs concepts from systems thinking to analyze dynamics that have been observed in software development and acquisition practice.
The Acquisition Archetypes, as we call them, are based in part on the idea of the more general systems archetypes. Acquisition Archetypes describe recurring patterns of failure observed in acquisition programs with the intent of making people aware of them and their effects and provide people with approaches to mitigate or avoid them. (See some of the previous SEI work in Acquisition Archetypes.)
In the majority of cases, the incentives at work in acquisition programs don’t change much from program to program, and so tend to drive similar behaviors across a wide range of acquisition programs. Taken together, these incentives are analogous to the laws of physics for nature in that they drive the behaviors of all organizations.
The archetype I present in this post is related to the introduction of a new technology and method. I illustrate it in the context of using DevSecOps because it is a newer portfolio of technologies that is being applied to key DoD acquisition programs. However, this archetype would apply equally well to many other new, disruptive technologies—underscoring the point that despite the many changes in technology and the substantial differences across programs, the ideas underlying this archetype still apply.
Clinging to the Old Ways
Description
There is a different tension happening within acquisition programs that try to adopt new technologies and methods: the technologists and engineers are thrown into conflict with functional organizations that are unfamiliar with and unaccustomed to doing business differently to support the new technology or method. These functional organizations often resist the changes that would increase speed and security. There may be some legitimate reasons for this resistance. For example, the current interpretation of the regulations under which they operate may prohibit certain decisions or actions.
A culture of doing things the usual or traditional way instead of embracing newer approaches and technologies can create schisms within the program. These schisms are not surprising since it’s a major culture change to significantly evolve the methods and policies of any organization. Changes are being driven by a number of different new methods and technologies—not just DevSecOps, but also model-based systems engineering (MBSE), digital engineering, artificial intelligence/machine learning, and others. I focus on DevSecOps in this post because it has demonstrated unprecedented improvements in DoD fielding times and security, but also introduces more engineering complexity and requires more coordination and skill.
Some engineers may expect everyone to jump onboard with the new technology and are surprised when they don’t and won’t. Some may think the functionals (the finance, legal, security, and contracting experts) are out of date and stuck in their ways, or some of the functionals may think the new technology or method is a passing fad that has little to do with the way they perform their role. These opposing points of view represent a cultural conflict that stems from the technology. The more the engineers try to force change on the functionals, the harder those parts of the organization are likely to push back against those changes.
An important aspect of this conflict is that there are two chains of command for functionals: one that goes to the program they’re working for, and one that goes back to the larger organization they’re a part of (e.g., finance, acquisition, etc.). The more revolutionary the technological change, the greater the impact on the functionals who will have to support its business aspects. For example, in the context of cybersecurity, instead of the security functionals adapting the security approach to the new technologies, the technologists are often forced to use the older technologies that the security people are more accustomed to. This aversion to newer technologies also has to do with the traditional approaches of decades ago versus the approaches being used by engineers today. The tension manifests in various ways, such as in the shift from waterfall to Agile/DevSecOps, or from traditional security approaches to more streamlined automated methods, from monolithic certification at the end of development to continuous certification, and so on.
This conflict is ultimately resolved in one of two ways: Either some degree of change is eventually effected to get functionals’ buy-in and support for adapting the existing processes to the new technology, or the technology adoption is deemed unsuccessful and may be discarded (Figure 1).
Reports from the Field
One program that was adopting DevSecOps ran into a variety of issues in supporting that approach with the functional support of acquisition, personnel, purchasing, and finance. As one official put it, “Part of the frustration on the acquisition side is the lack of DevSecOps understanding.”
Similarly, another staff member said, “Some people have no experience with DevSecOps before, so they struggle. The way they approach programs, they’re functionally aligned, and matrixed to them, so there’s a struggle sometimes to translate their work of finance and contracts to the technical people.”
Another went so far as to say, “The predominant risk in the DoD space is people who don’t understand DevSecOps and DevSecOps contracting, saying that this way of building software is unlawful—and I’ve been on calls with three government lawyers about that, where they were arguing that it was illegal to build software that way. There’s a DoD publication for building software, and how DoD buys software. It’s all about waterfall—but no one builds software like that anymore. New memos have come out that make DevSecOps purchasing approaches lawful now, but there’s still a lot of concern out there, and it’s hard to convince people that it’s OK.”
One officer pointed out that, “We have Federal Acquisition Regulation (FAR) policies that haven’t changed in years, and acquisition has local policies as well, and people become frustrated.” Another said “In acquisition it is so difficult to make something happen, you become happy with anything you can get. They leave contractual stuff in place for multiple years without evolving it—but we’d never do that on the technical side. People are afraid to ask to do something differently.”
While the rate of technical change seems to be increasing, one staff member said that many of the functionals are “…still living in a world where people are more comfortable with the old way of doing things, and not as comfortable with doing things in a new way with new technology. So, it’s the lack of willingness to use digital technology that concerns me.” As another acquisition official summed it up, “No one is looking at how acquisition must change to support DevSecOps”—and so there is a large and growing gap between the technical staff and the functionals who support them.
With security, “It comes off as a 1980s security approach. Instead of adapting the security to the new technologies, they force you to use the older technologies that they’re accustomed to instead.” Another admitted that while “We want implementations to be robust in terms of security, we’ve tried to implement security that people either don’t understand, don’t care about, or both. Most PMs (program managers), most SPDs (system program directors) don’t understand, and neither do the SCAs (security control assessors) or AOs (authorizing officials).”
In terms of finance, there are some obvious issues in supporting DevSecOps. As one functional noted, “We accept money from other programs, 3400 (O&M) and 3600 (RDT&E). We couldn’t mix colors of money, but you almost have to with DevSecOps.”
Regarding contracting for expert DevSecOps staff, a contracting official said “[The contractor] drives us crazy. They’re the most expensive, they think they’re unicorns, and so they’re difficult to negotiate. They know it, and they come in high on their rates. As a PCO (procuring contracting officer), I must determine if the price is fair and reasonable—and you have to justify that. Technical ability is always more important than price. Technical people don’t understand having to justify the use of a particular vendor.”
Despite the clear need to do things differently, one acquisition professional stated that “There are few acquisition people who are true advocates of or champions for change. That growth piece is missing.” The larger problem is that “Everyone just accepts the way things are. How can you change your processes so that you can do it better and faster? We can’t be content with what we have. We have to be thinking, what’s next, and what can we make better?”
In trying to answer that question, one officer admitted that “The [functional] career field is more about checking boxes to get promoted. It will take an overhaul in talent management and career-field management to do that better. You can also help to retrain some communities, but probably not all.” For one officer, a key starting point was acknowledging that “We must come up with a strategy to support DevSecOps. People need a baseline understanding of DevSecOps.” Going further, another officer acknowledged that an Agile-based and DevSecOps-like approach could be applied to the work of functionals as well, saying “We should be using DevSecOps for functionals in the same way that we’re already using it for engineers/developers, by doing more in the way of automation of repetitive tasks, and establishing a different culture that’s more innovative in using the mechanisms that already exist. We could be doing for acquisition what DevSecOps is doing for software development.”
Solutions and Mitigations
Due to the dual-reporting structure of functionals in the military, some changes required to enable full support of a new technology, such as DevSecOps, must occur well above the level of the program office attempting to adopt it.
Part of the problem is that each service has slightly different takes on how they interpret the FAR and Defense Federal Acquisition Regulation (DFAR) rules—and those rules are longstanding and rigorously enforced, even though they’re only interpretations of the original regulations. Revisiting the original regulations often shows that they aren’t as restrictive as the subsequent policy interpretations were—but years later those interpretations are still being rigidly applied even when they no longer serve either the current changing environment or the original regulation they were meant to enforce.
One example is the need to do correct budgeting five years in advance of every planned piece of work divided across research, development, test & evaluation (RDT&E) versus operations and maintenance (O&M) expenditures, which feature almost paralyzing rules regarding which type of funding would have to be used for things such as direct replacements versus replacement upgrades. Another example is the purchasing of software licenses, where there is uncertainty regarding the allowed use of RDT&E versus O&M colors of money in the first versus subsequent years of use. The cumulative effect is to constrain programs trying to move to more flexible development models, such as Agile and DevSecOps, and put their success at risk.
As alluded to earlier, contracting for expert DevSecOps staff can be difficult. Likewise, staffing also plays a role in the successful, or unsuccessful, adoption of DevSecOps. There are comparatively few DevSecOps engineers available in the DoD, and DoD is directly competing with industry in terms of salaries and work environment when hiring that type of skilled talent. Programs have difficulties staffing government billets with DevSecOps expertise, lacking appropriate job categories and well-defined career paths with sufficient compensation, and forcing programs to backfill with contract staff—which presents its own challenges. When military and civilian employees are able to be hired and trained to work in DevSecOps roles, retention becomes an issue as commercial companies work to poach them from their government roles into what are often more lucrative commercial positions. The government even acts against its own interests by rotating highly skilled military personnel out of DevSecOps positions to more traditional (and often less interesting) acquisition billets requiring more routine skills where their hard-won DevSecOps expertise may not be applicable, and rapidly declines.
To address the policy restrictions imposed on acquisition, finance, and contracting functionals, these staff need to be trained in the use of key new technologies such as DevSecOps even if they’re not directly using them, so that they’re aware of the issues, understand them and the goals, and are thus better equipped to promote and enable the use of the technology. Technical staff should also become more aware of the different aspects of acquisition. Some of this training content should come from collecting together the insights from the experiences of personnel in software factories about how to best use and leverage existing policies. A training curriculum along the lines of a DevSecOps for Managers should be the result, focusing on
- software lifecycle processes, acquisition strategies, and the full range of different types of contracting vehicles
- how existing mechanisms and contractual vehicles can be applied in innovative ways to support DevSecOps
- making existing training on DevSecOps more relevant
- addressing the cultural and process implications of DevSecOps adoption pertaining to acquisition
- involving DevSecOps experts in innovative training roles to teach and build new coursework
Another useful approach would be to institute an exchange program among acquisition, finance, and other functional staff working in different software factories, so that they could share and learn about different approaches that have been developed and applied by other staff to address similar issues and situations.
As a more strategic fix, DoD should continue to test more of the policy changes that may be needed on actual programs, based on the types of key issues they face. An example of such a policy experiment already going on is the Budget Appropriation 8 (BA-8) software funding single appropriation pilots, in which a single new appropriation category (color of money) is created that can be used for both RDT&E and O&M appropriations. Such an appropriation would mean that programs wouldn’t have to budget specific amounts of RDT&E and O&M funding years in advance, potentially restricting their ability to spend funding as needed in a DevSecOps development, where the development and maintenance activities are tightly intertwined and difficult to separate.
To address the issues of DevSecOps staffing over the long term, as the program workforce initially grows and then starts to turn over, the program must engage in a significant workforce improvement and training or retraining activity, and evolve toward a culture that can retain such an advanced workforce:
- Mentor military officers in DevSecOps organizations with successful industry DevSecOps leaders to learn new leadership styles for high-tech teams.
- Survey the government and contractor staff regularly (and report to leadership) on their morale and the degree to which the desired DevSecOps culture is being achieved, and take additional steps to promote the culture if the metrics are not moving in the direction and at the speed required.
- Actively engage with local and regional universities to create a pipeline of future software engineers with the DevSecOps skills to support the needs of the program during its lifespan.
- Institute externship programs or rotations between government and defense or commercial industry partners to regularly advance the skill sets of key software development staff.
- Advocate for new compensation rates that are more appropriate for hiring and retaining highly skilled DevSecOps positions (similar to what is done for physicians, surgeons, pilot flight pay, etc.).
- Advocate for dedicated DevSecOps officer and civilian career tracks beyond the traditional software career fields.
- Relax or obtain waivers for military rotations for skilled DevSecOps officers and enlisted personnel to improve continuity in teams.
Finally, a more controversial approach might be to align additional financial or performance incentives to functionals who successfully field their programs within time/budget/quality goals, incentivizing overall program performance as well as policy compliance.
The Outlook for DevSecOps Adoption
In this post, I’ve looked into one recurring program behavior related to the introduction of DevSecOps into the context of acquisition programs: a conflict between developers and their supporting functional areas that are not accustomed to supporting this new way of developing software.
While it has many substantial benefits, DevSecOps has been—and for the foreseeable future will continue to be—a powerful but disruptive technology with cooperation problems that are pervasive throughout acquisition. Some of these problems cannot be dealt with at the individual program level and may require some significant policy changes across the DoD enterprise. The importance of DevSecOps to DoD software development means that making the changes to policy to be able to fully support it must be a priority.
In my next blog post in this series, I will discuss in detail another recurring archetypal problem related to DevSecOps adoption: vendor lock-in and the high cost of switching vendors.