Introduction
Senior acquisition professionals encounter both challenges and opportunities when incorporating commercial off-the-shelf (COTS) products into software-intensive systems – acquisition professionals know how to manage traditional custom-development software programs; this article emphasizes differences in managing software-intensive, COTS-based systems (CBS). It focuses on 11 keys for CBS success, aimed at the level of those with responsibility for more than one system, but it's useful for almost anyone new to COTS-based systems.
To set the record straight, a COTS product is a product that is:
• Sold, leased, or licensed to the general public
• Offered by a vendor trying to profit from it
• Supported and evolved by the vendor, who retains the intellectual property rights
• Available in multiple, identical copies
• Used without modification of the internals
This definition is generally true, but there are gray areas. These qualities should at least provide a means to examine the variability between products. A COTS product can be either software or hardware. While our focus here is on software COTS products, we have found that most of the issues and advice are equally applicable to hardware COTS products.
The development and sustainment of COTS-based systems present some challenges that are not generally found in custom development:
• New and improved products are continuously offered by vendors.
• Changes to products are driven by the vendors' market share goals, not by the unique needs of your programs.
• Each product has its own model of how the user will use it, which may be quite different from the end user's actual processes.
• Licensing secures the use of products that you need; data rights and warranties protect you (and the vendors) in the long-term use of those products.
• Vendors release product upgrades (and new products) at times that suit their marketing strategies, not your priorities.
• As consumers, we have little insight into how a COTS product has been put together and how it behaves and why.
• Just as each product has its own idea of user processes, each also has its own architectural assumptions and paradigms, making products hard to integrate.
• Some COTS products have built-in dependencies on other COTS products, which multiply the impacts of product-change events.
Meeting these challenges requires a new mindset and new practices. These eleven keys to CBS success are applicable not only up-front or only for new starts: they can and should be applied no matter where a program is in its life cycle.
The Keys
The eleven keys to CBS success are grouped into three major topical areas as follows:
Make COTS-Based System Tradeoffs
• Reconcile Products and User Operations
• Leverage the Marketplace
• Engineer an Evolvable Architecture
• Make Tradeoffs Simultaneously
• Avoid COTS Modification
Think More Like a Business
• Live by the COTS Business Case
• Negotiate Licenses & Supplier Relationships
• Realign Budgets for COTS Realities
Establish Evolution as a Way of Life
• Evolve COTS-Based Systems Continuously
• Take the Long View on System Acquisition
• Change the Culture
Reconcile Products and User Operations
Mismatch between the processes practiced by end users and those envisioned by the product's creators is likely. It is complicated by the fact that those responsible for development of the system are often not in a position to dictate end-user processes and so must rely on commitments from independent stakeholders. These mismatches must be discovered as early as possible to avoid potentially catastrophic effects.
Recognition of what is and is not negotiable is essential for reconciling the mismatches. Sometimes an organization does not have control over its circumstances and, in particular, its processes. For example, processes that are subject to regulatory control are hard to change, regardless of how different they are from a product’s features. So it is not always feasible to change end-user processes to be more compatible with product capabilities. However, there is usually more room for flexibility in end-user processes than end users think.
Adapting the end-user organization’s processes to those embodied in selected COTS products may allow for a greater number of COTS products to be considered. The more closely the end-users’ processes mirror the inherent COTS product processes, the greater the chance of success for a COTS-based system solution. Trying to change the product to fit existing end-user processes is generally not a good idea.
Leverage the Marketplace
Leverage is important to allow programs to capitalize on the marketplace — where it makes sense — as one strategy to manage limited resources. To gain leverage, consumers need information about products and suppliers. The information ranges from functional features, to licensing alternatives and costs, to ease of integration with the system, to the health of the supplier, and its prominence in the marketplace.
Another help in leveraging the marketplace will be knowledge of marketplace drivers. In different market segments, different dynamics determine how the players interact. Without knowledge of what motivates vendors in different market segments, you cannot determine how or where you have leverage. Finally, determine where you can have influence (and where you can’t) and then exercise you influence where you are able.
You have limited ability to control the marketplace and therefore need to develop instead, ways in which you can influence it. The extent of influence you can have will vary with the domain, the size of the user base, and you position in the market niche. Then plan how to exercise the influence you have.
Engineer an Evolvable Architecture
Change drives COTS-based systems
Figure 1 illustrates some of the typical lifetimes that affect COTS-based systems. Not only are the systems long-lived, but they are, in some cases, orders of magnitude longer-lived than the things that go into creating and sustaining them. You may find for you systems that some of these numbers are greater or lesser than those shown on the chart, but the important point to understand is the sources of change across the life of a system and their relative likelihood of occurrence. The bottom line is that COTS-based systems must be able to evolve. An evolvable architecture is the foundation for system evolution and the key asset for its accomplishment.
A system architecture (hardware, software, and people) provides a framework for structuring the “parts” or components (regardless of their source) — COTS or NDI (non-developmental items, custom, or legacy)) that make up a system and how they will interact.
As with other systems, the system architecture becomes the foundation for COTS-based systems. But there are two important differences for COTS-based systems. First, the formation of a COTS-based system architecture must be done in concert with the system requirements and marketplace decisions. Marketplace trends and available products and technologies will of necessity influence and constrain the architectural and design alternatives. In addition, the architectural decisions may eliminate some products and technologies from consideration.
Second, with COTS-based systems, continuous, rapid changes driven by new mission needs, product upgrades, and technology advances are facts of life. An architecture that can retain its structure and cohesiveness yet allow the system to respond easily to these changes — an evolvable system architecture — becomes an important strategic asset to the organization.
Make Tradeoffs Simultaneously
There is a fundamental change that comes with COTS-based systems. If a program tries to use the traditional custom-development approach for a COTS-based system — defining their requirements and design, then looking around the marketplace for products that will provide the needed functionality and fit within the defined architecture — they are likely to find no available COTS products.
The most fundamental change necessary with COTS-based systems approach is the simultaneous definition and tradeoffs among the requirements, the architecture and design, and in the marketplace, as shown in Figure 2. Not only must the engineering activities support this change, but the acquisition strategy, contractual activities, and business activities must all support as well.
Finding a fit of system requirements, which include functional and non-functional requirements, end-user processes, business drivers, operational environment, constraints, policy, cost, schedule — everything that determines the shape of the system — the marketplace and the architecture — means making continuous tradeoffs throughout the system’s lifetime, not just during initial development. It is an approach that is very compatible with a spiral or iterative approach.
Avoid COTS Modification
There are many temptations for making changes to COTS products. You might feel you have an inability to change end-user processes. There may be some tailoring required for any use, and there’s likely to be some work required to integrate the product into the system, so it’s tempting to just “keep going.”
When faced with a product that does not quite meet the system need, there are basically four choices: (1) Use it as-is (i.e., change something else, such as the end-user process); (2) Tailor non-surgically; (3) Modify surgically; and (4) Custom develop.
All have near- and long-term consequences. In particular, for some of them the sustainment costs may be prohibitive. Many products provide mechanisms for adapting them without surgery; we call these built-ins: parameters and tailoring options that are provided as part of the product. Tailoring does not involve modifying the product’s source code.
Other non-surgical adaptations are bolt-ons (which add functionality) and camouflage (which involves adaptation or “glue” code that is often required when multiple COTS products are composed to form a system).
Surgery — modifying a vendor’s product without the vendor’s cooperation — may buy some shortening of the development time. But it requires understanding something created by an unknown person and will generally void any warranties or support agreements. If the modifications are not coordinated with or fed back to the vendor, the resulting software will not keep pace with the advances in the vendor’s product. The program will be responsible for all future maintenance: You have bought someone else’s legacy system.
Live by the COTS Business Case
Analyses must drive “make versus buy” decisions for COTS-based systems. A COTS business case is not done very differently from other business cases, but there are some differences in detail. For example, in addition to considering the mission, a COTS business case must also consider the market: products and suppliers. Also, when addressing total cost of ownership, it must include the costs associated with marketplace volatility.
In analyzing alternatives, consider the diversity of products offered by a supplier, the size of the market share of those products, and the maturity of both the supplier and the products. Note also that the evaluation of every single product is different — you are always in an apples versus oranges situation because each product interacts with the system context differently.
In addition, because of the rapidity of change in the marketplace, timeliness of information is important, as a single product will change over a short period of time.
Negotiate Licenses and Supplier Relationships
The first thing to realize is that programs can and should negotiate product licenses. There are many different kinds of licenses; for example, they may be enterprise-wide, per-seat, per-user, development time or user. It is incumbent on a program to understand the choices and their implications. For example, the choice of license can greatly affect system architecture and cost. License options are also important in building the COTS business case.
Licenses should be set up to be the basis of a sound partnership between the acquirers and vendors. The benefits of partnering with suppliers include the chance to have an influence over product directions, to obtain insight into supplier plans and motivators, and to share with suppliers important information on the program’s future plans, thus affording suppliers the opportunity to contribute to the system’s progress and evolution.
Realign Budgets for COTS Realities
Budgeting is not very different for COTS-based systems, but there are new realities to accommodate. One such difference is the volatility of the marketplace. This includes such events as product feature reduction or bloat, vendor demise, and changes to license arrangements.
Another difference to consider is the infrastructure necessary to support a COTS-based system approach. This includes such necessities as establishing teams for technology and market watch; test beds; culture change and training; information collection and dissemination (such as lessons learned); guidance; examples; handbooks; and incentives.
Remember to take into account budgeting changes resulting from spiral or iterative development. Unfortunately, there are no models or magic formulas for estimating the cost of COTS-based systems development or sustainment. But normal approaches, such as deriving costs based on a work breakdown structure, are still applicable.
Evolve COTS-based Systems Continuously
Marketplace volatility is a fact of life with which COTS-based system projects must deal. It affects sustainment: It may cause rework, and it makes rework costly. The defense for this is to adopt an evolutionary mindset. This evolutionary mindset needs to be evident in all of the processes the program uses, including budget-related processes. It must also be part of the acquisition strategy and sustainment plans.
While an evolutionary mindset has always be useful — after all — requirements can change on any program — the marketplace realities make it critical for COTS-based systems.
An evolutionary mindset starts with proactively managing continuous change. It capitalizes on the use of spiral or iterative development and maintenance and employs continuous evaluation.
These evaluations will be of different kinds depending on the stage of the program and the nature of the information that is required. But some form of evaluation will be ongoing throughout the lifetime of the system. Funds and a knowledgeable staff will be required. With continuous re-evaluation comes continuous re-integration and regular testing. Being proactive is the key: Anticipate continuous change.
Take the Long View on System Acquisition
A COTS-based approach is a long-term engineering and business strategy. Expecting short-term payoffs may be unrealistic. Potential benefits, such as lower construction and sustainment costs, faster development, or the ability to leverage new technology and the efforts of others, may require some investment of money and time. For example, to create an evolvable system architecture will probably require greater expenditure in the short term, but the payoff is in the medium to long term.
Or it might be that a program will discover the ability to field a capability sooner because of the use of COTS products, but the cost for the initial construction and fielding is not substantially less than with a more traditional approach.
Make a long-term commitment, both strategically and technologically, to the COTS-based approach, keeping in mind that the real payoff of this approach may well only be realized in the mid- to long-term. A program needs to maintain the long view even in the face of such things as normal program “bumps in the road” and changes in the marketplace, in government personnel, and in contractors.
To achieve this will require an acquisition strategy with long-term viability, flexible contract vehicles, stable funding, and long-term relationships in the marketplace and contractor community. If the program fails to maintain its up-to-date investment in a COTS-based system, the program might not be able to recover. If you get too far behind in product upgrades you may lose your whole investment and have to start all over again.
Change the Culture
This is the key that we hope will no longer be necessary in the future as projects become more accustomed to doing things “the COTS way.” But for now, many try to treat the change to COTS-based systems as only a technological change. By now you’ve seen that it makes profound demands on people, and these cultural changes must be addressed as well to be successful.
The differences between how we have approached custom-developed systems in the past and how we need to approach COTS-based systems suggest some of the kinds of cultural changes that you may have to deal with as your organization transitions to a COT-based systems approach. These changes affect everyone, from the government (e.g., executives, program offices, procurement staff), to the contractors, even end users and other key stakeholders.
Changes in how people thing about and perform their jobs are often met with resistance. For example, owners of a legacy system have an incentive to resist or thwart a COTS-based migration. A COTS-based system approach may challenge an organic sustainment capability — fewer people and different skill sets are needed. For a COTS-based approach to be successful, you and everyone above and below you must provide strong business motivation and proactive leadership to transform resistance.
Conclusion
The change to a COTS-based system approach is a new adventure in systems development. It is one at which you can be successful, but to do so you need to heed these keys to success.
Patricia Oberndorf is a senior member of the Technical Staff (MTS) for Dynamic Systems/COTS-based Systems, Software Engineering Institute.
Lisa Brownsword is a senior MTS for Dynamic Systems/COTS-based Systems, SEI.
Carol Sledge is a senior MTS for Network Survivable Systems, SEI.
For more information about COTS-based systems, go to the Software Engineering Institute/Carnegie Mellon University website at www.sei.cmu.edu/.
References
"Managing Software Acquisition: Open Systems and COTS Products,” B. Meyers and P. Oberndorf. Addison Wesley SEI series in Software Engineering, 2001.
“An Activity Framework for COTS-based Systems,” P. Oberndorf, L. Brownsword and C. Sledge. Software Engineering Institute Technical Report CMU/SEI-2000-TR-010, August 2000.
“The Commandments of COTS: Still in Search of the Promised Land,” D. Carney and P. Oberndorf, CrossTalk: The Journal of Defense Software Engineering 10, 5 (May 1997): 25-30.