The previous article in this series provided an overview of the Capability Maturity Model IntegrationSM models (CMMISM). This article will focus on appraising organizational practices using the CMMISM.
There are several possible reasons for performing an appraisal: 1) Identification of improvement opportunities or weaknesses; 2) Evaluation of the performance risk of an organization; 3) "Certification" of a Maturity or Capability Level (i.e., determination of a rating for publicity purposes).
With regard to the latter, it is important to note that there is no official certifying body for the various CMMI appraisals. The strongest statement that should be made is that an appraisal was conducted by a specific team under certain conditions and a given rating was determined.
Beginning process improvement: An organization just beginning process improvement should do some sort of appraisal to determine where their major problems are so they can address the most critical issues first. This can be a fairly simple review of organizational processes relative to the CMMI, done either by the organization itself after study of the reference model, or led by an experienced process improvement professional.
Benchmarking: After an organization has been doing process improvement for a while it may want to verify its progress by doing a formal appraisal. This can result in the determination of a Maturity Level or Process Capability Profile if so desired.
Source selection: An organization considering using a supplier may want to determine the risk that the chosen supplier will not be able to meet its commitments. One way of doing this is by using an appraisal to determine the maturity or capability of the supplier's processes.
Monitoring: An organization may want to understand over time how its process improvement program is progressing. Or if it has selected a supplier, it may want to verify supplier performance.
There is a range of appraisal methods available, ranging from less costly techniques such as a self-appraisal or mini-appraisal to a full-blown SCAMPISM (Standard CMMISM Appraisal Method for Process Improvement). In choosing a method the organization should consider the appraisal objectives and desired outputs, the accuracy of the results, the cost to prepare for and conduct the appraisal, and the anticipated extent of organizational disruption.
The Appraisal Requirements for CMMI, Version 1.1 contain requirements considered essential to appraisal methods intended for use with CMMI models. It defines three classes of appraisal methods: A, B and C. Class A, SCAMPI, is suitable for benchmarking and comparison, while Classes B and C have more limited objectives. Table 1 provides a comparison of the methods.
How does an organization know which appraisal method to use and when? Choosing an appraisal method, like choosing metrics, should be conditioned by the questions you want to answer. Do you want to benchmark current processes (rate at a Level), develop a process improvement program, check on improvement progress, allocate improvement resources or select a subcontractor? Various appraisal methods have significantly different costs, accuracies, and impacts on the organization, as noted in Table 1. Table 2 describes the applicability of various appraisal methods.
What is Involved in Performing an Appraisal?
Any appraisal generally has at least two objectives:
1. Gather accurate data in an efficient, minimally disruptive way.
2. Help to identify and prioritize improvement opportunities or weaknesses.
These objectives can be achieved in a number of different ways, with varying degrees of cost and accuracy as noted above. Sometimes a third objective is appropriate:
3. Signal to the organization that a new way of life is beginning.
This third objective is particularly applicable when the organization wants to institute a change in its culture — its customary way of doing things. In this case, disruption is good.
Most appraisals have two major categories of outputs:
Findings
• Provide an accurate picture of processes, using the CMMI as a framework.
Recommendations
• Provide guidance on process improvement activities appropriate to the current state of the organization’s process.
• Provide a framework and catalyst for action.
• Build ownership of results.
• Develop organizational commitment and energy.
• Sustain sponsorship and establish commitment.
• Facilitate continued process improvement.
An organization embarking on a process improvement program should consider a phased sequence of appraisals to provide identification of appropriate improvement opportunities at a relatively low cost. Beginning with a self-appraisal (see Figure 1) followed by a mentored self-appraisal helps the organization establish its process improvement program. A mini-appraisal can be used to determine progress and readiness; a full Class A SCAMPI can be used for benchmarking and determination of the organization’s Maturity Level or Capability Profile.
Appraisal Activities
An appraisal is a series of planned steps designed to elicit information about organizational practices relative to a reference model. It has a life cycle similar to any other project. It typically includes a review of documentary evidence and interviews with managers and practitioners to ascertain how processes are implemented within the organization. Table 3 describes an appraisal project life cycle with the SCAMPI activities associated with the various appraisal steps.
Preparing For an Appraisal
The most critical issues in planning and preparing for an appraisal are establishing high level senior management sponsorship and carefully defining the objectives and scope of the appraisal. The business needs and goals of the sponsor have a major impact on how extensive an appraisal needs to be. The major determiners of appraisal scope are: 1) the questions the sponsor wants answered and the actions he is anticipating taking as a result of the appraisal; 2) how much he is willing to pay; and 3) how much disruption the organization can afford to support the appraisal. The appraisal team leader, organizational unit coordinator and appraisal sponsor need to meet fairly early (possibly by telephone or video teleconference) in order for the team leader to ensure that the sponsor fully understands what an appraisal involves and what kind of results he can expect.
The organizational unit coordinator is in charge of all the appraisal logistics for the organization being assessed and is a member of the organization being assessed. The organizational unit coordinator has responsibilities in areas of sponsorship, appraisal participants, members of the appraisal team, facilities, equipment and supplies, documentary evidence and team support.
A major decision to be made is which Process Areas will be reviewed in the appraisal, and how much of the organization will be examined. This is particularly critical if the organization is located at more than one geographic site. The form of results documentation also needs to be determined. Will just the final briefing slides be sufficient, or is a formal written report also desired? Does the sponsor desire recommendations as part of the final findings briefing?
There are two different approaches to collecting the data required for an appraisal. A verification appraisal requires the organization to provide a detailed mapping of documentary evidence to CMMI practices. That mapping is then verified by the appraisal team. In a discovery appraisal the team does most of the “digging” to determine evidence in support of the model practices. Choosing whether to do an appraisal in verification or discovery mode is a critical decision which has a major impact on the effort required by the organization and the team.
If a Class A SCAMPI is being performed, a number of site personnel receive appraisal team training provided by the Lead Appraiser. Training more members than are required will allow for backup in case some people are not able to participate in the on-site appraisal.
Appraisal team members should be:
• Very knowledgeable about the organization
• Well-respected within the organization (particularly those team members who are part of the organization being assessed)
• Motivated to improve the organization’s software process
• Willing to accept change and have the ability to help implement change (be a change advocate or change agent)
• Sensitive to people and able to address questions to appraisal participants in a clear and nonthreatening way
• Capable of making a positive contribution as appraisal team members
Team members should be opinion leaders (other people listen to what they have to say) as well as team players. Members should preferably have at least 8 to 10 years’ experience as software or system engineering professionals. They should represent a wide variety of process areas such as requirements, design, implementation, test, configuration management, metrics, quality assurance and process definition.
Site appraisal team members must not occupy positions within the organization that may lead to a conflict between the appraisal principles and their job function. For example, if appraisal participants believe that information they volunteer has the potential to affect them adversely after the appraisal, they may not speak freely or not speak at all. Nor should there be a conflict between their function on the appraisal team and their regular job function.
To help ensure a free flow of information, organization appraisal team members should not hold positions that involve any of the following activities: 1) Acting as manager of a project included in the appraisal, nor the manager of such a person; 2) Working on or directly involved with reviewing or supporting a project under assessment; 3) Currently serving in a software audit or quality assurance position for any of the projects under review. This last requirement might be waived, depending on the relationship between QA and the projects. However, keep QA representation to one or two people at the most. Too many QA personnel could create an audit atmosphere.
Note: Depending on the organization culture, the Lead Appraiser may decide it is possible to violate some of the above restrictions without impacting the integrity of the appraisal.
Action Planning
Action planning is a necessary follow-on to any appraisal and the lead-in to implementing changes. The organization needs to review the findings and recommendations and decide what actions it will take as the next step in the improvement process. The plan sets the stage and establishes the priorities for implementing the next set of changes. That action planning should be based on the organization’s strategic objectives, the critical “market drivers,” those factors which ultimately determine success or failure.
Business leaders determine critical business drivers and associated strategic objectives to answer the question, “What do we want to achieve as an organization?” Action plans based on business goals and appraisal findings and recommendations drive the improvement project. An improvement project should be managed like any other project (but not Level 1). Leaders should model the expected behaviors and prepare the organization for the upcoming changes. In developing the action plan three factors should be considered:
Results -What desired results do we want to achieve? How much improvement can we expect? Desired results should be prioritized by impact on the organization.
Needs -What do we need to change to effect this result? How soon do we need this result to improve? Needs should be prioritized by urgency.
Activities -What tasks do we expect to be done to effect the needed change? Can this be done in time to get the desired results? Activities should be prioritized by cost/feasibility.
These three factors can then be combined (algorithmically, if desired) to come up with a prioritized list of actions to be implemented.
Process Action Teams (PATs) are a good choice for actually defining and implementing specific process improvements. Getting PATs up to speed quickly is easier with a defined process. One such process is documented in ETVX (Entry-Task-Verification-eXit) format, which is also used by the team to document the model of the process they are working on. In addition to assorted templates and guidelines for both project outputs and for project planning and status reporting, each step in the process has entry and exit criteria, roles, measures, standards and tools.
Conclusion
The first two articles described the CMMI models and associated appraisal methods. The next article will deal with issues involved in implementing the CMMI and transitioning from the Software CMM to the CMMI.
Capability Maturity Model® and CMM® are registered in the U.S. Patent and Trademark Office. CMMSM Integration and SCAMPISM are service marks of Carnegie Mellon University.
Sources:
Appraisal Requirements for CMMI.Ver.1.1. CMU/SEI-2001-TR-034. December 2001.
Standard CMMISM Appraisal Method for Process Improvement (SCAMPISM). Ver. 1.1. Method Definition Document. CMU/SEI2001-HB-001. December 2001.
Richard B. Waina, P.E., Ph.D., Principal of Multi-Dimensional Maturity, has over 35 years of IT experience. He worked for five years at White Sands Missile Range, and worked on a number of missile programs at Hughes Aircraft Company, including Maverick for the USAF, Phoenix for the DON and TOW for the USA. At EDS he was responsible for deploying process maturity assessment methodologies globally. Dr. Waina is a SEI-authorized CMM and CMMI Lead Assessor/Appraiser and Instructor for the Introduction to CMMI. He has conducted over 70 CMM/CMMI assessments in nine countries since 1990. He holds engineering degrees from Carnegie Mellon University, New Mexico State University, and Arizona State University.