Advances in Patient Safety: From Research to Implementation Advances in Patient Safety: From Research to Implementation Volume 3. Implementation Issues

Suicide Risk Response: Enhancing Patient Safety Through Development of Effective Institutional Policies

Laura Bonner, Bradford Felker, Edmund Chaney, Karen Vollen, Karen Berry, Barbara Revay, Barbara Simon, Lial Kofoed, Scott Ober, Linda Worley, John Fotiades, Scott Sherman

Abstract

A suicidal patient requires a prompt, coordinated intervention. In this paper, we describe a process for developing a suicidality policy, which may help clinics develop effective, locally adapted policies. We present the process in the framework of the Quality Improvement Plan-Do-Study-Act cycle. The process we describe occurred as part of a quality improvement project. Translating Initiatives for Depression into Effective Solutions (TIDES) is an evidence-based, quality improvement intervention for depression, implemented in seven Veterans Administration primary care clinics in five states. A multidisciplinary workgroup, the Collaboration Workgroup (CWG), created for this project supports the collaborative care process through evaluation and improvement of policies, including those for institutional response to suicidality. During the "plan" phase, the workgroup reviewed existing policies from each of the seven participating intervention clinics. This review revealed significant gaps and implementation difficulties. During the "do" phase, workgroup members developed or adapted site-specific policies as needed based on the initial CWG review, and assisted sites in implementing them. During the "study" phase, workgroup members reviewed what had worked and what had not worked in implementing policies for threatened suicide at each site, and identified a set of key features of successful policies. Features included a clearly defined chain of responsibility, well-defined followup procedures, and documentation of actions in the medical record. The workgroup developed templates that emphasized these key features but allowed for necessary local adaptation. Workgroup clinicians assisted clinics to implement site-specific policies. During the "act" phase, which is ongoing, site policies are in effect and are being evaluated.top link

Introduction

Overview

When a patient expresses thoughts of suicide, a potential crisis ensues for the patient, the provider, and the health care institution. In addition to the tragic death of the patient, the sequelae of a completed suicide include profound, negative outcomes for the patient's family, provider, and health care system. These outcomes affect emotional, financial, and professional domains. Prompt assessment of potentially suicidal patients by individual clinicians is imperative and requires knowledge of the following: risk factors for suicide, skill in establishing a rapport with an extremely distressed patient, and clinical experience in assessing the degree of risk. However, skilled clinical assessment, while necessary, is not sufficient for effectively responding to a suicidal patient. In any health care organization, a suicidal patient's initial contact often will be whoever picks up the phone at the practice. The progress from that individual to a clinician who can assess the level of urgency of the situation is part of a complex chain of events. Once the clinician has decided on a course of action, he or she must activate a response within the context of the system as a whole, involving other providers, disciplines, and departments, such as emergency room staff, the on-call physician, the admitting officer, the inpatient unit, the primary care provider, and the mental health treatment team. Appropriate response by the institution thus involves a complex process of clinical assessment and administrative support. Not surprisingly, gaps and deficits in suicide prevention systems within medical care organizations are common and are often discovered only after suicide. In this paper, we describe the process used by a multidisciplinary team of clinicians and researchers to develop suicide prevention policies. We present the team's consensus about important policy features and summarize the lessons learned about the critical aspects of developing such a policy. This work is not presented as a rigorous evaluation of suicide policies; rather, we demonstrate the application of a quality improvement methodology to an important problem facing health care institutions.top link

Evidence of need for a new approach

Current medical literature reflects a good understanding and general consensus about how to clinically assess and manage a suicidal patient. For example, numerous studies document the significant risk factors for suicide, 1  -- 5 including older age, male sex, presence of depression (especially with hopelessness), presence of substance use disorders, and absence of social support. 6  -- 8 Knowledge about risk factors helps determine the procedures to be used in clinical assessment. Hirschfeld, 2 for example, recommends asking the patient directly about his or her risk of suicide and, at the same time, determining the availability of lethal means to commit suicide, the presence of financial or other serious problems, and the possibility of comorbid disorders, such as personality disorders.

Knowledge about risk factors and about recommended approaches to clinical assessment is helpful in decisionmaking, but there is less research-based support for the institution's role in supporting the provider's treatment plan. Little research has focused on assessment and development of institutional policies for this part of the task, leading to frustration among clinicians and policy experts who search in vain for published guidelines. Dlugacz et al. 9 describe a root cause analysis of patient suicides in a non-VA health care system. By identifying deficiencies in current policies, Dlugacz et al. provide a starting point for improvement in suicide prevention. The deficiencies they found include poor communication among different providers and departments, difficulty in locating appropriate placements, and lack of knowledge about the role of related factors such as alcohol abuse. The authors recommended a comprehensive remediation process, including staff education, improved assessment of at-risk patients, and attention to seemingly simple matters such as prohibiting patient access to dangerous items. 10 Hirschfeld briefly addresses the details of outpatient prevention 2 and makes recommendations about practical matters, such as supervising the patient during transport to the hospital and monitoring the patient once hospitalized. However, such pragmatic concerns have not been widely addressed in available literature, and instructions on how to develop institutionwide policies are lacking. The gaps in the literature reflect the state of the field, where basic clinical management is well-established while institutional policies are beginning to receive more attention.

Our workgroup addressed this knowledge gap, using the Plan-Do-Study-Act cycle to develop detailed, pragmatic policies for suicidal patients. The policies we developed are based on clinical experience, research knowledge, and group consensus.top link

Background information

Translating Initiatives for Depression into Effective Solutions (TIDES), sponsored by The Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI) and Well-being Among Veterans Enhancement Study (WAVES), are companion studies that implement and evaluate collaborative care for depression in Veterans Affairs (VA) primary care clinics, using evidence-based quality improvement (EBQI) methods. TIDES implements the evidence-based collaborative care intervention, while WAVES is a rigorous research project that evaluates the TIDES quality improvement effort. The TIDES project engaged clinical management in three large VA regions (Veterans Integrated Service Networks or VISNs) in redesigning depression care in two to three medium-sized primary care outpatient clinics per region (serving thousands of patients). These redesigns were based on evidence about both appropriate treatments and effective depression care delivery methods. The TIDES quality improvements are fully implemented and ongoing. The WAVES project has selected a representative panel of patients from practices participating in TIDES, as well as from comparable usual care practices, and will determine whether the TIDES intervention is more effective for these patients than is usual care.

As implemented by the clinical leaders and managers who participated in the TIDES project, depression is managed initially by primary care providers, with nurse care manager support for patient assessment and education. The primary care clinician then decides, in collaboration with the patient, whether to prescribe antidepressants in primary care or to refer to a mental health specialist. If the patient is treated with antidepressants in primary care, the care manager follows patient symptoms and encourages treatment, with supervision from a mental health specialist. If a patient is referred to a mental health specialist, the care manager helps coordinate depression treatment among the primary care provider, who continues to manage other aspects of the patient's care, the specialist, and the patient. The collaborative care team thus involves the patient, the primary care clinician, the care manager, and mental health specialists. The team also involves the senior leaders in primary care, mental health, nursing, and administration from the participating VISNs. These leaders provide overall direction for the project and designate necessary clinical resources.

VISN leaders identified collaboration between mental health specialists and primary care clinicians as the single most important element of depression care improvement. TIDES therefore assembled a multidisciplinary workgroup ---Collaboration Workgroup (CWG) ---composed of VA primary care clinicians, depression care managers, mental health clinicians, and researchers to focus on fostering effective collaboration. The first major collaborative care issue brought to the CWG was how to manage patients who present with suicidal ideation (i.e., thoughts about potentially lethal self harm). The CWG recognized the importance of having well-defined policies for response to suicidal TIDES and WAVES patients, and worked together to develop such policies. The CWG also realized that although all health care institutions may encounter a patient with suicidal ideation, each institution is unique and will operate differently. The CWG therefore set out to develop policies that would be effective for this common problem while maintaining enough flexibility to adapt to unique local needs.top link

The Collaborative Care Workgroup approach to improving suicide prevention

The CWG's approach to development of a suicide response policy can be conceptualized within the framework of the quality improvement (QI) model of the Plan-Do-Study-Act cycle (Figure 1). This cycle is a well-known approach to quality improvement and does not require large-scale trials to yield valuable results. Therefore, it was a logical approach to use in the TIDES quality improvement intervention. The "Plan" stage involved development of the workgroup and review of the existing site policies. "Do" involved creation of a recommended response policy. "Study" involved analysis of study features and implementation and evaluation of the policy. The "Act" stage is ongoing and includes real-world implementation, dissemination, and evaluation of the recommended policy.top link

Figure 1. Plan-Do-Study-Act cycle of policy development

top link

Some health care sites lack integrated response systems; in other sites, such systems exist on paper and are neither functional nor integrated across service lines. In such cases, there is an additional layer of complexity in that the support staff must have some guidelines for assessment. This individual must then be able to locate a clinician to assume responsibility for the patient. Once the patient has been determined to be at high risk, the clinician must decide whether to hospitalize or to rely upon a less-restrictive form of management, such as a no-suicide agreement or safety contract in conjunction with close followup. When emergent hospitalization is the chosen course of action, the clinician must act to obtain help for the patient.top link

"Plan"

Workgroup development

The CWG included an average of 30 individuals, 22 of whom formed a core group that met monthly to address policy issues. The core group included nurse care managers (three), primary care clinicians (five), mental health specialists (six), and researchers (eight). Development of a useful suicidality policy emerged as an important issue during CWG discussions of the support required for effective collaborative care. In both TIDES and WAVES, the potential for encountering a suicidal patient necessitated careful planning of a suicidality response policy. Although researchers do not have contact with VA patients, depression care managers participating in the TIDES quality improvement project do face the probability of dealing with suicidal patients. In WAVES, research interviewers encounter VA patients in the course of conducting the evaluation of the TIDES project. Therefore, the CWG formed a suicidality workgroup composed of a subset of group members (including care managers, clinicians, administrators, and researchers). This subgroup developed a user-friendly general policy that has guided the study's response to suicidal patients and to research participants. The policy also can be modified by clinics and hospitals to guide their suicide-risk response. Below, we describe the process of policy development and discuss the team's consensus about important features of the policy.top link

Review of existing site policies

To ensure consistency of study procedures with pre-existing site policies, the workgroup researched and reviewed individual site policies at nine sites as a first step. Team members obtained policies from five participating sites (four of the nine sites did not have formal policies). Group members were asked to review their respective site's policies and to focus on the following steps: identifying the existing policy, implementing that policy, and understanding the core components of a successful policy. The group compared existing policies and documented gaps in those policies. The group pilot-tested the policies by attempting to follow them as written, and then evaluated how well the policies had worked and what needed to be changed for better performance. We used the results of this analysis to inform the study's suicide-risk response procedure.top link

Specific types of deficits found in existing policies

The initial planning step revealed that suicide risk assessment policies differed widely across sites. Although most sites believed that they had an adequate policy in place, when workgroup members tried to locate and implement the policies some challenging situations arose. Some sites had already developed institutional policies and were implementing them, while some had not. At sites with existing policies, some policies did not exist in written form, while some that were written contained outdated or inaccurate information, such as disconnected phone numbers for contact persons or incorrect descriptions of the chain of responsibility. In contrast, other sites had extremely detailed policies that described with precision who was responsible for performing which tasks under which circumstances (e.g., one policy specified the type of music to be played by the phone system while transferring a call). Group members found it was preferable to use an easily accessible policy that was well-known to staff members and that contained highly detailed instructions. This held true especially when the instructions took into account such pragmatic concerns as what would happen during business hours versus during off-hours. Such detailed policies also provided greater confidence during stressful situations.top link

"Do"

Policy development

Workgroup members developed or adapted site-specific policies as needed. Clinicians from the research team assisted the site clinicians with implementing these policies. Development and implementation of a policy required communication between the individual clinicians and administrators in order to gain administrative support for the policy and to ensure that it fit well with other site policies and resources. In some cases, implementation required educating the administrators about the strengths and weaknesses of the current policy and the real needs of the institution. Each site was different and had to tailor the general policy to its situation (encompassing staffing levels, after-hours support, etc.). Therefore, policies were not identical across sites. These differences highlight the flexibility and adaptability of the process.top link

"Study"

After researching policies at their respective institutions, workgroup members met to review and discuss each policy. This group discussion generated a list of policy features that group members agreed were important in responding to the suicidal patient. The issues discussed included what types of incidents were of concern (e.g., any spontaneous mention of suicide versus direct suicide threat); how to evaluate the severity of the incident; who should be involved in responding (e.g., is it necessary to include a mental health specialist; is it necessary to involve a physician in all cases); how to assess the patient (e.g., with a standardized instrument, a computer-generated screening tool, etc.); and how to document the process followed (e.g., in the electronic medical record, in a chart note, or in some other manner). Workgroup members identified a set of important features. To better prioritize the perceived importance of these features, CWG members designed a Likert-type scale (Figure 2) to allow group members to rate the importance of each agreed-upon feature (1 = lowest importance; 5 = highest importance). Because one of the group's goals was to address pragmatic concerns, members also were asked to rate the feasibility of each feature along a similar scale (1 = lowest feasibility; 5 = highest feasibility). The features were divided into five domains: Personnel (who should be involved in responding to a suicidal patient), Criteria for Intervention (how and when the policy is activated), Documentation (how assessments and interventions are documented), Responsibility (who is defined as having responsibility for actions), and Procedure (how the assessmenttop link

Figure 2. Sample page from rating form

Suicide Protocol Implementation

We would like your opinion about the requirements for successful development of a suicide protocol.

Please rate each item below on a scale of 1 --5, with 1=lowest importance and 5=highest importancetop link

and interventions are to take place). Four or five features were identified in each domain, and space was left in each domain for respondents to add "other" or additional criteria. 12 Rankings were performed by workgroup members (N = 8), including care managers, mental health specialists, and research team members. The rankings were used to guide final policy development. They are expected to differ, depending upon the organizational context.

Results of this survey identified a set of key features. The following labels were given the highest ratings on both importance and feasibility: "intervention personnel includes a clearly defined chain of responsibility," "follow-up procedures are defined," and "incidents are documented in the progress note." Results of these rankings are presented in Table 1.

Table 1. Mean ratings of policy features


CRITERIA Mean Importance Rating Mean Feasibility Rating

A positive screener is used to trigger an intervention 4.2 4

A computer calculates responses and automatically triggers the suicidality protocol 3.1 3

Spontaneous mention of suicide, self-harm, or persistent thoughts of death are criteria for protocol trigger 4.67 4.1

A standard assessment is used to determine the extent of the threat 4.55 4.4

PERSONNEL

Intervention/evaluation personnel includes a mental health specialist 4.78 4.4

Intervention/evaluation personnel includes an M.D. 3 3

Intervention/evaluation personnel consists of a mental health "team" 3.78 3.13

Intervention personnel includes a clearly defined chain of responsibility 5 4.75

DOCUMENTATION

A report of contact is used to document incident 2.4 3.5

A documentation form is individually created by the site/program 3.2 2.875

Incidents are documented in the progress note 5 4.625

Incidents are documented in a computer system designed by the site/program 3.88 2.625

RESPONSIBILITY

The initial contact is made by a clinician 4 4

A notification system is created to inform higher level personnel of incident 4.11 3.875

In-person assistance by a mental health specialist 3.44 3

Incident reports/progress notes require co-signatures 4.22 4.125

PROCEDURE

A standard battery of questions is asked of each patient 4.11 4

Standard protocol to involve police if necessary 4.67 4.5

Site protocol interfaces with VAMC protocol 4.67 4.375

Follow-up procedures (referral, note to primary care provider, etc.) are defined 5 4.75

Note: Bolded items were those rated most highly for both feasibility and importance.

Development of research study suicidality policy

Based on the recommendations of the CWG and additional research project staff, we developed a suicidality policy for WAVES research participants. This policy was reviewed and approved by institutional review boards at participating sites. In the WAVES project, 342 (approximately 3 percent) veterans interviewed expressed suicidal ideation. Of these, 49 (about 14 percent) were transferred to a study clinician for further assessment. Only one reportable adverse event ensued. 11 No revisions of the policy were necessary. We interpret these statistics as demonstrating the effectiveness of our policy for the WAVES project.top link


A copy of the entire rating scale can be found at http://www.va.gov/tides_waves/.top link

"Act"

The TIDES depression care managers have instituted suicidality policies that are compatible with the policies of the medical centers in which they work. The WAVES research project has implemented research-specific suicidality policies to ensure the safety of research participants. In both cases, the policies developed and implemented were tailored to the needs of the situation. Implementation at participating sites will continue to evolve as site structures change, but the recommendations made by the workgroup allow for measurement of how well policy features meet current needs.top link

Recommendations

Based upon the experience and findings of the TIDES project collaboration workgroup, we make the following recommendations about the development of a suicide-risk response policy:

1. It is important to involve clinicians and administrators from various disciplines.

2. The policy must take into account local site resources (hours of operation, different staffing levels at times of day, etc.).

3. The policy must clearly articulate who is to perform which tasks in what order.

4. The policy must have well-defined follow-up procedures so the person who initiated the procedure is assured that the patient receives the prescribed intervention.

5. Some provision for documentation must be made to facilitate communication with other members of the patient's treatment team.

6. We suggest the use of mock patients as a method for testing the policy. Preferably, the team should perform more than one test, at different times of day, to account for different staffing levels, etc.

Reviews also should occur after institutional changes such as increases in patient load or reorganization of departmental structures. It is almost inevitable

that such testing will reveal some problem with the policy. Therefore, the team must be willing to invest the effort required to address these problems, which may be as trivial as finding the new pager number for the resident on call or as complex as clarifying which department will ultimately admit the at-risk patient.

Table 2. Summary of recommendations for developing a suicidality policy


Step Recommendations

1. Plan Form working group Group composition: multidisciplinary stakeholder group: primary care providers, nurses, mental health specialists, administrators Establish goals for the team

2. Do Obtain existing policies, formal and informal Establish priorities for effective policies Evaluate existing policies to establish strengths and weaknesses (using rating form) Assess performance with "mock" patient Design policy based on evidence obtained

3. Study Implement policy Dissemination throughout medical center (CE, staff meetings, policy directives, etc.)

4. Act Sustain policies in real-world settings

It is important to remember that final policies require an approval process, and that approval should rest upon a plan for educating all relevant staff about the details of the policy and their roles in its use. This plan should ensure that suicide prevention is included in new employee orientation sessions, clinic manuals, and periodic employee education seminars. Because one-on-one education by opinion leaders is especially powerful, it is important to identify one or more workgroup members as local experts who can answer the inevitable questions and concerns as they arise. This is likely to improve results. It is best for this expert to periodically review the policy for feasibility, especially after institutional changes such as reorganization of departmental structures.

These results emphasize the high value put on a clear chain of responsibility, effective communication, and reliable documentation when dealing a suicidal patient. A chain of responsibility not only makes it clear who is ultimately responsible, but also ---and just as important ---assures staff that a plan has been thought out ahead of time and that support exists for them during highly stressful clinical encounters. Likewise, having defined follow-up procedures reassures clinicians that someone with specialized training will provide care for the patient even after the imminent crisis has been managed. Finally, the high value placed on documenting actions in the progress notes shows the importance of communication that allows all members of the treatment team to have access to the information. For example, if a patient is hospitalized over the weekend because of suicidal thoughts, the patient's clinician will benefit from having that information readily available on Monday morning.

It is encouraging that the items rated most highly on importance were also rated most feasible. Thus, in the experience of our group members, the existing VA infrastructure provides enough support to make feasible a clearly defined chain of responsibility, a well-thought-out follow-up plan, and a consistent form of documentation.

Policies for threatened-suicide management in VA clinics can show wide variations and implementation difficulties. Locally adapted policies sharing common key features can be developed using a threatened suicide-management policy development tool. Evaluation of the effectiveness of the final policies is ongoing, and should be a routine part of large-scale QI efforts such as this.top link

Limitations

This study was designed as an evidence-based QI method. We described development of a suicide prevention protocol as one important piece of TIDES/WAVES. This project was conducted by VA personnel within VA facilities. We would expect the process to occur differently and to have somewhat different results in other settings. However, the general process of deciding on priorities and adjusting policies on the basis of real-world performance is flexible enough to be adapted to a wide range of settings. The specific policy features are likely to be relevant to other large not-for-profit managed care organizations, and possibly to primary care practice networks. However, they are less likely to apply to small independent primary care practices. Our data on policy importance and feasibility are qualitative and are best used as a starting point for quality improvement.

Our results for the sites participating in the TIDES/WAVES projects represent a range of geographic and organizational characteristics. The personnel involved came from various facilities, ranging from relatively large inpatient medical centers to smaller outpatient clinics. Despite the differences in size and resources available to them, participants were able to reach a consensus on the important aspects of suicide policy development. This observation suggests that certain common features exist across a variety of settings. Suicide is especially pertinent to the VA population of older males, but it is by no means a problem unique to the VA. Some adaptations would have to be made in terms of the details of a suicidality policy (e.g., non-VA settings may not have access to an electronic medical record, so documentation would have to follow whatever process was most appropriate for the site). However, our general recommendations would adapt to other facilities.top link

Conclusion

The standard of care for today's health care system should include highly functional and integrated systems to respond to the suicidal patient, wherever he or she initially presents. Development of a working institutional suicidality response policy requires a cooperative effort among clinicians and administrators. When the policy has been established, it is imperative to test it for practicality before finalizing it. After the policy has been finalized, staff must be made aware of its existence and educated about their designated roles in implementing it.top link

Acknowledgments

The authors acknowledge the contributions of all members of the TIDES Collaboration Workgroup. In particular, we acknowledge the valuable contributions of Lisa Rubenstein, M.D., who contributed highly valuable comments and suggestions throughout; Alissa Simon, who helped with the preparation of the survey forms; Lisa Gregory, who assisted with development of graphics for this paper; and Carol Simons, who provided valuable coordination of the workgroup. The TIDES and WAVES studies were funded by VA Health Services Research and Development grants (MNT-01-027 and MHI-99-375 respectively).top link

Authors' affiliations

VA Puget Sound Healthcare System, Seattle, (LB, BF, EC). University of Washington Department of Psychiatry and Behavioral Medicine (BF). St. Cloud VA Medical Center (KV). Little Rock VA Medical Center (KB). Louis Stokes VA Medical Center, Cleveland (BR). VA Greater Los Angeles Healthcare System (BS, SS). VA Black Hills Healthcare System, Host Springs, SD (LK). VA Veterans Integrated Service Network 10, Cleveland (SO). University of Arkansas for Medical Sciences College of Medicine (LW). Bronx VA Medical Center (JF).

Address correspondence to: Laura Bonner, Ph.D., Northwest Center of Excellence, HSR&D (152), 1660 S. Columbian Way, Seattle, WA 98108; phone: 206-277-1780; e-mail: laura.bonner@med.va.gov.top link

References

1. Hall RC, Platt DE, Hall RCW. Suicide risk assessment: a review of risk factors for suicide in 100 patients who made severe suicide attempts. Evaluation of suicide risk in a time of managed care. Psychosomatics 1999. 40(1):18-27. Jan-Feb. (PubMed)

2. Hirschfeld R. When to hospitalize patients at risk for suicide. Ann N Y Acad Sci 2001. 932:188-99. (PubMed)

3. Peruzzi NB, Bongar B. Assessing risk for completed suicide in patients with major depression: psychologists' views of critical factors. Prof Psychol Res Pr 1999. 30(6):576-80.

4. Motto JA, Bostrom AG. A randomized controlled trial of postcrisis suicide prevention. Psychiatr Serv 2001. 52(6):828-33. (PubMed)

5. Vining R. Assessing risk of suicide. Samaritans' scoring system helps develop judgment. Br Med J 1995. 310(6972):126-7.

6. Lambert MT, Fowler DR. Suicide risk factors among veterans: risk management in the changing culture of the Department of Veterans Affairs. J Ment Health Admin 1997. 24(3):350-8.

7. Lehmann L, McCormick RA, McCormick L. Suicidal behavior among patients in the VA health care system. Psychiatr Serv 1995. 46:1069-71. (PubMed)

8. American Association of Suicidology. Available at http://www.suicidology.org/associations/1045/files/AAS%20Suicide%20Fact%20Sheet%20%2D%20Some%20facts%20about%20suicide%20in%20the%20USA%2Epdf . (Accessed 2004 Apr 16.)

9. Dlugacz YD, Restifo A, Scanlon KA, et al. Safety strategies to prevent suicide in multiple health care environments. Jt Comm J Qual Saf 2003. 29(6):267-78. (PubMed)

10. Logue EM, Parrish RS. Suicide precautions in a medical/surgical unit. Nurs Manage 1998. 29(10):33-4.

11. Simon B, Mittman D, Bonner L, et al. Risks to patient privacy in adverse event reporting for HSR&D studies. Poster presented at Annual Meeting, 2004 Mar; VA Health Services Research and Development, Washington, DC.