AN EVALUATIOtl OF THE RMP REVIEW CRITERIA Final Report on Contract HSM 110-72-329 Medical Care and Education Foundation, Inc. Boston, Massachusetts June 1973 This report was prepared in partial fulfillment of Health Services and Mental Health Administration Contract HSM 110-72-329, "An Evaluation of the RMP Review Driteria." Harold W. Keairnes,.MD Project Director Betty Bradshaw, MBA Project Manager Allan L. Willey, MBA Project Consultant Project Staff idward Miller, AB Patricia Kress, AB Maureen Fitzgerald,.MA Maureen Vachss, AB Patricia Miller, AB TABLE OF CONTENTS PAGE vii Acknowledgements ix Executive Summary Chapter I The Setting for the Evaluation Chapter II Approach to the Evaluation and 19 Reports of Component Activities A. Statistical Studies of RMP Review Criteria Scores 23 B. Reported Usefulness of the RMP Review Criteria Scores 29 C. Values Expressed in Desci4iptions of Local RMPS 43 D. Summa of Conferences with RMPS Staff ry 53 Chapter III Evaluation Findings 61 Chapter IV Re comen da ti on s 69 Appendix A RMP Review Criteria 87 Appendix B Description 'and Scope of Work 95 for Contract v ACKNOWLEDGEMENTS The collaboration and support of the Project Officer, Joseph de la Puente, contributed significantly to the completion of this contract. Muth of the work could not have been done without his support and the support of Roland Peterson, Director of the Office of Planning and Evaluation and Robert Walkington, Chief, Evaluation Branch. Ms. Judy Silsbee of the Division of Operations and Development provided perspectives that contributed to the analysis of the findings in terms of the needs of RMPS. Throughout this contract, RMPS staff, at all levels throughout the organization, were supportive and helpful. Fred R. Bookstein, Harvard University, consulted in the design of the statistical studies, supervised the computer analysis, and provided the primary interpretation of the findings. Consultations with James R. Miller, III, Stanford University and Alan Sheldon, Harvard University served to provide guidance in technical aspects of worth assessment procedures and conference techniques. Robert W. Murphy, Executive Director of the Medical Care and Education Foundation, Inc. was a ready resource for identifying the concerns Of RMP coordinators and for helping the project director maintain a practical perspective in this work. Patricia S. Keairnes reviewed the manuscript at multiple stages in its development and made many substantive suggestions. vij EXECUTIVE SUMMARY OF FINAL REPORT ON AN EVALUATION OF THE RMP REVIEW CRITERIA Contract No. HSM 110-72-329 ix Regional Medical Programs are a-federally funded program that provides grants to fifty-six local Regional Medical Programs "to improve generally the quality and enhance the capacity of the health manpower and facilities available to the Nation and to improve health services -for persons residing in areas with limited services" by promoting regional cooperative arrangements among medical schools, research institutions, and hospitals of research and training and regional linkages among health care institutions and providers that will make available to their patients the latest advances in the prevention, diagnosis, treatment and rehabilitation of heart disease, cancer, stroke, kidney disease and other related diseases. Because the legislation provides such a broad mandate, all applications for grant support are subjected to a formal review and apbroval process. The three year program grant application undergoes a site visit evaluation, Regional Medical Program Review Committee consideration, and National Advisory Council action. Annual continuation rant applications are reviewed by the Staff Anniversary Review Panel ?SARP). Supplemental grant applications are reviewed by SARP and the National Advisory Council. To lend uniformity to the review process, a set of evaluation criteria were developed and introduced into the review process in 1971. In 1972, after the Regional Medical Program Review Criteria had been in use for approximately one year, a decision was made that an inde ehdent contractor should eva p luate them and their contribution to the review process. The Scope of Work for that contract called for an evaluation of previous experience and recommendations regarding their continued use. This is a summary of the final report from that contract. Description of the RMP Review Criteria The RMP Review Criteria consist of three major criteria groupings,t,wenty individual criteria and more than one hundred questions. The questions under each individual criteria are intended to clarify the meaning of the criteria and to indicate the basis upon which a numerical score can be assigned. The numerical score reflects the reviewers' evaluation of a local regional medical program against the individual criteria. The numerical scores for individual criteria are summed to obtain scores for the major criteria and for all criteria. Weights assigned to each criteria specify their relative importance. Regional Medical Programs are classified as "A", "B" or 'IC" regions on the basis of their average weighted total scores. The Review Criteria play a role in the review process at several different levels. They are a general guide for the preparation of the historical profile in the staff briefing document, a summary report prepared by staff in the Division of Operations and Development to accompany the submitted triennial or.anniversary review grant appli- xi cation through the review process. Site visit reports related to the review of triennial grant applications are supposed to be formatted according to the Review Criteria and are expected to summarize all the relevant information in regards to each individual criteria. The Review Committee for triennial grant applications and the Staff Anniversary Review Panel for anniversary review grant applications are required to score the programs under review against the Review Criteria and to include the scores with their reports and r6commenda- tions. The scores may influence funding decisions by the Director of RMPS although no funding award may exceed that approved by the National Advisory Council. The notification of award and the advice letter sent at the conclusion of the review process to each applicant program summarizes the findings and recommendations, not uncommonly, in the language and conceptual framework of the Review Criteria. Description of the Evaluation Contract As the staff of Regional Medical Programs Service saw e RMP Review Criteria in use, they felt that significant benefits had accrued to the program, but that further improvements in the criteria and the ways in which they were used might lead to even greater benefits. They, therefore, prepared a Scope of Work for an evaluation contract which specified two purposes: 1. To assess the internal validity of the Review Criteria used in rating Regional Medical Programs, and 2. To identify the perceptions of those utilizing the criteria in the National review protest as to their effectiveness. The Scope of Work detailed a series of specific questions addressed to those purposes.and called for a final comprehensive report that would include "recommendations and suggestions (and basis thereof) for modifying the criteria, their application in the present review process, and that overall review process itself." Contract HSM 110-72-329 to evaluate the RMP Review-Criteria was awarded to the Medical Care and Education Foundation, Inc., in June, 1972. Based on the Scope of Work for the contract, the technical proposal submitted by MCEF and negotiations between the project office and the project director, three specific evaluation tasks were undertaken, statistical studies related to internal validity, 2. Interviews with ten selected people participating in the review process at different levels, and 3. Conferences with selected RMPS staff. These specific tasks were supplemented by analyses of various documents, xi i informal interviews with RMPS staff, and formal and informal consulta- tions with various-.experts. Findings from the Evaluation Contract The major findings resulting from the evaluation contract can best be reported in relation. to the two purposes specified in the Scope of Work. To assess internal validity The evaluation of the internal validity of the Review Criteria consisted of a series of analyses of scores prepared by individual members of the Review Committee and the Staff Anniversary Review Panel, the only individuals who were required to evaluate local Regional Medical Programs by rating them against the Review Criteria. Each member of the two groups was.given a rating form and asked to score all programs considered during a particular review cycle on that form. The scoring consisted of assigning a numerical value of 1, 2, 3, 4. or- 5 for each of the twenty individual criteria to indicate the reviewer's judgement as to the program s quality. "5" indicates hig hest quality; "1" indicates lowest quality. The values are assigned independently by each reviewer after the presentation and discussion of a program had been completed and the recommendations had been specified. The scores of all reviewers were then summed with and without weighting weighting points are assigned to indicate the.relative importance of each criteria -- and the summed scores were reported with the Review Committee or Staff Anniversary Review Panel recommendations. The major questions to be addressed by these analyses were: 1. To what extent is. there variance in total scores, scores on major criteria groupings, and scores on -individual criteria among programs classified as "A", "B" and "C" regions. 2. To what extent do the RMP Review Criteria constitute a scale or measure? 3. To what extent is there variance in scores by the same der differe observer un nt conditions o r by different observers under the same conditions. The,analyses indicated that there,were statistically signifi- cant differences in the raw and weighted total scores and the scores on major criteria groupings among programs classified as "A", "B" and "C" regions. Because prograwis.were classified on the basis of their total scores and because the correlation between scores on the major criteria groupings and the total scores exceeded 0.9, this finding was not unexpected. xiii the individual criteria An analysis of the scores for showed exceptionally high inter-criteria correlations. When the correlation matrix was subjected to principal components factor analysis, only one factor was identified. If the major criteria re should have been three factors. groupings had been valid, the The fact that only one factor was identified indicated, according to the consulting statistician, that the Review Criteria did not constitute a scale or measure that was in any way superior to a single rating on Overall Quality. Scoring on the individual criteria did not, in his opinion, contribute to an identification of those specific qualities which differentiate between good and poor RMPS. More specifically, a good program gets good scores on all criteria and a poor program gets poor scores on all criteria. The analyses also showed Exceptionally high inter-observer correlations. There was no significant bias of the primary reviewers, as a class, nor of any individual reviewer. The absence of significant bias was shrouded in such an extraordinary degree of agreement among the scores of all the reviewers for each program that independence of judgement appeared to be absent. Stated another way, the scores for a given program were so uniform as to t the at least tacitly, ore the atings. In the absence of scalarity and independent peer judgement, the assessment of internal validity becomes meaningless. The internal consistency.of the scores and the consistency among different observers indicated that the Review Criteria scores express the consensus of the review groups as to the quality of the programs rather than the convergence of independent opinion. To identify perceived effectiveness in assessi ng comparative quality Conferences and interviews were used to identify the perceived effectiveness of the Review Criteria in assessing comparative quality. Five of the six specific questions addressed by.these activities were concerned with the relative importance of major criteria groups and individual criteria, the-availability of information and the rationale for@adding and deleting criteria. The sixth question related to perceived improvements in the review process that had resulted from use of the criteria. A conference that was held to answer.the first five questions failed to achieve that objective because, under the conditions of the conference, selected RMPS staff manifested an 6xtraordinatory diversity of opinion on the subject. Although they agreed on the importande.of the three major criteria groupings: Performance, Process, Program Proposal --, they were unable to cluster the specific questions from the Review Criteria into an agreed uponset of individual criteria I categories. As.a result, there was no basis or describing the relative lo@' V-4 xiv 'a or a rationale for adding or deletin importance of criteri g criteria. The announced phase-out plans for RMP made it unfeasible to pursue further activities designed to develop a basis for answering those questions. There were, in that conference as well as in another con- ference with selected RMPS staff and in ten interviews with selected participants in the review process, many criticisrsregarding the lack of appropriate information for evaluating programs against the existing criteria. The second conference with RMPS staff and the ten interviews indicated that the Review Criteria had been a very effective manage- ment tool f or -4 t on of Fegi-onal Mei ams erv c ess. The Review Criteria were developed and introduced into the review process over a one year period spanning the latter part of 1970 and the first part of 1971. This occurred during the transition to the Triennial Award and Developmental Component grant application. procedure from the original project grant application procedure, a period of fiscal cut-backs in the total RMPS program and the intro- duction of the modifications in RMPS program called for by the RMP renewal legisl ation of 1970. The evidence from the conferences and interviews suggested that RMPS had, at least, four specific purposes for the development and introduction of the Review Criteria into the review process: 1. To facilitate a change in the review process from a project-oriented to a program-orien ted approach; 2. To provide additional structure to the technical as'sistance provided by RMPS to local programs to foster better program planning, development and operation; 3. To establish a mechanism for preserving continuity in the review process in the presence of changing personnel on the various review bodies; and 4. To serve as part of the rationale for selective funding decisions made by the Director of RMPS. It was reported that the RMP Review Criteria had made sighi- ficant contributions to the achievement of these purposes -- all of which were achieved -- and that these contributions resulted primarily from the framework that the Review Cri e organization of infoma Ss on, reports, v s and advice letters. xv The participants in the interviews and conferences, while reporting the effectiveness of the Review Criteria as a management tool, identified many weaknesses that would need to be resolved if the criteria continued to be used. These included criticisms of: 1) the excessive number of criteria, overlap of criteria and absence of precise definitions; 2) the artificiality of the scoring process, .71, and 3) the excessive number of questions, their redundancy, and their lack of explicitness. These criticisms did not, however, detract from their support of continued use of the Review Criteria in the review process and their strong endorsement of efforts to correct the identified weaknesses. Recommendations The evaluation of the RMP Review Criteria although reported in the findings, assumes a more coherent form in the recommendations. The seventeen recommendations are based on findings, criticisms and comments as reported in the three specific studies, the reports of the two conferences with RMPS staff and additional data derived from analyses of RMP-related documents, informal interviews and consulta- tions. The full rationale for each of the recommendations is reported in Chapter I V. All recommendations are submitted on the premises that Regional Medical Programs will continue to operate as a federal program and that the Review Criteria, in modified form, will continue to be used by RMPS as an adjunct to future RMP activities. The first six recommendations related to the pur oses for p which the RMP Review Criteria should be developed and utilized in the future. Recormnendati on #1: The Review Criteria should be used to develop and maintain a balance ective in the review Drocess on performance, proce Comment: The review process, according to the findings, has been transformed from an emphasis on discrete projects-to an emphasis on programs as whole units. Unfortunately, this has resulted in a focus on accreditation" with little attention, particularly in the rating procedure, to past performance or program proposal. Recommendation #2: The Review Criteria should continue to be used as a device for promoting continuity in the review process. Comment: The Review Criteria'were reported to be helpful in promot ng continuity in the review process. Thereobviously, is a continuing need as long as there continues to be changes in personnel involved in that process. xvi Recommendation #3: The RMP Review Criteria should be used to assure consistency of evaluativ ements within all levels of the review process and among different review cycles. Comment: An unexpected diversity of opinion and viewpoint expressed by staff members from different units within RMPS at one of the con- ferences from this contract has been identified and reported. To a lesser, extent, the same diversity was illustrated by the inter- views with members of the National Advisory Council, Review Committee, and Staff Anniversary Review Panel. Although diversity is probably essential to the development of exciting and innovative 'programs such as Regional Medical Programs, it can create problems in review and evaluation processes. Review and evaluation implies a single set of expectations uniformly applied to all applicants Evaluative judgements, ideally, should be based on explicit standards that are not dependent uporf the composition of the review body, the particular interests of individuals, or changing emphases over time. T'he Review Criteria, if the appropriate procedures are developed, can be used to assure consistency of evaluative judgements through the application of management rihciples to the review process. p Recommendation #4: The RMP Review Criteria should continue to serve as part of the rationale for selective funding decisions. Comment: Selective funding is based, in part, on the use of incentives to promote the development and operation of high quality local Regional Medical Programs. The use of the Review Criteria, according to reports, encouraged more consensus as to which programs were of "high" quality and of "low" quality a:nd, therefore, provided a useful rationale for selective funding decisions related to the use of incentives. Recommendation #5: The RMP Review Criteria should be used to provide a consistent framework for local prc pment and operations. Comment: The Guidelines for Regional Medical Programs plus various publications, statements and letters have served as the primary frame- work,for local program planning, development and operations. As RMPS staff began to be more active in providing technical assistance to 1 s, they.found, according to reports, that the Review ocal program Criteria were being used by some local programs as program guides.' Although the RMP Review Criteria were designed primarily for the review process, it can be assumed that more programs will have good scores and higher quality programs when they consciously use the Criteria as program guides. To be used as such program guides, the Review Criteria should be modified in their content and widely publicized in official RMPS documents. xvii Recommendation #6: Th Review Criteria should be used to facilitate the e development of information that can be used in RMPS program evaluation and in the justification of Reqional e ica lic. Programs to t e on Comment: In early 1973, according to RMPS staff, many local Regional Medi significant improvements in the planning, cal Programs were making development and operation of their programs and some as indicated by high Review Criteria scores and "A" ratings, were ju@ged to be of high quality. In spite of these favorable evaluations reported out of the review process, recommendations to terminate the Reg ional Medical Programs were made by the Administration. The recommendation, as re orted on p. 383 of the Appendix to the Budget Proposal for Fiscal p Year 1974, includes the following statement: ... there is little evidence that on a nationwide basis the RMP's have materially aff cted the health care delivery system..." . That statement implies that the Adminis tration had objectives related to changes in the "health care delivery system" that were not addressed by the evidence presented Modifications of the Review Criteria suggested in subsequent recomenda- tion§ might have been addressed earTier if the Review Criteria had been considered as part of the program evaluation and program Justification .effort of RMPS. The next three recommendations deal with changes in the content and design of the Review Criteria and are concerned with: A. Compatibility of content with Administration objectives for Regional Medical Programs; B. Broader participation in the development of the content; and C. Technical changes in design to eliminate criticisms regarding: 1. Excess focus on I'accreditation",, 2. Excess number of criteria, 3. Confusion with the numerical scoring process, and 4. Weaknesses of questions. Recommendation #7: The content of the Review Criteria should be related, whenever possible and appropriate, to specific guide ines be egtabl. station and the 1 gitlative roceedings. xviii Recommendation #8: a should be modified The content of the Review Criteri arti ipation under the leadership of RMPS staff with the p of National A( We-mbers, RMP coordinator and staff, and Regional Advisory Group members. Recommendation #9: The design of the Review Criteria should be significantly modified as follows: a. The major criteria should be retitled Achievements, nd Proposed Plans; Organizational Processes a b. The individual criteria should be reduced in number from twenty to twelve or less; c. The numerical values of 1, 2, 3, 4, and 5-should companied by statements that indicate the be ac precise meaning of each numerical value for each individual criterion and each major criteria grouping; and d. The questions should be reduced in number and should be translated into a form that can be answered by Present/Absent or that calls for specific qualitative descriptions or quantitative measures. The last eight recome 6dations are based on the prior recommendations and call for significant modification in the operations of RMPS, especially those operations related to the review process. In these recommendations, the terminology of Levels is introduced.. The proposed Level One Criterion is OverallQuality. The proposed Level Two Criteria are Achievements, Organizational Processes, and Proposed Plan. The Level Three Criteria correspond to the present individual criteria and are suggested in Table 4-1. The Level Four Descriptors are proposed as the replacement for the specific questions of the present Review Criteria Examples are shown in Table 4-2. Recormnendation #10- The Review Criteria should be included in the Guidelines for Re ional Medical Proqrams and/or in an at ons and Pr at local programs understand their importance for local program planning, development and operation. A complete description of how the Review Criteria are used by RMPS in the review process and their technical assistance activities should also be provided in the same way. xix Recommendation #11: t should be RMPS staff in the Office of @ystems Managemen responsible for the design c in lieu of the sp cific ques Dlication so that the infomat on appropy is available. for the analysis of the information related to these des e and/or can ed, and for the tion t h the MAnaqement Infomatio s em of rel;ted to tne@escr cess. Recommendation #12: RMPS staff in the Division of Operations and Development should be -responsible for Criteria with the particip National Council, t e Review Committee and local RMPs and for preparing the scores with accompanying statements of justification for eac individual criterion for the staff briefing document. Recommendation #13: site visit teams should study the Level Three Criteria score ents of justification as prepared by the Division of Operations and Development and the Level Four Descriptor information as prepared by the Office of System Management and should descr be, in their site v.' it report their evaluation of the subi atz , and data. T-heir Id include@@ fica he ubmitted Level Three Cr teria scores if indicated by the stte visi't findings. Rec,omendati on -#14: The Review Committee and Staff Anniversary Review Panel shoul-d including modifications from the site visit team and shou'ld include evalu e comments in their report, but they fication only n occuring in the present t gt it is recommended that flect the consensus t ho'uld be included recommendations. xx Recommendation #15 The Nationa Council should study the scores 1 Advisorv and statements of justification on Level Two and Three Criteria in staff briefing document, site visit report, Review Committee and Staff Anniversary Review Panel reports and should include eve mments in their report. In addition-, it is re( nat @he-riational Advisory Council identify by consensus a score and an acco stater ification that reflects their eval uation is with Recommendation #16: RMPS staff in the Office for Planninq and Evaluation should monitor the scores, justifli d evaluative comments reported by the Office of Systems Management, Division of Operations and Development, site visit teams, Review Committee, Staff Anniversary Review Panel and National Advisory Council and should analyze them for consistency of evaluative a o@ng-the review bodies, for perceived ade e submitted information, and for redundant a Recommendation #17: The Review Criteria data, scores and statements of justification from each of the review bodies should be submitted to the local-Reqional Medical Pr rams as a verification procedure rther st e development Concl usi ons The be'lief that the RMP Review Criteria had made significant contributions to the operations, specifically the review procedures, of RMPS was confirmed by this evaluation. Continued use of the Review Criteria, however, should be accompanied by significant improvements in their content and design and by alteration of the ways in which they are used. Seventeen specific recommendations regarding the purposes of the Review Criteria and their content, design and appli- cation have'been presented. These recommendations should lead to improvements in the effectiveness and acceptability of the Review Criteria in the review process in the merging of local program efforts with national guidelines and objectives, and in the development of mechanisms for improving the management of the review process. xxi CHAPTER I THE SETTING FOR THE EVALUATION -239, Regional Medical Programs were authorized by PL 89 the."Heart Disease, Cancer, and Stroke Amendments of 1965 The legislation authorized the Surgeon General of the United States to grant funds to area-wide organizations for the planning and opera- tion of local Regional Medical Programs for the following purposes: a) Through grants, to encourage and assist in the establish- ment of Regional Cooperative arrangements among medical schools, research institutions, and hospitals for research and training (including continuing education) and for related demonstrations of patient care in the fields of heart disease, cancer, stroke and related diseases,, b) To afford to the medical profession and the medical institutions of the Nation, through such cooperative arrangements, the latest the opportunity of making available to their patients advances in the diagnosis and treatment of these diseases; and C) By these means, to improve gene rally the healt manpower and facilities available to the Nation, and to accomplish-thes.e ends without interfering with the patterns, or the methods of financing, of patient care or professional practice, or with the administration of hospitals, and in cooperation with practicing physicians, medical center officials, hospital admin istrators, and representatives from appropriate voluntary health agencies. Fifty-six local Regional Medical Programs serving the entire United.States and the Commonwealth of Puerto Rico have received grants under this legislation. Each grant application funded by Regional Medical Programs Service has had the approval of a local advisory group and the National Advisory Council. At the National level,' National Advisory Council action and recommendations are preceded by site visits and Regional Medical Program Review Committee considera- tioh of grant applications for three year periods and grant appli- cations from new local organizations aspiring to operate a regional medical program. Annual continuation grant applications that fall within the National Advisory Council approved plan and budget are reviewed and approved by the Staff Anniversary Review Panel without further consideration unless there is a request for supplemental funds or a change in program. To facilitate some needed changes in the review process, a set of Review Criteria were developed and introduced into the review process of Regional Medical Programs Service (RMPS) as a routine procedure in the summer of 1971. The development of the RMP Review Criteria began in the summer of 1970 in respons e to the perceived need by RMPS staff to facilitate a transition from a project oriented to a program-oriented review process, to provide a more structured framework for technical assistance to local programs 3 than existed at that tirm, to establish a mechanism for preserving continuity in the review process with turnover- of personnel on the various review bodies, and to serve as part of the rationale for selective funding decisions by the Director of RMPS. The initial set of criteria were developed and pre-tested on eighteen site visits in the fall of 1970. The semi-final set of criteria were developed in a crash effort lasting three to four weeks in February, 1971, to meet administrative requirements of the Health Services a n' (HSMHA). Minor mod and Mental Health Administr tio ifications were made in the spring of 1971 so that they could be used on a routine @basis. The RMP Review Criteria consist of twenty individual criteria organized into three major groupings. The relative importance of each of the criteria, both major and minor, are specified in weightings--shown below in parentheses--which total to one hundred. The information to be used in evaluating a local Regional Medical Program-against these criteria is specified in a series of questions under each of the twenty criteria -- see Appendix A. A. Performance (40) 1. Goals, Objectives, and Priorities (8) 2. Accomplishments and Implementations (15 3. Continued Support (10) Eli 4. Minority Interests (7) B. Process (35) 1. Coordinator (10) 2. Core'Staff (3) 3. Regional Advisory Group (5) 4. Grantee Organization (2) 5. Participation (3) 6. Local Planning (3) 7. Assessment of Needs and Resources (3) 8. Management (3) 9. Evaluation (3) 4 C. Program Proposal (25) 1. Action Plan (5) 2. Dissemination of Knowledge (2) 3. Utilization Manpower and Fac'lities (4) 4. Improvement of Care (4) Short-Term Payoff (3) 6. Regionalization (4) 7. Other Funding (3) The Review Criteria play a role in the review process at several different levels. They serve as a general guide for the pre- paration of the historical profile in the staff briefing document, a summary report prepared by the staff in the Division of Operations and Development to accompany the submitted triennial or anniversary review grant application through the review process. Site visit reports related to the rev r iew of triennial g ant applications are supposed to be formatted according to the Review Criteria and are expected.to summarize all the relevant information in regards to each individual criteria. The Review Committee for triennial grant applications are required to score the programs under review against the Review Criteria and to include the scores with their reports and recommendations. In this scoring process, the members of these two review groups are asked to rate the programs under consideration by assigning a numerical value of I'2,3,4 or 5 for each of the twenty individual criteria to indicate the r6viewer's judgement as to the program s quality. The scores are then summed, averaged and reported with'and without weighting to the National Advisory Council. On the basis of the official scores from the Review Committee and the Staff Anniversary Review Panel, local programs were classified as "A" I "B", or "C" regionswith"A" regions being the most highly regarded. The scores and the classification may influence funding decisions by the Director or RMPS although no funding award may exceed that approved by the National Advisory Council. The notification of award and the advice letter sent at the conclusion of the review process to each applicant program summarizes the findings and recommendations, not uncommonly, in the language and conceptual framework of the Review.Criteria. 5 Des rii)tion of the Evaluation Contract T.i i As the staff of Regional Medical Programs saw the RMP Review Criteria in use during the period 1971-1972, they felt that significant benefits had accrued to the program, but that further improvements in the criteria and the ways in which they were used might lead to even greater benefits. They, therefore, prepared a Scope of Work for an evaluation contract which specified two purposes: 1. To assess the internal validity of the Review Criteria used in rating Regional Medical Programs, and 2. To identify the perceptions of those utilizing the criteria in the National review process as to their effectiveness. The Scope of Work - see Appendix B - detailed a series of specific questions addressed to those purposes and called for a final comprehensive report that would include "recommendations and suggestions (and basis thereof) for modifying the criteria, their application in the present review process, and the overall review process itself. Contract HSM 110-72-329 to evaluate the RMP Review Criteria was awarded to the Medical Care and Education Found6tion, Inc., in june 1972. Based on the Scope of Work for the contract, the techni- tal proposal submitted by MCEF and negotiations between the project office and the project director, three specific evaluation tasks were undertaken: 1. Statistical studies related to internal validity, 2. Interview with ten selected people partici ating in p the review process at different levels, and 3. Conferences with selected RMPS staff. These specific tasks were supplemented by analyses of various documents, informal interviews with RMPS staff, and formal and informal consultations with various f experts.- Some modifications were negotiated because o the scheduled phase-out plans of RMPS - see Appendix B. Although the essential content of the evaluation contract was specified in the Scope of Work, the work was approached by the contractor within the context that existed at the time that the Review Criteria were developed and formally introduced into the review process. This context included the history of the legislation for Regional Medical Programs, especially the renewal legislation of 1970, and the history of the review process within Regional Medical Programs, especially the changes that were introduced in the fall of 1970. - The work was approached with the knowledge that explicit standards for the evaluation of program performance were uncommon perhaps non-existent, within the Department of Health, Education and 6 Welfare at that time and that this attempt to develop such standards was complicated by the changes occuring within Regional Medical Programs in regards to the purposes for which RMP had been established and to the procedures for review approval and funding of local programs. Legislative History Ordinarily legislation is introduced into the House of Representatives and/or the Senate of the United States. The legisla- tion is then referred to a committee for hearings and a report. The committee report is introduced on the floor of each congressional body and the legislation is then discussed, possibly amended, and approved or disapproved. After the legislation is approved by both the House and the Senate, it is referred to a conference committee which resolves any differences and reports the final version of the legislation back to both bodies for final approval. The reports of the committees plus the discussion on the floor provide the basis for interpreting the language of the final legislation. The legislative history of Regional Medical Programs began in 1965 in the 89th Congress with H.R. 3140 and S. 596 " A'Bill to Armnd the Public Health Service Act to Assist in Combating Heart Disease, Cancer,-Strbke and Other Major Diseases." It continued with renewal legislation, hearings, reports and discussion all of which form the basis for the legislative history. The original legislation for Regional Medical Programs was stimulated by the findings of the Report of the President's Commission on Heart Disease, Cancer and Stroke, issued in 1964. The Commission, made up of distinguished physicians, scientists, and informed citizens, established eight subcommittees and a method of operation which called for: Ill. The collection of information from agencies, groups, and institutions concerned with these diseases .... 2. The holding of hearings at which expert witnesses... presented their views.... 3. The preparation of the report and its recommendations ... The Commission Report, which led directly to introduction of the legislation authorizing Regional Medical Programs, focused attention on a number of long-term trends including the development of a national biomedical research community of unprecedented size and gradual move- ment towards attempts at regionalization of.hbalth resources. The Report recommended the establishment of a nat ional network of regional heart disease, canter, and stroke centers for clinical investiga- tion, teaching and patient care in universities, hospitals, and 1 Report to the President: A National Program to Conluer Heart Disease, Cancer and Stroke, Vol. 1, December 1964, U.S. Government-Printing Office, Washington, D.C. - pp 89-90 7 rch institutes across the country. It also recommended a resea series of diagnostic and treatment stations in communities which would develop a close relationship with the regional centers. The third major recommendation was that a broad and flexible program of grant support be undertaken to stimulate the formation of medical complexes whereby university medical schools, hospitals, and other health care and research agencies and institutions work in concert. Legislation in response to the Commission Report was introduced in the Senate as S. 596 by Senator Lister Hill of Alabama. -The Senate version as reported came closer to the recommendations of the Commission Report than 'did the later House version. As written in the Report of the Senate Committee on Labor and Public Welfare on S. 596: "This legislation would implement recommendations of the President's Commission. The primary thrust of this bill is to provide for the planning, establish- ment, and operation of regionally coordinated medical complexes for heart disease, cancer and stroke, and other major diseases which will link together medical centers, categorical research centers, and diagnostic and treatment stations located in community hospitals 2 or other health facilities In the House, the bill was introduced as H.R. 3140 by Rep. Oren Harris of Arkansas. The Committee on Interstate and Foreign Commerce held extensive hearings during which it became apparent that some groups did not favor the concept of regional medical complexes. While testimony favorable to the legislation was submitted on behalf of the American Heart Association, American Cancer Society, and the Association of American Medical Colleges, testimony in opposition was submitted by the American Medical Association and several State medical societies. Because of the substantive objections of .some of the witn esses the bill was modified substantially in the House committee. As discussed by Rep. Harris in the floor debate: "There were great fears that there would be a major Government medical program set up with clinics, categori- cal centers, administrative centers, hospitals, and so forth operated by the Government. So we decided that instead of calling these by the term 'complexes,' which had developed an image.of that kind, we would refer to them in the bill as 'programs'. The bill provides for programs 2 Report of the Committee on Labor and Public Welfare, U.S. Senate, to accompany S. 596, U.S. Government Printing Office, Washington, D.C., 1965, p. 9. 8 s, hospitals and utilizing existing medical center rangements institutions. We provide for cooperative ar whereby medical schools in cooperation with clinical centers in the area and with the hospitals in the area, and other health activities, shall set up an advisory local committee. That advisory local committee will decide.,,3 Rep. Ancher Nelson of Minnesota, a member of the Committee on Interstate and Foreign Commerce, reflected the view of organized medicine that the bill should have had more discussion and been postponed for a year, during House debate on the bill: "I agreed that more time would be useful to allow the discussion to run its course and supported a motion to defer action until next session of the Congress. This did not prevail because the White House cannot let Congress do its work in an orderly fashion these days and apparently it was ready to settle for anything contuing the words, heart, stroke and cancer. Despite misgivings on the part of many of its members, the committee settled down to write some le gislation which could meet the objections and still make a start in the direction indicated in the original charge of the President's Commission. The result is the bill before you toqay, which came from the committee with full support."" Representative Nelson's remarks to the House, as recorded in the Congressional Record, provide a good summary of changes made in the proposed legislation by the Committee on Interstate and Foreign Commerce and provide a clear statement of Congressional intent, an intent which was subsequently modified by legislative amendments and program experience: "The changes are many and they are not mere clari- fications or exercises in semantics. They change an amorphous mass of objectives into a recognizable program which deals with units and controls thoroughly understood by those wh 0 must work with them. The bill now talks in terms of medical programs, put together by existing institutions under the eye of a local advisory board. it talks about cooperation and not coordination. The former means voluntary involvement and the latter infers an imposed plan. It talks of hospitals and not of diagnostic and treatment stations. The latter is an entity not familiar to practitioners, but we can all visualize a hospital and have a definite idea what it does, what it looks like and who runs it... 3Congressional Record- House of Representatives, Eighty-Ninth Congress, .First Session, Volume III, Part 22, U.S. Government Printing Office, Washington, 1965 . 24123, Sept. 1965. p 4 Ibid., p. 24126 9 "What did the committee substitute and what changes were the bill accurately states: made? The report on 'Numerous changes were made in the introduced bill by the committee designed generally to better define the scope of the program and to clarify the intent so as to guarantee that the legislation will accomplish its purpose without interfering with the patterns or methods of dinancing of patient care or professional practice or with the administea- tion of hospitals.' "This statement alone indicates the magnitude of the changes and the fact that the legislation as introduced was miles off the mark. Here are the specific changes: "First: Regional medical complexes were mentioned earlier in * statement. No one could even now define what they are or how they would operate. The committee substituted 'regional medical program' the term . At the same time, all authority to use funds for new construction, including replacement of existing buildings was removed. These are referred to in the report as primarily semantic changes. Do not believe it. They move the specter of huge, new autonomous institutions which receive their funds directly from the Government and quickly dominate every phase of medical practice and hospital practice in the fields of heart, stroke and cancer. "Second: The original legislation allowed for expansion or other major diseases. The committee restricted the scope of this legislation to related diseases. That too is something more than a refinement. We have no idea that plans devised by the various States will be the ultimate answer in conquering the three diseases named. This is experimental. It cannot guarantee success in the war on heart disease, stroke and cancer. It will do well if, from the many medical programs devised, we discover one or two which have real promise. There is little reason to leave room for expansion i-nto other fields. "Third: The term 'cooperative' was substituted for coordinated' wherever the latter appeared. This helps to.remove the prospect of domination of the program by one large institution. A program can be beautifully coordinated if all the power is concentrated at the head. What we are striving for here can only work if all elements participate through cooperative arrangements. 10 "Fourth: Grants will be used for planning, conducting feasibility studies and operating pilot projects for the establishment of regional medical programs of research, training and demonstration activities. "Fifth: Diagnostic and treatment stations have been eliminated. The bill now speaks of hospitals which participate in the program. This also demonstrates the basic character of the changes made in committee. Now the bill refers to the local hospital participating-in a cooperative program. We can explain to anyone what a hospital'is by merely mentioning its name. Any citizen has a definite idea of how a hospital operates, what it looks like, what kind of people service it, and who runs it. The fight against heart, stroke, and cancer will come to the local patient through the people he knows and trusts and throu h an-institution with which he is thoroughly 9 familiar. "Sixth: This legislation provides for advisory councils at both the local and national level and in each case the council must recommend a program before it can be imple- mented or funded. Of these two, however, the Council at the local.level is by far the more important. First of all, its membership is important. It must include practicing physicians, medical center officials, hospital administrators, medical society representatives, voluntary health agency personnel, as well as people from other organizations concerned with the program. But even the best council will not guarantee a sound program if that program is set up before the council is organized and has a chance to act upon it: For this reason the bill provides that the advisory council must be organized and must pass upon the local programs before it may be considered by the Surgeon General. This should guarantee that the plan worked out in the State will not be lopsided, concentrating till too heavily upon one area of activity or placing too m@ch authority or responsibility with any one institution." Although.Rep. Nelson described the House bill as including revisions which "chanqe an amorphous mass of objectives into a recognize able program which deals with units and controls thoroUqhly understood by those who must work them," it may be asked whether the new emphasis on voluntary arrangements and local cooperation,did not make the bill even more flexible and subject to varying interpretations. In either case, the Senate finally accepted the House version of the bill , with passage as Public Law 89-239 in October of 1965. 51bid., pp. 24126-7 In 1968, renewal legislation was introduced in the Congress. The report of the Committee on Interstate and Foreign Commerce sub- House of Representatives on June 10, 1968 indicated that mitted to the the development of cooperative programs under the control of local advisory groups had resulted not only in specific accomplishments, but also in greater understanding of what was needed at the local level and how it could be achieved: "During the hearings and from materials supplied for the records, the committee received many descriptions of 1 Medical Programs specific developments under Regiona which serve to illustrate the nature of the emerging program, the diversity of approaches being developed, the of regional cooperative arrangements in effectiveness bringing together in common effort the health resources Of the region, and the accomplishments already underway in making possible the wider application of the advances of medical sciences in these fields. All of these descriptions represent progress in the initial stages of the development in operational regional medical programs.11 "In many respects, the legislation presents an answer to the problems expressed in the saying that 'what is every- body's business is nobody's business'. There are a sub- stantial number of individuals, groups and institutions throughout the United States who are concerned with the prevention of heart disease, cancer, stroke and related diseases, and the treatment of patients suffering from these diseases. Until the enactment of this legislation, there were few mechanisms through which coordinated programs could be established relating the activities of the various individuals, groups and institutions concerned with these diseases to activities of the others. This legislation provides a means whereby the individuals, groups, and institutions concerned with these diseases are encouraged to plan their activities jointly so as to coordinate them, ui as well as to plan projects that otherwise might never get off the ground for lack of sponsor. In this fashion, everybody's business, in the fields of heart disease, stroke, cancer and related diseases become the business of a specific group, which plans and operates programs having the overall objective of reducing morbidity and mortality from these diseases." 6Report of the Committee on Interstate and Foreign Commerce, House of Representatives to Accompany H.R. 15758, U.S. Government Printing Office, Washington, 1968, pp. 11-12 12 The Administration, however, did not appear to be as pleased with the achievements of Regional Medical Programs as was the Congress. This was reflected, in part, by the Administration recommendation for a two year rather than a five year extension. The renewal legislation was passed in 1968 with the 8igni- ficant changes confined to authorization for multi-program services, that is, project involving more than one local Regional Medical Program and the allocation of 1% of appropriation for evaluation. During the next two years, almost all of the local programs became operational , but the Administration continued to manifest dissatis- @faction with the overall progress of the program. In 1970 this took the form of budgetary restrictions and legislative proposals that would remove the categorical mandate and merge the RMP with Comprehensive Health Planning, the National Center for Health Services Research and Development, and the National Center for Health Statistics. Congress,, however, continued in its support of RMP and proposed in S. 3355, an extension which preserved the categorical emphasis and extended i t to include kidney disease. The t estimony7 of Senator Edward M. Kennedy of Massachusetts at the hearings in 1970 on the renewal legislation epitomized those supportive attitudes: "Today, thanks in large part to the success of RMP, we now have far more doctors and health organizations working together cooperatively than anyone might have expected a few years ago. I believe that we must continue in our efforts to expand the program and.increase its funding ... Because of its involvement with the medical schools and the universities, I believe that RMP holds great promise for influencing the health delivery system in a beneficial way. Further, because it has encouraged the programs to develop as conceived by the indigenous medical forces in a State, or a metropolitan area, it holds equally high .promise for successfully regionalizing the health care system after years of relatively unproductive efforts towards this purpose." The legislation reported to the Senate as S. 3355 and sub- sequently passed as PL 91-515, October 30, 1970 reflected an accomoda- tioh to the views of both the Administration and the Congress. This accommodation to both a categorical disease and medical care system approach is clearly evident in the revised statement of purpose: 7 Hearings before the Subcommittee on Health of the Committee of Labor and Public Welfare, U.S. Senate, Ninety-first Congress, Second Session on S. 3355, S. 3443 and related bills, Part I. U.S. GOP, Wash.' 1970, p. 126-7 p 13 and assis "(a) through grants and contracts, to encourage t in the establishment o regional cooperative arranoements among medical schools, research institutions, and hospitals or research and training (including continuing education), for m exchanqe and for (related) demonstrations of patient care ields of heart disease, cancer, stroke and kidney disease , and other related diseases; (b) to afford to the medical profession and the medical institutions of the Nation, through such cooperative arrange- ments, the opportunity of making available to their patients the latest advances in the (diagnosis and treatment of these diseases) preventio . diagnosis, and treatment and rehabilitation rsons these diseases; (c) to Promote and foster regional linkages amona. health hen a care to strengt nd impro ship between specialized n@d p (d)' by these means, to improve generally the enha ce the capacity of the health manpower an@facilities aval e Nation (,) and to -rnprove health services ':@'ted heal for Persons residing in areas with ' imi th services, and ith these ends without interfering with the pat e methods@f financing of patient care or professional practice, or with the administration of hospitals, and in cooperation with practicing physicians, medical center officials, hospital administrators, and 118 representatives- from appropriate voluntary health agencies. Additional' changes reflected an accomodation to the divergent points of view about increased community control and increase central direction. Briefly stated, these changes: 1. Broadened the specified membership and obligations of the local advisory groups " to insure adequate community orientation." 2. Provided for "consideration of the application -- for each operational grant -- by each public and nonprofit agency or organization which has developed a comprehensive regional, metro- politan area, or other local area plan referred to in Section 314 (b) covering any area @in which the regional medical program for which the application is made will be located." 3. Authorized the Secretary to facilitate interregional cooperation, and develop improved national capability for delivery of health services, to assist in meeting the costs of special projects for improving or developing new means for the delivery of health services concerned with the diseases with which this title is concerned, and to support research, studies, investigations, training, 8Public Law 91-515-, October 30, 1970, "Heart Disease, Cancer and Stroke Amendments of 1970 14 and demonstrations designed to maximize the utilization of manpower in the delivery of health services. The Regional Medical Program legislation of 1970, in many respects, marked the end of a five year legislative history. At its inception, Regional Medical Programs were identified as a cooperative effort in regionalization of research, training and demonstration activities designed to improve the care of patients suffering from heart disease, cancer and stroke. After five years of operational experience and additional consideration by the Congress in 1968 and 1970, the legislation began to specify balanced working relationships between categorical disease care and the rest of the medical care system and between national and local organizations concerned with improving the health manpower and facilities available to the Nation. The successful negotiations of the divergent attitudes and approaches, as apparen t in the renewal legislation of 1970, was a mixed blessing. The changes provided the opportunity for the development of a more balanced approach to meeting -Che medical care needs of the Nation and of communities through regional activities, but obfuscated exactly what achievements were to be expected and who was in control of the program. In the absence of specific objectives or guidelines for Regional Medical Programs from HEW or HSMHA, RMPS staff, as they began the development of the Review Criteria in the summer of 1970, were forced to use a set of imprecise, overlapping and, often, confusing set of directions abstracted from sident's He the 1970 legislation and the Pre alth Message of 1971. History of the Review Process The legislation for Regional Medical Program specified that the activities would be carried out "through grants" to "public or nonprofit private agencies and institutions" with certain specified characteristics. Two types of grants were specified: (1) Grants for olanning and (2) Grants for Establishment and Operation. Grants could be funded only if they had the approval of the local advisory group and of the National Advisory Council. The com osition of both p groups was specified in the legislation. At the national level, only those grant proposals which had received approval from the local-now referred to as "regional" advisory group were accepted for possible funding. These grant proposals were entered into a review process patterned after that of the National Institute of Health. This included: 1. RMPS staff review for completeness and preparations of special analyses and reports called "staff briefing documents" for all grant proposals. 15 2. Site visits of experts, peers and public at the request of RMPS staff. 3. Panel reviews by experts in selected fields, e.g. heart disease, at the request of RMPS staff. 4. Review Committee consideration and recommendations based on technical considerations for all grant proposals. 5. National Advisory Council consideration and recommendations based on policy and program cdn- sideration for all grant proposals. The Director of RMPS could fund only those grant proposals approved by the National Advisory Council. Although the National Advisory Council usually followed the recommendations of the Review Committee, they were not bound by the Review Committee recommendations. The National Advisory Council had the final and only decision as to which grants could be funded. The Director of RMPS was obligated to fund grants approved by the National Advisory Council, but he could fund at a level less than that approved if there were limited funds or an apparent need to reallocate funds between grants or grantees. From 1967, the,first year that grants for establishment and operation of regional medical programs were received, until 1970, each component project of a grant application was considered as a separate grant. Operational grant applications consisting of one or more separate projects were submitted by some or all of the fifty-plus local regional medical programs at deadlines for review cycles scheduled three,or four times a year. RMPS staff, in 1969, and the Federal Assistance Streamlining Task Force (FAST), in 1970, identified the fact that the project- oriented review resulted in the excessive work load for the review process, significant variations in total funding levels among'the regions, and an inability to develop cohesive and logical programs at the local level. This led to recommendations for significant changes in the review process. The review process was changed for applications received on or after August 1, 1970. The new review process restricted local programs to a single operational grant application each year on a specified date and made provision for a triennial award which allowed for much greater'local program autonomy within an approved program plan. Previously a local regional medical program could support and carry out only those specific activities which had been approved by the National Advisory Council. Under the new procedure, a local regional medical program could carry out any activity which could be funded within the approved funding level. A triennial award included a developmental component Li which was money specifically set aside for the development of short term activities within the three year plan. In addition, a program receiving a triennial award was allowed to reallocate funds to any short term or long term activities judged by the local advisory group to 16 meet the programs objectives as specified in the approved three year plan. Reviewing a three year program plan whose approval reduces central contr nded activities is a significantly different ol over fu task from reviewing discrete projects. Not only would the various review groups have to switch from project-oriented to program- oriented review, but they would have to become concerned with the composition and decision-making ability of the local advisory groups so as to "insure adequate community orientation" as specified in the new legislation. Helping the Review Committee the National Advisory Council and the site visit@, terms to make i;he transition was an important issue for RMPS staff in the fall of 1970. Conclusion The RMP Review Criteria have been an integral part of the Regional Medical Program review process at the national level since 1971. They were developed and introduced during the latter part of 1970 and the early part of 1971, a period of transition in the purpose and methods of operation as indicated in the renewal legislation of 1970 and a period of transition in the review process. The request of RMPS for an eval uati on 1 of the Rev'ew Criteria called for an assessment of the adequacy and effectiveness of the Review Criteria within the review process and for recommendations that might increase their usefulness in the future operation of Regional Medical Programs. 17 CHAPTER II APPROACH TO THE EV ALUATION AND.REPORTS OF COMPONENT ACTIVITIES 19 A contract such as this one calls for close co laboration between the Project Officer and the Project Director in order to assure that the needs of the agency will be mt. Negotiations between the Project Officer and the Project Director as to the selection of specific work part of the activities and the level of qualitv expected is an important process. According to the Scope of Work for the contract, the first phase called for a statistical analysis to determine the internal validity of the Review Criteria as measures: the second phase called for an opinion survey of those using the criteria to determine their adequacy and effectiveness in assuming the comparative quality of local RMPS; the third phase called for the Dreparation of a comprehensive report with findings, recommendations and suggestions. The agreed-upon specific activities were: 1) statistical studies related to "internal validity" 2) interviews with ten selected users 3) conferences with selected RMPS staff The results of these activities are reported in the following sub- sections of this chapter. A. Statistical Studies of RMP Review Criteria Scores B. Reported Usefulness of the RMP Review Criteria in the Review Psocess from Interviews with Ten Users C. Values Expressed by Ten Users in Descriptions of Local RMPs D. Summary of Conferences with RMPS Staff on RMP Review Criteria 'The general theme to be found in the reports is that, in spite of many technical and methodologic weaknesses, the RMP Review Criteria were found to be very helpful to the operation of RMPS at a time of.great need and for guiding local programs for a new framework for the review process ry -changing intent of the RMP legisla- in the confusing task of car ing out the tion. The information contained in these reoorts plus the results of analysis of various document, informal interviews with RMPS staff and formal and informal consultations with.various experts from the substance of the evaluation and provide the rationale for the recommendations reported in@ Chapter IV. The studies reported in this chapter are merely a prelude to the formal Pvaluation and recommendations to follow. 21 STATISTICAL STUDIES OF RMP REVIEW CRITERIA SCORES 23 Members of the Review Committee and the Staff Anniversary Review Panel routinely score all programs that they review against the RMP Review Criteria. Each member submits a set of scores for each program that he reviews. All the scores for an individual program are averaged and reported as official scores to the National Advisory Council and the Director of RMPS. The computer data file with the scores of all individuals for all programs reviewed in the review cycles of November 1971, February 1972, and June 1972 was subjected to a series of statistical studies designed to answer the questions posed in the Scope of Work - See Appendix B. Of the 590 individual scores contained in the file, only 277 were selected for analysis. They were selected on the basis of completeness, that is, d for all scores recorde twenty categories, and on the basis that these five or more complete scores for an individual program. For many analyses, the regions were categorized into "A", "B" and "C" regions on the basis of information provided by RMPS. A complete report can be found in the Second Quarterly Report from the contract. Findings: 1. The averag-e raw scores of "A", "B" and "C" regions are significantly different from each other. TEST: Analysis of variance (F 25.67, p. 0.001) 2. The average weighted scores of "A'13 "B" and "C" regions are significantly different from each other. TEST: Analysis of variance (F 696.6, p. 0.001) 3. There is a consensus among reviewers regarding "A". 'B and "C" regions. TEST: Principal Component Factor Analyses of Regions Centered Scores - Only one factor identified. EXPLANATION: When the scores of each reviewer a.re expressed as deviations from the mean score for each region and subjected to principal components factor analysis, only one factor was identified. This means that all reviewers followed a single pattern in scoring regions, i.e. there was consensus. 4-. The scores of no individual reviewer reflects an unusual range. TEST A: Bartlett's Chi-Square Test on Means Scores and Standard Deviation of.Individual Review (X2 4.6116, p = 70.5) 25 TEST B: Principal Components Factor Analysis of Review Scores - Only one factor identified. EXPLANATION: When the scores of each reviewer are expressed as deviations from his wean score for all regions and subjected to principal components factor analysis, only one factor was identified. This means that no reviewer or group of reviewers had a range or patterns of variation that differed from the other reviewers. 5. No primary reviewer has a greater tendency than other primary reviewers to pull the scores of other primary reviewers to a modal val ue. Primary reviewers scores in general, are slightly higher than the man s scores. TEST: Bartlett's Chi-Square Test on Mean Scores and the Standard Deviatiop of Regions Grouped by Primary Reviewer. ( X4 - 1 3. 21 . , p. 0. 21 2) 6. The same statements recorded in 1-5 above can be made for subset scores related to the criteria on performance, process and program proposal. 7. The correlation coefficients between overall weighted scores and the weighted scores for performance, process, and program proposal are 0.945, 0.948, and 0.938 respectively. 8. There are strong positive associations between overall weighted scores and the scores for all individual criteria except minority interests. TEST: Principal Components Factors Analysis. EXPLANATION: A quotation from the report of the statistician describes the strength of the relationships and the significance of the finding. "Our first hint that the data is simplistic comes when we examine the full correlation matrix on the twenty items. The entries in this matrix are astoundingly hi h. we exclude item four from the analysis (a decisio analytically corroborated later), every correlation but (8.3) is larger than .400. For some. variables, such as #18, every corre- lation is larger than .500. This consistency strongly suggests that (except for item four) differences in the face significance of items are not being reflected in the scoring. 26 1-component analysis of this matrix, In the raw principa column one of the printoutis a column of weights by which we multiply the separate items (after they are standardized to the same variaRce) before summing to form the best scale. This data analyst has never before seen a set of factor loadings so large and homogeneous. With the exception of #4, all are larger than .700. Here is a scale which does not correlate less than .7, and occasionally as much as .85, with each of nineteen items. The "latent r,,-Ot" at the bottom of the column is the total squared correlation of this optimal scale with the separate items. Its maximum is 20, achievable only if our items are all identical. We are nearly two-thirds of the way tb.that ultimately redundant situation." 9. The relationship between site visit findings as reflected in site visit reports and the scores for selected criteria was not examined because the site visit reports were not provided by RMPS. Conclusions: Although some of the findings were predictable, e.g. the signi- ficant differences in scores of "A", "B" and "C" regions. most of the findings were surprising. Ordinarily studies of peer judgement procedures show very poor correlation between scores of different judges, but that was not the case in this study. The correlations between scores of different judges was extraordinarily high. Similarly there is usually considerable variation in the correlation of individual item.scores and total scores, but that, again, was not the case There were extraordinarily high corelations of individual criteria scores with each other end with total scores. These unexpected findings, although appearing to show consistent judgement or extreme internal reliability to the uncritical reade'r,".'actually indicated two very serious flaws in the Review Criteria as measures. First, the extraordinarily high correlations between reviewers suggests that there was agreement between the reviewers as to what the scores should be. Each reviewer did indeed, score the programs independently and there was no formal discussion as to what the scores should be, but the scoring was done after a full presentation and discussion had been completed and reco ndations were made. The h,igh correlations between the scores of the various reviewers reflected the fact of group consensus as to the quality of the program, and indicates that group judgement rather than individual judgement determined the scores. 27 'l criteria Second, the high,correlations between individua scores and total scores lea to the identification of only one factor subjected to factor analysis. The separation of when the scores were the individual criteria into three major categories, if it was a valid separation, should have led to the identification of three factors. Ordinarily one might have expected that more than three factors would have been identified. The fact that only one factor was identified revealed that the Review Criteria function as a unidimensional measure, that is, a good program is good on everything and a poor bad on everything. This failure to identify more than one program is factor suggests that the stores on individual criteria were probably based on an " a priori" and unconsciously adjusted so that the to all judgement. The only exception was lated to minority interests. Significant changes in the scoring procedure would be needed to introduce, in an operational sense, independent peer judgement and independent judgement oh each criteria. Only when independent judgement occur,ds will statistical studies indicate the degree of rel i abi 1 i ty, validity, precision, objectivity and constancy that is Dresent. A better solution, when orf-considers all the problems reported in the literature on the evaluation of independent judgemental procedures, would be to transform the process from subjective to objective measures related to the criteria. A second thoice solution would be to provide r -i precise definitions for each of the scale points used in the scoring. 28 B REPORTED USEFULNESS OF THE RMP REVIEW CRITERIA IN THE REVIEW PROCESS FROM INTERVIEWS WITH TEN USERS 29 Determining "the perceptions of those utilizing the (RMP Review) criteria in the national review process as to their effectiveness" is one of the expressed purposes of the contract to evaluate the RMP review Criteria. A related question from the Scope of Work for the contract is, "Have the review criteria and their utilization resulted in a significant improvement, either substantially or in terms of credibility, in the decision-making process?ll This is the report on one of two activities designed to obtain information sufficient to answer that question. The first activity was a conference held in two sessions with RMPS staff who are involved in various uses of the Review Criteria. The results of that conference are reported in Section D - "Summary of Conferences with RMPS staff on RMP Review Criteria." The second activity consisted of a series of interviews with ten users of the Review Criteria whose opinions of and insights into their uses was considered important by the Project Officer and his advisors. These users included the Director of RMPS, three members of the Staff Anniversary Review Panel, three members of the Review Committee and three members of the National Advisory Council as follows: Director RMPS Dr. Harold Margulies Staff Anniversary Review Panel Mr. Richard Russell Acting Chief Western Operations Branch Division of Operations and Development Dr. Edward Hinman Director Division of Professional and Technical Development Mr. Cleveland Chambliss Director Division ofIOperations and Development Review Committee Dr. J.E. Kralewski Assistant Professor and Director Division of Health Administration University of Colorado Medical Center Dr. William G. Thurman Professor and Chairman Department of@Pediatrics Universi ty of Virginia 31 Dr. Alexander Schmidt Dean Abraham Lincoln School of Medicine University of Illinoi.s National Advisory Council Dr. Clark Millikan Consultant in Neurology Mayo Clinic Dr. Anthony L. Kareroff Beth Israel Hospital Boston, Massachusetts Mr. C. Robert Ogden President and General Counsel North Coast Life Insurance Company The interviews were basically open-ended, guided interviews with a section of structured, closed-Onded questions related to the relative usefulness of the RMP Review Criteria Scores in decision-making in comparison with the following items. 1 . Grant Appl i cation 2. Program File 3. Staff Briefing Document 4. Site Visit Report 5. Review Committee Report 6. Primary Reviewer s Presentation 7. Secondary Reviewer's Presentation 8. Discussion Each respondent was asked to rate each of the sources of information, including the Review Criteria Scores as Very Important, Important, Uncertain,. Not Important or No Value to decision-making in the.review process. This is a report on that rating exercise, all@ discussion and comment retorddd during the rating exercise, and any general comments made during the interview related to the usefulness of the Review Criteria.. It has been suggested that the Review Criteria scores should not be treated as sources of information. Only the National Advisory Council 32 iva any Review and the Director of RMPs rece Criteria scores and that is only in the form of total scores. The results of the rating exercise clearly indicate that all respondents were aware of the Review Criteria and the scoring procedure and that they were not constrained in their evaluation-by any reference to the Review Criteria scores as a "source of information". Table of Definitions 1. Grant Application The one year and/or three year plan of operation su mitted by the local RMP. 2. Proqram File - Previous grant applications, site visit reports, advice ther written information which serves as a background to the grant applications under consideration. 3. Staff Briefinq Document - A summary and analysis of the important points out of the tion and of current developments with the RMP prepared by the Operations Officer from Division of Operations and Development assigned to the local RMP. 4, Site Visit Report - The report prepared by the chairman of the:,site Vis-it team at the conclusion of the site visit. Such teams consist of members of the National Advisory Council, RMPS staff, Consultants and a chairman from the Review Committee. 5. 'Review Committee Report - The analysis of the local RMP's program -a-nd grant application and recommendations on program plan and funding, level for the next one year and/or three year period. 6. Review CommittteeIScores - The average of the scores prepared by each member of the Feview Committee on each@of the 20 criteria in the Review Criteria. This includes both average raw scores and average weighted scores. 7. Primary Reviewer's Presentation - The oral presentation of the primary reviewer, usually the chairman of the site visit'team as to his analysis of the local RMP's progress and grant application, the fundings of the site visit and the recommendations of the site visit team on program plan and funding level. 8. Secondary Reviewer's Presentation - The oral presentations of the second- ary reviewer based on analysis of grant application, staff briefing document and site visit report usually without benefit of partici' pation in the site visit. 9. Discussion - Any discussion in committee over and beyond the oral presentati- s of the primary and secondary reviewers. on 33 TABLE 2B-1 Availability of Various Information Sources to flembers of SARP, Review Committee, National Advisory Council and Director RMPS. X Supplied specifically SA R P REVIEW CO@ItlITTEE COU@ICIL beforehand V) V) VI Available but ui ix = Ln C) not specifically CD V) LU ci ui VI supplied Ln w w V) (X C, ui w CD ui ui ui ui ui ce uj:> t- I- ui NA= flot Available as ui t- < cn w CD ui C) yet (or applicable) CD U-i Ca cn 1. Grant Application x x x x x x 2; Program File 0 0 0 0 0 0 0 a 0 3. Staff Briefing, Document x x x x x x x x x 4. Site Visit Report NA NA x x x x x x x 5. Primary Reviewer's Presentation NA NA NA NA x x NA NA NA x 6. Secondary Reviewer's Presentation NA NA x x NA x NA NA NA 7. Discussion x x x x x x x x x 8. Review Committee Report NA NA NA NA NA NA x x x 9. Review, Committee Scores x x x NA NA NA NA NA NA 34 The Table of Definitions describes all of the sources of information referred to in the interviews. Table 2B-1 indicates, in part, e each of th the availability of the various sources of information to four classes of respondents. The following brief descriptions indicate how the various groups carry out their review function. Staff Anniversa Review Panel: This is composed entirely of senior RMP staff. O( r review is presented by the Operations Officer assigned to the program. The members of SARP base their discussions and recommendations on this presentation, the grant application, the staff briefing document and, if necessary, the program file. Review Committee: Thi-s is composed of health professionals selected on the basis of :Eheir technical competence in various areas related to RMP operations. The RMP under review is presented by the primary reviewer who, not uncomimnly, was chairman of the site visit team. Corroborations or additional findings are furnished by a secondary reviewer who was not on the site visit team. The discussion and recommendations are based on the primary and secondary reviewers presentation, the grant application, the staff briefing document, site visit report and, if necessary, program file - usually presented by the Operations Officer assigned to the program. National Advisory Council: This council is,composed of .those categories of people designated in the legislation plus other t of Congress experts and laymen who are chosen to preserve the inten as specified in the legislation. A local RMP under review is presented by a primary reviewer who, in most cases, was a member of the site visit team. Corroboration and additional findings are presented by a secondary reviewer. The discussion and recomniendations are based on the grant application, staff briefing document,, site visit report, Review Committee report and recommendations and the Review Criteria score - a total score only - submitted by the Review Committee. Director of Reqional Medical Proqram Service: The Director is constrain6d-in his discussioni-by the recommendations of the National Advisory Council, but, within those constraints,@he has the opportunity to make selection funding decisions, i.e. to fund at less than the National Advisory Council approved funding level and to provide specific direction for future program development. He usually is present at 'l and has all sources the deliberations of the National Advisory Counci of information plus the counsel of R MPS staff to guide him in his actions. 35 TABLE 2 B 2 Importance of Various Information Sources to All Respondents Decision-Making in RIIPS Review Process Percent of Res@Dnses 4J Information Sources Number of -P 4-) -P.-P w to @ 5- $- .I- 0 S- 0 4J Responses (D 0 o u = 0 ;= i0 1 >. r.L (U E E Grant Application 75 12 12 99 8 Program File 38 12 12 25 12 99 8. Staff Briefing Documents 62 12 25 99 8 Site Visit 57 43 100 7 Report Review Committee 40 60 100 5 Report Review Committee Scores 20 40 30 10 100 10 Primary Reviewer's Presentation 75 25 100 8 Secondary Reviewer's 12 12 25 38 12 99 8 Presentation I I r-@ Discussion 25 50- 12 12 99 8 TOTAL NUMBER OF RESPONDENTS ..................................... 10 36 FINDINGS AND DISCUSSION. General Opinions on the Review Criteria and Process The general opinions of the respondents on the review criteria tion of th and the review process provided a background for interpreta eir attitudes toward the various information sources. Positive statements focused on the idea that the criteria provided a consistent framework through which a wide range of components measuring the strengths 'and weaknesses of an RMP can be considered. They felt that such a uniform approach was needed to create a basis for comparison between RMPS. In addition, they felt the existence of the criteria in written form gave the local @lPs a better idea of how they would be reviewed and a better lannin nde framework to use in their own p q and evaluation. Those respb nts who felt positively about the criteria generally agreed there were so'me areas of weakness as they stand now, but that they do provide a good overall approach to evaluation. Reservations oh the r'eview criteria centered in two areas. First, respondents felt the weakness of the present definition of what an RMP should do made it generally difficult to evaluate the organization. They stated that no specific goals had been laid out for RMPs so that detailed outcome measures could not be established. They felt that the existence of the review criteria had not solved this problem, but had merely exemplified that it did exist and that until a new definition could be worked out, effective review was impossible. Second, some felt that the numeric analysis, particularly the weighting system, applied to the criteria was artificial or at least needed reexamination. Others disputed the idea that an RMP should be measured numerically at all. They questioned whether such a distillation of details really reflects the significant features of an @IP. General Opinions on the Importance of the Various Information Sources to Decision-MYki Process On the basis of the analysis of all responses, the following observations on the relative importance of various information sources can be made. The respondents thought the primary reviewer's presenta- tion was the most important source. Between these two extremes, the information sources fell into two groups in terms of perceived importance: the "more significant" groups and the "less significant" group. In the limore significant" group were the grant application, the site visit report, the staff briefing documents, and the review committee report. Respondents placed review committee scores, along with discussion and the program file, into the "less significant" group of sources. These observations were discussed further below. Table 2B-1 presents the interview results upon which these observations were made. 37 tion was generally viewed as The primary'reviewer's presenta the most important source of information in evaluating an RMP. Of the eight individuals who discussed the primary reviewer s presentation, six said it was very important, and two said it was important. However, while saying this, several respondents insisted on discussing inherent weaknesses in this source of information. These stemmed chiefly from the feeling that, through a primary reviewer technique, a great deal of significance became attached to the opinions and presentation of a single person. Two respondents thought that, if the reviewer was particularly impressed by the RMP and made a very positive presentation, the importance of the rest of the committee procedure is lessened (Hinman, Russell). The fact that some reviewers made more effective presentations than others with more influence on the group was also pointed out (Russell). Another respondent felt that, because the presentation was oral rather than written, not all the collected information was communicated in some instances (Therman). In general,. respondents thought that the quality of this source of information was dependent on the amount of effort the reviewer had put into the project and his personal attitude toward the region. Despite these weaknesses, committee and council members apparently relied heavily on the primary reviewer's presentation for an analysis of the region that they do not have time to do themselves. Several res- pondentt discussed features of committee and council procedures which helped explain the situation. In most cases, the primary reviewer is the only individual who was well informed on the RMP under reviewi "I think the weakness of the Review Committee or (Advisory) Council is that most of the people don't do their homework. It (written material) hasn't been read by the time we've arrived (Thurman)". Primary reviewers seemed to have more extensive and detailed information than other committee and council members. "If you're not the primary reviewer you get briefer documents... Review Committee recommendations is what it amounts to, andthose are accepted somewhat non-critically b'y members of the NAC who say 'well, we're relying on the entire process'(Ogden)'. The fact that the primary reviewer's presentation was thought to be most important should also be examined in relation to the fact that the secondary reviewers presentation was viewed as the least important source of information. There was a range of opinion oh this item, but, of the eight who addressed the subject, only one found it very important (Chambliss) and one found it important. Six people chose the lower three ranks to labe1i'the secondary reviewer's presenta- tion. Some of the respondents expressed the feeling that these pretenta- erms of being a double check on the prima tions were useful only in t ry reviewer's presentation'(Russell) and gained importance only where there was an obvious difference of opinion (Millikan). The remaining documents fell less easily into a. clear position within a rank ordering according to importance, but tended to cluster into groups of "tdore significant" and "less significant" sources. 38 Included in the "more significant" group were: the site visit report, review committee report, grant application and staff briefing documents. All of these documents had a higher percentage of responses in the very important category and higher cumulative percentages in the very important and important categories. A higher degree of consensus on importance was also expressed concern- ing documents in the "more significant" group than those in the "less significant" group. The Orant application was regarded as a source which can vary greatly in terms of usefulness, depending on the quality of the document. "In certain instances where things are beautifully put together and told (in terms of) exactly how they are going or how they have gone off, the grant application can be very important There are other instances, that I was personally involved in, where I see the grant application was of very little value other than as a kind of reference document from which one could begin to ask some questions (Millikan)." "It's not because it's necessarily factual$ (everybody makes a grant look,better than it is), but it gives-you a feel for the entire program and unknowingly it points out a great many of the weaknesses of the program as well as the strengths of the program (Thurman)". Use of the grant application also depends on the individual's role in the review process. For primary reviewers or members of the site visit teams, the document became more important (Hinman). Another respondent indicated, "As a member of the NAC, I never see a grant application (Kamaroff) A great deal of time was required to study a'grant application thoroughly, (Russell) and this tends to limit the number of reviewers who used it effectively. When not studying the - particular RMP in detail, the grant application could become mi.xed with the staff briefing documents (Margulies). One respondent particu- larly noted the value of the RAG report within the application: "We look in the RAG report to see how they have addressed the issues that we raised in the previous review or whether they have really addressed the, or whether we are getting a snow job (Russell)." Staff,briefing documents were seen as a source for developing Ilikan, Thurman, a historical perspective on RMPS (Mi Rustell).' Eight res ondents discussed the documents which were prepared by individuals p who were "unifomly involved almost all the time" with the RMP and have collected a large volume of information. However,one respondent added that information can be presented in a biased manner (Thurman). Documents do Vary considerably, the best being those which related what the RMP has done without trying to convey any opinion as to whether the RMP was good or bad. Site visit reports were labeled very important by four people and important by three people, but few respondents gave explanations for 39 (Kamaroff) added the fact that site these opinions. One respondent visit reports became particularly important to committee or council review when a member of the committee or council was on the site visit team. Respondents also commented that the national review criteria served to encourage greater consistency and clarity of format in the site visit reports. The site visit itself was obviously one of the key steps in the review process; however, one respondent (Ogden) expressed the concern that more preparation by team members for the site visit is needed. He felt that site visits could become highly controlled by the RMP staff, not allowing the team to explore weaknesses or other s ecific areas which must be covered under the guidelines. The p respondent thought that the team should meet beforehand, discuss the program,, and divide up different areas of investigation and reporting to insure that the team examined all necessary aspects 'of the RMP. Another respondent added that small informal group meetings between RMPS staff and the local RMP could often be relied on to be more informative than the site visit because of these weaknesses in the site visit procedure. Review Committee reports were cited as either important or very important by the fi e respondents who discussed the source. . One respondent (Millikan) particularly noted that the report was nearly always automatically accepted as valid by the NAC. He felt that the council assumed that the Review Committee had done a careful job and seldom reversed its recommendations. Only when new information was introduced after review, when information from the site visit had not been weighed by the committee, or where a matter of policy or philosophy was involved would the council consider changing the recommendation. The "less significant" group of sources are: the discussion, the review criteria scores and program files. Respondents felt discussion becomes important only when various other sources of information were in disagreement. At SARP, NRC and NAC meetings, when the information sources presented do not make similar recommendations for the RMP under review, discussion on differing opinions became significant. Two respondents cited discussion as livery important" and four said it was "important". There was a variety of opinion on the importance of the proqram file. Again the program file, like the grant application was used more .Ey some individuals more than others 'in the review cycle because of the study time involved. Its analysis was most often undertaken by .staff members. Two individuals labeled review committee scores livery important" and r IMP owever, four seemed to think they we in discussion a number of reservations were voiced on the scoring design and procedure. 40 One respondent voiced strong opinions that RMP evaluation could not midt). Another said there were.failures "be reduced to numbers" (Sch reexamination (Ogden). in the weighting scheme which called for Another respondent said committee members are reluctant to score programs when they don't have strong comparative information or site visit experience (Kralewski). Other sources of information, which were not directly probed by the interviewer, were described by the respondents as being important. Several said they used the Management Reporting and Evaluation System (MRES) to some extent. One added that because the summaries were in the form of computer p@intouts they were difficult to handle (Russell). A variety of less formal sources of information were thought to be important to different degrees depending on circumstances. Accumulated personal knowledge and peripheral information picked up while working on various tasks, often unrelated to RMP, were picked out as being influential factors (Thurman; Russell). Respondents also noted small meetings and seminars as being means by which information was directly obtained from an RMP (Margulies, Hinman). CONCLUSIONS On the basis of the analysis of the interviews, certain common attitudes toward different information sources became clear. Review Committee Scores were not generally thought to be as important as several other sources of information; they fell into a group of "less significant" information sources. Other sources i n this group ar e discussion and program files. The primary reviewer's presentation seemed to be regarded as the most important. The grant application, the site visit report, the staff briefing documents, and the review committee-report also were thought to be more important than scoring. The respondents view the secondary reviewer's presentation as the least important source. However, there was also a@strong feeling that the significance of an information source dependent on its quality is highly variable. The grant application and staff briefing documents were particularly singled out to be sources where this is'true. Other variables were mentioned which make the use of documents dependent on specific circumstances of the review cycle. For example, if there were discrepancies between the recommendations suggested in what were generally the "most significant" sources of information, the "less significant" sources become much more valuable. Conversely, if there was strong agreement on recommendations for the RMP to begin with,. the "less significant" sources dropped from concern. Clearly@there were variables, particularly the quality of the information sources@@@@pr , @ details of the specific situation, which prevent rigid rules on th6@degree@ of usefulness-from being applicable. c IN DESCRIPTIONS OF LOCAL RMPS VALUES EXPRESSED BY TEN USERS 43 'ia were designed to assess the The RMP Review Criter n the contract comparative quality of Regional Medical Programs. I for the evaluation of the RMP Review Criteria, the contractor was asked to "determine the adequacy and effectiveness of the Review Criteria in assessing the comparative quality of Regional Medical Programs as perceived by those utilizing them. In this process, he was asked to address his efforts towards answering a series of questions,-.-but he was not limited to those questions. In the approach to this assignment, the Review Criteria were assumed to represent a series of values which could be used to describe the quality of a local RMP. It was further assumed that the Review Criteria would be adequate and effective for that purpose only if they coincided with the values that those involved in,the review process-used in making judgements during t,.. review process. In the ideal situation, all users would have iden- tical values and that set of values would correspond to those of the Review Criteria. Ordinarily one would expect differences among the users and between the users and the Review Criteria. The degree of correspondence or lack of correspondence between the value sets of the users and that of the Review Criteria is an index of the ade- quacy and effectiveness of the Review Criteria in assessing the comparative quality of Regiona 1 Medical Programs as specified above. Three sources of information were developed in an attempt to respond to this item in the contract: 1. Analysis of the Review Criteria in a worth assess- ment model; 2. Conferences with RMPS staff - see Section D; and 3. Interviews with ten selected people participating in the review process at different levels. This is the report on the interviews with ten selected respondents. The general design of the interviews and a description of the respondents has been presented in Section B. The data from the interviews for this analysis has been drawn from the answers to the following open-ended lead questions: Describe, in terms of accomplishments, a specific RMP that has done well. 2. Describe, in terms of accomplishments, a specific RMP that has not done well-. 3 . What kinds of things can a good RMP achieve that a poor one cannot? 45 TABLE 2C Number and Percent of Respondents Making Statements in Twenty Categories of Concern Percent Number 1. An RMP should be involved and work closely 80 with the communities in the region (Ex through subregionalization, out@each, etc.) 2. An RMP should be involved in activities T-7 7 70 dealing with categorical diseases. 3. An RmP should be involved in activities 7 70 dealing with training and education of all types of health manpower. 4. The key health interests should be 5 50 accurately represented on the RAG. 5. An RMP should serve as a forum to brina to- 50 gether all the key people in the 5 health care system.. 6. An RMP should serve as a helper and 5 .50 advisor to other health planning agencies, especially CHP. 7. The RAG should actively establish the program's goals and 5 50 priorities. V@' 8. An RMP should be involved in pro- 4 40 ducing meaningful relationships between providers and consumers. 9. An RMP should be a flexible organi- 4 40 zation which tan serve as a catalyst for action in a variety of settings. io. An RMP should have an effective 4 40 evaluation mechanism. 11. An RMP should have a strong 4 40 management team. 12. An RmP should concern itself with deter- 3 30 mining the needs of the region'and IT'7@ developing goals and objectives to meet these needs. 13. An RMP should sponsor activities which will 3 30 be able to support themselves after the withdrawal of RMP funds. 14. An RMP should be involved with activities 3 30 dealing with patient care appraisal. 15. The-coordinator of an RMP should direct the program towards achieving the long- 3 30 range goals and major objectives that the program has established. 16. An RmP should have a capable and competent .3 30 staff. develop close relationships 17. An RMP should 2 20 and open communication with RMPS. 18. An RMP should be actively involved in 2 20 addressing the interests of minority groups. 19.@ An RMP should be involved in 2 20 data collection. 20. The RAG should determine which specific I 10 activities should be initiated in the region. 46. The verbatim answers to these questions were transformed into a series of statements. The statements from the answers to all the questions from all respondents were then categorized according to the concerns that were being expressed. As a verification procedure, a list of the categories of concern was sent to each respondent. Each was. asked to sort the categories into quartiles by order of relative importance to local RMPS. All data was subjected to qualitative and non-parametric statistical analysis. FINDINGS Twenty categories of concern were found to encompass all the statements made in response to all three questions - see Table 2C-1. Seven categories included statements by 50% or more of the respondents",, sixteen ca togories by 30% or more. The total of 188 statements reflected an average of 2.35 statements per respondent per category. Because a-chi-square analysis showed no relationship between the average number of statements per respondent per category and the number of respondents citing each category, a decision was made to base all subsequent analyses only on the number of respondents citing each category. Ten respondents participated in the interviews; only seven participated in the verification procedures. Of the seven categories including statements by 50% or more of the ten respondents, five exceed this 50% level for, the seven respondents. Table 2C-2. shows that the proportion of respondents in the verification procedures .Citing each category concerned was similar to that of all ten respondents. Table 2C-2 indicates that there was no strong relationship between the number of respondents citing a category of concern in the original interviews and the number who specified that the category belonged in the first or first and second quartile of importance. Although not displayed, the data shows a similar lack.of relationship between the data on importance and.the number of statements in each category. Discussion The discrepancies between the results of the original interviews and the verification procedures were unexpected. Careful re-reading of the original i nterview suggested that the context in which the respondents described the qualities of good, poor and ideal RMP's might help to explain those discrepancies. This rereading 47 TABLE 2 C - 2 Number of Respondents Making Statem6nts in Each Category of Concern and Ranking Categories in Quartiles of Importance NO. OF RESPONDENTS Categories ortance importance of Concern artile Quartile I I & II 8 5 4 7 2' 7 5 0 0 3 7 5 0 2 4 5 4 0 2 5 5 3 2 4 6 5 4 3 7 5 2 4 6 1 8 4 3 3 9 4 2 4 7 10 4 3 4 5 11 4 2 1 3 12 .3 3 4 6 13 3 2 0 4 14 3 2 0 15 3 2 5 6 4 16 3 1 3 17 2 0 0 1 0 2 18 2 2 2 19 2 0 1 20 2 3 (10) (7) (n) (7) (7) *Qnly seven respondents returned the verification questionnaire. 48 also re-emphasized the fact that the ten people selected because of their participation in the RMP review process interpreted the questions about "accomplishments" in a manner quite different from that intended by the interviewer. The interviewer asked the respondents to think of RMPs that had done well, that had not done well and that were ideal and to describe them in terms of "accomplishments". The intent was to obtain descriptions of the impact that the three types of RMPs had had on the medical care system of the regions that they served. Analysis of the complete text of all interviews revealed,that the respondents chose to describe the RMPS in terms of: 1. CaDability - the organizational structure of the RMP a6-d the perceived competence of its staff and onal advisory group regi 2. Direction - the,@type of activities undertaken-by the program, ana 3. Charisma - the characteristics of the leadership videnced by the program and its activities. A Regional Medical Program, according to the respondents, may be said to have @ability if it has: 1. A Regional Advisory Group which is broadly representa- tive of key health interests in the regions. which actively establishes the prdgram's goals and objectives, and which determines which ,specific activities should be initiated 'd co petent 2. A coordinator who has assembled a capable an m staff who directs the program towards achieving the long-range goals and m;jor objectives'that the program has established, and who has an effective management and evaluation mechanism -- 3' A program which serves as a forum to bring together all the key people in the health care system and at a helper and advisor to other health planning agencies. The direction of a Regional Medical Program can be considered satisfactory, accor ing to the respondents, if its activities are: 1. Concerned with meeting the needs of the region 2. Concerned with long term funding of activities 3. Actively addressing the interests of minority groups 4. Involved with.activities dealing with patient care, especially those that will improve the health status iof the population in regards to the categorical diseases 49 5. Involved in activities dealing with training and education of all types of health manpower. A regional medical program, according to.the respondents, has charisma if it is: 1. Responsive 2. Flexible 3. Aggressive 4. Facilitative, that is, work with and through others 5. ExpressiveIof leadership, that is, effective in getting the health care system to solve the problems that need to be solved 6. Catalytic 7. Regional The following descriptions of the same RMP by two different respondents illustrate the importance of the directional and character components to the identification of a qood program: "There were some good (programs) who were categorically oriented, but who have not done well since the alteration of the mission statement. Others who weren't very good categorically came back strong with that alteration. I can think of several that I would consider to be good RMPS. For instance, let's take RMP-A. There are lot's of things wrong with RMP-A,, but, on the basis of looking at their initial premise which was that Regional Medical Programs should try to do something about@the quality of care for the majority of citizens in a state...,@they moved very smoothly. They were particularly strong in heart disease because the coordinator was a heart di@tease man but he also was able to have enough vision to think about cancer and stroke and kidney disease. He's also been able to makd an alteration in reference to thinking about delivery of health care patterns in that area. In talking about what they've accomplished, I'd say that, when you talk about RMP in Region A, you're talking about something that all the physicians would recognize and that much of the population would know about because of the tremendous publicity they've had for various projects. Th e Regional Medical Program existed as an entity that tried to relate the person who'5 receiving care to the providers of care 50 hing that has been successful and I think that's one t in some RMPs and RMP-A is a good example. I think it has Established quality of care particularly in heart disease, almost as much in stroke, not so much in cancer@ but part of that relates to the capabilities of provi- ders in the area, not necessarily the weaknesses in the RMP. And lastly, I think that they have detected weaknesses on the base of significant data and have done their best to encourage programs to cure the weaknesses. And I guess, for me, that's a very succinct summary of what I think RMP is all about." "I'll talk about what has fascinated me to a very great extent in RMP-A and that is that they have tried to work through and with the hundred or so hospitals in the state in a way to upgrade hospital care and to improve their training programs so that better health care could be delivered throughout the state as a whole. They have worked in terms of developing a-very competent and capable staff. They've always had strong, competent, enlightened leadership. They have been able to make changes in their program direction, I think, in a very favorable way. They've shown evidences of really trying to respond to the needs throughout the state. They have not concentrated in the large population centers of the state, but have tried to move their activities out to the far reaches of the state and to the small community hospitals and to develop training programs for a wide range of physicians, not just specialists in the (major metropolitan area).Il The quoted descriptions illustrate the emphasis on the qualities of "direction and charisma", an emphasis that was more pervasive in the original interviews than in the structured analysis of "categor ies of concern". It is an emphasis that was almost totally eliminated in the verification procedure. 11 The fact that capability" was the dominant concern on the verification procedure may have resulted from the design of the pro- cedure. The original interviews allowed the respondents to talk about those qualities -- although the questions were asked in terms of "accomplishments" -- that they felt were important. They were able to speak as individuals to an interviewer trained in guided and supportive interview techniques. The verification pro- cedure, however, was a highly structured,'closed-ended mail-telephone survey. It-is,hypothesized that the structured instrument presented an examination-like situation which caused the respondents to answer not just as individual, but Ias individuals whose answers were going to be compared with those of others. Knowing that the respondents were all part of a review process that focussed on "accreditation". 51 iie. assessing the capability of an RMp see Section D one can assume that they were trying to give answers that others participating in the review process would give. Why there should be discrepancies between the concerns of the respondents as members of review bodies and their concerns as individuals raises questions that go beyond the scope of this report, but it can be suggested that the current emphasis in the review process on "accreditation" or I'capability" reflects astate in the development of the review process which differentiates between "satisfactory" and "unsatisfactory" programs. As the review process continues to develop, one could anticipate attempts to differentiate between "superior" and "tatisfactory" programs that would consider the qualities of "direction" and "charisma". The evidence suggeststhat considerable effort would be required to further modify the review process to be concerned with accomplishments". Conclusion Interviews with ten selected participants in the review process plus a verification procedure completed by seven of them provided a basis for identifying three general qualities that can ional Medical Programs: be used to evaluate local Reg 1. Capability 2. Direction 3. Charisma The fact that there were marked discrepancies in the emphasis between the interviews and the verification was construed to suqqest that there is a.possibility that the review process will evolve from its present concern with "accreditation" as a device to differentiate "satisfactory" from "unsatisfactory" programs to concern with differentiating "superior"-from "satisfactory" programs. Evaluating Programs in the review process in terms of the "accomplishments" probably will continue to be deferred until there are substantive improvements in the concepts of "accomplishments" and in the information base for describing them. C-7i 52 D. SUMMARY OF CONFERENCES WITH RMPS STAFF ON RMP REVIEW CRITERIA 53 SUMMARY OF CONFERENCES WITH RMPS STAFF ON RMP REVIEW CRITERIA February 27, 1973 INTRODUCTION Selected RMPS staff were convened in two separate con erence sessions to answer the following general questions: 1. How could the process of RMP Review Criteria development be improved? 2. How could or should the use of the RMP Review Criteria be altered? 3. How could or should the content of the RMP Review Criteria be improved? These lead questions were designed to promote a thorough process by this group evaluation of the entire Review Criteria of RMPS staff. This is a summary of the major points, conclusions and recommenda- tions flowing from each of the sessions. SESSION I Development Process Although the first efforts to develop review criteria for use in the review and Approval of local programs was undertaken in the summer of 1970, accelerated development was undertaken in the early spring of 1971 in response to HSMHA's criticism of RMPS' selective funding program. HS@IHA, in essence, felt that there was no systematic basis for decisions made under the proposed selective funding program. In an effort to overcome this objection in time to influence the administration of'a major budget reduction becoming operative at that time, a staff of eight or nine people began and completed the task of developing the RMP Review Criteria in a period of two weeks. The effort, however, was not sufficient at that time for that purpose. Subsequent modification of the criteria and the development of a complete process for evaluating and scoring local RMPs against the criteria was successful in leading to approval by HSMHA of RMPS' selective funding approach. The development process was carried out by RMPS staff'61one on the basis of the RMP legislation, the RMP mission statement'@the ' 9 President's Health Message of 1971, the advice of the National Advisory Council and the accumulated experience of the RMPS staff. The' major 55 cri.ticisms, both of which resulted from the very short time interval over which the criteria were developed, related to the absence of input from other than the senior staff of RMPS and from the coordinators and other local program staff Recommendations for future development of revised review criteria caii( ipation of many more members of RMPS staff and of staff and r(g advisory group members from local programs over a sufficient tire period so that the criteria better reflect the concerns of all involved in the RMP.. It was also recommended that the criteria should focus on those items that tend to discriminate between good and bad RMPs and should specify the objective data to be used in making or contributing to the evaluation of the programs against the criteria. Finally, it was recommended that the Review Committee and the National Advisory Council, the primary users of the results of such a review against criteria outside of AMPS staff, should be intimately involved in any future development of criteria and/or scoring procedure. Utilization of Review Criteria The Review Criteria were reported to be 'helpful to the review process, but not necessarily easy to use. The Review Criteria formed a structure for the preparation of staff briefing documents and for the site visits that was found to be useful to varying degrees largely determined by the general quality of the work done by the staff person of the site visit chairman. When the information in the staff briefing document and/or the site visit report was well organized in relation to the Review Criteria, the users of those documents were quite content' if not well organized and concise, the users tended to be dissatisfied. The sub ective nature of most of the criteria, the absence of precise definitions, and the qOneral lack of objective data related to the criteria were significant deterrents to any uniform benefit to the structuring of information in the review process. The fact that selected individuals were able to use the criteria in a helpful way suggested that improvements in the design of the criteria might lead to a more uniform benefit. The participants in thi s session felt that the local RMPs had not been unhappy with the criteria per se, but that they were uncom- fortable with the rating system which, in general, they di.d not under- stand. They, the local RMPS, tended to ignore the fact that the rating system was a device for comparing the 56 programs with each other and to make the erroneous assumption that somehow the scores of ratings were equated to dollar funding. The criteria were acceptable as evaluation concepts or guidelines, but the 'scores were unacceptable as an evaluation measure. Their confusion was further compounded when they learned that an RMP with a reasonable rating might receive less money than an RMP with a low rating. They failed to understand that selective funding based on Review Criteria scores was not a mechanism for ironing out what might be historical inequities. p@- I 56 Most of the objections focused on the use of the Review Criteria for the development of scores and ratings6 Many of the people who were asked to store programs were reported to object to the lack of precise definitions and to the lack of precise or complete rted that information on which to base their judgements. It was repo some low scores resulted from the absence of information rather than from any deficiency in the programs. No specifit recommendations for changes in the utilization of. the criteria were made, but most of the respondents hoped that any revisions in the criteria would make them easier to use, primarily, in relation to the obtaining and organizing of information and, secondarily, to the scoring and rating. Content of Review Criteria When the Review Criteria were.originally developed, it was the reported intention of RMPS to periodically modify and/or update them. Such an intent was based both on periodic changes in RMPS poli Cy and on increasing insight into those factors associated with the success of a local RMP. The participants felt that modification of the Review Criteria to reflect changes in policy might be innappropriate, but that the content of the Review Criteria should somehow be able to accomodate such policy changes without changing the fundamental precepts and definitions on which the criteria were developed. They felt that the Review Criteria was not a proper way to communicate changes in RMPS program policy. The content of the Review Criteria was judged to be oriented more towards the structure and organization of an RMP than towards its actomplishments. The criteria tended to focus on where An RMP was rather than where it was going or what it had done, but that focus twas based on a rather unstructured and unvalidated model. Although there was some interest expressed in developing more accomplishments or outtomes-oriented criteria, this was thought to be extraordinarily difficult. The major recommendation for changing the content of the Review Criteria related to those changes needed to make them more useful to the users. It was suggested that some means be established to monitor the problems which users of the Review Criteria experience so that such problems can be addressed in any revisions. Such revisions would include changes in content as well as changes in definitions and specifications of necessary data. Summary In general, it was reported that the review criteria worked relatively well , but that certain improvements would be desirable. The. participants felt that the Review Criteria was a more rational. 57 on judgements than previously existed and but limited basis for evaluati that those involved in the review process had demonstrated a willingness to work with criteria. More acceptance of the use of the Review Criteria could probably be anticipated if they were revised by the processes specified towards the resolution of the problems identified. SESSION II Development Process The participants stated that the review process was directed towards the determination of whether or not the local goals and objectives of the program were being met and how well they related to national goals and objectives. They also stated that the precipitbus-developm6nt of the RMP Review Criteria in a thirty day t.ime period by a grou6 of 10-12 RMPS senior staff was in response to the desire of RMPS staff to administer a major cut i'n fundin b ' selective reduction of funds for local RMPs 9 y rather than an across-the-board reduction. The content of the Review Criteria was thought to reflect the perception of the RMPS staff as to the essential elements of a local RMP. The fact that no performance objectives for RMP as a whole had ever been articulated in the Review Criteria as they were developed. Later modification of the Review Criteria to deal with performance against national objectives as they were specified had been anticipated. The following recommendations were made: 1. Members of the steering committee of coordinators, coordinators of local RMPS, and/or evaluation staff of local RMPs should be involved in any future revisions of the Review Criteria. 2. RMPS should contract with a management consultant firm to provide unbiased guidance to the development of the Review Criteria from the management point of view. 3. Adequate time and effort of talented staff should be invested in the development of a conceptual model of the ideal local RMP and of criteria related to that model. .4. Future development of the Review Criteria should provide for major input from the users, e.g. DOD staff, Review Committee, National Advisory Council. 5. Future de velopment of the Review Criteria should deal clearly with the interface between national goals and local programatic goals. 58 6. Future development of the Review Criteria should ample, if the include precise objectives for their use. For ex Review Criteria are to be used as guides for local programs in improving their performance, then they should be i)rocess-oriented, but, if they are to be used in an award-penalty system, then they should be outp6t-oriented. 7. Involvement of both RMP and non-RMP related people in future developments should minimize any unreasonable biases in the Review Criteria. Utilization of Review Criteria The participants chose to discuss the utilization of the Review Criteria and their content primarily in terms of the development process. A series of questions were raised: 1. How does the use of the Review Criteria influence the by which they are developed? process 2. How does the development of the Review Criteria take into account existing or changing national policies, goals and objectives? 3. What are the Review Criteria trying to measure? 4. What future uses of the Review Criteria can be anticipated? in the ensuing discussion, it was felt that the use of the review criteria had and should influence the process by which they are developed. The group pointed'out that "questions" will influence "answers". As a result, the nature of the questions was felt to be just as important as the portent of the answers. The participants felt that the relation between national RMP objectives, the Review Criteria and the measures were not clear. They further felt that the specific objectives for the uses of the measures were sufficiently objective. These were thought to be problems antici- pated in the original development process because of the brief develop ment period; they were also problems which were intended to be solved by revisions in the Review Criteria which did not occur. The present Review Criteria are primarily a device for accrediting local programs, not for measuring their accomplishments and/or the outtome of their activities. The following recommendations were made: 1. Local RMPs should be evaluated on the basis of their accomplishments in relation to national goals and objectives. 59 2. Because the Review Criteria are used primarily by RMPS and its committees for the comparison of local RMPS, the development of the Review Criteria should be done primarily by RMPS personnel. Such Review Criteria should then become an integram part of the RMPS management process in relation to local programs and should not be restricted to the review process alone. 3. Measurements or scoring against Review Criteria should be done primarily by RMPS staff, but should be subjected to review by the local RMPS. 4. The translation of objectives and criteria into measures should involve both RMP and non-RMP personnel. Summary The present Review Criteria are useful in the review process as an accreditation device. More -intensive and sophisticated development of the Review Criteria would increase both their usefulness to the review process and the scope of their use. 60 CHAPTER III EVALUATION FIIDINII '61 This evaluation of the RMP Review Criteria was conducted at the request of the Regional Medical Programs Services and was designed to obtain information which would "include, but not necessarily be limited to answering'' the specific question raised in the Scope of Work - see Appendi-x B. That information has, in large measurealready been presented in the history of the legis- lation, the history of the review process- and the reports of component activities. The evaluation, however, requires that the information be organized and reported in a manner which responds to the manifest concerns of RMPS. Findinqs Regardin'q "Internal Validity" The first area of concern was the "internal validity of the Review Criteria" and the contractor was instructed to "assess the manner in which the established Review Criteria are, in their application, sensitive tools in adequately and objectively assessing the effectiveness of Regional Medical Programs for the purposes of Regional Medical Programs Service review at the National level by both the Regional Medical Program -Review Committee and the Staff Anniversary Review Panel." The statistical analyses relative to this concern and 611 the questions specified in the Scope of Work have been reported on pages 23-28 of this document and in the Second Quarterly Report from this contract. Those analyses were based on scores prepared by individ al u members of the Review Committee and the Staff Anniversary Review Panel. They indicated that there were statistically Siqnificant differences in the raw and weighted total scores and the scores on major criteria groupings among programs classified as "A", "B" and "C" regions. Because programs were classified on the basis of their total scores and because the correlation between scores on the major criteria groupings and the total scores exceeded 0.9, this finding was not unexpected. An analysis of the scores for the individual criteria showed exceptionally high inter-criteria correlation. When this correlation matrix was subjected to principal components factor analysis, only one factor was identified. If the major criteria groupings had been valid, there would have been three factors.. The fact that only one factor was identified indicated, according to the consulting statistician, that the Review Criteria did not constitute a scale or measure that was in any yay Superior to a single rating on Overall Quality. Scoring on the individual criteria did not, in his opinion, contribute to an identification of those specific qualities which differentiate between good and poor RMPS. More specifically, a good program g score s on it ts 00' all criteria and a poor program gets poor scores on a 1 cr eria . 63 The analyses also showed exceptionally high inter-observer correlations. There was nostgnificant bias of the primary reviewers, as a class, nor of any individual reviewer. The absence Of significant bias of any significance was shrouded in such an extraordinary degree of agreement among the scores of all the reviewers for each program that independenceof judgement appeared to be absent.. Stated another way, the scor n program were so uniform as to suggest that h ed to before thes6reviewers completed their@atings. In the absence of scalarity and independent peer judgement, the assessment of.internal validity becomes meaningless. The internal consistency of the scores and the consistency among different observers indicated that the Review Criteria scores express the consensus of the review groups as to the quality of the programs rather than the convergence of independent opin$bns. These conclusions to the effect that the Review Criteria are a very weak tool for adequately and objectively assessing the effectiveness of Regional Medical Programs was substantiated by criticisms and recommendations for improvements reported in the conferences with RMPS staff, pp. 53 - 60 and the interviews, pp. 29-41 . These criticisms can be summarized as follows: 1. The scoring against the Review Criteria is too subjective. 2. Adequate information for making subjective judge- ments against the criteria is not uniformly available. 3. The scoring does not contribute significantly to decision making in the review process. Many of the recommendations from RMPS staff called for making the scoring process more objective and for improving the availability of appropriate objective information. None of the weaknesses identified during this evaluation, however, appeared to have disillusioned.the users who, uniformly anticipated further use of the Review Criteria and who, therefore,, hoped that the weaknesses would be eliminated. Findings Regarding "Effectiveness" The second area of concern to RMPS was "perceptions of those utilizing the criteria in the national review process as to their effectiveness The contractor was instructed to 'determine the adequacy and effectiveness of the.Review Criteria in assessing the comparative quality of Regional Medical Programs as perceived by those utilizing them (e.g. Review Committee members, Regional r, A Medical Programs staff). The information obtained was to be aimed at, but not necessarily limited to,answering a set of six specific questions. Five of the six specific questions concerned the relative importance of major criteria groupings and individual criteria, the availability of information and the rationale for adding or deleting criteria. The sixth question related to perceived improvements in the review process that had resulted from use of the Review Criteria. No information that would answer the first five questions has yet been reported because of an unexpected occurrence. Following principles of worth assessment as described in the book, Professional Decision Making by Jamet R. Miller, III, and as articulated in consultations with Dr. Miller, a conference to answer these questions was held with selected RMPS staff. The objective of the conference was to identify those qualities that the participants, as a group, felt were important in evaluating the-local RMPS. The intent was to compare the identified qualities with the Review Criteria so that the five questions could be answered. The a enda of the conference called for the participants: 9 1. To individually sort the questions contained in the Review Criteria into clusters of similar or related questions and to title those clusters -- 2. To compare and discuss the clusters and titles of individual participants -- 3. To develop, as a group, a single set of cluster titles 4. To individually sort the questions into clusters under the agreed upon titles. The agenda was excessively long for the scheduled time of the conference and, therefore, the objectives could not be achieved but a totally unexpected phenomenon that occurred du@ing the comparison and discussion segment suggested that days rather than hours would have been needed to develop a consensus on cluster ti.,tles. The phenomenon was two-f6ld: a. Marked differences of opinion about qualities that were relevant to the evaluation of RMPs--differences both among individuals and among groups of individuals from different Divisions and Offices of RMPS; and b. An unwillingness of the participants under the circum- stances of this conference to negotiate their differences and come to consensus on cluster titles. 65 e he the questions clusters pr pared at t An analysis of start of the conference by each individual failed to show any consistent clustering patterns and gave further evidence of the diversity of opinion apparent in the discussion. Because the development of a consensus would require considerably more effort on the part of the participants then they would be willing to provide due to the announced phase-out plans for RMP, this approach to answering the five questions was discontinued. Subsequent conferences -- see pp.53-60 and analysis of interviews -- see pp. 29-41 suggested that the major criteria groupings were satisfactory in concept-, that is that Regional Medical Programs should b' evaluated on their Performance (Achievements), Process e (Organizational Capability) and Proposed Program (Proposed Plan), but that significant modifications in the individual criteria and their clustering would be needed to-make that concept operative. Many felt that, in spite of the major criteria titles, the Review Criteria tended to focus, at least in terms of how they were applied, on Process in an "accreditation" concept: "The criteria tend to focus on where an RMP is rather than where it is going or what it has done." Specific criticisms were levelled at the individual criteria, the scoring process and the detailed questions. u taff were particularly concerned about Individ al Criteria: RMPS s the number of criteria and the overlap between them. The interviews identified cbnt@@n"'6b6ut thei'@"I'a'ck of @ '@eci'sion p and RMPS staff reported that site visitors, in particular, did not always have a clear understanding of what was intended by each individual criterion title. Scoring Process: Numerical scoring was disliked by many of those interviewed because, according to the impressions of the interviewer, they felt that the numbers did not allow them to express their judgements in a way with which they were comfortable. They apparently felt that numerical scoring was an artificial process. They reported that the assign- ment of numerical values was a subjective process that, as currently being practiced, required each reviewer to com are the program under consideration with other p programs with which he was familiar. They apparently would have preferred some'rechanism for getting around the need for making comparisons and would have preferred to have reliable information uniformly available upon which they would base their judgements. 66 Questions: The questions were criticized for their number, their f explicitness and the inavail- redundancy, their lack o f relevant information. ability o In spite of these criticisms, RMPS staff and selected members of the National Advisory Council , Review Committee and Staff Anniversary Review Panel felt that the Review Criteria had been an effective management tool for the following purposes: 1. To facilitate a change in the review process from a project-oriented to a program-oriented approach; 2. To provide additional structure for technical assist- ance in program development provided to local programs by RMPS; 3. To establish mechanism for preserving continuity in the review process in the presence of turnover of pe rsonnel on the various review bodies; and 4 To serve as a partial basis or rationale for selective funding decisions made by the Direttor of RMPS. The Review Criteria contributed to or facilitated the change in the review process from a project-oriented to a program- oriented approach by providing a framework for organizing and/or soliciting information. Although many of those involved in the review .te critical of the requirement for scoring process were qui programs agains the criteria, that requirement probably forced them to pay more attention to the details of the criteria than would have occurred if there had been no such requirement. The scoring exercise, although of low repute among the users, was probably an essential ingredient to the facilitation of the changes in the review process. The Review Criteria, according to RMPS staff, began to serve as the structure for advice letters to local RMPS in 1971. Prior to that time,,local programs received only a notification of award, a notice which may or may not have provided an assessment of their strengths and weaknesses as identified in the review process and, therefore, varied considerably as a useful guide for future program development. As the review process shifted to the program-oriented approach, RMPS began to play a more supportive role in local program development. This took the form of reporting to the local programs those areas within the framework of the Review Criteria in which they had been judged to be doing poorly and advising them to make specific improvements. Because the Review Criteria were, publicized and formed the framework for the advice letter, local programs quickly began to understand how they would be reviewed in the future and, as a consequence, they had a better framework for 67 for their own planning and program development. Although :no specific information was available to evaluate the Review Criteria as a mechanism for preserving continuity in the review process, RMPS staff, in informal conversations., did report that the criteria had been helpful in this area. The basis for selective funding decisionsmade by the Director@ of RMPS has not been explicitly stated in a public document, but it appears, on the basis of conversations with RMPS staff and relevant RMPS memos, that the decisions are based on the concepts of @u incentives, and other unidentified factors. ty sTo n Of ated implies that t ere to the existing to parity of funding on a per capita basis and quantity of available resources and magnitude of need within regions. Those components of the concept of equity plus other still to be identified components have not been transformed into a funding formula; they are, however, considered in selective funding decisions. .The concept of incentives, that is, programs that do well should be rE do poorly should not be rewarded,also contributes to the selective funding decisions. The Review Crit eria contribute to the implementation of the concept of incentives in selective funding decisions in the following manner. On the basis of Review Criteria scores, each local program receives the designation of "A", "Bl'i "C", with "A" being the highest. Almost all "A" regions are funded at I00% of the award level approved by the National Advisory Council. Only a few of the 'IC" regions are funded at the 100% level. This is based on the premise that an "A" program is more accountable and, therefore, better able to get full value from their funds than a "C" program. The variations of funding levels within each class of programs reflects the influence of the concept of equity as well as variations in amount of money available for awards during different parts of the fiscal year. Conclusions The evaluation findings indicate that the Review Criteria have been a very effective "management tool" within the operations of Regional Medical Programs Service, but that considerable improvements in their design, content and application are needed to make them. function as an and objective measure ively assesses the "c I-P rams. 68 CHAPTER IV RECOMMENDATIONS 69 The evaluation of the RMP Review Criteria, although already n findings, assumes a more coherent form in reported in the evaluati.o the recommendations. The seventeen recomendations are based on the findings, criticisms and comments as reported in@-the three specific studies, the reports of the two conferences with RMPS staff and add!- tional data derived from analys6s of RMP-related documents, informal interviews and consultations. In many ways, the findings presented as a rationale for the recommendations are more understandable than the findings presented as information related to the purposes and questions posed in the Scope of Work. The rationale for the recom- mendations, therefore, serves as a partial reiteration of previously reported information and as a clarification of the con-cepts underlying the approach to the evaluation. The recommendations fall into three general groupings. The first six recommendations outline some uses for the Review Criteria in 1) the review process, 2) selective funding decisions 3) local RMP program planning, development and operations, and 4) iMP program evaluation. Recommendations #7-9 describe suggested methods for im roving the content of the Review Criteria and call for some specific modifications in their design. The last eight recommendations deal with specific changes in the ways in which the Review Criteria are developed and utiltzed fn RMPS operations. All the recommendations are made on the premises that Regional Medical Programs will continue to operate as a federal'program and that the Review Criteria, in modified form, will continue to be used by RMPS as an adjunct to future RMP activities. The announced phase-out plans for Regional Medical Programs and their subsequent reversal makes the first premise tenuous. The s econd premise, although conditioned on the first, is reasoned in terms of the support for the Review Criteria evoked from RMPS staff and various participants in the review process during this evaluation effort. Should either or both of these premises prove to be false, the recommendations should be reviewed for their applicability to other federal health programs that choose to develop expltcit criteria for use in review and approval decisions and in other aspects of program operations. Recommendation #I: The Review Criteria should be used to develop and maintain a balan tive in the review process on performance, pro rogram pro The RMO Review Criteria are organized into three major groupings: 1) Performance, 2) Process, and 3) Program Plan. These groupings have weil ghts of 40, 35 and 25 respectively and imply that they follow that order of relative importance. 71 Comments from the-conferences and interviews and the analysis of the findings from the interviews and the verification study indicate that. Irl 1. The review process focusses on llaccreditation" and emphasizes criteria related to "Process,.,Il 2. The Review Criteria in spite of their design, emphasize "Process" over I'Periormance" and "Program Proposal." Although the statistical studies indicated that scoring against the Review Criteria resulted in a rating against "Overall Quality" and not in measurements involving a series df different qualities, it can-be assumed that "Overall Qualtty" is really a proxy for those qualities that describe the capability of an RMP, i.e. "Process", in terms of its organizational structure and perceived competence of its staff and regional advisory group. Many of the participants in the evaluation activities expressed dissatisfaction with the focus on "accreditation" and advocated the balanced approach that underlay the original design of the Review Criteria. Specifically they advocated an increased emphasis on "achievements" or performances, but felt that such an emphasis would require the develop- ment of better concepts and information for making judgements in this area. The difficulties in focussing on "Program Plan" were described as the result of the absence of any National objectives or guidelines .against which a local program plan could be judged. In no case, however, did the participants withhold support for the balanced concepts in the Review Criteria related to Perfomance, Process and Program Plan and they generally supported further efforts to make those balanced concepts operative. Recommendat4on #2: The Review Criteria should continue to be used as a device for in the review process. ifidicated that the Review Crit7e The evaluation findings ria had been effective in promoting continuity in the review process. As long as there continue-to be changes of personnel in the review process, there will be a need for the new people to know the bases for decision- making that have been established. It was reported that some RMPS staff and site visit chairmen had used the Review Criteria quite effectively to indoctrinate new members of the site visit teams in@the information to be collected on the site visits. It was also reoortod. that the Review Criteria had been used to facilitate the incorporation: of new members into the Review Committee and the National Advisory Council. Such uses, obviouslycan and should be continued. 72 Recommendation #3: The Review Criteria should be u ed to 6ttUre consistency s of ithin all levels of t e review pro t review cycles. ntify the The conference with selected RMPS staff to ide relative importance of the various criteria and the interviews and ten participants in the review process suggested verification study with that there was considerable possibility for variations in the bases for d decision-making among the different individuals evaluative judgements an and groups involved in the review process. Although the present use of the Review Criteria may mitigate against this possibility, more active efforts to pre vent it and, thereby, assure consistency in the review process should be undertaken. Diversity, while essential to the develop- mont of innovative programs, is anathema to review and evaluation. Review and evaluation implies a single set of expectations uniformly applied to all those undergoing those types of judgements. Evaluativ6 judgements, ideally, should be based on explicit. standards whose application should not be dependent upon the composition of the review bodies, the particu lar interests of Individuals, or changing emphases over time:that have not been reflected in well-publicized formal policy modifications. The Review Criteria, properly used, can function both as the explicit standards for judgement and as the basis for evaluating the consistency of the performance of groups making those judgements. Recommendation #4: The Review Criteria should continue to serve as part of the rationale for selective fundjna decisions. RMPS has been and will continue to be faced with the need for making budgetary decisions within the funding levels approved by the National Advisory Council. These decisions will be required-when the total funds approved for award by the National Advisory Council exceed the total funds available and/or when there are unexpected reductions in the funds available. The budgetary decisions may-call for uniform percentage reductions in awards to all programs or selective.reductions in awards to some programs. Historically, RMPS has tried to use the selective funding approach. Initially, their selective funding decisions were found unacceptable by HSMHA because there was no explicit basis for such decisions. Subsequently, selective funding decisions were in part; on the basis of progra allowed m classifications derived from Review 6riterta scores. The full basis for current selective funding decisions has not been published, but the concept of incentives does appear to play a role in these decisions. The concept of incentives means that programs that receive an "A" classification as a result of their review criteria scores are more likely to receive 100% funding than those classified as "C" regions. As currently practiced, this rationale does encourage programs to try to meet the standards implicit' in the Review Criteria, but the objections to the scoring process, the 73 reported weaknesses of the criteria and the potential for variability of @P@l@c tton opens the use of this rationale for selective-funding a decisions, to criticism. The fact that the users of the Review Criteria tn the Rev@ew Committee and Staff Anniversary Review Panel reported their contributions to the development of consensus as to which Programs were of "high" quality and of ow quality suggest that the classification as "A", "B", and "C" may be.more valid than the objections would indicate, but further use of the Review Criteria tn selective funding decisions should be dependent upon eliminating the bases for the objections. It can be anticipated that further use of the classification of programs based on Review Criteria scores, particularly i,f the bases for the objections can be eliminated will promote the development and operation of high quality local Regional ?4edi,cal Programs. Recommendation #5: The Review Criteria should be used to provide a consistent fra local program planning, de ment and operati s. The fundamental unit of organization within Regional Medical Programs is the local Regional Medical Program.. The organization whch receives the planning and/or operational grant award is charged with the responsibility o? organizing a regional advisory group that develops the goals and objectives for the program and approves all grant applica- tions. They are also charged with the responsibility of assuring that any proposed activities have been reviewed by the appropriate Compre- 'llity with hensive Health Planning Area-wide Agencies for compatib any local plans for health care systems development. The local autonomy implicit in the local organization of Regional Medical Programs and the requirements for local approval and/or review belies the fact that there are some uniform purposes specified in the legislation for Regional Medical Programs. To date, the processes related to local autonomy have received more emphasis than achievements related to National objectives, but there is an increasing tendency to shift the emphasis. Local programs, thus-'far, have been allowed to develop and operate wlthi,n the loose framework supplied by Guidelines for Re'glonal Medical Programs &Ius various publications, statements and letters. The promulgation by MPS of grant programs for Emergency Medical Services and A'rea Health Education Centers signified the transition towards a greater degree of central direction. The fact, as reported in the conferences, that some local Regional Medtcal Programs, wtth and without the prodding of RMPS staff have begun to use the RMP Review Criteria to guide their program planning, development and operations,suggests that the Review Criteria may serve a role in guiding local programs towards the achievement of National,objectives. Although the summary-o@ the RMPS staff conference indicated that the Review Criteria are an inappropriate mechanism for communicating changes in RMPS'policy, modifications in the Review Criteria that emphsize the 74 importance of RMPS policy and National objectives for RMP tn local program activities Eight lead to greater compiance with them. If widely publicized, the Review.Crtteria could serve more unifomlv as a gu@de to local programs and would probably have the effect not 0 that score well against the criteria, only of developing more pr grams developing more programs that make substative contributions but also of to the achievement of Nattonal objectives for Regional Medi.cal Programs. Recommendat@on'#6: The Review Criteria should be used to facilitate the development of information that can be used in RMPS program evalua on and'ih'the Justificati6h'df'RO!Oibh@l me ica i-ograms to t C. Regional Medical Programs have been criticized by the Administra- tion for their failure to make significant changes in the health care system of the Nation: "...there is little evidence that on a nationwide basis the RMP's have materially affected the health care delivery system... That criticism implies that the Administration had objectives related to changes in the "h'ealth care delivery system" that were not addressed by the available evidence. In contrast to the judgements of the Administration are the evaluations of local RMPs developedtlhrough the review process. Many local programs, according to RMPS s ff and the reports from the review process, were making significant improvements in the planning, development and operations of their programs and some, as indicated by their high Review Criteria scores and ratings, were judged to be of high quality.. The discrepancies between the evaluation of the Administration and that of RMPS could be accounted for in a number of ways" but accounting for those discrepancies is not nearly as important as eliminating them. Modifications of the Review Cri7Nria to emphasize the importance of National objectives and utilizing the Review Criteria as a tool for developing information to be presented to the Administration, the Congress and the public that would document achievements related to those objectives seem indicated. Such modifications might have been considered earlier if the Review Criteria had been considered as part of the program evaluation and program justification effort of RMPS and not just as a device for evaluating local programs in the review process. Recommendati,on #7t The content of the Review Criteria should be related, whenever possible and appropriate, to specific guidelines and ObJecttves -to-be established 4y the Administration for the Regional Me2ical Programs at the ational Tevel on the basis of the legislation and the legislative proceedings. The absence of specific National guidelines and objectives';,for Regional Medical Programs from the Review Criteria reflects a reluctance I I Budget Pr cal Year 1974, Appendix, p. 383 oposal for Fis I 75 of Health, Education and Welfare to establish within the De artment onent programs. As long as guidelines performance objectives for its com ves for Regional Medica@ Programs are deduced'by WPS staff and objectt from the legislation and Prestdential Health Messages @- the process currently followed according to RMPS staff "- there remains the opportunity for DHEW to second@guess the staff.and, after"the-fact, outline a set of objectives for evaluating the program that, in fact, were never available to guide the program in its activities. s is Establishing reasonable objectives for social pr'ogram extraordinarily difficult. Delegating the responsibility to lower levels in the admnittrattve hierarchy is a common ploy for the development of objectives and one that implies acceptance of those objectives by the htgher levels. Whether the objectives are established at a higher level !7i or a lower level i's not the issue. Regardless of where they are established, the objectives should be explicitlystated and accepted at all levels and should adequately reflect the intent of the Congress as recorded in the legisla tton and the legislative proceedings. The objectives need not be all-inclusive. They should specify the standards against which cost-effectiveness studies can be done, but they need not limit the range of benefits identified i@ cost-benefit studies. According to RMPS staff, DHEW or HSMHA has not, in the past, provided Regional Medical Programs with performance objectives. Neither has RMPS established and publicized such objectives. The Review Criteria are the nearest proxy to performance objectives, but'they fall short' The RMPS Mission Statement is a similarly imprecise proxy for performance objectives. If and when National objectives are established for Regional Medical Programs, relating the Review Criteria to them should significantly increase the probability that they will be achieved. Recommend tion #8-. The content of the Review Criteria should be modified leadership of RMPS staff with--the Dartii f cil members, Review Committi 0 staff, and Regional Advisory Group members. The development of Review Criteria can be a tedious,, time- consuming and expensive process . Their'initial development was done by a small group of senior RMPS staffi Selected RMPS staff in conference indicated that the original methods of development was undesirable and that future developments of the Review Criteria should allow for broad participation of National Advisory Council members, Review Committee members and RMP coordinators, staff and Regional Advisory Group members. They old however, indicate that future development of the criteria should continue to be under the direction of RMPS staff with advice of expert consultants and that it was of sufficient importance so that cost factors should be disregarded. 76 The question-clustering conference with RMPS staff and a consultation wi'th James R. Miller, III, an authority in this field, made it quite clear that such broad participation would complicate the process of developing the Review Criteria and would not necessarily increase thei teness or acceptability. Nevertheless, broad r appropri,a participation was so strongly recommended that, regardless of any anticipated problems, it should be carried out. A suggested approach to this recommendation that include facets of subsequent recommendations arepresented later in this chapter. Recommendation #9: The design of the Review Criteria should be significantly modified as follows: a. The major criteria should be retitled Achievements, Organizational Processes and Proposed Plans; b. The individual criteria should be reduced in number from twenty to twelve or less; c. The numerical values of 1, 2, 3, 4 and 5 should be accompan ied by statemen a ts that indic te the precise meaning of each numerical value for each individual criterion; and d. The questions should be reduced in number and should be translated into a form that can be answered by Present/Absent or that calls for specific qualitative descriptions or quantitative measures. The RMP Review Criteria consist of twenty individual criteria organized into three major criteria groupings: Performance, Process and Proposed Program. Neither the individual criteria nor the major' criteria groupings have definitions. The intended,,but unexpressed definitions of the individual criteria are supposedly expresse d in the specific questions, but the fact that those questions could not be re-sorted into groupings Under those titles by RMPS staff suggests that the questions do not function adequately as definitions. This recommendation responds to a series of-criticisms identified by the evaluation studies, but, in additionlis based on the concept of the Review Criteria as a hierarchy of related values. Level One of the hierarchy is a value entitled: Overall Quality. Level Two, according to this recommendation, is a set,-of values entitled Achievements, Organizati'onal Processes, and Proposed Plans. Level Three calls for three sets of values, not to exceed a total of twelve individual criteria,,that indi.cate the components of the three criteria-at Level Two. Level Four specif@es the related objective information that should be used in making evaluative judgements against Level Three criteria. Table 4-1 is a sample of how such a hierarchy might appear. This recommendation,,however, is concerned with detailed changes in design that go beyond the simple development of a hierarchy. Table-4-2 illustrates most of these changes as they relate to the Level Two criterion of Achievements. 77 TABLE 4-1 sample RMP Review Criteria To Illustrate Four Level Hierarchy LEVEL THREE LEVEL FOUR .VEL ONE LEVEL TWO List regional networks Regionalization of by type Health Care System List subregional networks by type and population served Ltst deficiencies Resolution of Identi- Achievements fied Deficiencies in List activities by deficiencies Delivery System List impact data by deficiencies List activities by objective Achievement of National Objectives List impact data by objective for Delivery System Coordinator Staff Regional Advisory Group verall Quality Organizational Processes Management Planning Evaluation Related to Identified Deficiencies in Delivery System Related to National Proposed Plan Objectives for Delivery System Utilizes Concepts of Regionalization TABLE 4-2 Sample Criteria Definitions, Scoring Definitions and Descriptors Related to Second Level Criterion of Achievements LEVEL TWO LEVEL THREE LEVEL FOUR Achievements: The degree to The degree ionaliozafti,0n lescri tor' which funds expended by the lization of List f nctio nal regional local RMP have resulted in the health care system of the networks by type of support identifiable improvements region has been attained. service, categorical disease, in the health care system or elements of comprehensive of the region. Scoring Definitions: services I- No identifiable regionaliza- 2. List of functional sub-regional networks by type and size of Scoring Definition: tion 1- No identifiable improvements 2- Sub-regional or regional population served 2- Few identifiable improvements collaborative efforts estab- 3- Few identifiable improvements lished in high priority areas 3- Sub-regional or regional 4- Many identifiable improve- networks established for ments in high priority areas supportive services to 5- More identifiable improve- health care system, e.g. ments in high priority areas continuing education than night be expected from 4- Sub-regional or regional the level of funds expended networks established for health care delivery in one or iitore catl-yurical disease areas 5- Sub-reainnal or recional networks established for comprehensive health care delivery including the categorical diseases, preventive, primary, ency and rehabilita- emerg tive care Deficiencies: The degree to Deficiencv Descriptors: which significant deficiencies 1. List ireficiencies in health in the health care delivery care system of region and system of the region have been their priority, preferably resolved. as supplied by Comprehensive Health Planning Areawide and Scoring Deficiencies: Statewide Agencies or as I- ijpficiencies not identified Supplied by tne Regional 2- t:o apparent resolution of Advisory Group and reported identified deficiencies in the-last submitted triennial 3- Partial or complete resolu- plan h deficiency, list the tion of identified deficiencies 2. For eac of low priority activities undertaken since the 4- Partial or complete resolu- last triennial plan was approved tion of identified deficiencies that purport to resolve the of high priority deficiency 5- All significant deficiencies 3. For each deficiency, summarize resolved that might indicate any reported data or descriptions resolution of the deficiencies National Objectives: The degree to National riDtors: which National objectives for 1. For ea list activities improvements in the health care undertaken in last grant period delivery system have been achieved that purport to achieve that within the region. objective 2. For each objective, summarize any Scoring Deficiencies: reported data or'descriptions that 1- No activities related to National might indicate partial or complete objectives achievements of that objective 2- No identifiable achievements related to National objectives 3- Partial or complete achievements of some National objectives in one or more sub-regions 4- Partial or complete achievement of some National objectives in entire region 5- Partial or complete achievement of all National objectives in @qtire region -Implementation of this recommendation should make it possible to eliminate some of the reported criticisms including 1) excess focus and associated problems, on liaccredttatton", 2) excess number of criteria 3) confusion w ari.cal scori.ng process, and 4) weaknesses of the _itli@ num specific questions. Recommen d be inclu ed'iii,the Guidelines The Review Criteria shoul and/or in local programs under program planning development and operation., n of how the Review Cri-teri'a are used by ocess and their technical assistance ided in the same way. activities should also be prov profound effect on program Evaluation systems can have as, the evaluation methods and performance when the evaluation standard t'on findings are clearly understood by the managers the uses of the evalua i and personnel of the programs. Publishing that information in official documents is one of the first steps leading towards such understanding. When the Information about the evaluation system is promulgated through ble evidence that the evaluation systems are official documents with visi being used, the evaluation standards then begn to function as program guides. Only when they function as program guides will programs stand any reasonable chance of receiving high ratings when evaluated against the standards. Recomend6t!on #11: RMPS the Office of Systems Management should be staff in responsible for the de lieu of the Of e e@scriptors related to t reparative standards can be eve ope a u th@ nt Information System of on related Lack of information was one of the oft-repeated criticisms of the. Revtew Cri'teria. It was also one of the factors cited in the Admintstrations recomendation that Regional Medical Programs be phased out. Obtatning, analyzing and presenting information, however, is no trivial task and, therefore, should be guided by the precepts of relevancy. At the present time, the Office for Systems Management processes certain information from the grant applications in a management information system. Unfortunately,, thit system is not closely linked either to the Revtew Cr@terila or to the implied, but unpublicized, objectives of the Administratton for Regional hedical Programs. This recommendation, which assumes that the previously recom- mended alterations in the Review Criteria will be made" is based on the 80 existing responsibilities of the Office of Systems Man'agement for the management'information system. It calls for that Office to take the grimary'responsibili.ty for developing the appropriate Level Four escri.ptors as a replacement for the specific questions of the Review Criteria and to obtain, analyze and present that information 1) for the review process, 2) for program justification to Congress, Acknin- istration and the publ@c, and 3) for other management .purposes as needed. If closely tied to the modified Review Criteri:a', the information related to the Level Four Descriptors should eliminate the expressed criticism about lack of information. Recommendation #12: RMPS staff i'n the Division of Operations and Development should be responsible for the design of tne Level Three (;ri eria f the National Advisory Council, the with the participation 0 Review Comittee and local RMPs and for preparing the scores with accompanvinq statements of justfficatton for each indivi- dual criterion for the staff briefing document. The staff of the Division of Operations and Development provides the primary interface between RMPS and local RMPS. They have a significant responsibility for helping local RMPs to plan, develop and operate their programs at a high level of quality and effectiveness. This recommendation is bas f ed on the ollowing assumptions 1. The staff of DOD has the best first-hand knowledge of those factors that indicate a high level of quality and effectiveness; 2. The factors that indicate a high level of quality and effectiveness should be translated into Level Three Criteria; 3. Review Criteria, if used as program guides, should be used. as guides for technical assistance to local programs by DOD staff; 1 4. Scoring programs against Level Three Criteria by DOD staff will emphasize to the staff and to the local RMPs that-the criteria are to be used as program guides; and 5. Scoring program against Level Three Criteria by -DOD staff can serve as an internal evaluation of the effectiveness of their technical assist ance activities. Recommendation #13: Si.te visit teams should study the Level Three Criteria scores an of justification as prepared by@@the Division of Operati.ons and Development and the Level @Four Descriptor information as prepared by the@Office of Systems Management and should describe, in their@@itt,vitit report, their evaluation of the submitted scores justification - $ @s, a n@ r report shou include modifications of su mitted Leve'@ Thrie Criteria scores if indicated by the site visit findings. 81 to accomplis Site vi.si,t teams have a great amount of work n a very short,time. 'Accomplishing that work can be facilitated by providing w eadtly understandable information related to Level Four Descri tors them 'ttli r p and wi.th scores and statements of justification for Level Three Crtteria in the staff briefing document. Their ti'sk, by being transformed into a verification procedure, should become considerably easier. Their evaluation of the submitted Review Criteria Information and scores should make weaknesses in the Review Cri'teri'a, the Level Four Descriptor Information and the evaluative judgements of DOD staff readily apparent, and, therefore, amenable to corrective action. Recommenda ion #14. The Review Committee and Staff Ann@er, Panel should stud@ the sub@l , justific data including modifications from the site visit team and should Include evaluative comments: in"their report, but they should prepare scores with s atements of justification only-for e ree- ce of independent judgemi rring in the present patterns of Review Criteria score it is recommended that the scores and their justifications of the qro 0 and that scorin by indiv nated. 9 The scores and Justif riteria should 4r be Included in their ions. The RMP Review Committee and the Staff Anniversary Review Panel have had the sole responsibility for scoring programs. against the Review Criteria. These recommendations ;call for different groups to score programs at different levels: Nunber of Criteria Level Criteria Review Gro@ Level One Criterion (i Nati.onal Advisory Council Level Two Criteria (3) Review Committee and Staff Anniversary Review Panel Level Three Criteria (12) Staff of Division of Operations and Develop- v ment; erified by site visit teams Level Four Descriptors Staff of Office of Systems Management This means that, in the future, the Review Committee and Staff Anntversa Revi'ew Panel would have responsibility for scoring programs ry, against the three pro osed Second Level Criteria: Achievements,, Organiza- tibhal Processes and @roposed Plan -- a significant reduction oad in work I from the present required scoring against twenty criteria. This reduction In work lbad would @ake 1 ossible for their scoring process to be following c modified in a way that would resolve the @iticism and adverse evaluation findi.ngs: 82 1. The scoring process fails to function as independent peer -nent; judger 2. The scoring against twenty individual criteria functions as a rating against a single criteria of Overall Quality and not as a true measure; 3. The scori.ng process depends on information that varies considerably in its availability; and 4. The. scori.ng process is dependent upon comparative experience with many programs. The proposed modifications in the scoring process will: 1. Eliminate the pretense of independent peer judgement by calli,ng for scori,ng by concensus; 2. Increase the likelihood of rating against more than one criteria by reducing the number of required scores to three and, thereby,allowing the members to .focus on the differences between the criteria; 3. Assure that the appropriate information is available by calltng for an evaluation of the submitted information and for the preparation of scores with statements of justifica- tions by concensus, a procedure that is intended to eliminate the differential understanding and weighting of information that occurs with individual scoring without required statements of justifications; and 4. Allow the comparative experience of the group to influence the scoring and, thereby, mitigate the effect of variations in the experience of individual members in reviewing multiple programs. nd the Staff Although the scoring by the Review Committee a Anniversary Review Panel may be influenced by the descriptive data submitted from the Office for Systems Management and the Level Three scores submitted by the staff of DOD with or without modifications by a site visit team, the scoring should be based primarily on the judgement of the members. If there are discrepancies between Level Two scores and Level Three'scores,. it can be Assumed that corrective actions need to be taken in terms of the- Level Three Criteria or the scoring process within DOD.. Recomme-nda on #15-. The National Advisory Council should study the scores and s a emen Level Two and Three Criteria 1 staff briefing document, s@te visit report, Review Committee and Staff Anniversary Review Panel reports and should include evaluative comments in their report. In addition, it is recom- mended that the National Advisory Council, identify, by cbncensus a score and an accompan ing statement of jus i ica ion that reflects -their evaluation of the n uveraii quality aniT inciu i ii'ith their report and recomen@a ions. 83 Councills role in Regional Medical Prog The Nati,pnal Advisory rams -in the legislation In carrying,out that role,'they must is !4pectfted m nd a funding level before funds approve all grant applications and reco me can be-awar'ded. 'At'the present time'. they make no formal evaluation judgements other than those recorded in their recommendations for approval and funding level. Because of the use of Review Cr@teria scores in the classification of programs for selective funding decisions by the Di@rector of RMPS, a formal rat@.ng from the Nati.onal Advisory Council on Overall Quality seems appropriate. This rating would serve as additional guidance from the group designated by- the legislation as responsible for approved funding levels to the Di-rector of RMPS in his selective funding decisions. This rating would also serve as a check on evaluative judgements against Level Two and Three Criteri:a. Recommendation #16-,- RMPS staff in the Office for Planni uation should monitor the sco tive comments reported by t s Management, Division Of Operations and Development, site visit teams, Review Committee, Staff Anniversary Review Panel and National Advisory Council and should anal-vze'them for consistency of evaluative judgement among tted information, and f ria. The evaluation studies suggest considerable diversity of judgements values among individuals and groups of individuals involved in the review process and between the same individuals under varying conditions. At present, there are no methods for assuring consistency within the review through their eva uative process and it is possible for RMPS staff judgements, to-direct .programs towards go;ls that are not considered important by the Review Committee and the National Advisory Council Centralization of responsibility for evaluative judgement is one approach to the elimination of diversity, but one that is noi feasible in a program that is the size of Regional Medical Programs. This recommendation proposes a management systems approach that would assure consistency by identifying for the management of RMPS when consistency is not present. The necessary management actions, when consistency is found to be absent, could include modifications of the criteria, education of personnel, and/or other reasonable tactics. Analysis of the Review Criteria scores according to methods described in Profe ision Making by James R. Miller, III, will not only make it po@ssi t and missing criteria, but also p eria will make it ' ossible to dev, s,for Level Two and Three-drit scores by 6 rational method. Recomendation #1 7: The Review Cri.teria data scores and statements of justification from each of the review bodies should be submitted to the I Regional Med@cal POoqtamt at a electrification Proce e o e nd 84 Th@s recommendation i.s designed to promote "maximum feasible understanding of the cormon..@.Purposes for RMP@ and local RMPT. Secrecy breeds susplcton; openness engenders trust. Thi.s recommendation symbolizes the openness necessary to foster the collaborate've working relationship betwe'en RMPS and'local RMPS- that is advocated as the first step in success- ful regionali.zation at the local level. Throughout these recommendations is t e assumption that the Review Criteria wi,ll be modified and will continue to be used. The f this evaluate ed sequence experience o on leads to a suggest of activities that would, at a reasonable cost and effort, result in the recommended modifications in content and design'. Modifications in RMPS operations related to the Review Criteria and the review process, as proposed in these recommendations, are the prerogative of RMPS management to accept or reject,, but any such decisions are contingent upon many unilue factors beyond the scope of this study. The procedures for modifying the content and design, however, are not only independent of many of those factors, but also generally applicable to Regional Medical Programs and other federal programs that 'desire to develop review criteria: Suggested Ac the Content and Des@n 1. Review, modifications and acceptance of criterion definition and scoring definitions for Level One Criteria: Overall Quality -- by National Advisory Council based on draft document prepared by Staff Anniversary Review Panel and consultants. 2. Review, modification and acceptance of criteria definitions and scoring definitions for Level Two Criteria: Achievements, Organiza- tional Processes, and Pr osed Plan by RMP Review Committee based on O' . draft document prepared by Staff Anniversary Review Panel and consultants. 3. Publication and distribution of Level One and Two criteria, criteria definitions and-.scoring definitions to RMPS staff,, coordinators and Regtonal Advisory Group chairmen for review and comment. 4. Modificatton, finalization and approval of Level One and Two criteria, cri.teri.a definitions and scorin@' definitions by National Advisory Counci.1 RMP Review Committee and PMPS Di.rector. 5. Revtew, modification and acceptance of Level Three'criteria titles, cri.teria definitions and scoring definitions by Steert.ng Committee of COordtnators based on draft prepared by staff of Dtvis-ion of, Operations and Development and consultants. the Level Three criteria ti,tles.' should relate directly to the Level Two criteria definitions and scoring ;definitions. 6. Publtc4tton and,idistribution of Level Three criteria, criteria dp-ftntttons and scoring definitions to RMPS staff, coordinators and - R.egi,onal Advisory Group chalmen for review and comment. 7. Modtftcation, finalization and approval of Level Three crtteria, crtteria definitions and scori.ng definitions by National Advtsory- Council, RMP Revtew Commlttee,.Steering Comtttee of Coordinators and RMPS Director. 8. Design of Level Four Descriptors by staff of Offfce of Systems Management with consultants. 9. Revtev, modification and acceptance.of Level Four Descri,ptors by Staff Anni'versary Review Panel. 10. Publi,cati,on and distribution of Level Four Descriptors to RMPS staff, coordinators and Regional Advisory Group chairmen for review and comment. 11. Modi,fi,cation, finalization and approval of Level Four Descriptors by National Advisory Council, RMP Review Committee, Stetrlng Comittee of Coordinators, and Director RMPS. 12. Redesign and pretest of grant application and management thfomation system by staff of Office of Systems Managements. 13. Publication and distribution of redesigned grant application and data to be generated by management information system to RMPS, staff, coordinators and Regional Advisory Group chairmen for review and comment. 14. Modification, finalization and approval of redesigned grant application and data to be generated by management information system by National Advisory Council , RMP Review Committee, Steering Committee of Coordinators and Department of Health, Education and Welfare. 15. Implementation of redesigned Review Criteria, grant application and management information system according to procedures developed by RMPS management. This sequence of activities should be able to be completed fn less than six months and should result in an acceptable set of Review Criterta. The need for further revisions should be detected by the recommended activities of the Office for Planning and Evaluation and such revisions should be made on an annual basis. Conclusions The seventeen recommendations that flow from this evaluation of the RMP Revtew Criteria would result in the resolution of most of the criticisms and adverse evaluatton-findings. They call for significant changes tn the content, design and application of the Review Criteria changes that would influence the review process as well as many of the technical assistance, information and evaluation-.activities of RMPS. The ultimate objective of these recomendattons is to facilitate the achievement of Nattonal objectives and the resolution of regional needs for improvements in the health care deli'very system. 85 APPENDIX A RMP REVIEW CRITERIA 87. December 28, 1971 RMP REVIEW CRITERIA A. PTRFORMANCE (40) 1. GOALS. OBJECTIVES, AND PRIORITIES (8) been developed and explicitly stated? a. Have these b. Are they understood and accepted by the health providers and institutions of th e Region? C. Where appropriate, were community and consumer groups also consulted in their formulation? d. Have they generally been followed in the funding of opera- tional activities? e. Do they reflect short-tem, specific objectives and priori- ties as well as long-range goals? f. Do they reflect regional needs and problems and realistically take into account available resources? 2. ACC S AND IMPLEMENTATIONS (15) a. Have core activities resulted in substantive program ac- complishments and stimulated worthwhile activities? b. Have successful activities been replicated and extended throughout the Region? C. Have any original and unique ideas, programs or tethniques- been generated? d. Have activities led to a wider application of hew knowledge and techniques? e. Have they had any demonstrable effect on moderating cotts? f. Have they resulted in any material increase in the availa- bility and accessibility of care through better utilization of manpower and the like? g. Have they significantly improved the quality of care? h. Are other health groups aware of and using.the data, expertise, etc., available through RMP? i. Do physicians and other provider groups and institutions look to RMP for technical and professional assistance, consultation and information? If so, does or@will such assistance be concerned with quality of care standards, peer review mechanisms, and the like? 3. CONTINUED SUPPORT (1,O) a. Is there a policy, activOly pursued,' aimed at developing other sources of funding for successful RMP activities? b. Have successful activities in fact been continued within the regular health care financing system after the withdrawal of RMP support? 89. 4. MINORITY INTERESTS (7) a. Do the qoals, objectives, and priorities specifically deal with improving health care delivery for underserved minorities? b. How have the RMP activities contributed to significantly increasing the accessibility of primary health care services to underserved minorities in urban and rural areas? c. How have the RMP activities significantly improved the quality of primary and specialized health services delivered to minority populations; and, have these services been developed with appropriate linkages and referrals among in- patient, out-patient, extended care, and home health services? d. Have any RMP-supported activities resulted in attracting and training members of minority groups in health occupations? is this area included in next year's activities? e. What steps have been taken by the RMP to assure that minority patients and professionals have equal access to RMP supported activities? f. Are minority providers and consumers adequately represented on the Regional Advisory Group and corollary committee structure; and do they actively participate in the deliber- ations? 9. Does the core staff include minority professional and supportive employees and does it reflect an adequate consideration-of Equal Employment Opportunity? h. Do organizations, community groups, and institutions which deal primarily with improving health services for minority populations work closely with the RMP core staff? Do they. actively participate in RMP activities? ii What surveys and studies have been done to assess the health needs, problems, and utilization of services of minority groups? B. PROCESS (35) 1. COORDINATOR (10) a. Has the coordinator provided strong leadership? b. Has he developed program direction and cohesion and established an effectively functioning core staff? c. Does he relate and work well with the RAG? d. Does he have an effective deputy in name or fact? 2. CORE STAFF (3) .a. Does core staff reflect a broad range of professional and discipline competence and possess adequate administrative and management capability? 90. 2. CORE STAFF (3) (continued) b. Are most core staff essentially full-time? C. Is there an adequate central7 core staff (as opposed to tional components)? ihstitu 3. REGIONAL ADVISORY GROUP (5) a. Are all key health interests, institutions, and groups within the region adequately represented on the RAG (and corollary planning committed structure)7 or 4 times annually? b. Does the RAG meet as a whole at least 3 c. Are meetings well attended? sented on,.,the RAG and corollary d. Are consumers adequately repre el participate in the committee structure? Do they activ Y deliberations? e. Is the RAG playing an active role in setting program Policies, and prioritie , and prov'd4ng overall establishing objectives s guidance and direction of core staff activities? f. Does the RAG have an executive committee to provide more frequent administrative program guidance to.the coordinator and core staff? g. Is that committee also fairly representative? 4. GRANTEE ORGANIZATIOR (2) a. Does the grantee organization provide adequate administrative and other support to the RMP? b. Does it permit sufficient freedom and flexibility, especially insofar as the RAG's policy-making role is concerned? 5. PARTICIPATION (3) a. Are'the key health interests, institutions, and groups actively participating in the program. b. Does it appear to have been captured or co-opted by a major interest? C. Is the Region's political and economic power complex involved? 6. LOCAL PLANNING (3) a. Has RMP in conjunction with CHP helped develop effective local planning groups? b. Is there early involvement of these local planning groups in the development of program proposals? c. Are there adequate mechanisms.for obtaining substantive CHP review and coment? 91. CES (3) 7. ASSESSMENT OF NEEDS AND RESOUR a. Is there a systematic continuing identification of needs, problems, and resources? b. Does this involve an assessment and analysis based on data? c. Are identified needs and problems bci g translated into the Region's evolving plans and priorities? d. Are they also reflected in the scope and nature of its emerging core and operational activities? 8. MANAGEMENT (3) a. Are core activities Well coordinated? b. Is there regular, systematic and adequate monitoring of projects, contracts, and other activities by specifically assigned core staff? c. Are periodic progress and financial reports required? 9. EVALUATION (3) a. Is there a full-time evaluation director and staff? bi Does evaluation consist of more than mere progress reporting? c. Is there feedback on progress and evaluation results to program decisions and modifications; specifically have unsuccessful or ineffective activities been promptly phased out? C. PROGRAM PROPOSAL (25) 1. ACTION PLAN (5)' a. Have priorities been established? b. Are they congruent with national goals and objectives, including strengthening of services to underserved areas? c. Do-the activities proposed by the Region relate to its stated priorities, objectives and needs? d. Are the plan and the proposed activities realistic in view of Iresources available and Region's past performance? e. Can the intended results be quantified to any significant degree? f. Have methods-for reporting accomplishments and assessing results been proposed? g. Are priorities periodically reviewed and updated? 2. DISSEMINATION OF KNOWLEDGE (2) a. Have provider g.roups or institutions that will benefit been targeted? 92. 2. DISSEMINATION OF KNOWLEDGE (2) (continued) b. Have the knowledge, skills, and techniques to be disseminated been identified; are they ready for widespread implementation? c. Are the health education and research institutions of the Region actively involved? d. Is better care to more people likely to result? e. Are they likely to moderate the costs of care? f. Are they directed to widely applicable and currently practical techniques rather than care or rare conditions of highly specialized, low volume services? 3. UTILIZATION MANPOWER AND FACILITIES (4) a. Will existing community health facilities be more fully or effectively utilized? b. It it likely productivity of physicians and other health manpower will be increased? c. Is utilization of allied health personnel, either new kinds or combinations-of existing kinds, anticipated? Is this an identified priority area; if so, is it proportion- ately reflected in this aspect of their overall program? e. Will presently underserved areas or populations benefit significantly as a result? 4. IMPROVEMENT OF CA RE (4) a. Have RMP or other studies (1) indicated the extent to whic ambulatory care might be expanded or (2) identified problem areas (e.g., geographic, institutional) in this regard? b. Will current or proposed activities expand it? c. Are comunications, transportation services and the like being exploited so that diagnosis and treatment on an out- patient basis is possible? d. Have problems of.access to care and continuity of care been identified by RMP or others? e. Wil,l current or proposed activities strengthen primary care and relationships between specialized and primary care? f. Will they lead to improved access to primary care and health services for persons residing in areas presently underserved? g. Are health maintenance and disease prevention components included in current or proposed activities? h. If so; are they realittic in view of present knowledge'. state-or-the-art,.and other factors?. 5. SHORT-TERM PAYOFF (3) a. Is it reasonable to expect that the operational activities proposed will increase the availability of and access to services, enhance the quality of care and/or moderate its cotts, within the next 2-3 years. 93. 5. SHORT-TERM PAYOFF (3) (continued) b. Is the feedback needed to document actual or prospective pay"offs provided? upport can be withdrawn c. Is it reasonable to expect that RMP s successfully within 3 years? 6. REGIONALIZATION (4) a. Are the plan and activities proposed aimed at assisting multiple provider groups and institutions (as opposed to groups or institutionssingly)? b. Is great.er sharing of facilities, manpower and other resources envisaged? c. Will existing resources and services that are especially scarce and/or expensive, be extended and made available to a larger area and population than presently? d. Will new linkages be established (or existing ones strengthened) among health providers and institutions? e. Is the concept of progressive patient care (e.g., OP clinics, s, home health services) reflected? hospitals, ECF' 7. OTHER FUNDING (3) a. is there evidence the Region has or will attract funds other -than RMP? b. If not, has it attempted to do so? c. Will other fundsl.(private, local' state, or Federal) be available for the activities proposed? d. Conversely, will the activities contribute financially or otherwise to other significant Fedorally-funded or locally- supported health programs? 94. APPENDIX B DESCRIPTION AND SCOPE OF WORK FOR CONTRACT HSM 110-72-329 TO EVALUATE THE RMP REVIEW CRITERIA 95. ARTICLE I DESCRIPTION AND SCOPE OF WORK A. The purposes of this study are (1) to assess the internal validity (e.g., repeatability, descrimination) of the Review Criteria used in rating Regional Medical Programs; and (2) the perceptions of those utilizing the criteria in the national review process as to their effectiveness. B. In performance of this contract, the Contractor shall: 1. Assess the manner in which the established Review Criteria are, in their application, sensitive tools in the adequately and objectively assessing the effectiveness of Regional Medical Programs for the purposes of Regional Medical Program Service review at the national level by both the Regional Medical Pro- gram Review Committee and the Staff Anniversary Review Panel. This assessment should include, but not necessarily be limited to answering the following questions: (a) Are the average raw scores of "A," 'IB," and 'IC" Regions significantly different from each other? ("A" Reg ons are the most highly re@arded while "C" Regions are the least highly regarded. (b) Are the average weighted scores of "A," "B." and "C" Regions significantly different from each other? ("A" Re gions are the most highly regarded while "C" Regions are the least highly regarded.) (c) Is there a consensus among reviewers regarding "A, BD and 'IC" regions? (Use of standard statistical methods such as analysis of variance to test pertinent hypotheses.) (d) Are there individual reviewers wh ose scores reflect an unusual range? (e) Are there particular primary reviewers who tend to "pull" the scores of the other reviewers towards a model value? (f) What are-the answers to questions (a) - (e) considering each broad grouping of criteria (ie., performance, process, and program proposal) as well as the overall or total criteria? (g) What is the relationship between the scores for each of these broad groupings of criteria and the overall.scores? (h') How do the weighted mean scores based Upon the individual criteria compare with the-single overall assessment scores? (i) What is the relationship between site visit findings as reflected in site visit reports (and summarized at the meeting) and the scores for selected criteria (e.g., coordi- nator, 'regionalization)? *Question not addressed by contract activities by agreement .with Project officer 97. (j) What is the relationship between scores and the percent- ages of requests recommended for award? 2. Determine the adequacy and effectiveness of the Review Criteria in assessing the comparative quality of Regional Medical Programs as perceived by those utilizing them (e.g.i Review Committee members, Regional Medical Programs staff). The information ob- tained should be aimed at, but not necessarily limited to answer- n i g the following questions: (a) What is the relative importance of each of the major criteria groupings? (b) Which individual criteria are the most important and credible,--that is, should be given the greatest weight in assessing the comparative quality of regions? (c) For those criteria singled out as most important is the data and information made available generall adequate to y make judgments? If not, what kinds of data would be desir- able and/or required? (d) What additional criteria, if any, should be included; which if any of present ones might be eliminated? f @71 (e) Of those suggested for deletion, what is the basis there- fore Unimportant, not too relevant? LF (2) Difficult to judge, intrinisically or because of the unavailability of data? (f) Have the review criteria and their utilization resulted in a significant mprovement, either substantially or in terms of credibility, in the decision-making process? kg) In the opinion of a sample of Regional Medical Program coordi nators what.is their assessment of the Review Criteria as an aid in the communication to reviewers of the characteristics and quality of Regional Medical Programs? 3i Prepare a comprehensive report regarding the Review Criteria that would include but not be limited to: (a) Findings based upon the analyses and opinion surveying undertaken. (b) Recommendations and suggestions (and basis therefore) for modifying the criteria, their application in the present review process, an t at overall review process itself. *Question not addressed by contract activities by'.agreement with Project officer 98.