*EOOI August 3, 1971 @TS REVIEW CRITERIA AND RATING SYSTEM INTRODUCTION Several important factors have contributed to the need for and development of an @S rating @ystem. Foremost among these is the recognition that the PIVS program is a mature, complex national activity whose processes are deserving of and accorded scrutiny by the public at large, Congress, and others. Additional factors include the need to assess the degree to which Regional Programs' strength and activities are consonant with evolving national priorities as reflected in the Council-endorsed Mission Statement and the growing gap in recent years between the grant funds actually made available to the program and the dollar level of Council approvals. Irrespective of fluctuations in the levels of funds available, it is important that the Review Committee and Council continue to base their assessments and recommendations on the overall merit of individual programs and to leave to fhe Director, @S, the responsibility for implementing the judgments of the Council. To assist him in discharging this responsibility, and with the encouragement of HSMHA, a rating,system has been designed and tested. Subsequent to the May Council meetingr an RMPS staff committee .was formed to develop and apply on a trial basis a rating system for the applications currently under review. The Committeels approach was to develop a further elabor ation of the 17 criteria included in the @S Mission Statement , a weighting scale for the criteria, a 1-5 point evaluation scheme, and associated forms and instructions-for use by the RMPS Review Committee in applying the criteria. The materials developed were revie@7ed by the Administrator, HSMHA. The purpose of this report is to summarize@the reaction of the Review Committee to their initial experience in using the criteria and rating system, and'to present an analysis of the results. THE REVIEW COMMITTEE'S REACTIONS The Review Committee met in executive session after completing the review of 13 triennial.applications in the current cycle for the purpose of providing feedback on the'rating system. In general, Committee members were quite favorable in their reaction to the criteria themselves. They apparently felt that these,comprehensively covered the relevant and salient points that need to be taken into account in assessing and ranking RNP's.. They also appeared to feel that the sub-criteria that had been developed in the form of questions were useful in clarifying the broader criteria and helping them to score these. The principal drawback in applying the criteria was lack of specific information relating to some of the individual items. A number of suggestions relating to the arrangement of the material also were received. 3 ANALYSIS OF RESULTS 1. The criteria and ratings clearly discriminated between the best, average,and poorest regions. 2. The reviewers were tough graders. The top Region received a score of only 327 out of a possible 500. If, as expected, they become more lenient in the future, it will be necessary to apply weighted means to insure comparability from one review cycle to the next. 3. Reviewers were asked to circle. ratings about which they were uncertain due to insufficient evidence or for other reasons. Three criteria, Continuity of Cate, Prevention, and Ambulatory Care were the most troublesome. There was virtually no uncertainty about Goals and Objectives, Organizational Viability and Effectiveness. In general, criteria relating to organization, management.and objectives appeared to be least troublesome. F'OLLOW-UP Certain minor modifications have been made in the criteria, however,. as the result of Committee suggestions and staff analyses. Some interim steps also have been taken to provide additional information in selected areas (e.g., continued support, evaluation, other funding) so that the uncertainty of reviewers with respect to their criteria may be reduced in the future. CONCLUSION The trial use of the review criteria and'rating system by the Review Committee, their reactions to it, and the analysis of the e and workable basis results strongly suggest that it is an'effectiv for assessing the quality and performancelof Regional Medical Programs. INSTRUC,71ONS: Using a one through five scoring scale (5-outstanding, 4-good, 3-satisfactory, 2-Fair, 1-Poor) rate the Region in accordance with the criteria set forth under: I - Performance, II - Process, and III - Program. Subcriteria or elements in the form of questions have been included in order to make the broad, general criteria more specific and understandable. These are designed to be'of help to the reviewer in assigning a score to each of the criteria. Multiplication of scores by the assigned weights and the necessary addition will be done by staff; reviewers need not make those computations. Reviewers should provide their overall subjective assessment of the Region and its application by rating on a one to five basis in Item IV, OVERALL ASSESSIENT. Use a check (01) in Item V, DEVELOPWNTAI, COWONENT, if in. your best judgment this Region has achieved sufficient program maturity and status to warrant award of ,t developmental component. In Item VI, BASIS FOR EVALUATION, indicate for each Region the basis for your evaluation. When appropriate more than one item in IMM VI may be checked for each Region. AD(]III ISTPPTIVELY (OPTIO CG-IFIIEITIAL RATI NGS: @oLrrSTANDI,'IG 4-GOOD ?--FAI R @SATISFACTORY I-POOR @,i n m rn 0 m z 0 _q ig cn U) > rTi cn PD 2 - G) C/) 0 > C/) m z GIITERIA > 0 Date of @st Pecent Site Visit 5/nt/69 771 /07'1 IV71 L?/70ilO/70 6/71 Y71 CPI 10016/716/71112/70 1. PEPFOF,"'Y@!,ICE- 1. Goals, Objectives & Priorities 12 2. Accomplisln@nts &ImT,)]-e,@nentation 16 3. Continued Support 12 11, P'R'@OCESS 1. Org. Viability Effectiveness 2. Participation 5 3. Loml Planning 4 4. Assessjiient of Needs @ Resources 4 S. @,@iage,-nent & Evaluation 5 PPM. hill Pr lnlrwpL 1. Action Plan 5 2. Dissemination of Ynoiiledue 2 3. Util: @lanpoi@er & Facilities 5 4. Prevention 2 S. Ambulatory Care 2 6. Continuity of Care 2 7. Short-Terni Payoff 5 8. Regionalizatioii 4 9. Other Funding 3 IV. O,RLPPJJ- PSSE-SSP V. -@tLop" VI, P.4S!S FO 1. Current Site Visit 2. Previous Site Visit 3. Application 4. Coiiniittee Discussion S. Other 6. Primary Revi.c;.,,c-r Fk7PS TO BE CJISIDM @,'Illl TtiE POIEN CPITEPJA Ca IORITIES h. Are unsuccessful or irrelevant.activities being phased out? a. Have these been developed & i. Are other heaith.groups aware of and using the explicitly stated? data, expertise, etc. available through @IP? b. Are they understood and accepted by j. Do physicians and other provider groups and institu- the health providers & institutions tions look to M for technical and professional of the Region? assistance, consultation and information c. Where appropriate, were comtmity and consumer groups also consulted in their k. If so, does or will such assistance be concerned with formulation? quality of care standards, peer review mechanisms, d. Have they generally been followed in and the like? the funding @' operational activities? e. Do they reflect short-term, specific CWTINLF-D SUPPORT objectives and priorities as well as long-range goals! a. Is there a policy, actively pursued, aimed at f. Do they reflect regional needs and developing other sources of fmdingfor successful problems and realistically take into RMP activities? . ' account available resources' b. Have successful activities in fact been continued within the regular health care financing system 2. ACCOWLIS@S PND I@PLEWW TION after the withdrawal of W support? a. Have core activities resulted in substantive pro am accomplishments and stimulated gr wort@,hile activities? b. Have successful KNIP activities been replicated and extended throughout the region? c. Have any original and unique ideas, programs or techniques been generated? d. Have activities led to a wider application of new knowledge and techniques? e. Have they had any demnstrable effect on erating costs? f. Have they resulted in aV material increase in the availability and accessibility of care through better utilization of manpower and the like? DRAFR 6/29/71 - AMNIGMTIW USE CNLY g. Have they significantly improved the quality of care? FACTOPS TO BE OD(SIDM I-'IT]i TIE PEVIEW CPITEPIP, 3. @IIIZA.TIC91AL VIA131LITY Al a. lias @IP help develop in conjunction with ChP effective local planning groups? a. Is the coordinator effective; has he provided strong b. Is,there early involvement of these local planning groups leadership, developed program direction and cohesion in the development of program proposals. and established an effectively functioning core staff c. Are there adequate mechanisms for obtaining substantive with due re,@ard to equal employment opportunities and CHP review-and cement? udnority group interest. b. Does core staff reflect a broad range of professional 4- ASSESGENT OF NEEDS AND FESO@S and discipline co:,.petence and possess adequate a&dnistrative and management capability? a. Is there a systematic, continuing identification of needs, c. Are rost core staff essentially full-time? problems, and resources? d. Is there an adequate central core staff (as b. Does this involve aA assessment and analysis based on data? opposed to institutional components)? c. Are identified needs and problems being translated into the e. Does tne grantee organization (1) provide adequate region's evolving plans and priorities? administrative and other support to the R4P and d. Are they also reflected in the scope and nature of its (2) permit it.sufficient freedom and flexibility, emerging core and operational activities? especially insofar as the RAG's policy-making role is concerned? S. I-AMGEMENT AND EVALUATION f. Are all key health interests, institutions, and groups within the region adequately represented a. Is there regular, systematic and a'@eq@te of on the P,@G (and corollary plarming committee projects, contracts, and other activities by specifically structure)? assigned core staff? g. Does the RAG meet as a whole at least 3 or 4times b. Are periodic progress and financial reports required? annually? c. Is there a full-time evaluation director and staff? h. kre meetings well attended? d. Does evaluation consist of more than T-,ere progress reporting? j. Are minority groups a-.4.consuners adequately e.. Is tielce feedback on progress and evaluation re-,- ts to represented an the RAG and corollary committee program management, RAG, and other appropriate ;7.,;S? structure, and do they actively participate in f. Have negative or unsatisfactory results been converted into the deliberations? program decisions and modifications; specifically h-ave j. Is the P,-%G playing an active role in setting program unsuccessful or ineffective activities been promptly phased policies and establishing objectives and priorities? out? k. Does the RAG have an executive or steerin committee .9 to provide more frequent administrative program guidance to-the coordinator? 1. is that conuttee also fairly representative? 2. PARricipATiou a. Are the key health interests, institutions, and DRAFR 6/29/71 - ADNUNISTRATIVE USE ONLY groups actively participating in the program b. Does it appear to have been captured or-co-opted by a major interest? c. Is the region's political and economic power complex involved? FACTOP,S TO BE CrtSIDEPS VIITII 11E FEVIEN CRITEPIA PPDGPAtl PMN X71 ON PLM a. Have priorities been established? a. Have R,@ or other studies (1) indicated the extent to b. Are they congruent with national goals and objectives? which ambulatory care might be expanded or (2) identified c. Do the activities propose.d by the region relate to its problem areas (e.g., geographic institutional) in this stated priorities, objectives and needs? regard?, d. Are the plan and the proposed activities realistic in b. Will current or proposed activities expand it? view of resources available & Region's past performnce? c. Are communications , transportation services and the like e. Can the intended results be quantifi to any being exploited so that diagnosis and treatment on an significant degree? outpatient basis is possible? f. Have methods for reporting accomplishments and assessing results been proposed? 6. COWINUITY OF CARE g. Are priorities periodically reviewed and updated? a. Have problems of access to care and continuity of care 2. pISSF identified by,@IP or others? _IMTNATICV been 1 b. Will current or proposed activities strengthen primary a. Haveprcrvider groups or institutions that will care and relationships between specialized and primary benefit been targeted? care? b. Have the knowledge, skills, and techniques to be C. Will they lead to improved access to primary care and diss@ated been identified; are t@iey-ready for -health services forpersons residing in areas presently widespread implementation? underserved? c. Are the health education and research institutions of the Region actively ijvTolvc-d? 7. S@FC-T-rg,', P@,Y'-EE d. Is better care to more people likely to result? e. Are they likely to moderate the costs of care? a. Is it reasonable to expect that the operational activities f. Are they directed to widely applicable and currently proposed will increase the availability of and access to practical techniques rather than care of rare services, enhance the quality of care and/or moderate its conditions or highly specialized, low volume services? costs, within the next 2-3 years? b. Is the feedback needed to document actual or prospective 3. UTILIZATION MMpoqEg NM FACILI= .pay-offs.provided? c. Is it reasonable to expect that W support can be withdrawn a. Have areas or populations been specifically targeted? successfully within 3 years? b. Will presently underserved areas or populations benefit significantly as a result? 8. REGIONAL-IZATION c. Will existing community health facilities be more fully or effectively utilized? a. Are the plan and activities prolosed aimed at assisting a. Is i't likely productivity of physicians and other multiple -,rc,,-ieer groups and institutions (as opposed to health manpower will be increased? groups or institutions singly)? e. Is utilization of allied health personnel, either new b. Is greater sharing of facilities, manpower and other kinds or combinations of existing kinds, anticipated? resources envisaged? f. Is this an identified priority area; if so, is it c. Will existing resources and services that are especially proportionately reflected in this aspect of their scarce and/or expensive, be e)ctended and made available overall program? to a larger area and population than presently? d. Will new linkages be established (or existing ones strengthened) %ENTION among health providers and institutions? e. Is the concept of progressive patient care (e.g., OP clinics, a. Rave specific diseases, areas, or populations been hospitals, ECF'S, home health services), reflected? targeted? b. Are health maintenance and disease prevention components 9. NC, included in current or proposed activities? c. If so, are they realistic in view of present knowledge, a. Is there evidence the region has or will attract funds other than @IP? state-of-the-art, and other factors? b. If not, has it attempted to do so? d. Is early detection included? C. Will other funds (private, local, state, other Federal) be e. If so, has adequate provision been made for follow- available for the activities proposed? through treatment? d. Conversely, will the activities contribute firancially or otherwise f. Is health related education of the public or patients to other significant Federally-funded or locally-supported health included, where appropriate? progranm? DMn 6/29/71 AMNISTPATIVE USE ONLY