B&B ImFoRmikreom & 300 @omoz M=Rncon ROULZVAAD Ulnonicn p4AIt 040 Z0772 0 UMA 0 taol) 249001 10 10/29/70 RMP REVIEW CRITERIA AND PRIORITY MECHANISM The FAST Report on Regional Medical Programs Service recommended that the "anniversary or triennial review system" previously proposed by RNPS "be adopted . . . 4so as to focus on.7 program review at the Federal level (Underscoring supplied.) The specific implementa- tion action required by it was spelled out in Dr. Wilson's response in terms of "Criteria for evaluating programs and a priority mechanism, i.e., what staff and Council need to evaluate in recommending a general level of support What has been developed in preliminary and draft form are: (1) A set of proposed criteria to be employed in assessing overall program progress and prospects of individual regions in connection with the review and approval of grant applications, especially requests for developmental components. (2) A suggested priority system for comparatively ranking programs for what is similar to accreditation. What has not been worked out, however, is an allocation method. Criteria The proposed review criteria set forth in Attachment A are both:, Retrospective - aimed at assessing the region's progress and current status; their performance'to date. Prospective - aimed at assessing the overall programmatic merit of what is proposed including operational projects and core -2- activities as w@ll as a, d2vc-lol-@iii@nE--,l time potciit--al for desirable achievement - as well as the Re,,ion's plans to build in previous experiences. The criteria are a synthesis of (1) general criteria specifically suggested (and employed) by several of the Review Committee members at their recent meeting; (2) review criteria developed by an RMPS staff group several months ago and recommended to the Director; and (3) "the seven essential elements by which regions might be judged set forth in the "RMP Guidelines" over two years ago. rit System The priority system or mechanism briefly outlined in Attachment B is a rather traditional and common model. It is, however, a priority or ranking system utilized by some regions (e.g., Michigan, Northeast Ohio) and discussed by Dr. Besson at the last Review Committee meeting. The simplicity which has been sought may serve as a reminder that quantification of essentially qualitative characteristics and subjective judgements does not elevate the process to "science." On the other hand, there does seem to be considerable opinion that not only does the review process need to become somewhat more systematic and disciplined (e.g., criteria), but some way must be found that permits comparison between regions (e.g., ranking system). Attachments Attachment A REVIElq CRITERIA These criteria can be utilized for both retrospective and prospective assessments. Retrospectively-, they can be used to measure progress, accomplishments, effectiveness, and impact. Prospectively, they can be utilized to assess new directions and goals, exploitation of past experiences, innovative approaches, and methods to deal with problems and barriers, among others. 1. Goals, Objectives and Prioritieg explicitly stated reasonable and relevant based on assessment of needs, problems, and resources congruent with national priorities ..understood, disseminated, and accepted 2. Organizational Effectiveness a. Coordinator b. Core staff * administrative and management capability * professional and discipline competence * relationship of central - institutional field components c. Regional Advisory Group, including committee structure . participation of key groups and interests . policy control over program . credibility ' . adequate technical review of project proposals d. Subregionalization geographic and functional 3. Regional Resources and Involvement . community hospitals, hospital boards . nursing . allied health personnel groups . medical school and center . practicing physicians and organized medicine . voluntary and official health agencies . Comprehensive Health Planning, Model Cities 0 consumers . tie in with political and economic power structure 2 4. Assessment of Needs, Problems and Resources identification and analysis based on data relationship to objectives and priorities 5. Implementation a. Core activities action-oriented planning development of community organization and planning capability at local level; subregionalization coordination, cooperation, and conjoint activities with others b. Operational projects * contribution.to present objectives and priorities * strengthening and utilization of linkages; regionalization * quality and results generally * turnover of funds * phasing out unsuccessful projects and irrelevant activities c. Developmental component 6. Evaluation staff and other resources program assessment as well as project evaluation relationship to goals 7. Starting Conditions: Consideration would be given in applying the above criteria to the "starting conditions" within regions so that program progress and current status would be viewed in both relative and absolute terms. 8. Requirements: Certain minimum standards would have to be met with respect to the following before a region could qualify for a developmental component. . fiscal and management capability and policies so as to in- sure adequate control of, and accountability for funds expended by affiliated and other institutions carrying out core, operational, or developmental activities. . regional review process and structure to insure the adequacy of technical,-scientific peer review at the regional level. Attachment B PRIORITY SYSTEM The minimum essential elements of an operational priority system (or mechanism), are (1) a scoring scale, (2) an agent such as the Review .Committee to apply it, and (3) the resultant ranking or grouping of programs. This, of course, is in addition to (4) the review 'Criteria (or priorities) themselves against which individual prog rams would be assessed. Other elements such as (5) the weighting of each individual criterion, may also be involved. One priority system, using whatever criteria might be agreed to, is outlined below in terms of,each of these elements. Scoring Scale: A five-point scale to permit some shading of 'Iquality." Specifically: 5 - Excellent, outstanding 4 - Good, superior 3 - Average, fair 2 - Poor, weak 1 - Unsatisfactory gent: Whether or not a number of sub-agents apply the criteria and in effect grade programs for example, staff, site visit team, prin- cipal and secondary reviewers the final scoring should be an action of the Review Committee as a whole. This does not imply that each member would necessarily complete a scoring sheet or that assessments of others might not be presented to the Committee. 2 Ranking: Although a numerical scoring scale, if applied, would permit the sequential ranking of regions (1 through 55), it is unlikely that any good purpose would be served by this. Moreover, to do so would suggest greater accuracy in the judgments and precision in the scoring than warranted. It might be better just to group regions according to quartiles. For example: 4.25 and above . . . .Upper range 1.76 4.24 . . ... . .Middle range 1.75 and below . . . .Lower range. Criteria: Whatever criteria are employed, it might be desirable to include or make allowance for the "undefineable" or "immeasurable.tf For example, provision might be made for adding, up to a specified maximum (5), to the overall score if it was felt that the criteria and scoring somehow failed to adequately and/or fully reflect the quality of the regional program. Weighting: This provides a way for giving greater influence to certain criteria which are regarded as more significant or important than others. For example: Goals,, Objectives and Priorities 1 Organizational Effectiveness 3 Regional Resources and Involvement 1 Assessment of Xeeds 1 Implementation and Accomplishments 2 Evaluation 1 Subtotal 9 Proposed Activities 2 Developmental Component 1 Total 12 3 Either some subdivision of criteria or weighting probably is called for. It should be kept quite simple, as should all aspects of any priority system initially. Operating experience may suggest the need for certain changes; any embellishments might also be postponed. 10/29/30 REGIONAL REVIEW STAI\TDARDS AND REQUIREMENTS The,FAST Report on Regional Medical Programs recommended that "RMPS-HSMHA phase out project-by-project technical review (and) develop criteria to permit each Regional Medical Program to seek quali- fication as its own project review agency." Dr. Wilson in his response stated that "Implementing standards for local RMP review process" would be established. RMPS staff currently is surveying the review process and structure in about.one-third of the regions. This survey should be completed and the information from it be available by November 10. The information will provide a rather specific indication of the characteristics of regional review, which in turn will be helpful in establishing review standards and guidelines. It may, on the one hand, indicate review characteristics in certain regions that are especially noteworthy; characteristics that should be translated into minimum standards appli- cable to all regions. On the other hand, it may reveal that universal standards are currently unrealistic for many regions. Some general observations and comments about the standards to be established and the decentralization of the technical review of projects to the 55 regions may be in order. (1) It is not anticipated that all regions initially will be able to meet the minimum standards to be established. Conversely, it is expected that all should meet them within a reasonable period. 2 (2) To be eligible to receive a developmental component, a region's review process would have to meet the minimum standards. ' (3) The standards to be established for project review must include technical, scientific peer review. (4) Thus, the Regional Advisory Group, the Regional Medical Programs Service staff, Review Committee, and Council would continue to review project proposals from a programmatic standpoint--for example, their relationship one to another and to stated regional needs, priorities, and goals as an integral part of evaluating the quality of the overall program. The following are among the kinds of minimum standards and/or require- ments under active consideration already. (1) The presence of an actively functional technical review structure, which drawls upon expertise from outside the Region when appropriate. (2) The inclusion on those committees and groups of the breadth and depth of competence and expertise appropriate to the subject area of their review responsibilities. (3) Explicit criteria and/or priorities against which proposals are reviewed and assessed; and the general availability of such to all prospective applicants. p (4) Formal provision for the referral of applications to the appropriate CHP agencies for review and comment; and how such comments are taken into account in the regional review process. 3 (5) A functioning system that takes into account regional priorities and the technical or substantive merit of proposals in deter- mining which are to be funded when approvals exceed monies available to the region. (6) Availability to RMPS and the national review.process of appropriate summary reports, meeting minutes, etc, reflecting review deliberations and actions of regional review committees and advisory groups. Decentralization of the technical review of projects to a region is an established procedure. However, the effectiveness of the local review process becomes one of the multiple measures to be used when examining a Region's request for a developmental component. The.capability of a Region to achieve and maintain autonomy and self-sufficiency can, ...in part, be examined in terms of the quality of its local review mechanism.