I a i v B&B INFORmikirlom & IMAGE MAkm^13EmE@ 300 @IVSCC M=M=C"O BOULXVAMO Upfaca K.%RLUOgto, P4^ &AD Z077Z & u a^ 0 Cao 1) 24@ 1 1 0 OF EV-kLU)I'!L"-', Cil@FEEEICE iill-D A National @fP Conference and I,!orksliop on Evaluation was held September 28 - 30 at the University of Chicago Centcz for Continuing Education. Attendance Over 250 persons attended. Of these about 160 were lUIP Staff (coordina- tors, evaluation directors, and others); another 70 or so were participants and special i-nvitees; with the remainder largely accounted for bv P@IPS staff. Several National Advisory Council (Mrs. Wyckoff and Dr. Cannon) and Review Committee.members also were present. Background and Objectives of Conference The Conference was planned by an ad hoc steering committee of PIT evaluation directors and coordin t@o s@in conjunction with the MIPS Office of Program Planning and Evaluation. A survey of regional problems, needs, and expertise in evaluation resulted in the committee's decision that the primary objectives of the conference should be to: Present from various perspectives the differing needs for, and uses of, @IP evaluative material. Re-emphasize (or introduce) the concept of program evaluation. Present philosophies and methodologies of evaluation in broad programmatic areas. Present methodologies of evaluation in narrower subject and project-oriented areas. Provide both formal and informal opportunity for consultation with evaluation experts. Provide @ representatives concerned with evaluation an opportunity to become familiar with other RMP personnel in similar positions and with RMPS and outside evaluation resources. Discuss with participants th@ need and means for further formal mechanisms to improve RMP evaluation. Program A copy of the program, listing all speakers, is enclosed. A copy of the paper presented by Dr. Schon in the Monday plenary session is enclosed elsewhere in your agenda book. Enclosed herewith are copies of the remarks of Dr. Peter Fox (Office of Management and Budget) and Mr. John Blamphin (MEDICAL l@TORLD NEWS). Page 2 Pi:ocLcd4-r@-s of tl:e conference, -'-ncludin.,4 tl@e re@.aris o@' I)r,,7. 11 and t'7ilbur at tlic- ficnar\' D,-. iu-.-iclicun remarks and those of Dr. l@ilson on I%lidnesday, as well as the papers presented in the workshop and special interest sessions on Tuesday, will be available in about six to eight weeks. Copies will be sent to Council members then. PROGRAM NATICNAL REGIONAL MEDICAL PROGRAM CONFERENCE AND WORKSHOP ON EVALLIATION The University of Chicago Center for Continuing Education 1307 East 60th Street Chicago, Illinois 60637 SUNDAY September 27 6 10 p.m . . . . . . . . .REGISTRATION MONDAY - September 28 1:30 - 9 a.m . . . . . . . REGISTRATION' Lol)b@ 9 a.m . . . . . . . . . . PLENARY SESSION Assembly Room Presiding John Glas @w, Ph.D. Associate L;oordinator, Research and Evaluation Comecticut Regional Medical Program Welc(xne Morton C. Creditor, M.D. Prograin Coordinator Illinois Regional Medical Program Introduction and Purpose Roland L. Peterson Assistant Dl'r@ctor tor Program Planning and Evaluation Regional Medical Prograins. Service "Program Evaluation and Systems Transformation" Donald A. Schon, Ph.D. resi nt Organization for Social and Technological Innovatim 10:20 a.m . . . . . . . . .COFFEE BREAK 10:45 a.m . . . . . . . . .PLENARY SESSION (cmtinued) "Haw Others see Regional Medical Programs and Evaluation" Alexander M. Schnidt, M.D. Dean, Abraham Lincoln School of Medicine University of Illinois Member, National Re@iew Comnittee Regional Medical Programs Peter Fox, Ph.D. Senior E mami-st - Health Evaluation Office of Management and Budget Office of the President John Chase, M.D. Assistant @e Medical Director for Professional Services Veterans Administration MONDAY - September 28. (continued) Richard S. Wil-bur, @I.D. Assistant Executive Vice President American Medical Association John M. Bl!mhin Assistant Bureau Chief - Washington Office MEDICAL WORLD NEWS 12:30 p.m. LLJNCHECN Dining Room Speaker Harold ies, M.D. t@ Director Regional Medical Programs Service 2 p.m . . . . . . . . . . .DISCLISSION GRC)UP MATINGS As Assigned Eight separate assigned groups will be asked to react, discuss and critique the strategies and policies of evaluation against the background of morning presentations. (See assignments in registration packet.) 3:30 p.m . . . . . . . . . COFFEE BREAK 4 p.m . . . . . . . . . . .DISCLISSION GROUP MEETINGS (continued) 5:30 p.m . . . . . . . . . SOCIAL HOUR Second Floor Lounge TLIESDAY - September 29 8:30 a.m. - 3 p.m . . . . .WORKSHOPS See Appendix A Workshops in five specific areas will be offered in th-ree.separate sessions. (See Appendix A - WORKSHOP TOPICS AND ROCMS). These five work- shops will run concurrently, but attendance may be divided at the option of the participants. 8:30 a.m . . . . . . . . . SESSION I 10:15 a.m . . . . . . . . .OC)FFEE BREAK 10:30 a.m . . . . . . . . .SESSICN II 'RMSDAY - September 29 (continued) 12 noon . . . . . . LT]NCHFON - Not axranged 1:30 p.m . . . . . . . . .SESSICN III 3:15 p.m . . . . . . . . .COFFEE BREAK 3:30 p.m . . . . . . . . .SPECIAL INTEREST M=INGS See Appendix B Meetings covering 12 selected areas of special interest will be offered and run concurrently. (See Appendix B - SPECIAL INTEREST M=ING TOPICS AND RDCHS.) 5 p.m . . . . . . . . . . ADJO - sign up for consultation sessions in second Floor Lomge area WEDNESDAY September 30 9 a.m . . . . . . . . . . .PLENARY SESSION Assembly Roan Introduction Harold ies, M.D. Acting Director Regional Medical Programs Service An Instrment for Improvement In Health Services" Vernm E. Wilson, M.D. Administrator Health Serv-ices and Mental Health Administration 'IRMP Evaluation - Where Have We Been; and Where Do We Go Fran Here?" (Panelists to be announced.) 10:20 a.m . . . . . . . . .COFFEE BREAK 10:30 a.m . . . . . . . . .FOLIOW-THROLUH MEETING Room 2-EF Members of the Ad Hoc Steering Committee which planned the conference and selected @S staff and conference discussion leaders will met to discuss plans for follow-through. Others are welcome to attend. WEDNESDAY - Septembcr 30 (continued) 10:30 a.m . . . . . . . . . .INDIVIDLJAL CONSULTATION SESSICNS Participants may request brief consultation sessions with conference presenters or any other participants. Scheduling may be done anytime Monday or Tuesday mtil 8:00 p.m. at the consultations desk in the Second Floor Lounge area. Requests may be made for specific consultants or by area of evaluation expertise, in which case an RMPS staff member will help in selecting an appropriate consultant. 12 noon . . . . . . . . . .ADJO Appendix l-,l WORKSHOP TOPICS AND Tuesday, September 29, 1970 8:30 a.m. - 3:15 p.m. 8:30 10:30 1:30 to to tc WORKSHOPS SESSIONS 10:15 12:00 3:15 A - DATA (Room 2-D) I. The Values and Limitations of National Data x II. Data for Ambulatory Care Planning x III. The Computer and Health Information System x B - @URING CHANGES IN BEHAVIOR (Rom 1-AB) I. Measuring Changes in Knowledge x II. Measu@ Changes in Attitude x III. Measuring Changes in Clinical Perfo ce x C - EVALUATION OF AMGING HEALTli STATUS (Rom 2-EF) 1. The Value of Health Status Measures x II. How to Measure Health Status x III. The Relation of Process and End Result Evaluation x D - PROGRAM EVALUATION (Assembly Room) I. Concerns in Program Evaluation x II. Design of Program Evaluation x III. Implementation of Prograrn Evaluation x E - RESOURCE ALiOCATION/ECONCMICS (Roan 2-BC) I. Social Scientists and W Evaluation: Role, Techniques and Usefulness x 11. Cost Benefit - Cost Effectiveness x III. Resource Allocation and Management x (See Appendix A-2 A-10 for details of each Workshop Session) Appendix A-2 Workshop A - DATA Room 2-D Relevance of data to @ planning and evaluation will be discussed. The Workshop will be carefully structured to provide for a free and informal exchange of ideas among @ personnel concerned with data, rather than to teach statistical and epidemiological techniques. Moderator: Arthur R. Jacobs, M.D., M.P.H. Director of Statistical and Evaluation Unit Rochester Regional Medical Program Associate Moderator: Sarah J. Peterson tiostat stician Session A-I The Values and Limitations of National 8:30 Data (U.S. Census Data and National to Health Survey Data) 10:15 Room 2-D Charles A Metzner, Ph.D. r@o essor of Medical Care Organization University of Michigan Session A-II Data for Ambulatory Care Planning 10:30 to J. William Gavett, Ph.D. 12:00 @sociate Professor, College of Room 2-D Management afid Department of Preventive Medicine and Community Health School of Medicine University of Rochester Session A-III The Computer and Health Information System 1:30 to David E. Reed, M.D. 3:15 @sistant Director of Evaluation Room 2-D Western Pe@ylvania Regional Medical Prog-Fam Katherine G. Bauer Research Ass6cia-te Harvard Center for Community Health and Medical Care John E. liennberg, M.D. @or inator, Northern New England Regional Medical Program A Regional Health Planning and Management Data Base Appendix A-3 Workshop B - MEASURING AMGES IN BEMVIOR Room I-.kB Moderator: John S. Ll@d, Ph.D. Assistant Coordinator, Evaluation Area V - California Regional Medical Program Session B-I Measuring Changes in Knowledge 8:30 to The discussion of techniques for measuring 10:15 changes in knowledge -- some well known and Room 1-AB some not so well known. Topics included will be paper-and-pencil measures as well as measures which have been developed for situations in which paper-and-pencil technqiues are inappropriate. William R. Crawford, Ed.D. Associate Evaluation Studies Section Center for Study of Medical Education University of Illinois Session B-Il Measuring Changes in Attitude 10:30 to Display of methods and materials - the how- 12:00 to-do-it of attitude assessments of medical care 'ders, including ways Room 1-AB prove attitude change can be objectively formulated, evaluated, and documented to show project and program impact. Opportmity for questims, problems, and needs in area of attitude evaluation. Mitchell Schorow Assistant oor tor for Educational Plaming_and Evaluation Intermountain Regional Medical Program Session B-III Measuring Changes in Clinical Performance 1:30 to Discussion of various direct and indirect 3:15 techniques of assessing clinical performance. Roan I-AB Advantages and disadvantages of each as well (continued) Appendix A-4 Session B-III (continued) as criteria for the selection of specific measurement instrments will be discussed, with special attention given to the use and construction of "patient management problems.'' Barbara J. Andrew, Ph.D. Assistant Protessor Medical Education Research Division of Research in Medical Education University of Southern California Appendix A-5 IVorkshop C - ME FVUUATION' OF CHANGING @lTi STATUI; Roan -7-EF Moderator: Robert R. Carpenter, M.D. erector, estem Pennsylvania Regional Medical Prograin Session C-1 The Value of Health Status Measures 8:30 to This Workshop will consider the measurement 10:15 of the "end result" of health care efforts Room 2-EF in te= of patient and population well being. The difficulty of such measurements and their value is to be discussed. Examples of methods required to make and interpret measures of health status will be considered. Sani Shapiro Director of Researcii and Statistics Health Insurance Planof Greater New York Maureen M. Henderson, M.D. Professor, Preventive Medic3-ne Departnent of Preventive Medicine and Rehabilitation University of Maryland Sessim C-II How to Measure Health Status 10:30 to Howard R. Kelmm, Ph.D. 12:00 Departnent of Preventive Medicine Rom 2-EF and Public Health New York Medical College Session C-Ill The Relatim of Process and End Result Evaluation 1:30 to Charles E. Lewis, M.D. 3:15 Professor and Head Rom 2-EF Health Adninistratim Division School of Public Health Appendi-x A-b Workshop D - PROGR41 EVA'-UATION Assembly Room This workship is planned for those who have an appreciatim of the need for program evaluation but who are not familiar with the ccamon methods or approaches. By participating in this Workshop, each persm should: (1) know the methods for detemining program evaluatim priorities, (2) know the cam- mon method for obtaining program evaluatim information, (3) be more cmfident of his ability to cmceptuali2,e program eval- uation design, and (4) be able to design with or without the help of consultants a program evaluation design for his own program. Moderator: Harold W. Keaimes, M.D. Coordinator for Evaluation Tri-State Regional Medical Program Session D-I Concerns in Program Evaluation A. Approaches to Program Evaluatim: Lecture 8:30 and Discussion. to 10:15 This lecture will introduce the workshop Assembly Room participants to: (1) The role of program evaluatim in program development and funding procu'rement, (2) The comon methods for obtaining and reporting program eval- uatim informatim and, (3) Method of as- sessing the utilizatim of program eval- uatim infonnatim. It will create the foundation for me of the workshop sessions. Harold W. Keairnes, M.D. Coordinator for Evaluatim Tri-State Regional Medical Program B. Questions to be Answered by Progrm Evaluation: Panel Discussion. Through th6 technique of panel discussion, a series of questions will be developed which reflect the progym evaluation needs of an illustrative MT. The process will clarify the method by which evaluators can begin the task of designing their program evaluation activities. Harold W. Keairnes, M.D. Coordinator for Evaluation Tri-State Regional Medical Program (continued) Appendix A-,- Session D-1 B (continued) Avery M. Colt Field sociate of Eastern Massachusetts Tri-State Regional Medical Program Claire G. Farrisey pecial oiects Coordinator Tri-State Regional Medical Program Osler L. Petersen, M.D. sociate erector for Data Collection Tri-State Regional Medical Program and Acting Chairnan Departnent of Preventive Medicine Harvard Medical School Dean J. Siebert, M.D. sociate oor ator for Dartmouth Medical School Tri-State Regional Medical Program Ruth B. Mott Research Associate for Data Collection Tri-State Regional Medical Program Session D-II Design of Prograrn Evaluation 10:30 to The Workshop will divide into three small 12:00 discussion groups. Each group will consist Assembly Rom of one or two staff members from the illus- trative RV, and three consultants with specific interest in, and special concern with, program evaluation. (Up to ten workshop participants and an unlimited number of observers. Advance -registration of participants is recamended.) Priorities for evaluation activities will be developed by discussions in each small group. The consultant will provide detailed infomation about methods-and techniques for obtaining the information that answers the high priority questions. Some emphasis will be placed upon the problems of program evaluation and planning that have been precipitated by the new Anniversary Review Guidelines. Appendi-x @-8 Session D-II (continued) Se@,en people li-sted above in Session I Plus James J. Dunl2p, Ph.D. Ar ur . Little, Inc. Pobert K. Ausm@,i, M. D. Deputy Director Florida Regional Medical Program Jack E. Thomson, Ed.D oo ator ot Evaluation California Regional Medical Program Robert Bec@, Ph.D. Fire-ctor ot Research Nassau-Suffolk Regional Medical Program William R. Thcmpsm D uty irector Washington/Alaska Regional Medical Progrmn Paul E. White, Ph.D. Division o e vioral Sciences Johns Hopkins School of Hygiene and Public Health Jeannette For@h, Eh.D. Project Achninistrator, Intormation Support System Tri-State Regional Medical Program Session D-III Implementation of Program Evaluation 1:30 to A. Making program evaluation useful. 3:15 Assembly Room The small groups will continue discussion by focusing on sme of the operation problems of program-evaluation: e.g., funding, reporting tu-rn-aromd-ti.me, special requirements, etc. B. The range of solutions in the small groups will join together to compare the results of two discussion sessions and to demonstrate the range of possible solutions to comon problems. All consultants from the first two sessions will participate. Appendix A-9 Workshop E - RESOURCE kUOCATION/ECONCMICS Room 2-BC Moderator: John Glasgow, Ph.D. Associate Coor inator, Research and Evaluation Connecticut Regional Medical Program Session E-I Social Scientists and @ Evaluation: 8:30 Role, Tec @ ques, and Usefulness to 10:15 Panel discussion to clarify purpose of Room 2-BC evaluation, the potential role of the social scientist in terms of available tools and their applicability, and devel ent of a OPM consensus cmcerning framework within which a socioecmomic analysis may be profitably applied. Panel Members A. A. Florin, M. D. Coordinator, New Jersey Regional Medical Progrmn Michael Zubcoff,.Ph.D. Head, Health EC onucs Section Temessee Mid-South Regional Medical Program James R. Jeffers, Ph.D. erector, Medical Ecmanics Research Center University of Iowa John E. Wennberg, M.D. oo 3.nato-r, Northem New England Regional Medical Program Cmrad Seipp, Ph.D. Associ t@e Protessor, Health Services Research Center University of North Carolina Sessim E-II Cost Benefit - Cost Effectiveness Studies 10:30 to Explanatim of the purpose, objectives and 12:00 methodology of such studies with particular Rom 2-BC attention to the values and defects of these studies when applied to health care issues. (cont@d) Appendix A-10 Session B-Il (continued) .Robert L. Be@, INI.D. ro essor and Chairman, Department of Preventive Medicine and Camunity Health University of Rochester Sessim E-III Resource Allocation and Management 1:30 to Consideration of the ecmoTnic allocation of 3:15 scarce resources within the contest of , Rocrn 2-BC alternate planning approaches of resource allocatim, particularly PPBS. Panel Members @e A. Kimel Program Analyst, uttice of the Assistant Secretary for Planning and Evaluation Departinent of Health, Education and Welfare Charles W. Caldwell, M.P.A. sociate ordmator, Iowa Regional Medical Program Charles L. Joiner, Ph.D. i)irector, j3ureau of Reg'earch and Commity Service Alabama Regional Medical Program Appendix B-1 SPECIAL INTEREST MEETING T'OPICS AND ROCMS Tuesday, Sept @ er 29, 1970 3:30 - 5 p.m. 1. Statistical Models and Operations Research ............... Conference LoLmge 2. Coronary Care Training Evaluation ........................ Room 1-A 3. A Systems Approach to Coronary Care Evaluation ........... Press Rom 4. Evaluation of Instructional Technology Projects .......... Room 2-EF S. Evaluation of Physician Bducation ........................ Roan 1-C 6. Evaluation of Multiphasic Screening ...................... Room I-B 7. Evaluation of Stroke - Rehabilitation Projects ........... Roam 2-A 8. Evaluation of Cancer Registries .......................... Library 9. Evaluation of Regional Advisory Groups ................... Room 2-B 10. Medical Care Evaluation .................................. Roan 2-C 11. Evaluation of New Categories of Manpower ................. Room 2-D 12. Training for Evaluators ................................... Assembly Pocm Appe,-idix B-1- SPECI.AL INTEREST N=INGS Tuesday, September 29, 1970 3:30 - 5:00 p.m. STATISTICAL MODELS AND OPERATIONS RESEARGI Conference Loun-e Moderator: Francis C. Ichniawski Acting Oiief, Systems Management, R4PS Paper: David H. Gustafson, Ph.D. Assistant Professor Industrial Engineering Division University of Wisconsin Vernon E. Weckworth, Ph.D. Director, Systems Deve@opment Project University of Minnesota Member, Regional Advisory Group Northlands Regional Medical Program Discussion 2. EVALUATICN OF CORCNARY CARE TRAINING Room 1-A Moderator: Rodger M. @erd, M.D. Assistant Program Director, Continuing Education Presbyterian Medical Center San Francisco, California Report: Daniele Deverin Cybem Education, Inc. Report on Xerox Study of Twelve National Coronary Care Training Centers Presentation: Mariella Larter, M.S. Coordinator, Washingtm/Alaska Coronary Care Unit Nurse Training Program Discussion 3. A SYSTEMS APPPOACH TO CORONARY CARE EVALUATICN Press Pom Moderator: Morton Robins Acting e , Study Design and Analysis Staff, R4PS (continued on next page) Appendix B-3 Report: M.A. Rockwell, M.D. Director, ealth Program Rand Corporation A Study of Coronary Care Unit Effectiveness Progress Report and Proposal for Purther Work Presentation: Morton Robins Acting ChieF,- Study Design and Analysis .Staff, WS Evaluatim of Mobile Coronary Care Ambulances Discussion 4. EVALLLKTICN OF INM=IONAL TECHNOLOGY PROJECTS Rom 2-EF Moderator: James E. Dyson, Ph.D. Director, Gmtlnuing Education Division Colorado-Wycming Regional Medical Program Short Presentations: Cecelia C. Ccnrath Chief, Continuing ucation and Training Branch, RMPS IMS Policy on Instructional Technology James Barrett, Ph.D. Gmt:Ln@ Education Division Colorado-Wyming'Regicnal Medical Program Assessment of Interest of Participants M. Gene Aldridve Cmtinuing @tion Division Colorado-Wyoming Regional Medical @gram . Feedback Projected Consultations Via Audio and Visual Telephone Conference: William J. Pais@, Ph.D. 15irector,@ic Clearinghouse on Educational Media and Tec@ology Stanford University Elizabeth Norman, Ph.D. NoriYeastern University College of Nursing Frederick Breitenfeld, Jr., Ph.D. Ex-ecutive Director Ma-ryland'Center for Public Broadcasting (continued on next page) Appendix b-'t EVAULKTICN OF INSTRUCTI@ TECI-NOLOGY PROJE(,TS (cmtinued) Cmsultations Via Audio an3 Visual Telephone Conference: Gerald W. Gaston, D.D.S. Project Supervisor, Computer Assisted Instruction (CAI) Ohio State Regional Medical Program David L. Bell Fi@er Alta&-na, California Summary: William @bretsm, Ph.D. President, Liove-rnor@s@tate University Park Forest, Illinois EVALUATICN OF PHYSICIAN EDUCATION Rom I-C Moderator: Daniel Fleisher, M.D. Director ot Health Professions Temple University Resource: William B. Mmier, M.,D. Statt Assistant, Con@ing Education and Training Branch, RMPS Audience will study and discuss three surrogate projects. 6. EVALUATICN OF MULTIPHASIC SCREENING Rom 1-B Presentations: Donald N. Logsdon, M.D. Associatc, Director, Multiphasic Health Screening Center Department of C ty Health Brookdale Hospital Center State of the Art of Multiphasic Screening Evaluation Frank R. Mark, M. D. Chief, Operations @search and Systems Analysis,- @S Suggested Evaluation of NPS Projects Discussion of predistributed case history 7. EVALUATICN OF STROKE - REHABILITATION Room 2-A Moderator: Jerome Tobis, M.D. Chaiman Department of Physican Medicine and Rehabilitation University of California at Irvine (continued m next page) Appendix B-5 EVALUATICN OF SMKE - ILITATICN (cmtinued) Presentations: Charles M. Wylie, Ph.D. Professor, PZEliL- e=a th Administration University of Michigan Overview of Stroke and Rehabilitation Evaluation Bertram L. Tesman, M.D. Assistant or toy, Stroke Prog-rmn California Regional Medical Progymn - Area VIII A Comprehensive Stroke Program, Irvine California B. Lionel Truscott, M.D. irector; Stroke Program North Carolina Regional Medical I>rogram A Ccomprehensive Stroke Prograin, North Carolina Philip A. Klieger, M.D. Assistant Director toy Organizational Liaison for Stroke and Rehabilitation, RMPS Stroke and Rehabilitation - Tlle National Pictur(-- Discussion EVALUATICN OF CANCER IZEGISTRIES Library Moderator: Charles R. Key, M.D. A-Ssistant DiFect-or for Cancer New Mexico Regimal Medical Prograin Presentations: Charles R. Smart, M.D. Ullrector, Tn-t -nnamtain Tumor Registry Salt Lake City, Utah Methodologies for Evaluating Effectiveness arid Value of Registries GeoEge Linden . Chief, Califomia Tunior Registry State Department of Health Berkeley, California Alternative Methodologies for Evaluation of Registries Discussim Appendix B-6 9. EVALUA.TION OF REGICNAL ADIIISORY GROUPS Roan 2-B Moderator: James P. Harkness, Ph.D. Deputy Coordinator, New Jersey Regional Medical Program Presentations: David A. Pearsm, M.P.H. T&oo o Pubil c Health Yale University RAGs as a Factor in the Regimialization Process Homer @edom, Ph.D. Arthur 1). Little, Inc. Findings of the A.D.L. Study of @s Related to the PLmctioning and Characteristics of RAGS. Paul E. White, Ph.D. Division 0 vio-ral Sciences Johns Hopkins School of Hygiene and Public Health Regional Advisory Groups and Decision @ - Prel ry Fin@s Fran a Cmtractual Study Discussion 10. MEDICAL CARE EVALUATION Room 2-C Moderator: Glen E. Hast @ s, M.D. Z!oordinator, Nassau-Suffolk Regional Medical Program Resource Consultant: Jobn W. Williamson, M.D. Proħessor, Departnent @f Medical Care and Hospitals Johns Hopkins School of Hygiene and Public Health Discussion: Group will cbmider end results evaluation and its applicability to assessing quality of health care, including definitim of categories of appropriate measures, examples of end results measures, and implementation of end results evaluation. Appendix B-7 11. EVALUATION OF NEW CATEGORIES OF MANPOWER Room 2-D Panelists: 'Harriet Kitzman Departnent oT7ediatrics Strong Memorial Hospital University of Rochester . Expanded Role of the Nurse Sarah Mazelas Evaluatim Di-rector Califomia Regional Medical Progrmn - Area I 'Ihe Physician Assistant Program Resource: Veronica L. Conley, Ph.D. FFeacL, Aliled Health bectim, Cmtinuing Education and Training Branch, RMPS Discussim 12. TRAINING FOR EVALUATORS Assmbly Rom Presentations: George E. Miller, M.D. Director for Study of Medical E@tim University of Illinois Donald Po@ly, Ph.D. chiet ot 'I'rain@, L;enter for Study of Medical Education University of Illinois Resource: Marian E. Le FFead, Educat s Sectim, Cmtinuing Educatim and Training Branch, @S This interactive: ses,;im w4 11 focus on two kinds of training: a) that which prepares operational staff with basic mder- standings about evaluation and, b) that which is applicable to persons with special training and already highly qualified in their own areas. Examples of the Center's work in evalua- tion and training with individual-Regions will be presented. 10/26/70 sT.t@s RE-ro@r EAR,NftPxnD MTALUNTIO.-'\' @DS Background The 1968 legislative extension of Recyional Medical Pro(Traris earmarked Z> up to one percent of appropriated funds for evaluation to be utilized at the discretion of the Secretary of HEIV. This legislative proviso has become increasingly common to most health, education, and other DfiBV service-oriented grant proo-rams. It is expected that this DrO- vision will remain in the legislation for the foreseeable future and that tlae Regional Medical Programs Service will be asked to program approximately one-half of the one percent earmarked funds, i@Ti tli Di D@ V and IIS,@ffiA splitting the other one-half for programming by them. Available Funds and ITplementation It is estimated that during FY-71 approximately $700,000 will be available to the Regional Medical Programs Service for the earmarked evaluation studies. 7lie final amount available is, of course, dependent upon the appropriation now -under Concyressional consideration. Depart- mental policy requires that the funds be used for grants, contracts and consultant services; they may not be used to support RIPS staff and in-house evaluation activities. In this connection, the new multiprogram grants (Section 910 of the Regional P-ledical Programs Service legislation) may be one possible mechanism to support these activities, and the recently-issued guidelines on these grants included a statement to this effect. The Office of the Secretary, rliBV, must approve the use of all evaluation earmarked funds, and each Program has been asked to develop an evaluation plan on how it tentatively anticipates to program the funds. Departmental guidelines on the development of the plans were broad and suggested that the evaluation activities be primarily (1) retro- specti-k,e-, C2) results oriented and, (3) of assistance to the decision- making levels of the Federal establishment and perhaps to the grantees (collectively) themselves. "Quick and dirty" studies as well as long- range studies should be considered. In addition, evaluation methodologies, information systems, and some types of trainina were considered appropriate if they were judged to be priority areas. On-going and Proposed Activities A. oin Of the estimated $700 000 available to the Regional Medical Programs Service in FY-71 for this activity, approximately $200,000 may be considered already committed to tivo contracts initiated in late FY-70 and an additional $20 1000 has been' spent on the evaluation conference held in Chicago in late September. 'nie two on-going contracts include: (1) Information Support System for Regional Medical Programs -- Harvard Center for Community Health Director: Harold Keairnes, NI.D. Amount: $1105,000 This contract is testing and developing a socio-political information support system for management and self-ap'raisal p to assist a Region in assessing regional status, identifying chanues in the medical system resulting from P\.euional Medical Program activities, and comparing the effectiveness of organizational processes in promoting chan-e. Four Regional Medical Programs are now participating in this stud),, begun July 1, 19701, including one Region which "bought into" the stud),.on an independent basis. The first series of interviews i,;ith RecTional Adirisor), Group members, selected key regional health leaders and core staff members have been held and feedback to the Program Coordinators is now underway. Document analysis is Lu-ider-. way and the field intei-xrieiv@r for each P@ff) has been selected from the core staffs. (2) Training of Health Professionals in Characteristics of Effective Evaluation -- Llniiersity of Illinois College of Medicine Director: George Miller, M.D. Amount: $855,000 This contract supports short-term training for an estimated 250 evaluators and health professionals in Regional Medical Programs on the use of special evaluative techniques. One workshop has already been held and four more are planned for the fiscal year. B. Proposed The @IPS evaluation plan-covers a wide range of subject areas, including (1) the development of management information systems, (2) evaluating the extent and effectiveness of cooperative arrangements, (3) evaluating the role of R\U?s i:n the implementation and adoption of the heart guidelines, and (4) assessing demonstration activities. This plan is now under review by the Department and final clearance is expected shortly. Several of these proposed evaluation activities (items 1 and 3) are under active consideration. 2