evaluation workshop '72 qw lakes area regional medical program Inc. November 2, 1972 Tre adm ay n n, gara Falls, New York Sponsored by Lakes Area Regional Medical Program, Inc. Title The Process of Program Evaluation Purpose To acquaint those persons associated with the Lakes-Area Regional Medical Program with the evaluation mechanism, its complexity, and the need for overt commitment to its implementation. The Lakes Area Regional Medical Program gratefully acknowledges the support of the Division of Regional Medical Programs Service, Health Services and Mental Health Administration, U.S. Department of Health, Education and Welfare. The findings and conclusions in this publication do not necessarily represent the views of the sponsoring agencies. con en 3 Foreword by John R. F. Ingall, M.D. 4 Welcome by Irwin Felsen, M.D. 5 The Agenda 7 Executive Director's Address 9 Highlights of Secretary Richardson's Remarks 1 1 Keynote Address 14 Model of the Process of Program Evaluation 16 Discussion Groups 17 1 nteraction 1 9 Reaction 21 Participants reiNO The Workshop vividly portrayed evaluation as an integral process of all Lakes Area Regional Medical Program activities. It provided participants with a keen understanding of the evaluation process, its methods and procedures, and its overwhelming contribution toward the improvement of health care delivery systems. In addition to assessing program and project effectiveness, participants were shown how the evaluation pr (41 ass provides opportunity for learning, how.it identifies strengths, and how it pinpoints weaknesses - all'of which serves to improve program and project ope@rations. The information contained on the following pages expresses the pride of Lakes Area Regional Medical 3 Programstaff inconductingthisWorkshopaswellasthe sincere interest shown by its many participants. John R. F. lngall, M.D. Executive Director ICUFne.i, Felsen, M.D., President, Lakes Area Regional Medical Program Regional Advisory Group, Inc. "I would like to welcome you to our Second Annual Regional Medical Program Workshop. The topic of this conference is 'The Process of Program Evaluation.' At our Regional Medical Program in Western New York and Northwestern Pennsylvania, we are pledged to plan and to coordinate programs to strengthen the health care system capable of responding to the needs of all our people. That is our major objective. At last year's workshop we established our goals and objectives. During the year we formed the Program Committee to serve as a long range planning group. This committee is responsible for today's program. I would like to give special thanks to Patricia Hoff, Joseph Reynolds, Tony Zerbo and Dr. James McCormack for helping in the mechanics necessary for the conference. We make policy and when we make policy we are betting on the future. The great danger of any policy 4 sent into the future. Today we is to project the pre seek some objective yardsticks for measuring our Regional Medical Program's actual impact on society. We are, interested mainly in estimating the program's ultimate health benefits. We ar@@ xploring ways to assess the health impact of our projects, and to develop a meaningful method of evaluation. We seek 'workable' machinery. Can we do anything about it? I will confess that I am an optimist; one who draws confidence from facts as well as from hopes. I am sure we will find means of dealing with the problem, and be able to narrow the gap between promise and performance. We seek participation from all of you present. We are not too proud, too rigid, too blind, or too complacent to change. I am sure that we are all looking forward to a valuable learning experience. e a en a Thursday, November 2, 1972 9:00-10:00 2:40-3:15 Registration - Ballroom Small Group Discussions, Niagara Room, coffee Ballroom 10:00-10:10 Discussions will be based upon the Welcome , objectives and activities proposed in local Irwin Felsen, M. D. funded grant applications. A skillful leader President, Lakes Area Regional will be assigned to direct the discussion of Medical Program the group around Doctor Cassel's Regional Advisory Group, Inc. presentation of his evaluation model. The 10:10-10:30 participants will use Doctor Cassel's model to work toward the creation of an "Program Evaluation and the Mission of evaluation approach for the peoposal. Lakes Area Regional Medical Program" Participants wil I be given an opportunity John R. F. Ingall, M.D. to become directly involved in'the process Executive Director of program evaluation. Lakes Area Regional Medical Program 3:15-4:15 10:30-12:00 Group Reports "A Design for Program Evaluation" John C. Cassel, B.Sc., M.P.H., 4:15-5:15 Panel Reactors M.D., B., Ch., I nt. Med. Moderator Department of Epidemiology University of North Carolina J. Warren Perry, Ph.D. Chapel Hill,.North Carolina Dean, School of Health Related Professions State University of New York at Buffal 12:00-1:30 Buffalo, New York Luncheon Panel Harry A. Sultz, D.D.S., M.P.H. Thursday Afternoon Professor, Department of Social and 1:30-2:35 Preventive Medicine Introduction of State University of New York at Buffalc HEW Secretary Elliot L. Richardson Buffalo, New York John R. F. lngall, M.D. Elsa R. Kellberg, M.A. Assistant Director for Evaluation Elliot L. Richardson Lakes Area Regional Medical Pro ram 9 Secretary Buffalo, New York United States Department of Health, Robert O'Shea, Ph.D. Education and Welfare s6r Ais6ciate@ Profes Washington, D.C. Department of Sociology St @t6 @u h@iv@rsit'v of N ew York at'B uf John M. Glasgow 'Ph.D. @s@is@@t@Izint -Prot--essor, Universit@ of Connecticut - Health Cehto Hartford, Connecticut executive director's address John R. F. lngall, M.D., Executive Director, Lakes Area Regional Medical Program Program Evaluation and the Mission of Lakes Area Regional Medical Program "The Lakes Area Regional Medical Program is responsive to a large constituency. We have the capacity to catalyze efforts towards their stated goals and objectives, which in time become ours. In serving this body of people we want to enable them to attack the deficits in our nine county health market area. To accomplish this, we need to measure respectively what we are doing and what we have achieved. It is an obligation for those in receipt of our funds to appreciate what evaluation means. It is important for programs to clearly under- stand where they are; where they are going; and how they are going to get here. It is important for project directors to establish what measures they can apply, in the time frame they have set for themselves, to p @irtray hievement - particularly the ability to become self-supporting. We have developed and will continue to construct an evaluation c6,mpO7 nt into everything we do. Evaluation is essential for diagnosis; crucial to wise clinical judgements; and essential in measuring outcome. This process is without punitive overtones; it is a mechanism 7 which will allow a diagnosis of performance and impact. We can, and must, refine this mechanism in order to produce an effective effort toward correcting well-stated deficits in the health market area to which we are nsible. The Regional Medical Programs of this country have, by industrious involvement with. the grassroots community, made things happen. People of various disciplines are meeting for the first time. An informed, representive constituency of responsible individuals are sensitive to community deficits in the' health services. The involvement of this informed community must be augmented. The fact that it is being developed, that it is underway and has momentum, is a major achieve- ment and must be recognized. Involving intelligent people in the validation and interpretation of reliable data leads to intelligent decisions. These decisions are the first steps in regionalization which is implied in our title. Regional Medical Programs are 'enabling'. agencies. They recognize the industry required in bringing groups together and convening them for ecific purposes with specific challenges. Their unique role in maintai Sp momentum by judicious staff support, enhances the ability of-establi! agencies to deliver and to improve upon previous performanc6.,'@ They I facilitated many agencies now carrying out their mandated functi They have the ability to meet challenges whether they be in the professi arena or in the consumer population. A recurring theme is the ability to deliver and to measure the amount of ds t -o the"maximum amount of people. The funds to this purpose are dollars as venture capital which are invested in economically viable r.oiects. We have sought to involve all health professionals, particularly the private and voluntary groups. Regional Medical Programs had dramatic visibility in developing categorical disease components, such as the coronary care training program for nurses and physicians. The categorical killers are not in any way negated by our broader approach to the health problems. The first of our goals is prevention. It is an area where the biggest impact on human morbidity can be anticipated. In stimulating efforts towards prevention, the factor of dramatic visibility, which in turn releases tears and money, is absent. A crippled child will release more funds by being exhibited, than a normal child, who is so by virtue of preventive methods which are hard to render glamourous. Education of the consumer is important. The question of its feasibility, the economy of doing so, and the method of evaluating these efforts is a challenge to the Regional Medical Programs for the future. We must try to teach the difficult problems which will have impact on the categorical diseases. A problem well stated is half solved. It seems to me we have problems. It seems to me these can be given dimensions. In doing so, we can remeasure these programs. That we do so in an objective, clear, and responsible fashion is an essential ingredient in our evaluation. I anticipate today will be a learning experience and an exercise in working towards a lucid understanding of measures of achieve- ment@and their validity." Highlights of Secretary Richarcison's remarks We perceive very clearly that we cannot effectively improve the quality of health care delivery in the United States; we cannot rcome the gaps and improve access to good care; we cannot ove deficiencies in our delivery system by Washington decree. We cannot build from the top down without displacing in the process, not only the contribution of local resources, but the contribution of local concern. If we are genuinely interested in bringing about the kind of coalition of resources at the local level that is involved in putting the existing pieces together with optimal efficiency, we have to rely on people like yourselves to carry out that process. There is just no way in which federal bureaucrats can know enough about the capabilities of existing components of the health delivery @ ,system in any community to know how to fit them together in 6','More coherent whole. The initiative for this and the exercise of carrying it out must be in local community and regional hands. Awareness of need A very valuable quality of American life is the devotion of 9 -community concern to meeting the problems of people. How- ever, local response to problems can lead to fragmentation because so many well meaning citizens may be motivated to create community agencies or organizations that often do not communicate sufficiently with each other or do not participate in joint efforts to create a more total effort. But it is that kind of response, that kind of 'awareness of. need' within a community, that is part of what we are as a people. The very idea that you can rely on the government to solve these prob- lems and to meet these needs can only have the consequence of drying up the springs of local concern and compassion. There are two values at stake here. One is the capacity of local service to provide agencies that can fit their efforts together. e second is the preservation of community esponse to problems that derives from a sense of need. This is what leads us to resist building n the top down through a structure that would necessarily displace state and local initiatives. The planning capa- 'bility at the Federal level is so enormous, that, in order to get anything done at all, the solutions prescribed would have to be uniform, and they would have to displace local responsiveness and local concern. Initiatives Applauded We look to the kind of effort that you represent and the kinds of things that are being done in response to the problems of sparsely populated rural corhm unities in this nine county area. These include your efforts to make more accessible improved care for coronary and respiratory patients; to promote regional linkage among health care providers through your Telephone Lecture Network which enables them to keep their skills up to date; and to develop a data base that can contribute to the planning capabilities that are required in order to fit the pieces together. These are all the kinds of initiatives that we in HEW applaud and seek to encourage. Develop Planning Capabilities In my view, if we are to achieve the development of an adequate 10 network of services through building from the community level up, rather than imposing solutions from the top down, we can only do it through substantial improvement in our planning capacity. Under the leadership of Dr. Merlin DuVal, Assistant Secretary for Health Affairs, through whose general aegis the Regional Medical Program comes, we in HEW are now reviewing all of the present legislative authorities that would impinge in any way upon the planning process. We are seeking to strengthen the capacity as well as the authority and responsibility of planners at the local and regional levels. There is a limited amount of time and unless we move more quickly now to develop our planning capabilities, we may get left behind or be overwhelmed by the pressures which insist that you just are not going to get there from here, that you have got to let Washington do it. That, to me, would be profoundly wrong. o e a ress High r. John Cassel's Speech While there is a great deal of activity in the field of evaluation, most attempts have ended in rather dismal failures, producing very little in the way of data or results; and s the method those that have produced results have usually The third major deficit concern s led more to anger than to action. used to accomplish evaluation. It is too often thought that if you want to have an There seems to be three crucial reasons why current attempts at evaluation have been less evaluation, you must go out, hire an evaluator than successful. The first is a confusion and say to him or her, "Evaluate please, and give me a report." This is obviously an about the purposes of evaluation. Evaluation oversimplification, but I think you recognize is often confused with inspection, whereby how pervasive and insidious it is. There is a it is seen as a method of forcing program place for professional evaluators; but is a people to account for their actions. Results very specific place and it is not something of such accounting would be a change in budget, elimination of jobs, and downgrading one can turn over to an 'outside' agency or individual. or upgrading personnel. Such an unfortunate set of purposes leads to a stand-off in the personnel involved in the program and those involved in evaluation, with one trying to defend their position and the other trying to attack it. There has to be some method of public accountability for the use of public funds, but I choose for the purposes of clarity not to call that evaluation. It would be more precise to consider evaluation as a type of diagnosis. It is an integral part of practice, concerned with providing such information for program personnel as how to change, improve, or modify their actions. If we regard evaluation as a diagnostic probe rather than an inspection, we would be in a lot better shape. In addition to confusion as to purposes, there is confusion about the types of information needed for evaluation. Many evaluation pr rict ivities. How many eminated? How many blood counts did you do? How many patients did you see? I agree, there is a place for counting activities, but it produces no indication whatsoever of program outcomes, IL and program outcomes should be the heart and soul of any evaluative exercise. Keynote address argument is made that self-evaluation is It takes time. There is no instant solution. It takes a lot of dialogue because people have ry difficult because of personal bias. I counter that by saying that the most been unaccustomed to think in these terms. aluation is that And it takes professional Judgements important principle of any ev there be an on-going partnership between A very difficult and important question those people organizing and providing the service, and those people with the technical follows: "Given the objectives, and given that a certain set of information would be skills to conduct evaluations. If evaluation acceptable evidence, what needs to be changed is to be successful at all, there not only has to in order to accomplish these objectives?" be an acceptance that evaluation is desirable What should be done that is not being done and necessary, but a commitment on the part of those who have the responsibility now; what sorts of procedures and techniques for providing the service to be involved in are available; and what will be used in order this evaluation process from the beginning to accomplish these changes and lead to the through to the end, in partnership. desirable program goals? Evaluations will tell us how.accurate and Too frequently, people state their goals in "for-mother and against-sin" terms, such as useful our original formulations have been and "This program is supposed to improve the enable us to reformulate them in a second go around, or a third if need be. This becomes 12 health of all the people". This is relatively a feedback situation. It is the evaluator's useless, because the questions that follow responsibility to pick up the dialogue, to must be: "What aspect of health, what take the ideas that have been developed by evidence would you accept that this program will make any difference in anyone's health?' the program people and to translate them Such vague,'global statements are desirable, into measurable quantitative data. This step but unpractical or unrealistic in terms of the of translating information into data requires action take'. Another error often made in the skill of a professional evaluator. defining goal statements is to state them in There are three general types of data that terms of 'services provided'. are needed in evaluation. First, data must These are not goals. They are the means to show that the program is being successful; accomplish something. What they aim to that there has been a change in health status or change in health behavior. The question accomplish and what evidence they can is, "What sort of data do we need that will produce to measure that accomplishment measure sick people in need of care?" are the direct concerns of the program and its Objective and subjective data have to be evaluation. It is necessary for a dialogue to presented back to the people responsible take place between all those people who are for the program, who must then determine going to contribute to the evaluation, in order what would be appropriate sorts of information to specify what could be the feasible and labeling people as sick and in need of care. acceptable goals of such an activity. Having specified the goals, what information Homogenous groups of patients should be specified so that it would be possible to would you accept as evidence that you have develop a basis for describing the problem arrived there? Having stated the goals, unless that has to be changed in order to affect the there is clear understanding about the nature health status or health behavior of such people. of the evidence that would be acceptable, conducting an evaluation exercise that produces unacceptable evidence is not going to lead to any change. The second type of data needed pertains to The second strategy for program evaluation the changes that have been postulated as is a comparison of two or more different necessary. If you talk about preventing heart programs operating under the same needs disease by changing diet, you not only must or toward the same goals. If the programs know how to measure heart disease, but how are operating in the same or adjacent localities, to measure the diet. You should have some it is reasonable to compare just the after data information about the occurrence of the on the assumption that the sort of people and phenomena that are going to be changed and conditions being treated are relatively similar. whether one can see if they have changed. The third method is the randomized control The third type of data concerns the services trial. If there is the option of two programs or techniques that have been used. It does or treatments to be introduced and evaluated, become necessary in this whole picture to 'eligible patients can be identified and randomly measure whether the activities have been assigned to program A and prog @am B. The carried out, how frequently etc., in order to hope is that by random assignment the other find out if they have been successful in factors that influence outcome a ''re equalized. bringing about the change. The outcome of the treatment should remain There are three general types of strategy that unknown to the provider and the patient, in order to avoid bias. can be used in gathering and analyzing data: to compare the situation before and after the All of this sounds awfully formidable and 13 program has been introduced; to compare one long-term. It certainly has its complex program with another or with the absence of problems. I feel, however, that there would another program; and to engage in some sort be a lot more progress made in the area of of randomized trial. Common to all three evaluation if, in addition to this formalized approaches is that similar types of subjects approach that involves a lot of dialogue and a are to be compared. All factors likely to lot of cooperation, we made it desirable, affect the outcome should be similar as far possible, and expected for every professional as possible. person working within a pro-gram to ask himself, "What would I accept as evidence The before and after comparisons involve that I am doing my job well?" We can make finding out what the situation is, introducing it less formal, more personalized, and more the program or service, and measuring the involved. We can involve ourselves so that we,'v same phenomenon at the second point in time. The advantage of this approach is that as professional people providing services, can, begin to get some sort of feedback as to how it is simple and usually feasible. If there is no effective our actions are and where we need difference between the before and after I ,I @ T' '.;i @ to modify these activities. Again, evaluation i7 @: @, comparison, the program has been unsuccessful- is an integral part of practice, not an extraneous If you do find a difference, however, it is not set of.activities for inspection. It can lead t certain that it is due to the program or improvement in our knowledge and in our whether there were other factors not measured. ability to deliver services. We can learn fr6r ation would make this -:. r @ A control or situ our failures as well as our successes, ana we method more rigorous. Another alternative can begin to make knowledge cu(nulative.', is to collect as much data as possible to If we can do this, then I think we have a,' provide the program with influential factors. fighting chance of getting something Or there is the option to replicate, to repeat USK;Iul done. the program in other surrounding regions K6ynote address I by Dr. John Cassel e process of program evaluations MOde Statement of objectives in operational terms. Change in techniques, procedures or services necessary to accomplish the objective. ...,..ion of characteristics or variables measures of problem; measures of 14 process; measures of outcome. 3gy of data collection; choice of od, design of study. 1. Before and after study 2. Program of comparison 3. Controlled trial terpretations 1. Have changes occurred 2. Have objectives been met 3. What are implications for service .m,,.,,!,2,luation workshop '72 15 A AL Alk scussion groups discussed in separate group discussions e following projects were roup Project Directors 1. Telephone Lecture Network Joseph Reynolds, TLN Coordinator 2. Comprehensive Continuing Care Evan Calkins, MD, for Chronic II Iness Chairman, Department of Medicine, State University of New York at Buffalo; E. J. Meyer Memorial Hospital 3.. Pulmonary Rehabilitation 4. Pulmonary Home Care (The above two projects are a part of the John Vance, MD, Clinical Chronic Respiratory Disease Program) Associate Professor of Medicine, State University of New York at Buffalo; Millard Fillmore Hospital 16 5. Model Program for Comprehensive Ernest Haynes, MD, Director, Family Practice Family Health Center, Deaconess Hospital, Buffalo @@@ity Health Information Profile Harry Sultz, DDS, MPH, 6. Commun Professor, Department of (CH I P) Social and Preventive Medicine, State University of New York at Buffalo 7. Model Stroke Rehabilitation Proposal for William Kinkel, MD, Regional Community Hospitals Clinical Associate, Professor Neurology, Anatomy, State University of New York at Buffalo; Millard Fillmore Hospital ir erac @ion Seven groups met to evaluate individual LARMP projects, and found themselves QUESTIONING... We had better decide first what we are going to define as SUCCESS and if we do arrive at this success, WHAT ARE WE GOING TO DO ABOUT IT? HOW FAR DO YOU GO in a study? WHEN do you say it is SI GN I F ICANT? VI What measures are VALID in a particular case that could really measure CHANGE? We want to know if this kind of program will help people to BETTER COPE with the problems of living in contemporary society? WHY DO THIS? and questioning 17 We question the SCOPE of evaluation, the BOUNDARIES of evaluation ... WHERE DO YOU STOP? What caused the drop in USE of this program; Was it CONTENT or COST or both? Do other PEOPLE in tl OUT abo se program H'ow many times do you have to WORK THROUGH these steps and then WORK BACK? Are the MEANS getting mixed with the ENDS; are ACTION A physician must question, "Are there functions physicians have traditionally performed that programs getting mixed in with the OBJECTIVES? would better be performed by others? Is it a VIABLE program or is it NOT? Could we turn success into USEFUL ACTION; and, if not, we QUESTION the whole worth of the whole evaluation. @and concluding OTAL well-being of the patient, If you break We are concerned with the T not strictly a particular health problem. an objective down to a specific point, The most important thing is to determine whether there is it starts a NEED for a program or an ADVANTAGE in having one. to MAKE SENSE. difficult to ACCESS TOTAL IMPACT There are parts of health care that cannot be measured. Evaluation is like salt: every meal IMPROVES with salt; some foods NEED MORE than others; but there can be TOO MUCH salt! Are the objectives WORKABLE? Periodically there should be an assessment of data. Based upon the insights of the professionals, you could make sound decisions to BUILD ON. TherearemanyVARIABLEStoconsider: TIMING; PROGRAM; HOW and WHEN PEOPLE ARE INVOLVED; WHO they are trying to reach...you must CLARIFY your thinking. Data collection can be a valid INDICATION of CHANGE. This is al I based on the assumption that we BELIEVE it will WORK! ft reac ion program committee meeting on the EVALUATION of the evaluation WORKSHOP Father Cosmos Girard, L.A.R.M.P. Staff Chairman Dr. John I ngal I Dean Virginia Barker Patricia Shine Hoff Dr. Theodore Bronk Dr. James McCormack Dr. Irwin Felsen Joseph Reynolds Dr. Larry Gre6n Anthony Zerbo Dr. Bert Klein Elsa Keliberg Dr. Harry Su ltz Mr. A. Burton Kline Dr. LaVerne Campbell Dean J. Warren Perry WHY A WORKSHOP ON PROGRAM EVALUATION? Evaluation is an essential component in all of the Lakes Area Regional Medical Program's projects and program activities. It is a mechanism that promotes sound planning, productive operation, and innovative re-planning. The purpose of the workshop was to acquaint those persons associated with the Lakes Area Regional Medical Program @19 with the evaluation mechanism, its complexity, and the need for, overt commitment to its implementation. Evaluation efforts depend upon the cooperation of project-sponsors and administrators; and cooperation depends upon understanding. The workshop proved to be a workable way to accomplish. that understanding, for both project and Program people. WHAT DID THE WORKSHOP ACCOMPLISH? What was indeed accomplished was an attitudinal change toward evaluation. Those attending realized the importance for evaluation, and demonstrated an increased appreciation of its relevance to program goals and objectives. The Workshop produced rationales for 'why'-@, evaluation is necessary in terms of planning and outcome. It forced basic and direct questions about the need, motive and direction behind projects. It lead to the clarification of individual goals, demonstrating the need for relevant, workable, and specific objectives. In an attempt to evaluate particular projects of LARMP, participants were confronted evaluation problems. They found that as they shaped evaluation., they re-shaped the project. @ Evaluation began as an idea and emerged as a functional process that required work,'@ understanding, and unqualified commitment from every -facet "of,@ every program. 20 ar 410clis an Project Directors Virginia Barker, R.N., Ph.D. Jean Miller Director, Mobile Health Unit Information Dissemination Service Alfred, New York State University of New York at Buffalo Buffalo, New York Michael Carey, Director Lake Area Health Education Center John C. Patterson, M.D., Director Erie, Pennsylvania Tumor Registry Buffalo, New York Roger Fenion Emergency Medical Services Joseph Reynolds, Director Erie County Health Department Telephone Lecture Network Buffalo, New York Buffalo, New York Phyllis Higley, Ph.D., Director Harry Sultz, D.D.S., M.P.H., Director Plan and Articulation for Allied Health Community Health Information Profile (CHIP) Buffalo, New York Buffalo, New York Regional Advisory Group Members 21 Irwin Felsen, M.D., President LaVerne Campbell, M.D. Lakes Area Regional Medical Program New York State Health Department Wellsville, New York Buffalo, New York Father Cosmos Girard, OFM, Ph.D. Herbert E. Joyce, M.D., Past President Vice-President, Lakes Area Regional Medical Program Lakes Area Regional Medical Program Buffalo, New York St. Bonaventure University Olean, New York Bert S. Klein, D.P.M. Jamestown, New York Theodore Bronk, M.D., Secretary Lakes Area Regional Medical Program J. Warren Perry, Ph.D., Dean Mt. St. Mary's Hospital School of Health Related Professions Niagara Falls, New York State University of New York at Buffalo Buffalo, New York John C. Patterson, M.D., Treasurer Lakes Area Regional Medical Program Harry A. Sultz, D.D.S., Professor Director, Tumor Registry School of Medicine Buffalo, New York State University of New York at Buffalo Buffalo, New York Virginia Barker, R.N., Ph.D. Alfred University Alfred, New York PIrofessional Staff Members Participants Helen Applebaum John R. F. lngall, M.D. Executive Director Occupational Therapy State University of New York at Buffalo James McCormack, Ph.D. Associate Director for Planning Sister Mary Alberta and Evaluation (for Sister Gerard, Admin.) St. Jerome Hospital Batavia, New York Robert J. Beebee Staff Associate for Niagara, Genesee, llegany and Wyoming Counties Nancy L. Benson (for Sister Armiger) A Dean, School of Nursing Niagara University Helen Brown Niagara Falls, New York Hospital Library Consultant Donald A. Bradley, V.P. David Buck Millard Fillmore Hospital Staff Associate for Grant Development Buffalo, New York John Cady Carolyn Brayley Staff Associate for Evaluation Occupational Therapy State University of New York at Buffalo Floyd Cogley, Jr. 22 Staff Associate for Grant Development Raymond E. Brewster Jean Hanna, R.N. Mt. View Hospital Staff Associate for Erie (Pa.), Chautauqua, Lockport, New York CattaraugusIand McKean, Pa. Counties Thyra Charles, R.N. Patricia Hoff, R.N. Visiting Nursing Association Assistant Director for Nursing and Buffalo, New York Allied Health@Affairs Sara Marie Cicarelli Elsa Keliberg Medical Technology Staff Associate for Evaluation State University of New York at Buffalo Michael Miller Frances Cirbus Pulmonary Study Unit Staff Associate for inner City Affairs Millard Fillmore Hospital Robert J. Miller Buffalo, New York Staff Associate Louis Cohen Joan Philipps Health Association of Niagara County, Inc. Staff Assistant for Evaluation Niagara Falls, New York Joseph Reynolds Kay Collard, R.N. Director, Telephone Lecture Network Pulmonary Study Unit Millard Fillmore Hospital Mrs. Marion Sumner Buffalo, New York Consultant for Fiscal Affairs Margaret Connelly, R.N. Anthony Zerbo, Jr. Director, Allegany County Assistant Director for Communications Public Health Nursing Service 11 Angelica, New York Mary Lou Considine Stanley Grenn Niagara Coalition HANCI United Health Foundation Niagara Falls, New York Buffalo, New York Joan Cookfair Jerrold S. Greenberg, Ed.D. Mrs. Family Practice Assistant Professor Buffalo, New York Department of Instruction State University of New York at Buffalo Robert M. Cooper Department of Pharmacy Sister Margaret Ann Hardner, S.S.J. State University of New York at Buffalo Erie, Pennsylvania Buffalo, New York Robert W. Harris John A. Coulter Executive Director School of Pharmacy Western New York Hospital Association State University of New York at Buffalo Buffalo, New York William D. Crage Mrs. Elizabeth Harvey Niagara Coalition Health Association Family Practice Niagara County Inc. Buffalo, New York Niagara Falls, New York A, A. Hosinski, Director Bette Dale, Executive Director Veterans Administration Hospital,, Batavia, New York Lockport Senior Citizens Lockport, New York Phyllis Higley, Ph.D. Carolyn Daughtry Allied Health Planning Department of Mental Health State University of New York at Buffalo Buffalo, New York Mrs. Sylvia Helbert, R.D. Extension Home Economist 23 John Donnelly Erie, Pennsylvania Erie County Cooperative Extension Buffalo, New York Alan J. Drinnan, D.D.S., M.D. School of Dentistry Mrs. Fleeta Hill (for Mr. William Gaitor) State University of New York at Buffalo "Build" Buffalo, New York Barbara Edelman, R.N. Rick Horanburg Pulmonary Study Unit y P. Smith III Millard Fillmore Hospital Aide - Congressman Henr Buffalo, New York Rev. Edgar L. Huff Ruth Elder, Ph.D., M.S.N. Niagara Falls, New York Associate Professor of Nursing State University of New York at Buffalo Art Hundsburger comprehensive Health Planning of Roy Fabry, Rehabilitation Director Northwestern Pennsylvania, Inc. Meyer Memorial Hospital Erie, Pennsylvania Buffalo, New York Doris Jones Gerald Farmer, Director Niagara Community Action Program Inc. Comprehensive Health Planning Niagara Falls, New York Erie, Pennsylvania Robert Joss Dominic F. Falsetti, M.D. Community Psychiatry Vice-President Health Association E. J. Meyer Memorial Hospital Niagara County Buffalo, New York Niagara Falls, New York T. J. Kittle A. Burton Kline 'Family Practice Regional Medical Program Service Buffalo, New York Rockville, Maryland Hal Farrier Ruth E. Kocher Comprehensive Health Planning of Regional Nutritionist Western New York, Inc. New York State Health Department Erie, Pennsylvania George Lawn, D.P.M. James Felsen, M.D. WCA Hospital Indian Health Service Jamestown, New York Wellsville, New York Ralph Levy, retired business man Roger Fenlon Niagara Falls, New York Emergency Medical Services Erie County Health Department Victor Lapuma Buffalo, New York Comprehensive Health Planning of Northwestern Pennsylvania, Inc. John F. Fortune Erie, Pennsylvania Erie County Health Department Buffalo, New York Janet Lindner Visiting Nurse Association John R. Foster Buffalo, New York Martin Luther King Center Erie, Pennsylvania Robert A. Ludwig Comprehensive Health Planning of Western New York, Inc. C. M. Fuller American Cancer Society Buffalo, New York Erie, Pennsylvania Kathleen D. Mack, R. N. 24 Charles Garverick, Ph.D. Genesee Community College Behavioral and Related Sciences Batavia, New York State University of New York at Buffalo Suzanne Manson Martin Gerowitz Family Practice Comprehens Iive Health Planning Buffalo, New York of Western New York, Inc. Buffalo, NeOv York Emanuel Luper, R.N. West Seneca State School Geoffrey Gibson, Ph.D. Sociology Department Andrea McKown State University of New York at Buffalo Comprehensive Care E. J. Meyer Memorial Hospital John Glasgow Buffalo, New York University of Connecticut Carol McPhee Rev. Allen H. Goss Division of Community. Psychiatry Niagara County Migrant and Buffalo, New York Ministry Committee Robert M. Mercurio, Ph.D. Wilson, New York Gowanda State Hospital John C. Grabau Gowanda, New York comprehensive Health Planning Jean Miller of Western New York, Inc. Health Sciences Library Buffalo, New York State University of New York at Buffalo Larry J. Green, D.D.S. Mary M. Mogan, R.N. School of Dentistry New York State Helath Department State University of New York at Buffalo Alice Moordian, Executive Director Sister Mary Alberta (for Sister Gerard, Administrator) Golden Age Club St. Jerome Hospital Niagara Falls, New York Batavia, New York Michael Parsnick, Clinic Coordinator Niagara County Migrant and Rural congressman Henry P. Smith III mission Yo rk @@t?@ Smyth evv Buffalo. N a, R.N. M. t Buffalo D ew York 6 E osp @al Patterson insist Affairs fv Tu St@ie As N nnsylva a Niagara unty Migr6nt and of ,Y:Oll@i k R.1 en, V Health e@p. m@nt.. Euciene P. hardson Secretary of Health; Education and Welfare of Weste on, D.C. Buffalo, Ne N New 'York ohim Nevw 'it@ ea N ew,@ sa@ln6p B.S elated Professions iversity of New York at Buffalo