1 I$ 14111111*111111111111 CONFERENCE ON REGIONAL MDICAL PROGRAMS Washington Hilton Hotel, Washington, D.C. January 15-17, 1967 THE ISSUE: EVALUATION OF MEDICAL CARE UNDER PUBLIC LAW 89-239 Paul J.. Sanazaro, M.D. Director, Division of Education Association of American Medical Colleges Chairman, Health Ser-gices Research Study Section U. S. Public Health Service Consultant, Division of RegionAl Medical Programs This paper was prepared on request and is provided as background to the MOIQDAY, JANUARY 16, 1967 A f ternoon Discussion Session iEits first in collecting Evaluation in the field of medical care cons information on the operations and end-results of a programs, then making judgments regarding the effectiveness and efficiency of the programs or services under- study with respect to both individual patients and com- munities. On a short-term basis, evaluation identifies needed revisions and improvements in an operating program. Its long-term function is to provide a rational base for broad policy decisions governing the future directions of such programs or services. When conducted with a high order of technical competence, evaluation may also contribute substantive knowledge to the field of health services research and is then designated as evaluation research. A distinction exists between evaluating a Regional Medical Program and evaluating medical care. Public Law 89 -239 and the Guid6lines emphasize the livery of medical care, i.e. the personnel, facilities, services, and resources necessary to improve diagnosis and treatment. However, only in certain limited situations will increasing the capa- bilities for delivering medical care automatically assure an improvement in the it of care. For example, increasing the number of trained personnel or providing specialized facilities and services in areas where these are marginal or nonexistent constitutes, on the face of it, a distinct improvement in the quality of care. In this sense, evaluation of a Regional Medical Program can be directly comparable to evaluating the quality of care. The term "medical care" has several unique meanings depending on whether it is defined as a process, as a system, or as an area of study. It is also analyzed in different ways depending on whether individual patients, a community, or the entire Nation are the recipients. The following components of medical care are particularly relevant to the evaluation of a Regional Program: 1. Suppl or availability of health care personnel, facilities, and services,, including preventive measure 2. Utilization of personnel, facilities, and services, including preventive measures,by individual patients or population groups. 3. Process oorf care: accuracy of diagnosis, adequacy of treatment, and appropriate utilization of consultative resources and specialized technical services. 4. End results: the effectiveness of a treatment or program as determined by the consequences for the individual patient or population, including expressed views of patients and poten- tial patients toward the availability and acceptability of medical care. 5. Unmet neeeeldass: individual patients or population groups with identifiable diseases not yet diagnosed, or dia nosed but not 9 under treatment. 2 In a limited, technical sense the requirements for evaluating a Regional Medical Program in accord with the stated purposes of Public Law 89-239 can be met by limiting the evaluation of medical care to its first component, supply or availability. However,, in order to evaluate the effectiveness of the increased supply of personnel, facili- ties, and services and their improved distribution, it is necessary to include the other components of medical care: utilization, the adequacy of diagnosis and treatment, end results, and unmet needs. The assumption seems warranted that the law was passed with the implicit belief that there would be demonstrable improvement in the care, and in the results-o care, of patients with the specified diseases. It appears to be a legitimate responsibility of those conducting Regional Medical Programs to ascertain so far as is feasible the relationships between improved health manpower, facilities, and services and the other defined elements of medical care. As stated, evaluation is a dual process of data collection followed by judgment. Depending upon the particular program or services,, evaluation may be carried out at varying levels of precision and sophistication. These levels will be described separately. 1. Evaluation to determine whether the stated objectives of a particular program were met. If the stated objective of a program is to train ten rehabilitation aides, and this is accepted as the only objective of the program, then the evaluation of this program rests entirely on the fact that ten rehabilitation aides were or were not trained. By analogy, this level of evaluation applies to the establishment of specialized patient care units, demonstration programs, diagnostic or treatment ser- vices, and so on. The fact of their establishment provides the necessary 3 and sufficient information needed in judging s whether or not the objective were met. 2. Obiective descrip on and anal sis. For this level, descriptions of education and training programs, facilities, servicet, and capabilities of personnel are compiled in accord with prevailing professional concepts and standards. For example, a program for training nurses to staf f coro- nary care units should be described in terms of the functions nurses will be expected to perform as a result of their training. These functions will have been defined by appropriately informed and experienced experts. Evaluation of the training program will be directed At answering two questions: (1) Has the program been designed in accord with generally accepted principles of such training? and (2) Was the program carried out as planned? Descriptive data bearing on these questions must be collected before a judgment can be made. Similarly, with respect to the operation of coronary care units, the basis of judgment regarding their adequacy is simply an accurate description of the services which these units pro- vide, together with a description of their overall operation and adminis- tration. These descriptions are then compared with prevailing professional and administrative judgments of what constitutes proper staffing, organiza- tion, resources, and administration for coronary care units. 3. Evaluating utilizat: ulations. The question of whether or how the improved staffing, facilities, and services bring about improvement in medical dare cannot be answered without information concerning the utilization of such personnel, facilities, and services by patients. Two approaches are possible. Prior to the institution of the 4 program baseline data can be obtained on the utilization rates of various personnel and services by all persons with the specified diseases in the population served by the Regional Program. If baseline data are not available, a comparison group of patients to whom the new resources are not available must be studied in order to determine that other changes totally unrelated to the Regional Medical Program have not brought about equivalent changes in utilization. Both approaches require the use of epidemiologic methods applied to probability samples of general populations. It is inappropriate both in terms of the overall objectives of Public Law 89-239 and correct methodology to base evaluation on changes in the numbers or characteristics of only patients who receive care. Similar approaches are necessary to determine whether changes in frequency of duration of hospitalization for equivalent disorders or their complications are brought about by the program. Judgment of the Adequacy of utilization will rest on two comparisons: (1) between rates per 1,000 general population in control and experimental communities or before and after the introduction of a program in the same community, and (2) between utilization rates and known prevalence of the target diseases. 4. Evaluation of improvement in the patient care process. Direct comparisons on a controlled basis are required to determine changes attributable to the program in accuracy and completeness of diagnoses, adequacy of treatment programs, and appropriate referral of patients for specialized services. This level of evaluation encompasses the techniques of the medical audit in office, clinic, and hospital settings. 5 5. Evaluation of end results. This level constitutes the definitive measure of effectiveness of personal health services. By use of matched populations, data can be compiled on decreases in interval between onset of symptoms and receipt of.care; end results of care; prevention of com- plications; alleviation or reduction of disability; improvement in social functioning; increased longevity; and so on. Whereas techniques for the preceding four levels of evaluation are well worked out and can be applied in pre-tested form, the determination of end results is still under research and development. 6. Analysis of cost-effectiveness. This form of evaluation focuses on the efficiency of a program and questions whether the results of a given program or program element are achieved economically in terms of dollars, manpower, time, space, and resources. Competence in operations research and economics is required. Two or more training programs for aides might be compared to discover whether comparable skills can be achieved more economically. Appropriate economic bases are needed to compare these programs with training programs which produce fully qualified professional personnel. Similarly, the costs of establishing and operating different types of coronary care units need to be compared in relation to demonstrable improvements in the outcomes of care given in these units. It is also appropriate to compare costs and staffing economies or the functional efficiency of such specialized units with an at-large monitoring system dispersed throughout the hospital. The critical element in such - evaluations is an agreed-upon set of criteria of adequacy for services and end results. Only then can the relative costs be rationally analyze - -6 sures. This 7. Evaluation of the effectiveness of is the most difficult level of evaluation since it attempts to determine the extent to which diseases are being reduced, controlled, or eradicated from the population by the application of preventive measures. The use of epidemiologic methods is also essential for this form of evaluation. Evaluation is a sequential process, each step of which must be appro- priately planned and carried out before the next step can be taken. The sequence may be outlined as follows: 1. Collection of Information and Data. 1 . Specification in detail of the objectives of the programs, services, and end results which are to be evaluated. 2 . Establishing the criteria on which judgments will be based. 3 . Designing the instruments or records for data collection. 4 . Applying the appropriate methods for collecting the relevant descriptive information with minimal bias. 5. Statistical analysis and/or summary of descriptive information. 6 . Interpretation and comparison of results against agreed-upon criteria. II. Judgments Regarding Adequacy or Inadequacy of Program, Program Components, or Results. Quality, effectiveness, and efficiency of medical care cannot be measured directly in standardized units. They can be inferred from one or more objectively specifiable indexes derived from established professional standards. These indexes can serve as the base information or data for judging the degree to which a program 7 or its results meet or do not meet the criteria specified in 1.2 above. Judgments of quality are based on consensus of physicians and other professional personnel. Effectiveness and efficiency of a program or procedure can be defined somewhat more-objectively, because data can be collected on effectiveness, and the dollar and manpower investment can be objectively related to outcomes (cost- effectiveness analysis). However, even under the best of circumstances, evaluation is a difficult and demanding procedure, especially in'the field of personal health services. Section 908 of Public Law 89-239 states that the Report to the President and Congress will include "an appraisal of the activities assisted under this title in the light of their effectiveness in carrying out the purposes of this title.11 On page 65 in the first paragraph, the Guidelines stipulate that "special effort" is to be made to incorporate evaluation in the planning and operational phases. "Research into better means of accom- plishing the purposes and objectives of the Regional Medical Program" qualifies for support in an operational grant. In order to analyze the role of evaluation in the Regional Medical Programs, it will first be necessary to identify the intents and provisions of Public Law 89-239 which have implications for the purpose, scope, level and limitations of evaluation. Within Public Law 89-239 and the published Guidelines, the following major categories of objectives are defined: 1. making available to patients the latest advances in prevention, diagnosis, treatment, and rehabilitation; 8 2. developing more effective distribution and utilization of all types of medical resources; 3. establishing cooperative arrangements among medical institutions and professions to overcome fragmentation and insularity and meet the diversity of needs, resources, and existing patterns of education and services; 4. improving health manpower and facilities through education and training of health care personnel and demonstrations of patient care; 5. extending the productive interrelationships of extensive research, teaching, and patient care activities to community hospitals and practicing physicians; -apt-a@n, 6. creating an effective environment for continuing 84 innovation and modification without interfering with the patterns or methods of financing patient care or professional practice, or with the administration of hospitals. It is legitimate to question whether augmenting existing patterns for the organization and delivery of services will automatically bring about maximum possible improvements in the health of the population in proportion to available knowledge and techniques. The potential impact and the pro- jected total investment in Regional Medical Programs are such that consid- etable effort should be devoted to the development of standardized data on incidence and prevalence of the target diseases in the general population. (as described in paragraph 1, page 16 of the Guidelines). Furthermore, significant effort should be devoted to analyses of factors which determine 9 the degree of success achieved in improving the delivery of, medical care to all persons who could benefit from it. It is only by using techniques of evaluation which link together personnel, facilities, services, utilization, end results,'and cost- effectiveness analyses that an approach can begin to be made to the evalua- tion of the impact of any program on the medical care system and on the quality of care. Study of one component of the medical care system will not provide sufficient information to make possible wise decisions concerning needed modifications in other components and links. The evaluation of medical care within Regional Medical Programs must be comprehensive in scope and long-range in perspective. The most productive attack on this problem will result from cooperative efforts by universities and private organizations utilizing the resources of a number of units within the Public Health Service. IC Evaluation as Operational Research The particular form of evaluation which is undertaken and the technical competence of those who design and conduct the study are essential consid- erations. In addition, failure to properly utilize or apply the results of evaluation will defeat the basic purposes of evaluation, namely, to improve programs and their effectiveness and efficiency. Many circumstances may vitiate evaluation and prevent its effective contribution to the continual improvement of programs. The list of potential contaminating factors is long. It includes such factors as the introduction of undue bias and subjectivity by those administratively responsible for the program; resistance of professional personnel to evaluation; arbitrary restriction of the, limits of evaluation; changes in the program while it ropriate methods of data collection; failure is being evaluated; use of inapp to specify clearly the goals and end results to be evaluated; failure to onfusion of availability a before attempting evaluation; c establish criteri of services with utilization or with actual patient benefit; inadequate access to or lack of availability of standardized rates for prevalence and incidence of diseases. One approach of proven merit is the establishment of a health services research unit, a form of an operational and epidemiologic research unit, as an integral part of a health services program. By this means, an administrative mechanism is set up for feeding the results of evaluative studies to those who must make decisions governing the day-to-day operations of the program as well as future improvements. Given long-term responsibilities, such units are more likely to develop and maintain records which cumul-a- tively become more valuable and informative because of the documentation of changes over time. This resource is not likely to be developed when ad hoc evaluative studies are.carried out on a short-term basis by con- sultants who have no continuing responsibilities to the program. Even under the most advantageous circumstances, continuing evaluation of health services based on opIerational and epidemiologic research encounters certain problems with predictable regularity. These will be listed briefly: 1. One of the most important potential contributions of evaluation is the analysis of alternate approaches to the attainment of program objectives. Very often the decision at issue is not whether a particular program in operation is effective but whether an alternate program might be more effective. To base evaluation upon an all-or-nothing answer for an entire program is much less productive than providing alternate program components which can be independently evaluated with respect to their consequences and costs. 2. It may be that the major contribution of evaluative research is to determine whether the traditional ways of carrying on @r ofessional practices and delivering medical services are, in fact, the most effective. If arbitrary assumptions and unwarranted limitations are placed upon the scope of evaluation, even though some limitations are always necessary, the hope that continuing experimentation and innovation will lead to dramatic improvements in medical care is less likely to be realized. 3. There are several stages in the evolution of new health care programs., on a local, regional, or national level. Initially, decisions are made and implemented on the basis of best judgments of those responsible for the program. After a program has been established, a number of new, unrelated facts begin to influence decisions, but in the absence of an organized and definitive body of data, the administrators of the program require wide latitude in making decisions because factual guidelines are still imprecise. The third phase of such programs emerges when cumulative evaluation, studies, reports, and research have both defined the system and its component parts and related their operations to objectively specifiable effects. In this period, the data base becomes more important in supporting operational decisions than empirical judgments of administrators. Many Regional Medical Programg are in the first stage. It will be some time before the second stage is reached. The third stage can only be d3'.Mly glimpsed in the distant future, and will not be reached at all unless activities in acquiring appropriate data bases are promptly established. 4. Evaluation of demonstrations in which the purely medical aspects of the services rendered are assumed to be effective may be based on a - 12 false assumption. To the extent feasible,, evaluation should concern itself with all the factors that actually or potentially influence effec- tiveness, as it has been defined for the purposes of evaluation. These factors include the reliability and validity of the medical measures of diagnosis and treatment. In settings where such access is feasible, such factors should be identified as the objects of evaluation. If this is not done, programs may be evaluated as highly effective in terms of their operation and costs, although they may not be advancing the actual care of patients. 5. Finally, the question may properly arise whether a particular program is an appropriate one for the area or population to be served. Presumably this decision was made when the particular program was instituted. s, it is legitimate to subsume, under evaluation, questions con- Ndnetheles cerning the appropriateness of the program in terms of the cultural attributes of the area or population and the likelihood that elements of the program might be applicable to other areas and populations. The methods used must take into careful account the possibility that the unique circumstances operating in a particular program may make it impossible to achieve compa- rable effectiveness and efficiency in other areas. Sources and Resources for Evaluation A sound program of evaluation in the f iela of medical care requires the direct and cooperative involvement of a number of disciplines and competences. Background or experience in medical care is not essential for all contributors in order for them to make substantive contributions; the principles of evaluation can in many instances be transferred from other 13 - fields. Many individuals will have to be recruited into the medical care field to make possible the level and scale of evaluation that is called for. Potential sources of professional assistance or consultation include many departments in the university: Sociology, Social Psychology, Economics, Political Science, Business Administration, Administrative Science, Educa- tional Psychology. Schools of Public Health generally possess high-level competence in epidemiology and medical care organization. In several such Schools, as well as in several Medical School Departments of Preventive Medicine and a few other university departments, medical care research units have developed well-qualified faculties in medical care and patient care research, health economics, medical sociology, operations research and systems analysis, epidemiology, demography, health services statistics, and medical care administration. The national impact of Public Law 89-239 will best be evaluated through the cooperative efforts of the Public Health Service, other governmental agencies, the individual Regional Programs, and a number of other public and private resources. The National Institutes of Health, the Bureau of Health Services and the National Center for Health Statistics as well as other offices within the Public Health Service have unique sources for medical care research and evaluation. The task of evaluating the effectiveness and efficiency of Regional Medical Programs calls for the cooperativ e effort of staffs of universities, members of the health professions, and of units of governmental agencies. Onl then can the y requisite talent and competence be mobilized to provide the data essential to local and national policy determinations which must shape wisely the future of medical care for all our citizens. - 14 1-71 CONFERENCE ON REGIONAL MEDICAL PROGRAMS Washington Hilton Hotel, Washington, D.C. January 15-17, 1967 THE ISSUE: CONTINUING EDUCATION AND REGIONAL MEDICAL PROGRAMS Prepared by Staff of Continuing Education Branch of Division of Regional Medical Programs This paper is provided as background to the MONDAY, JANUARY 16i 1967 Afternoon General and Discussion Sessions. directly Continuing education and training address themselves quite to the primary purpose of the Regional Medical Programs -- to make more widely available to the patients of the Nation the latest advances in the diagnosis and treatment of heart disease, cancer, stroke and relIated diseases. Because the more successful continuing education and training programs are often dependent upon cooperative efforts of a number of individuals and organizations, the creation of regional cooperative arrangements by the Regional Medical Programs may provide significant new opportunities for the development of effective continuing education activities. The regional nature of the Programs can also provide other assets to continuing education and training - an opportunity for close relation of teacher and learner in development of programs, convenience and accessibility of programs, and opportunity to build together links between education and health care. Indeed one of the real potentials of continuing education and training within Regional Medical Programs is the opportunity toIintegrate these activities into the larger sphere of health care which they subserve. Relation of Educational Needs to Health Needs Although Regional Medical Programs have st lated additional attention to the problems of'continuing education, this new interest is only an additional increment in the extensive array of activities already underway along with widespread discussion of needs and solutions. Yet there is -I- cause for thoughtful concern and a hard look at past accomplishments and future prospects, for there are a number of knowledgeable persons who have entertained serious reservations about the effectiveness of current activities in continuing education in improving patient care. The approach to developing truly effective training programs must be viewed in the broad context of health care. Educational program design takes its origins in identification of the educational needs of the health professional. These educational needs in turn have their origins in the health needs of individual patients and in the patterns of medical care and the total health needs and resources of the particular region. The sequence of educational design commences then with the identification of the health needs.of the population accompanied by an analysis of the existing resources to meet those needs. out of these considerations, discrepancies between resources and needs become apparent. The challenge then becomes the design of methods to meet these discrepancies. Some of these discrepancies can be met by programs in continuing education and training. Often however, the human resources available within a region for continuing education are scarce. Conservation and appropriate utilization of these scarce resources requires close working relationships between all individuals, groups and organizations involved in continuing education in the region. Difficult judgements will have to be made as to which educational programs will receive priority, for all educational needs cannot be met at once. Strong consideration to the - 2 - health needs of the region should be given in setting these educational priorities. Design of Educatio rams The design of educational programs to meet these needs requ ires considerable creative thought. Based on previous experiences, however,, some of the important factors to be considered in effective educational design can be identified. Many educational experiences which have staying qualities are characterized by active participation of the learner in the learning experience. These experiences have also linked that participation to the ultimate focus of the educational process - - care of the patient. The clinical clerkship, internship and residency programs in medicine have recognized the importance of participation. Judged on this basis, the standard two-day program of sequential lectures may not be the most effective mechanism for continuing education. Although health care has become increasingly complex with resulting requirements for close collaboration among specialized personnel, our educational programs continue to be designed in a manner which suggests each health professional is functioning independently. Educational programs designed to meet patients' needs should give consideration to these areas of interrelated function. It is meaningless, for example, to design educational programs for physicians in the functioning and appropriate use of intensive care units without considering the education of the other personnel essential for the unit's operation as well as the availability of the necessary facilities and equipment. It is also 3 wasteful of scarce human and physical resources to carry out such programs where they will not be utilized optimally. Continuing education by definition implies some continuity to the educational process, yet the continuing education of most individual health personnel today is characterized by the lack, rather than the presence, of continuity. The framework of the Regional Medical Programs provides an opportunity for program design which can achieve better continuity. The challenge is to structure programs which relate not only to current educational needs but which take into consideration the previous educational experiences of the participant. Consideration must be given to other factors which have inhibited ef f ective educational activities in the past. such as the problems of time, distance, commitment of available effort to the actual delivery, of health care, financial loss, and established habit patterns. Modern technology offers potential for overcoming some of these problems. , The use of television, computers, teaching machines, or other applications of modern techniques and hardware is being explored in some place-; and many Regional Medical Programs are considering the effective-utilization of these educational tools. The comments in this document about design and evaluation are, however, very relevant for educational programs@ utilizing these techniques. By providing an opportunity to integrate the use of these techniques into a total educational program related to the real educational needs of the region, the Regional Medical Programs can help to avoid the danger that these techniques may be developed in isolation from those needs. - 4 Educational Evaluation Even if the design of educational programs gives careful consideration to the factors discussed, one may anticipate that the resulting programs will not be totally successful in meeting the educational needs. The successes and the failures must be evaluated and analyzed to serve as the basis for appropriate decisions about the improvement and continued renewal of the educational activity. Since resources for continuing eduation and training are scarce, continued evaluation of educational effectiveness is necessary to assure the efficient use of these resources. As discussed above, the ultimate criterion of effectiveness of an edpca- tional activity in health resides in measurements of change in health care. There are many components, however, of this effectiveness, including the success in reaching the desired audience, effectiveness of information transfer, effectiveness in bringing about behavioral change, and the effectiveness of the behavioral change in improving patient care. These factors need to be assayed at each step in the process in order to understand fully the relative significance of their effect on the ultimate goal of improved health care. The manpower resources of those who have competencies and experience in educational evaluation as it applies specifically to continuing education and training in the health professions are limited. One potential resource for advice, counsel, and training is the modest cadre of individuals who have established units of research in medical education in recent years. A resource exists in the colleges and schools of 5 education throughout the country where graduate activities in educational research are being carried out. Although few of these units have had direct involvement with education in health affairs the potentiality of their involvement is very real and should be encouraged. Cooperative Efforts in Educational Programs In addition to ongoing evaluation and modification of educational programs, consideration must also be given to the development of effective cooperation among the people, institutions, organizations and agencies already involved in the education of health personnel. The development, of improved programs requires utilization of their strengths and should, in turn, provide a mechanism for those strengths to expand and grow. Cooperative activity in continuing education and training should become a symbiotic relationship. If possessiveness by any single group occurs or if mono it :ic programs are attempted the benefits of symbiosis will be lost to the detriment of better health care. The necessity of cooperative efforts for effective continuing education is inherent in the nature of our medical system. It is determined both by the requirements of modern medicine and the patterns of our society. The Regional Medical Programs provides a mechanism for cooperative relationships between the medical environment primarily concerned with development and dissemination of new knowledge and the environment primarily concerned with the delivery of health services. Only if both environments are involved and cooperating will the full impact of continuing education and training programs be made on the health needs of the region. 6