*EOOO155* CONFERENCE ON REGIONAL MDICAL PROGRAMS Washington Hilton Hotel, Washington, D.C. January 15-17, 1967 THE ISSUE: EVALUATION OF NEDICAL CARE UNDER PUBLIC LAW 89-239 Paul J.. Sanazaro, M.D. Director, Division of Education Association of American Medical Colleges Chairman, Health Serwices Research Study Section U. S. Public Health Service Consultant, Division of Regional Medical Programs This paper was prepared on request and is provided as background to the MONDAY, JANUARY 16, 1967 Afternoon Discussion Session Evaluation in the field of medical care consists first in collecting information on the operations and end-results of a program, then making judgments regarding the effectiveness and efficiency of the programs or services under-study with respect to both individual patients and com- munities. On a short-term basis, evaluation identifies needed revisions and improvements in an operating program. Its long-term function is to provide a rational base for broad policy decisions governing t e uture directions of such programs or services. When conducted with a high /---N order of technical competence, evaluation may also contribute substantive knowledge to the field of health services research and is then designated as evaluation research. A distinction exists between evaluating A Regional Medical Program and evaluating medical care. Public Law 89-239 and the Guidelines emphasize the delivery of medical care, i.e. the personnel, facilities, services, and resources necessary to improve diagnosis and treatment. However, only in certain limited situations will increasing the capa- biliti6s for delivering medical care automatically assure an improvement in the quality of care. For example, increasing the number of trained personnel or providing specialized facilities and services in areas where these are marginal or nonexistent constitutes, on the face of it, a distinct improvement in the quality of care. In this sense, evaluation of a Regional Medical Program can be directly comparable to evaluating the quality of care. The term I'medical care" has several unique meanings depending on whether it is defined as a process, as a system, or as an area of study. It is also analyzed in different ways depending on whether individual patients, a community, or the entire Nation are the recipients. The following components of medical care are particularly relevant to the evaluation of a Regional Program: 1. @.Supp r availability of health care personnel, facilities, and services, including preventive measures. 2. Utilization of personnel, facilities, and services, including preventive measures, by individual patients or population groups. 3. rocess care: accuracy of diagnosis, adequacy of treatment, and appropriate utilization of consultative resources and specialized technical services. 4. End results: the effectiveness of a treatment or program as determined by the consequences for the individual patient or population, including expressed views of patients and poten- tial patients toward the availability and acceptability of medical care. 5. U-nmet needs: individual patients or population groups with identifiable diseases not yet diagnosed, or diagnosed but not under treatment. 2 In a limited, technical sense the requirements for evaluating a Regional Medical Program in accord with the stated purposes of Public Law 89-239 can be met by limiting the evaluation of medical care to its first component, supply or. availability. However, in order to evaluate the effectiveness of the increased supply of personnel, facili- ties, and services and their improved distribution, it is necessary to include the other components of medical care: utilization, the adequacy of diagnosis and treatment, end results, and unmet needs. The assumption seems warranted that the law was passed with the implicit belief that there would be demonstrable improvement in the care, and in the results of care, of patients with the specified diseases. It appears to be a legitimate responsibility of those conducting Regional Medical Programs to ascertain so far as is feasible the relationships between improved health manpower, facilities, and services and the other defined elements of medical care. As stated, evaluation is a dual process of data collection followed by judgment. Depending upon the particular program or services, evaluation may be carried out at varying levels of precision and sophistication. These levels will be described separately. 1. Evaluation to duetlre-rmine whether the stated objectives of a particular program were met. If the stated objective of a program is to train ten rehabilitation aides, and this is accepted as the only objective of the program, then the evaluation of this program rests entirely on the fact that ten rehabilitation aides were or were not trained. By analogy, this level of evaluation applies to the establishment of specialized patient care units, demonstration programs, diagnostic or treatment ser- vices, and so on. The fact of their establishment provides the necessary 3 s and sufficient information needed in judging whether or.not the objective were met. 2. Obiective description and analysis. For this level, descriptions of education and training programs, facilities, services, and capabilities of personnel are compiled in accord with prevailing professional concepts and standards. For example, a program for training nurses to staff coro- nary care units should be described in terms of the functions nurses will be expected to perform as a result of their training. These func tions will have been defined by appropriately informed and experienced experts. Evaiuation of the training program will be directed at answering two questions: (1) Has the program been designed in accord with generally accepted principles of such training? And (2) Was the program carried out as p annea? Descriptive data bearing on these questions must be collected before a judgment can be made. Similarly, with respect to the operation of coronary care units, the basis of judgment regarding their a equacy is simply an accurate description of the services which these units pro- vide, together with a description of their overall operation and adminis- trationo These descriptions are then compared with prevailing professional and administrative judgments of what constitutes proper staffing, organ za- tion, resources, and administration for coronary care units. 34 Evaluating utilization- tions. The question of whether or how the improved staffing, facilities, and services bring about improvement in medical care cannot be answered without information concerning the utilization of such personnel, facilities, a nd services by patients. Two approaches are possible. Prior to the institution of the 4 program, baseline data can be obtained on the utilization rates of various personnel and services by all persons with the specified diseases in the population served by the Regional Program. If baseline data are not available, a comparison group of patients to whom the new resources are not available must be studied in order to determine that other changes totally unrelated to the Regional Medical Program have not brought about equivalent changes in utilization. Both approaches require the use of epidemiologic methods applied to probability samples of general populations. It is inappropriate both in terms of the overall objectives of Public Law 89-239 and correct methodology to base evaluation on changes in the numbers or characteristics of only patients who receive care. Similar approaches are necessary to determine whether changes in frequency of duration of hospitalization for equivalent disorders or their complications are brought about by the program. Judgment of the adequacy of utilization will rest on two comparisons: (1) between rates per liOOO general population in control and experimental communities or before and after the introduction of a program in the same community, and (2) between utilization rates and known prevalence of the target diseases. 4. Evaluation of improvement in the patient care process. Direct comparisons on a controlled basis are required to determine changes attributable to the program in accuracy and completeness of diagnoses, adequacy of treatment programs, and appropriate referral of patients for specialized services. This level of evaluation encompasses the techniques of the medical audit in office, clinic, and hospital settings. 5 e 5. Evaluation of end results. This level constitutes the d6finitiv measure of effectiveness of personal health services. By use of matched populations, data can be compiled on decreases in interval between onset of symptoms and receipt of-care; end results of care; prevention of com- plications; alleviation or reduction of disability; improvement in social functioning; increased longevity; and so on. Whereas techniques for the preceding four levels of evaluation are well worked out and can be applied in pre-tested form,, the determination of end results is still under research and development. 0 This form of evaluation focuses 6. sis 0 on the efficiency of a program and questions whether the results of a given program or program element are achieved economically in terms of dollars, manpower, time, space, and resources. Competence in operations research and economics is required. Two or more training programs for ol aides might be compared to discover whether comparable skills can be achieved more economically. Appropriate economic bases are neede to compare these programs with training programs which produce fully qualified professional personnel. Similarly, the costs of establishing and operating different types of coronary care units need to be compared in relation to demonstrable improvements in the outcomes of cake given in these units. It is also appropriate to compare costs and staffing economies or the functional efficiency of such specialized units with an at-large monitoring system dispersed throughout the hospital. The critical element in such evaluations is an agreed-upon set of criteria of adequacy for services and end results. Only then can the relative costs be rationally analyzed. 6 asures. This 7. Evaluation of the ef is the most difficult level of evaluation since it attempts to determine' the extent to which,diseases are being reduced, controlled, or eradicated from the population by the application of preventive measures. The use of epidemiologic methods is also essential for this form of evaluation. Evaluation is a sequential process, each step of which must be appro- priately planned and carried out before the next step can be taken. The sequence may be outlined as follows: I. Collection of Information and Data. 1. Specification in detail of the objectives of the programs, services, and end results which are to be evaluated. 2. Establishing the criteria on which judgments will be based. 3. Designing the instruments or records for data collection. 4. Applying the appropriate methods for collecting the relevant descriptive information with minimal bias. 5. Statistical analysis and/or summary of descriptive information. 6. Interpretation and comparison of results against agreed-upon criteria. II. Judgments Regarding Adequacy or Inadequacy of Program, Program Components, or Results. Quality9 effectiveness, and efficiency of medical care cannot be measured directly in standardized units. They can be inferred from one or more objectively specifiable indexes derived from established professional standards. These indexes can serve as the base information or data for judging the degree to which a program 7 or its results meet or do not meet the criteria specified in I.2 above. Judgments of quality are based on consensus of physicians and other professional personnel. Effectiveness and efficiency of a program or procedure.can be defined somewhat more objectively, because data can be collected on effectiveness, and the dollar and manpower investment can be objectively related to outcomes (cost- effectiveness.analysis). However, even under the best of circumstances, evaluation is a difficult and demanding procedure, especially in the field of personal health services. Section 908 of Public Law 89!-239 states that the Report to the President and Congress will include "an appraisal of the activities assisted under this title in the light of their effectiveness in carrying out the purposes of this title." On page 65 in the first paragraph, the Guidelines stipulate that "special effort" is to be made to incorporate evaluation in the planning and operational phases. "Research into better means of accom- plishing the purposes and objectives of the Regional Medical Program" qualified for support in an operational grant. In order to analyze the role of evaluation in the Regional Medical Programs, it will first be necessary to identify the intents and provisions of Public Law 89-239 which have implications for the purpose, scope, level and limitations of evaluation. Within Public Law 89-239 and the published Guidelines, the following major categories of objectives are defined: 1. making available to patients the latest advances in prevention, diagnosis, treatment, and rehabilitation; 8 g more effective distributio and utilization of n all 2. developin types of medical resources; ents among medical institutions 3. establishing cooperative arrangem and professions to overcome fragmentation and insularity and meet the diversity of needs, resources, and existing patterns of education and services; ies through education and 4. improving health manpower and facilit training of health care personnel and demonstrations of patient care; 5. extending the productive interrelationships of extensive research, teaching, and patient care activities to community hospitals and practicing physicians; 6. creating an effective environment for continuing innovation and modification without interfering with the. patterns or methods of financing patient care or professional practice, or with the administration of hospitals. it is legitimate to question whether augmenting existing patterns for the organization and delivery of services will automatically bring about maximum possible improvements in the health of the population in proportion to available knowledge and techniques. The potential impact and the pro- jected total investment in Regional Medical Programs are such that consid- erable effort should be devoted to the development of standardized data on incidence and prevalence of the target diseases in the general population- (As described in paragraph 1, page 16 of the Guidelines) . Furthermore, significant effort should be devoted to analyses of factors which determine 9 the degree of success achieved in improving the delivery of medical care to all persons-who could benefit from it. It is only by using techniques of evaluation which link together personnel, facilities, services, utilization, end results, and cost- effectiveness analyses that an approach can begin to be made to the evalua- tion of the impact of any program on the medical care system and on the quality of care. Study of one component of the medical care system wil not provide sufficient information to make possible wise decisions conc erning needed modifications in other components and links. The evaluation of medical care within Regional Medical Programs must be comprehensive in scope and long-range in perspective. The most productive attack on this problem will result from cooperative efforts by universities and private organizations utilizing the resources of a number of units within the Public Health Service. Evaluation as Operational Research The particular form of evaluation which is undertaken and the technical competence of those who design and conduct the study are essential consid- erations. In addition, failure to properly utilize or apply the results of evaluation will defeat the basic purposes of evaluation, name y, to improve programs and their effectiveness and efficiency. Many circumstances may vitiate evaluation and prevent its effective contribution to the continual improvement of programs. The list o potential contaminating factors is long. It includes such factors as the introduction of undue bias and subjectivity by those administratively responsible for the program; resistance of professional personnel to evaluation; arbitrary restriction of the limits of evaluation; changes in the program while it is being evaluated; use of inappropriate methods of data collection; failure to specify clearly the goals and end results to be evaluated; failure to fusion of availability establish criteria before attempting evaluation; con of services with utilization or with actual patient benefit; inadequate access to or lack of availability of standardized rates for prevalence and incidence of diseases. One approach of proven merit is the establishment of a health services research unit, a form of an operational and epidemiologic research unit, as an integral part of a health services program. By this means, an administrative mechanism is set up for feeding the results of evaluative studies to those who must make decisions governing the day-to-day operations of the program as well as future improvements. Given long-term responsibilities, such units are more likely to develop and maintain records which cumula- tively become more valuable and informative because of the documentation of changes over time. This resource is not likely to be developed when ad hoc evaluative studies are carried out on a short-term basis by con- sultants who have no continuing responsibilities to the program. Even under the most advantageous circumstances, continuing evaluation of health services based on operational and epidemiologic research encounters certain problems with predictable regularity. These will be listed briefly: 1. One of the most important potential contributions of evaluation is the analysis of alternate approaches to the attainment of program objectives. Very often the decision at issue is not whether a particular program in operation is effective but whether an alternate program might be more effective. To base evaluation upon an all-or-nothing answer for an entire program is much less productive than providing alternate program components which can be independently evaluated with respect to their consequences and costs. 2. It may be that the major contribution of evaluative research is to determine whether the traditional ways of carrying on professional practices and delivering medical services are, in fact, the most effective. If arbitrary assumptions and unwarranted limitations are placed upon the scope of evaluation, even though some limitations are always necessary, the hope that continuing experimentation and innovation will lead to dramatic improvements in medical care is less likely to be realized. 3. There are several stages in the evolution of new health care programs, on a local, regional, or national level. Initially, decisions are made and implemented on the basis'of best judgments of those responsible for the program. Aft6r a program has been established, a number of new, unrelated facts begin to influence decisions, but in the absence of an organized and definitive body of data, the administrators of the program require wide latitude in making decisions bec2use factual guidelines are still imprecise. The third phase of such programs emerges when cumulative evaluation, studies, reports, and research have both defined the system and its component parts and related their operations to objectively specifiable effects. In this period, the data base becomes more important in supporting operational decisions than empirical judgments of administrators. Many Regional Medical Programn are in the first stage. It will be some time before the second stage is reached. The third stage can only be dimly glimpsed in the distant future, and will not be reached at all unless activities in acquiring appropriate data bases are promptly established. 4. Evaluation of demonstrations in which the purely medical aspects of the services rendered are assumed to be effective may be based on a 12 false assumption. To the extent feasible, evaluation should concern itself with all the factors that actually or potentially influence effec- tiveness, as it has been defined for the purposes of evaluation. These factors include the reliability and validity of the medical measures of diagnosis and treatment. In settings where such access is feasible, such factors should be identified as the objects of evaluation. If this is not done, programs may be evaluated as highly effective in terms of their operation and costs, although they may not be advancing the actual care of patients. 5. Finally, the question may properly arise whether a particular program is an appropriate one for the area or population to be served. Presumably this decision was made when the particular program was instituted. Nonetheless, it is legitimate to subsume, under evaluation, questions con- cerning the appropriateness of the program in terms of the cultural attributes of the area or population and the likelihood that elements of the program might be applicable to other areas and populations. The methods used must take into careful account the possibility that the unique circumstances operating in a particular program may make it impossible to achieve compa- rable effectiveness and efficiency in other areas. Sourcds and Resources for Evaluation A sound program of evaluation in the field of medical care requires the direct and cooperative involvement of a number of disciplines and competences. Background or experience in medical care is not essential for all contributors in order for them to make substantive contributions; the principles of evaluation can in many instances be transferred from other 13 fields. Many individuals will have to be recruited into the medical care field to make possible the level and scale of evaluation that is called for. Potential sources of professional assistance or consultation include many departments in the university: Sociology, Social Psychology, Economics, Political Science, Business Administration, Administrative Science Educa- tional Psychology. Schools of Public Health ge nerally possess high-level competence in epidemiology and medical care organization. In several such Schools, as well as in several Medical School Departments of Preventive Medicine and a few other university departments, medical care research units have developed well-qualified faculties in medical care and patient care research, health economics, medical sociology, operations research and systems analysis, epidemiology, demography, health services statistics, and medical care administration. The national impact of Public Law 89-239 will best be evaluated through the cooperative efforts of the Public Health Service, other governmental agencies, the individual Regional Programs, and a number of other public and private resources. The National Institutes of Health, the Bureau of Health Services and the National Center for Health Statistics as well as other offices within the Public Health Service have unique sources for medical care research and evaluation. The task of eva uat g the effectiveness and efficiency of Regional Medical Programs calls for the cooperdtiv e effort of staffs of universities, members of the health professions, and of units of governmental agencies. Only then can the requisite talent and competence be mobilized to provide the data essential to local and national policy determinations which must shape wisely the future of medical care for all our citizens. 14