!ill . III *EOO I I( i@1116 iii(ilill Proceedings REGIONAL MEDICAL PROGRAMS National Conference and Workshop on EVALUATION,, 1970 U.S. DEPARTMENT of HEALTH, EDUCATION, AND WELFARE Public Health Service Health Services and Mental Health Administration Regional Medical Programs Service Regional Medical Programs National Conference and Workshop on Evaluation Chicago, Illinois - September 28-30, 1970 Center for Continuing Education University of Chicago PREFACE The Regional Medical Programs Conference and Workshop on Evaluation, h eld in September 1970, marks the first time that coordinators and staff members from all 55 Regional Medical Programs met to exchange views on evaluation and to assess their own activities and programs. A number of factors and circumstances prompted the idea of such a conference- workshop. Most of the 55 Programs were at least three or even four years o . t was a natural time for stocktaking. Changes had been slowly taking place within rograms and were subtly emerging; goals and objectives, and means and methods for achiq, ing these ends, were being examined; and national priorities and budgetary restrictions were leading the Congress and the Administration to scrutinize federal programs more closely than ever. This current of events emphasized the need for greater self-assessment. The impetus for the Conference lay largely in the Regions themselves, and most of the Conference planning and development was undertaken by the Regions. Moreover, the content of presentations and discussions were drawn directly from the evaluative work of the Regions. This fact illustrates more clearly than anything else the considerable strides that Regional Medical Programs have made in the past several years - not only in building up their evaluation capability, but also in putting it to good use. The Conference was significant in its purpose, development and content. Some of the issues posed were broad and generic to the program itself, such as is "change" really the mandate? Others were more specific to evaluation, e.g., how much should be spent on evaluation? Still others were directed to specific aspects of the Regional Medical Pro- at is the Regional Advisory Group's role in evaluation? grams: Wh if there was a central issue posed by the Conference-Workshops, it must, I believe, have been capsulated by Dr. Donald Schon's presentation. If the whole Regional Medical Program is greater than the sum of its parts, those specific activities supported by it - as its proponents have long argued - then the total program must be a primary object of evaluation or assessment. The Conference-Workshops provided few solutions to the great gamut of issues and problems that were raised. It did, however, make more explicit than ever before those questions that had to be answered. That in itself is a considerable accomplishment and an auspicious beginning. Any measure of the relative success (or failure) of a conference such as this one must of course be deferred. Its major impact, its final contr ibution, will only emerge in the actions and changes which will follow. I hope these "Proceedings" will be useful to those many persons who are concerned with, and who will carry out and evaluate, the Regional Medical Programs, their activities and their efforts. This volume itself provides a fair index of the range of both the interests and work of the Regional Medical Programs to date. HAROLD MARGULIES, M.D. Acting Director Regional Medical Programs Service iii CONTENTS Page An Approach to Evaluation for The Regional Medical Program ............. I Donald A. Schon How Others See Regional Medical Programs and Evaluation Alexander M. Schmidt .................................... 21 Peter Fox ........................................... 23 Richard S. Wilbur ..................................... 25 John M. Blamphin ..................................... 27 Luncheon Remarks ........................................ 31 Harold Margulies HSMHA - An Instrument for Improvement in Health Services ............ 35 Vernon Wilson Workshop Sessions WORKSHOP ON DATA Participants ......................................... 41 The Values and Limitations of National Data ................... 41 Charles A. Metzner Data for Ambulatory Care Planning ......................... 44 J. William Gavett Information Systems to Meet Common Data Needs of Health Agencies 46 Katherine G. Bauer The Northern New England Regional Medical Program Health Planning Data Base .................................... ; ...... 51 John E. Wennberg WORKSHOP ON MEASURING CHANGES IN BEHAVIOR Participants ......................................... 55 Measuring Changes in Knowledge ........................... 55 William R. Crawford Measuring Changes in Clinical Performance ...................... 55 Barbara J. Andrew WORKSHOP ON THE EVALUATION OF CHANGING HEALTH STATUS Participants ......................................... 61 Remarks by Moderator .................................. 61 Robert R. Carpenter v Page The Value of Health Status Measures 62 Sam Shapiro ........................................... I . Maureen M. Henderson ................................ 67 Discussion ........................................ 72 How to Measure Health Status Howard R. Kelman .......... ......................... 78 Discussion ........................................ 83 The Relation of Process and End-Result Evaluation 85 Charles E. Lewis ................................... Discussion ................................ I........ 90 WORKSHOP ON PROGRAM EVALUATION -99 Participants .......................................... 99 Approaches to Program Evaluation .......................... Harold W. Keairnes 104 Program Evaluation Workshop - A Case Study ................. Harold W. Keairnes WORKSHOP ON RESOURCE ALLOCATION/ECONOMICS Participants ............................................ 109 Cost-Benefit and Cost-Effectiveness Analyses in the Health Field ......I John Glasgow Role of Social and Behavioral Scientists in RMP Evaluation ....... I116 Michael Zubkoff The Application of Economic Analysis to Regional Medical Programs 1 17 James R. Jeffers Summary of Remarks .................................... 122 John E. Wennberg Social Scientists and the Process of Evaluation ................... 122 Conrad Seipp Cost Benefit - Cost Effectiveness Studies, and Their Application to Allocation of Resources ................................. 126 Robert L. Berg Accountability and Decision-Making in the Iowa Regional Medical Pro-., ......... 130 gram ..................................... Charles W. Caldwell Resource Allocation and the Evaluation Process ................. 141 Charles L. Joiner vi Page Special Interest Meetings S RESEARCH STATISTICAL MODELS AND OPERATION Participants ......................................... 147 A 'Weighted Aggregate' Approach to R & D Selection ............. 147 David H. Gustafson, Pai and Kramer Comments on an Evaluation Model for the Regional Medical Program ... 157 Vernon E. Weckwerth On Evaluation: A Tool or a Tyranny ......................... 159 Vernon E. Weckwerth EVALUATION OF CORONARY CARE TRAINING Participants ......................................... 167 Evaluation of Coronary Care Training: Some Direct Observations of Per- formance in Hospital Practice ............................. 167 Rodger Shepherd Report on Xerox Study of Eleven 'National Coronary tare - Training Centers .............................................. 167 Daniele Deverin Evaluation of Coronary Care Unit Nurse Education in Washington and Alaska ........................................ I ..... 170 Mariella Larter A SYSTEMS APPROACH TO CORONARY CARE EVALUATION Participants ......................................... 175 A Study of Coronary Care Unit Effectiveness ................... 175 M. A. Rockwell EVALUATION OF INSTRUCTIONAL TECHNOLOGY PROJECTS Participants ........................................... 177 Summary of Session ..................................... 177 Cecelia Conrath EVALUATION OF PHYSICIAN EDUCATION Participants ......................................... 179 Summary of Session ..................................... 179 William B. Munier EVALUATION OF MULTIPHASIC SCREENING Participants .......................................... 181 via page Evaluation of Multiphasic Health Testing ...................... @ 181 Donald N. Logsdon EVALUATION OF STROKE - REHABILITATION PROJECTS Participants ......................................... 189 Evaluating Stroke and Rehabilitation Programs: An Overview ......... 189 Charles M. Wylie An Evaluation of a Stroke Program in California ................. 192 Bertram L. Tesman North Carolina Comprehensive Stroke Program .................. 194 B. Lionel Truscott EVALUATION OF CANCER REGISTRIES Participants ......................................... 197 Use and Evaluation of Cancer Registries ....................... 197 Abraham Fingel Alternative Methodologies for Evaluation of Registries ............. 198 George Linden Methodologies for Evaluating Effectiveness and Value of Registries ..... 200 Charles R. Smart EVALUATION OF REGIONAL ADVISORY GROUPS Participants ......................................... 207 Regional Advisory Groups as a Factor in the Regionalization Process ... 207 David A. Pearson Bases for Regional Advisory Group Evaluation .................. 211 Paul E. White and Van Hove MEDICAL CARE EVALUATION Participants ......................................... 219 ABCD Strategy of Patient Care Assessment .................... 219 John W. Wflllamson EVALUATION OF NEW CATEGORIES OF MANPOWER Participants ...................................... I ... 225 Evaluation of New Categories of Manpower .................... 225 Harriet Kitzman viii Page TRAINING FOR EVALUATORS Participants ......................................... 227 Summary of Session .................................... 227 Marion Leach ATTENDANCE BY ORGANIZATIONAL AFFILIATION .............. 229 ix AN APPROACH TO EVALUATION FOR THE REGIONAL MEDICAL PROGRAM DONALD A. SCHON, President Organization for Social and Technical Innovation Introduction Within this framework evaluation serves three distinct The questions in which we are primarily interested purposes: are these: Justification: to defend what's planned or what has * What are the criteria, methods, and measures per- been done. We justify in order to assign reward or tinent to evaluation of the activities of the Re- punishment (as in "grading"), to decide what re- gional Medical Program? sources to commit to an activity, or simply to place * How can evaluation be linked most effectively to an activity on a scale of excellence. In any case, justi- the planning process? fication concerns itself with identifying what has * What are the appropriate roles for those engaged in been done, or what is proposed, and appraising it evaluation at project, regional, and national levels? against some standard. These questions have a deceptively simple ring. They Control: to monitor an ongoing activity in order to raise, in fact, not only the special problems stemn-dng make it conform to standard. from the nature, context and history of RNT but several more fundamental questions of theory concerning the Learning: to change activity, to do it better. Learning evaluation of any activity. may be limited to the selection of means to achieve goals or to conform to standards, or it may en- compass change in the goals and standards them- Section I selves. Toward a General Theory of Evaluation For any program such as RMP, there are always demands Evaluation is an essential part of intelligent individual for justification, control and learning. But it is not and organizational behavior. always recognized that these several purposes have It is the process through which individuals or OTganiza- different implications for'methods and systems of eval- uation. tions perceive the consequences of action, assess their meaning for future action, and reformulate plans and We are accustomed to think about evaluation from policies. the point of view of a rational manager who supervises Action by Individual (Work) @ (implementation in organization) Perception of Consequences by Individual Reformulation of Action by Individual (Planning) (Judging) (Evaluation by and of organization) (Policy formulation by management of organization) the business of an organization or program. The rational formance of the components the information is intended manager takes as his reference point a systems rationale to characterize. - that is, a set of formal objectives, operations for AU variants of the rational manhger's model and the achieving them, and methods for appraising the effec- evaluation systems that flow from it suffer in practice tiveness of operations in achieving objectives. In a from an overriding constraint. Characteristically, systems business firm, the systems rationale makes reference to do not behave as they are supposed to. Even the most profits and return on investment; in the 'public housing bounded organized activities result in social systems that system, to the provision of standard housing for persons do not behave exclusively in terms of the rational pur- cut off from access to the market; in the health care poses assigned to them. As distinct from the rational system, to improvement in people's health, in the manager's model, there is. always a real system of actors quality of care. or in equitable access to care. and agencies which interact with one another in the According t.- - the rational manager's model of evalua- ways they are found to do and with the interests they tion, the systems rationale is fixed and given. Justifica- are found to have. Their discovered interactions and Level 1 Top Management Level 2 oductio ales Finance Level 3 tion consists, then, in assessing the impact of past or interests may have li ttle to do wi th the interactions and proposed activity on established systems objectives. How interests imputed to them under the systems rationale. effective are these activities in meeting objectives? How The "discovered systems" of organizations and pro- efficiently do they use resources? Control consists in grams tend to have certain features in common. Regard- monitoring ongoing activity to make it - conform to less of systems rationale, individuals tend to be in- established standards. Learning is limited to the selection terested in: of means for achieving objectives. o their own survival in their positions; o independence of action; The evaluation process appropriate to the rational o local conditions and needs (as opposed to '.'cen- manager's model depends on the assumption that every- t@al's" view of them); body in the system is to some extent a rational manager. protecting and extending territory; People's accountabflities for activities within the system maintaining stability. are supposed to mirror the systems rationale. These interests characterize the informal, homeostatic structure of organizations and programs. But discovered I Within the organization or program, as within the systems tend also to be open-ended, associated with systems rationale, activities are organized hierarchically. emergent obj ectives and swift changes in goals which Each person is accountable for the activities of his com- correspond to individual interests in creativity and re- ponent, whose goals are keyed, in turn, to the objectives sponsiveness. Often the rational manager's model con- of the system. The job of evaluation is to compare strains creativity, responsiveness and freedom of action accountabflities with the actual behavior of individual in ways that run directly counter to the interests of components within the system. Evaluation tends, then, actors and agencies within the system to become an auditing process in which a third party Within any on-goirig program, the rational manager's assesses behavior in terms of the systems rationale, and model and the, discovered system always co-exist. The sends information toward the top of the system. On the state of their relationship critically detern-dnes the basis of this information, decisions flow downward to nature of evaluation. influence the behavior of the components below. At When the two systems have little overlap and little each successive step of the way, the primary use of infor- interaction, evaluation is lin-dted to retrospective justifi- mation is in justifying and then in controlling the per- cation, 2 Performative Retrospective The System of the Rational Manager The Discovered System stem produces 3. The discovered system and the rational manager's In this condition, the evaluation sy system may fight one another more or less openly until statements believed neither by the producer nor by the of view of the consumer, which are generated ritualistically in response they reach a compromise. From the point to formal demand. Rational managers produce justifying discovered system, this is paying a price. Those in the statements at regular intervals, expressed in the language system do some of what the rational manager wants in of the systems rationale, and resources continue to flow order to preserve considerable abflity to satisfy the into the system. Evaluation processes have no other interests of the discovered system. From the point of output than justification. They are used neither to view of the rational manager, the discovered system is modify the systems rationale nor to force the real social merely distorting system objectives in the direction of its system to conform to it. own 1 terests; but he has to put up with it to get any Where there is little overlap, but the rational manager response at all. seeks to impose a systems rationale on the discovered system, several things may happen: In none of these dissociated cases is there any interest 1. The discovered system may respond verbally in producing or using information that r. without other changes in behavior, by offeringprofonna . the strategy of evaluation as justification. Where the retrospecdv-e-j'-ustificatio.n long on language but short on systems are operating in parallel- but without much substance, a process generally known as "conning." The contact, there is common interest in avoiding informa- two systems operate substantially in parallel. tion that threatens dissociation. In the other two cases, 2. The discovered system may respond to the controls there is common interest in information that supports that the rational manager seeks to impose by adapting to the systems rationale; since justification rests on the the evaluation measures he prescribes but continuing to systems rationale, and resource allocation rests on justifi- operate as much as possible as before. Measures of per- cation. The discovered system is content to generate formance are always different from performance itself. information that conceals how great the discrepancy is For example, in an effort to control'expenditures of the between the goals of the rational system and the be- vocational rehabilitation system, Congress demanded to havior of the discovered system in order to protect the know how many "rehabilitations per yeae' the agency resource allocation they need to continue doing more or effected for a given investment. "Rehabflitations" were less what it is they want to do. defined as job placements lasting three months or more. However, where the whole activity is conceived as a As a consequence, the vocational rehabilitation system learning system, then relationships between rational.and began to "cream" its clientele for those most likely to discovered systems can be fundamentally different from graduate to job status leaving out those who were most those just sketched. The opportunity for learning is in need and least able to qualify; to select low4evel jobs primarily in the discovered system. The discovered for graduates so as to facilitate entry; systematically to system offers the most vital basis for reformulating avoid distinguishing between a "case" and a person, so systems objectives and redesigning systems theory. that a graduate who had achieved job status, lost it and Discrepancies between the rational manager's system and returned to training, could be counted as another the discovered system as perceived by its inhabitants rehabilitation"; and systematically to avoid follow-up become the basis for-progressive modification of the of clients after three months. system's rationale, of modifying the real interests of 3 individual participants, and of developing relationships manager's efforts to control it, may mean that the between the total activity and its constituencies. rational manager is simply precluded from learning It is critical that an evaluation system aspiting to an what's actually happening in the discovered system. But i.mportant role in intelligent management recognize the rational manager may be able to bargain for this rather than bury discrepancies between systems rationale information by exchanging information about resources and the discovered system. The evaluation system itself and ongoing administrative changes to which he is privy must become a vehicle for continuing interaction and for accurate information about what's really happening continuing mutual 'influence of the two. Its ability to in the social system. Even more powerful, when central support intelligent, direct interactions between the rational management gains some freedom to modify rational manager's system and the discovered system systems rationale to take account of real local interests becomes @ central function and a central criterion of and activities, the basis for withholding or distorting in- adequacy in an evaluation system oriented to learning. formation may disappear. The way may then be clear for While these considerations are important at all times, central rational management and local people to bargain they become essential in a period of development or effectively and directly over changes in systems ration- mstabflity, when new kinds of activity must be devised ale, local behavior modification, and information flow. to meet established objectives more effectively and when As in all such cases, the bargaining will depend on estab- program environment changes so as to lead to shifts in lishing and maintaining good faith. objectives, as well. Several additional consequences for the evaluation system flow from these considerations: Learning-Oriented Evaluation Information intended to modify behavior must m Discovered Systetns Hooked flow upward to influence systems rationale as well to Rational Systetns as downward to bring the discovered system into line with pre-existing systems rationale. When planning begins to incorporate a mutual modi- The evaluation information that is gathered should fication of objectives and activities, evaluation includes be limited to amounts, complexities, and preci- much more than mere measurement of the extent to sions determined by the capability and willingness which activities conform to specification. The evaluation of actors within the system to learn from it, as system that is oriented to learning has special features: experienced in actual practice. Nobody in the The 'conceptual framework for evaluation has to system should be presented with more information include a description of the discovered system as than he can handle, nor information laid out in well as the rational manager's statement of systems more precision or complexity than he can respond rationale. This includes a description of key actors to. Analyses should not present actors with a and agencies, actual relationships and modes of greater breadth of alternatives than are real for interaction among them, and the several interests them. As a corollary, the evaluation system needs of all of them. It must include also a description of to be able to detect the changing capability and the real (if informal) evaluation system as dis- willingness of actors to use information, and covered - the information that actors in the should itself be capable of responsive modification system in fact produce, are interested in pro- in turn. ducing, and how they use it. The evaluation process shoul.d be structured to An analysis of discrepancies and overlaps between accomodate the different kinds of learning ap- the systems rationale and the behavior of the dis- propriate to different roles and levels within the covered system. This analysis takes account of the system (rational managers, project pushers, evalua- differing perspectives of actors in the system. tors, planners, etc.). Strategies for responding to discrepancies between the discovered system and the rational manager's The learning objective should also determine the system. Mere analysis is not enough; learning must content, extensiveness, duration, and accessibility be capable of application. of information in die evaluation system memory. These factors focus on gathering accurate information This requirement places high priority on accessi- about the discovered system. The discrepancy between bflity and retrieval capability on behalf of many the rational system and the discovered system, or the different levels Nvithin the system in addition to response of the discovered system to the rational that of the rational manager. 4 Since the learning derived from evaluation may be attei-npt through its own evaluation processes to applied to evaluation processes themselves, the accomplish justification, control, and learning that conceptual framework for evaluation may itself be is downgraded. Accordingly, the evaluation infor- expected to change (sometimes rather rapidly); so mation flowing to central from the local regions information needs to be gathered and formulated normally reflects the nature of the processes devel- in ways that make it more or less equally usable in oped for raising and answering evaluative questions terms of a broad range of systems rationales. Prior- in the localities rather than the answers to any ities should be given to those bits of information specific questions thought up by central manage- that are likely to retain high relevance across a ment. range of manager's rationale and discovered Central also takes on the role of building a net- systems. work learning system, facilitating information- transfer from locality to locality and encouraging specific local experiments. Cases in Which There is No Explicit Systems Rationale What if the activity to be evaluated is itself recog- Section 2 nized as so diverse, diffuse, swiftly changing, and open RMP in the Context of that no overall systems rationale is credible? This situa- Evaluation Theory tion may occur with respect to public problems urgently To place the Regional Medical Program in the evalua- requiring solution but for which there are no clear policy tion context developed in the previous section, some of answers, where national willingness to devote resources RMP's principal characteristics should be recited. to their solution is high, though the credibility of 1. There is no single organization corresponding to proposed rational solutions may be low. Agencies may RMP. RMP is a broad-aimed Federal program concerned be -funded to work on. such problems, constrained only with introducing changes of various kinds into a within very broad limits as to what their work should number of more or less interconnected systems of actors be like. What are the implications here for evaluation and agencies involved in health care. Within these systems? systems, RMP attempts to play a variety of related roles Each region or subregion (or other entity) saddled with respect to other actors and agencies; but for the with a whole problem becomes a center of its own most part it cannot directly control them. RMP does problem-solving process. The number and location not, therefore, have to do with a single rational will depend on the number of centers that turn "system," in the sense used earlier, and its boundaries out to be capable of functioning under their own are vague and shifting. individually developed systems rationales. In this From the point of view of evaluation, this assertion situation the distance between information and has several implications. RMP's scope and turf do not analysis is minimized , and responsibility for have sharp boundaries. We cannot go about analyzing designing and conducting the evaluation process is RMP as though it were a unitary organization, like the very close to the actors who are accountable for Veterans' Administration, for example. And while RMP the activities under evaluation. has formulated broad objectives for itself, its funda- In this case central management's evaluation func- mental activity in relation to these objectives rhust be tion is changed with respect to that of the regions. understood for the most part as "influencing" or "facili- Central management may now impose on the tating" rather than direct control. localities criteria for the evaluation process, but it 2. There is no single, established systems rationale is no longer in a position to impose criteria for either for the health care syste m as a whole or for RMP substantive evaluation of concrete activities. For in particular. There are various rationales, held at various example, central management can still ask whether times and in various contexts by different actors in the regional evaluation processes are differentiated in system. terms of justification, control, and learning; but 3. The larger health care system and the RMP are the central evaluator will accord just as high marks changeable. They are not in a stable state. The character to a region displaying one workable form of and functions of these systems are themselves in process differentiation as to a region displaying another of constant change. Within them, the key actors are form. It is only the region that does not explicitly often unsure of their principal, functions or of how best to carry them out, and they tend to shift behavior as The authorizing legislation made no attempt to they learn and as the system around them changes. rationalize these elements or to resolve potential con- 4. Ne-v-ertheless, as a federal program RMP is locked flicts among them. It was understood by many of the into a structure of controls and demands for justifica- key actors that, as the program matured, the specific tion. At the national level these include regular reviews meaning of its legislative provisions would develop and by the Congress, the Bureau of the Budget, and the clarify. Department of HEW. These demands for justification It is not surprising, then, that there have been per- and for controls over the expenditure of funds are, of ceptible shifts over time in the dominant systems ration- course, passed on to the regional program le@el. ale for RMP, even though no element originally con- The problem of devising approaches to evaluation for sidered as the legislation evolved has altogether ceased to RMP is essentially that of meeting what may well be exert some influence. conflicting requirements for learning, on the one hand, Let us be explicit about an evaluation scheme that is and for justification and control, on the other. The generally accepted as appropriate to one of the simplest vagueness and changeableness of objectives, lack of pro- and accordingly most easily rationalized interpretations gram control over components to be influenced, and of RMP. We refer to the center-periphery regionalization sources of methodological uncertainty all argue for a model based on the diffusion of technology and infor- flexible, process-oriented approach to evaluation-as- mation that is assumed to be stored in the great medical learning; whereas the agents of rational administrative centers. In this instance, it is seen as desirable to judge control tend to press for firm, quantitative measures of the program initially, at both national and regional rogram impact. levels, by its effectiveness in reducing rates of mortality p Like most broad-gauged federal programs, the legisla- and morbidity for heart disease, cancer, stroke, and tion establishing RMP represented a series of compro- related diseases. Individual projects are seen as means to mises among the diverse interests of various concerned these ends, and fall basically into the following cate- oups. The authorizing legislation is, therefore, a kind gories: deployment of new facilities (for example, coro- gr of mosaic of objectives, values, and constraints. Ajnong nary care units); establishment of new linkages between the more important elements of the mosaic are these: medical centers and peripheral care-providing centers Emphasis on the provision of means to improve (for example, exchange of personnel); the development the treatment of the three "categorical" diseases - of new working relationships (for example, changes in heart disease, cancer and stroke. referral patterns); continuing education (for example, Emphasis on the transmission of advanced tech- training of physicians and other medical personnel); and niques and knowledge relating to these diseases. information dissemination (for example, DIAL access). * Emphasis on the method of continuing education The major kinds of evaluative questions under this as a device for this transmission; and on the major interpretation of the'RMP system are these: academic medical center as the principal source of 1. What are the kinds of baseline data and measures expertise. of performance by which the impact of diffusion pro- jects on mortality and morbidity can be assessed? * Emphasis on maintaining or improving the quality of medical care. 2. What is the relative effectiveness and,-efficiency in relation to cost of the various technologies diffused, * Concern with the region as the principal unit of seen as means of achieving reductions in rates of mor- activity; concern, that is, that the program be a bidity and mortality? regional one, with regional centers of activity 3. What is the related effectiveness, for particular throughout the country; concern with recognition technologies and for particular regional situations, of the of regional diversity of problems and resources; various methods of diffusion? This question leads, in and concern with "regionalization" as a process of turn, to questions about the optimal "regions" for diffu- knitting together or building regional resources to sion, the forms of greatest "diffusion impact" for a given realize the purposes of the Act. investment of dollars and other resources, patterns of Emphasis on the establishment of voluntary utilization of new facilities and the like. arrangements among regional institutions as the Other aspects of the activities within the center- dominant mode of program activity. periphery model of RMP - for example, the manage- Specific warning against "interference in the ment of new institutional arrangements at the regional interface between patient and doctor." level - must be judged in terms of their effectiveness in 6 f the quality Of care through The effects of substantial investment in Medicare and leading to enhancement o Medicaid have begun to convince observers that no d technoloaN the more effective diffusion of advance - I amount of investment in payment for care will suffice to with the ultimate effect, of course, of reducing mortality vider system. tified diseases. introduce necessary changes in the pro , and morbidity from the categorically iden There is clea rly need for some forms of interIvention on In the minds of many key actors in Wasliinaton and in the regions, the DeBakey model came to dominate the the provider side as well. r to be overriding objections conceptual climate of the early phases of RNIP. But it There continue to appea was not always or everywhere the dominant ,iew of either to the development of nationalized systems of RMP activity. In the discovered systems of some of the care or to such decentralized solutions as community- regions, regional coordinators and other key actors took based group practice, on a large scale. Shortages of as primary the sorts of changes in institutional arraizoe- scarce resources of medical manpower suggest that rnents which, from the point of view of the DeBakeli) changes in the system will have to work with existing model, figured only as secondary means to an end. personnel and, very largely, with existing institutions. In this interpretation: This means. to a great extent, attempting to facilitate aRjIIP's central concern may be expressed through voluntary re-arrangements,of existing institutions. categorical diseases or @vith the diffusion of ad- Of the available program instruments (Neighborhood vanced medical technology, but RMP consciously Health Centers, Comprehensive Health Planning, Com- concerns itself with overall improvement in quality munity Mental Health Centers), kMP presents itself as of care and equity of access to care. perhaps the most promising candidate for intervention &But these sorts of improvements require changes in of this kind. What RMP has been doing, initially en route the structure and modes of interactio?is of care- to the DeBakey model in some regions or in other providing institutions which no single agency con- regions as a matter of primary though informal agenda, trols - changes that can be generally described as now is emerging as a more dominant (though not exclu- knitting together components of the system that sive) rationale for the program as a whole. It must be are now fragmented so as to permit more effective added, or course, that by no means all regions regard and rationalized planning and action. themselves as primarily involved in systems transforma- #These systems changes are necessary conditions for improvement in quality or equi.ty of care. They tion. Some RMP's still regard themselves as solicitors and must precede any significant improvement along screeners of proposals, and do not yet conceive of them- these lines. selves as "programs" in any sense other than as clearing- In the past year, systems transformation* has be,,un houses for projects. And in nearly all regions, there is the 0 residue of the view of RMP as a conglomerate of projects to dotwnate among competing systems rationales for centering around continuing education, training, coro- RMP (without, of course, completely displacing other views) at national as well as some regional levels. NVhile it nary care units, and the like. At the very least, then, is to some extent a subject for guesswork why this shift coordinators face, as part of the task of systems trans- has occurred, certain factors suggest themselves. formation, the problem of what to make of and what to There has been a movement into good currency of do with the projects initiated under earlier views of certain basic concerns about the national system for pro- RMP. viding medical care - concerns about rising medical Under a systems transformation model for RMP: costs, about the effective exclusion from the health care The primary unit for evaluation becomes the pro- system of large numbers of disadvantaged people, about gram; and since RMP is conceived as an essentially shortages of medical manpower, about the difficulties of regional enterprise, this means the regional pro- negotiating the medical care system even for ordinary gram. It will be necessary to reach both "above" middle class people. this level to the national program and "below" it to the project; but the regional program is pri- mary. *"RMP as process," "RMP as facilitator," "RMP as oppor- Every element of RMP takes on a dual aspect. As tunistic change agent" were expressions heard as early as 1967 we seek to assess projects, regional program and and conveyed the underlying idea behind systems transformation national program, we must ask both about sub- before this rationale became as significant as it now is. Recent stantive changes in the provision of care - changes legislative proposals convey the idea even more explicitly. in the quality and configuration of services, 7 e changes in access to services, changes in h alth - levels of change in health, access to health care, quality and about systems transformation. Seen as systems of care, configuration of health resources, as well as transformation, RMP functions in two ways: changes in the institutional arrangements, interactions through the direct efforts of the regional co- and attitudes cl:a:acte'ristic of the health care system. ordinator and those he works with to knit together The issue of justification raises sharply the problem of or otherwise influence elements of the medical what it is possible to know about these matters, and at care system of his region, and through the.shaping what level of generality it is possible to know it. and selection of projects which become occasions The remainder of this paper "411 be taken up with to effect systems transformation. questions (1) and (2), above. We will focus on the view Evaluation must take account of regional diversity. of RMP as systems transformation and will attempt to The starting conditions of the region, the array of spell out the bases on which, in spite of regional diver- resources, the problems to be attacked, the level of sity and open-endedness,judgments about regional per- development, the regional strategy - there may be formance may be made and learning about systems as many of these as there are regions. From the transformation may be fostered. point of view of evaluation, therefore, the content of regional programs should be expected to be different. There is no 'model" of a regional pro- Section 3 gram to be applied to all regions, although we The Central-Regional Dialogue should be able to develop a conceptual framework There is a conceptual framework for systems transfor- which will allow assessment of diverse regional mation in RMP from which we can derive criteria and models. questions useful in undertaking and assessing systems Evaluation must not only take account of this transformation, without violating regional differences regional diversity; it must also take account of the and without second-guessing particular regional answers fact that regional programs are in critical ways to the substantive questions of medical care. open-ended. The essential elements to which attention must be Regional programs undertake systems transformation paid are these: by engaging the emerging issues of medical care in the 9 Starting conditions (What is to be changed?). region. These are only partly, if at all, within the co- 9 Ends-in-view (Changed to what end?). ordinator's control; to be effective he must use them and o Processes and techniques (How can change be build on them. Evaluation must take account of the accomplished?). open-ended or existential character of regional activity; Broad regional strategies for systems transformation except within a very broad range, it cannot second-guess express directions for the process through which the the issues to be encountered in a particular region at a region may be brought to move from its starting condi- particular time; and it must not impose on the region a tions (as they are conceived in a particular instance) to model of sequential activities independent of the issues particular ends-in-view. Characteristically, such a process of medical care which in fact arise. proceeds in stages of: The central questions of evaluation now become o Diagnosis (getting started, casing the region). these: o Involvement (engaging these individuals and agen- 1. How can we facilitate learning about systems trans- cies whose interaction is taken to be critical). formation, at all three levels, but with emphasis on the 9 Planning and goal-clarification (discovering feasible regional program? processes and choosing and testing specific ends-in- 2. Given regional diversity and open-eiidedness, on view). what basis can we control regional activities or hold These stages are apt to be cyclical rather than sequen- them to standard? tial. The passage from diagnosis through implementation 3. Given the several levels of change relevant to eval- leads to a revised picture of starting conditions, and uation of RMP, how can we go about the justification of through the cycle again. Because several streams of past or projected regional activity? activity often proceed concurrently, the region may at a The questions of justification demand separate treat- given time engage simultaneously in all stages. As the inent. Given the multiple impacts of RMP activity, justi- region moves through stages of systems transformation, fication requires methods for identifying baseline data, in its developmental cycle, it may extend the scope and ends-in-view, and indicators of change at the several depth of the issues it tackles. 8 Given this skeletal view of systems transformation, an All likely emphases have been tried out by the evaluative process oriented toward learning must take evaluator in an effort to test and understand how the form of a dialogues continuin- process of inquiry the starting conditions fit together dynamically. in which the regional coordinator* and RMPS both raise An adequate response constitutes a diagnosis of the and respond to questions. This is for several reasons. regional health care system. It also furnishes the eval- Given the open-endedness of '@systems transformation," uator with some beginning hypotheses about how prospective systems rationales for RMP must be inferred skillful the regional core staff is in casing the region. from (rather than imposed on) regional activities. When well explored and laid out in the dialogues, the Systems rationales and systems activities must modify diagnosis includes the data crucial to workina out strate- one another. The evaluative process must detect discrep- gies of systems transformation, both those which define ancies between systems rationales and discovered health issues and health needs and those which define systei,ns, and tactics for responding to those discrep- the organizational and political character of the health ancies. Moreover, project and program goals shift over care system. time. That is often a si-n of progress; and the evaluative C, 9 What is the character of the principal health prob- process should help to discover whether it is, and in lenis of the reaion? What is their distribution? appropriate cases both reflect and encourage it. z' o What is the character of the present configuration In what follows, we list guidelines for the kinds of, of health care facilities and resources? What is the questions to be raised in such a dialogue; criteria for nature of the health care delivery systems that are systems transformation, from which these questions dominant in the region? flow; and, in some instances, illustrations of response. 0 What are the patterns of access to care among the One test of the dialogue is that both coordinator and principal population groups? central" become able, on the basis of it, to form con- The foregoing questions aim at establishin- starting tinning, grounded judgments of regional program per- C, formance. A second test is that as a by-product of the conditions at the level of health, access to care, and re proficient at configuration of care-providing resources. dialogue the coordinator becomes mo designing and carrying out the process of systems trans- Who are the key actors and powers within the formation. A third is that the national staff is enabled to health care system of the region? How do they formulate progressively more adequate "systems ration- relate to the power structure and to the politics of ales" for RMP. the region as a whole? The dialogue follows what we have identified as the What is the nature of the linkages, the relation- three main elements of systems transformation and, ships, the patterns of referral, the tensions and within them, the stages of development. conflicts, among these key actors? 1. Starting conditions. The coordinator should be What do the central actors perceive as the major capable of articulating a regional diagnosis which holds issues of health care for the region-whether these water, at7d which provides the basis for the formulation are identified in disease-specific terms, in terms of of directions of systems transformation. access to care, quality of care, or in terms of costs, The subject has to be probed to the point that both manpower, patterns of dominance and distribu- participants in the dialogue are convinced that: tion, or other facets of the health care system? * The evaluator understands the spokesman's view Ths next set of questions aims at an understand Iing of of the region and has stated enough of it clearly the "political" forces that can be used or that must be enough to reassure himself and the spokesman. dealt with in any strategy for systems transformation. * The spokesman has stated whether he believes this Out of responses to these questions come regional particular array of starting conditions is tough, average, or a bit simpler to deal with than average diagnoses which provide the material for designing (assuming for the moment the accuracy of what strategies of systems transformation for the region. the spokesman has said). At the point of establishing agreement on starting conditions, the evaluative dialogue has to involve: o Feed-back to a widening circle. We will use "regional coordinator" as shorthand for those o Testing the perceptions of those who first describe agents involved in formulating and carrying out RMP strategy at starting conditions, strategies, or other aspects of the regional level. RMP and the territory in which it functions. 9 Some appraisal (i.e., development of a more or less RMP has to make its way among a number of giants, acceptable description) of the way the local RMP all zealous defenders of quality medical care, each with went about data selection and gathering. its own tradition of constructive innovation, each with Gradual clarification, through the dialogue itself, its own considerable institutional inertia and sense of of the specifics on which detailed information is independence. needed. Y Region. In the region's largest city there is one The following are excerpts from regional diagnoses large medical school and one large community hospital. which illustrate something of the variety of starting The region consists of five quite different counties. conditions to be discovered. Three counties made common cause with RMP from the outset. Two are left. In one, a private physician has his X Region. X is a prosperous, relatively homogenous own comprehensive health plan; prepaid medical care society. Good medicine is practiced here, and the profes- has been attempted under his auspices; success is be- sion is in relatively good repute with the local political- lieved to be uncertain; critics prophesy failure. The other social establishment. As yet medicine and the other county is simply cut off and disinterested. It is difficult health professions are facing only tentative questions to get medical or consumer representatives from either about the "relevance" of where sub-specialization and county even to meet for reasons that pre-date RMP, but bigger-better hospitals take us. But something very real is embrace it: several of the major counties are joined in brewing in the state leoislature's effort to force a 0 uneasy alliance, with many rivalries, all felt particularly "Family , Practice" Department on the distinguished strongly in the smaller cities. specialists of the University medical faculty. Additional Z Region. The major hospitals and associated medical intimations exist in the reluctance and opposition of the schools are all in the major city and dominate the region. Academy of General Practice to the way the medical These are set against the smaller community hospitals, faculty had first planned to go about teaching family each of which in turn is trying to be a medical center. medicine. Not surprisingly, there is relatively thin patient use of Layer on layer of competent, skilled, devoted people these expensive facilities in suburban hospitals. Not sur- working in hospitals and other health care institutions all prisingly, too, there are parochial and compartmen- over the state, all of which tend to emulate or somehow talized referral patterns disturbed by conflicts among the react or respond to the presence of the internationally several large medical schools and hospitals. There tend to famous institutions: the Central Clinic, the University, be economic and social distinctions drawn between the and Rehabilitation Foundation. There is an apparent largest and the other medical school complexes, though shortage of manpower willing and able and wanting to these may be decreasing, and certainly keep changing. perform health care services on the level of ordinary care With all, the distribution of physicians to patients is for ordinary conditions. Town-gown issues are real, but highly inequitably spread over the region. because "gown" somehow includes Central City as well 0 ghetto areas: 1/3000 to 115000 as "The U," and because "everybody" was trained at o center city: 1/200 "The U," the issues take a special form. Centralization * suburban: 1/700 to 1/800 of the Clinic and decentralization of the University 0 rural: 1 /I 000 to 1/2000 complicates their association, whenever joint commit- The 5 medical centers have limited goals. All are ments are required or contemplated. Good acute care under great financial pressure, pressure relative to in- general hospitals are a dime a dozen, and coming to view come, to student load, and pressure to pay attention to one another as competitive whether they are or not. the ghettoes. They are beginning to believe that is where Many are trying to become referral centers both in big the money is. In the meantime, the cultural institutions specialist consulting staffs and many high technology of the major urban center continue to tend to turn services. inward, there is very little that can happen "unless you Generally the establishment, medical and non- own it." So the tendency is rather stronger than average medical, exhibits a touc-,h-minded, "show me" con- to want to turn RNIP and training dollars to the enhance- servatism, tempered b@, a very active consensus and ment of existing institutions and departments. willingness to try out credible ways of improving the Rivalry conditions all attempts to regionalize or situation (e.g., 40% of X-State private physicians have otherwise bring about constructive associations between tried out group practice. They and their patients like it people in the somewhat depressed cities of the North well enough to stick with it.) and the rich primary city. 10 2. Preliminary strategies. - Proponents of the re- both for changes in the delivery system and for gional diagnosis should be capable of meeting chaflen(,es changes in their own position within the system9 as to the accuracy or relevance of their analysis. But the What are the critical "starting issues," and how analysis need be' neither exhaustive nor entirely accurate. might these be used to move toward systems trans- It is of greater importance that it be capable of shifting fori-nation? in response to challenge and that there be, in the inquiry But the specific forms of these questions must Cot-ne undertaken by the coordinator, a continual source of from the regional diagnoses, and must elicit the ways in challenge to be met. In particular, it is important that which prelin-Linary strategies address themselves, or fail judgments about major issues of health need, quality of to address themselves, to the issues raised in these diag- care and access to care, facilities, manpower, cost Of noses. care, and the political and organizational structure of the The following are examples of some of the prelimi- health care system, all be subject to the continual test Of nary strategies emergent from the fragments of diagnoses the multiple perspectives of key actors in the health care listed above, and questions that the evaluator can or system. Where important conflicts of perspective arise, should raise about these strategies, to push the dialogue they should be confronted explicitly and actively. Where a step further: they cannot be resolved, these conflicts of view them- selves become issues for continuing work and inquinr. XRegioti Based on the regional diagnosis, the coordinator should have formulated preliminary directions of The primary problem is the isolation of many small strategy which reflect defensible judgments about communities, especially rural communities from which crucial substantive issues of health care, issues relating physicians are slo@vly disappearing, and their disinclina- to the political and organizational structure of the tion to collaborate. Corollary to and underlying this is health care system, and key actors and initiators of the past success of medical education in selecting and innovation in the health care system. training physicians to want to work in sophisticated hospital settin-s, thus creating strong impetus for hospi- While the coordinator should be capable of ar,,W'ng C, 0 tals to compete, even within communities, and to attract for these directions of movement, on the basis of the physicians by offering ever more highly differentiated regional diagnosis, these preliminary views about strat- and costly services, without careful, credible investi- egy should remain developmental, in two senses. They gation of community needs and how they are satisfied. should take account of the issues they do not address, The function of RMP should be (and is) through and there should be some thought as to the means by which these other issues may come to be addressed. And projects, membership on advisory committees, and core- they should be responsive to changes in the regional staff activity to facilitate connections,and collaborations diagnosis which come to light in the course of RMp among elements of the medical care system, particularly activity, among small communities and particularly among physi- The basic question is "How have you gone about cians. The connections and collaborations should be formulating preliminary strategies for systems trans- multiple and small-scale, so as not to ruffle too many formtion?" feathers. * Through what process have you gone? I So RMP, for example, should serve as broker and * What is the substance of the strategy as so far supplier of seed money for the merger of hospitals in developed? adjoining rural market towns; should support short-term * Why this far, and no further - or why so far in in-residence programs for GPs at the Clinic; shotdd dot this direction? coronary care programs around the State; should Often, the best way of gettinc, at these issues in the promote outreach programs from the Clinic and the dialogue is through questions such as these: University; should use the RAG and its committees to 0 Where are the outstanding strengths and weak- involve all elements of the medical care system and nesses among key agencies and actors in the medi- representatives of its consumers, in order to connect cal care system? small communities with one another and with the 0 What are the patterns of alliance and conflict, and centers. how are these changing? The object is to build larger movements toward colla- a For key actors in the system, and for the issues boration and more ambitious ends-in-view from the they regard as critical, what are the ends-in-view success and the fallout from many small-scale efforts, in 11 the process of learning what is feasible and helping the The coordinator should have found ways of includ- various interests and groups involved to assume as con- ing actors and eleniei2ts of the region's medical care structive leadership roles as possible. system identified as key in the regional diagnosis; Some questions: where some of these cannot be included at the outset, Will the small-scale collaborations ever get big the problems about their inclusion should be explic- enough to make an impact on medical care in X itly confronted and strategies developed for over- Region, and will they happen so slowly that one is coming these problems over time. forgotten before the next happens? What is the "Inclusion" may. be indicated by participation in a threshold level of scale and pace for facilitation if range of RMP-related activities, including involvement in it is to have a building effect? RMP coi-nmittees, in project work, or in ventures initi- Have you taken into account what needs to ated or supported by RMP., The difference between happen in order to get the Clhiic and the Univer- significant and pro fon?7a inclusion must be resolved by sity really involved in the medical problems of the tests that vary from case to case. smaller communities? How much "involvei-nent" do you want and why? Can you do that without What is to be appraised includes: confronting the "family practice" issue, and help- Whether there has (or has not) been a real attempt ing to attain a viable resolution to the conflict to arrange for specific people to be included in among the Academy of General Practice, the Uni- RMP. (Was the labor union representative really versity medical faculty department heads, and the invited to RAG meetings? Did he feel invited? Was legislature? Would sponsoring more activity within there anything for him to do?) the allied health manpower field force or encour- o How well the attempt is related to the co-ordi- age a i-nore valid solution to the general practice- nator's sense of starting conditions and his strategy family practice problem - or just convince the and objectives (which depends on having learned MD's that RMP is against doctors? those things first). How do you propose to respond to the conserva. 9 How explicit the coordinator can be about who is tive stand of many GPs, particularly in southern not to be included, and under what circumstances areas who don't see what RM-P has in it for them, those persons would or should be included. and who feel threatened by or disagree with what o How much the coordinator and core staff learns they hear? about the process of including people from the o What stance will you take toward groups currently experience of doing it. (If they had it to do over left out of the strategy - for example, hospital would they do it another way? Are they increas- a dministrators, dentists, mental health practi- ingly imaginative and increasingly direct in their tioners? Are there parts of the State in which it approaches to people?) would make sense to do so? o The ii-npact on others of the co-ordiiiator's at- o Does the current mix of efforts respond, at the tempts at including people (clumsy or skilled, level required, to the serious problems you have relevant or irrelevant, useful or useless, well identified - i.e., to the problems of rural medi- planned and well understood or otherwise). cine, isolated communities, care for the small but A case in point is the following: clustered populations of minorities, the deficien- cies associated with the (otherwise desirable) pro- Y Regio,z liferation of specialist physicians and the dis- appearance of family physicians, both in the The RMP has taken the position that it is a clearing- central parts of the large cities and in rural areas? house for projects; it solicits and processes applications If you cannot envisage any adequate response in from elements all over the region. RMP is, therefore, a flrst-round activities, how do you plan to build conglomerate of projects; how can it have a program toward such a response? If manpower shortages strategy for systems transformation or anything else? seem to you the central questions about the But there is the sense of need to involve the two response, how do you plan to attack the question counties currently disengaged from the program. The of manpower over time? preliminary strategy has impacted on the starting condi- Often the formulation of preliminary strategies de- tions in a way that permits, encourages, and partly pends upon the involvement of key actors and agencies. specifies a revision in approach. 12 One county, medically under the leadership of a from the interaction, planning, bargaining and negoti- strong physician, has no involement in the RMP pro- ating of the key actors. grarn. And there are 250,000 people there. The belief in These ends-in-view are the specific rearrangements the county is that the big city always wins, and that's sought in systems transformation. They, too, have many where the money is. qualities that are subject t o evaluation. The emphasis, In spite of its apparent role as a "clearinghouse for again, is first to discover what attempt has been made to projects," the @NIP turns out to be operating on a strat- identify these qualities, and to deal with them. Evalua- egy which says, "Get every major actor and every tion of specific content makes sense only after its clear county active in RMP." Their tactics are based on this and more or less agreed what has been attempted, and strategy. the context for attempting it. The major physician in the isolated county is con- The following are examples of appropriate questions: cerned about diagnosis of cancer, and about the Have the issues earlier identified as crucial in the 100-mile round trip required to get specialized diag- region found their way into the formulation of nostic screening in the large city. He is encouraged, ends-in-view? therefore, to propose the establishment of a diagnostic This is an illustration of what such a list of issues center in his county. might look like: Some of the relevant questions, especially appropriate Guidance to get people into the health professions. to -early involvement phases: - Coordination and involvement of the voluntary 1. Is the investment worth it? How much does it take agencies. to "purchase" involvement? as a percentage of the over- - The urgent need for dental care in the north. all budget? compared to the costs of confronting other - The lack of out-patient care centers except for urgent health care issues? Are there other excluded or emergency rooms. isolated elements of equal importance (geographical - Essentially no preventive medicine is done in the areas, professions, voluntary associations, health depart- State. ments, medical societies, hospitals, or a combination)? - Too many community hospitals trying to become What are the potential future consequences (enmity, medical centers. retribution, etc.) of failing to try to involve somebody -- There is no weekend and almost no night-time now? How does an effort to include Dr. H. relate to the medical coverage now in a major rural county regional diagnosis? area." 2. What are the signs that investment has been sue- Is the RMP engaging some of these issues through the cessful in involving Dr. H and his county? How do you deliberations and interactions stimulated among distinguish pro forma from significant involvement? For elements of the health care system? "Engaging" means, example, visibility at RMP meetings? Attitudes of Dr. H. here, facilitating the formulation of ends-in-view and toward the proposals of others? Willingness to permit strategies adapted to them. some "teaching days" in the area? Other projects coming 9 Certain general criteria cut across regions and out of the county? Willingness of Dr. H and others in the across possible activities within regions. Questions county to lend voices in support of RMP activities? Will- about "relevance". of particular activities apply not ingness of Dr. H. to share his emergent strategies for only to the match between ends-in-view and development of medical care system in his county, or to judgments about issues, but to the need for some participate with others in formulating such strategies? attention to these criteria. 3. Ends-in-View. - Out of interactions of key actors, - Costs of care, particularly for hospitalizatiotij ends-in-view should have been established. These must extended care, and costs as experienced by confront at least some of the key issues earlier identified lower- and lowe.r-middle income persons as wet as crucial in the region. On the level of substantive as others. health care, they must confront at least some of the - Quality of care, and the distribution of quality constant health problem themes, or emergent issues in of care across the region. health care. - Access to care, and equity of access to care, At a zone in time, attention shifts from the problem across socio-econon-dc strata, minority and mal of "getting all the key actors active in RMP" to the prob- jority groups, and geographic subregions. lem of formulating the more specific ends-in-view and Have the processes making for inclusion, discussed the strategies for achieving them which are to emerge earlier, extended beyond formal membership in 1 3 RMP activities, to formulation of ends-in-view and - An outreach center, as a way of involving strategies for achieving them? major hospital and medical school in the problem,5 How are priorities formulated?* Are priority issues of an adjacent ghetto? Who will make it work@ being confronted explicitly at all? By whom? Do Who wants it? priority considerations enter explicitly into the - A joint coronary care project as a way of deliberations and interactions of elements of the encouraging collaboration and rationalization of medical care system, or are they handled by the planning among a set of community hospitals? coordinator or core staff alone, or ostensibly or What will make it transcend its original focus? really left to Washington? If there are conflicts Questions about such strategies will focus on a number among elements judged to be crucial to the region of dimensions: - for example, conflicts between major hospitals Adequacy of scale of the "solution" to the and medical schools, between town and gown, problem. between professional providers and representatives Feasibility of the methods proposed. of users - are these conflicts allowed and en- Appropriateness of the strategy to objectives on couraged to enter into the fon-nulation of multiple levels of the activity (e.g., substantive priorities? Does the coordinator intend to attempt health impact, as well as systems transformation to build clusters of these elements into working ends-in-view; clarification of ends-in-view as well as groups, through explicit confrontation of these involvement). questions? If he is not doing this, is it a matter of Appropriateness of the strategy to the constraints deliberate intent? Is he working - temporarily, or and problems perceived to be underlying the issue. as a matter of continuing strategy - on a model of One of the questions to arise at this point is the compartmentalization, in which conflicts over question of "teeth." Is the issue one that will yield priorities and ends-in-view are not allowed to come best, or at all, to voluntary involvement on the up, except within limited subsets of elements? Is part of the key actors concerned? Or does it he "sub-regionalizing" in this sense? If so, does it require some forms of sanction and complusion? make sense to do so? This is a question of ideology, strategy and legisla- Is conflict of ends-in-view being handled as a tive mandate for RMP, as well as of propriety: matter of "dividing up the pie" among competing possibly some other agency is more appropriate. actors, or is there also an attempt to relate such Where the focus is on learning, attention will go not only judgements to shared judgements about the to questions of this kind but to questions about the urgency of health issues, or about the usefulness of. ways in which the development of strategies is handled: issues as ways into systems transformation in the - Is there evidence of the active consideration of region? alternative ways of achieving the same ends-in- Major themes of RMP activity should be developed view? and stated. These should be iiot merely a reflec- - Does the deliberation over strategies carry with it tion of what is con7moii to ongoitig activities, but consideration of effectiveness of the strategy in a source of guidance for the gei7eration of new relation to the costs of carrying it out, and con- actii,ities. Questioiis of priorities an7ong eiids-in- sideratioii of the cost/effectiveness characteristics view should have beeii confronted, through a of alternative strategies? process iii which key actors iii the region work oii - Are there timetables for accomplishment? How their conflicting interests i?ot only oil the level of realistic are they? ownership of RMP resources but oil the level of - Has there been consideration of ways of deter- substantive health issues at7d strategies. ininin- over time how effective strategies are in How appropriate, acceptable and feasible are the achieving ends-in-view? Tests for their achieve- strategies being developed for achieving the ends- inclit? in-view adopted? For example, Where the focus is successfully placed on learning, the impact of such questions will not be to "grade" the This may be the first time that themes of RNIP activity strategies at this zone in time where emphasis is on the become explicit and that questions of priorities become real development of specific ends-iii-view, but to influence issues (often first stimulated by conflicts over ownership of their development positively, by "accelerating" and limited funds). "enricliing." 14 4. Implementation. - The process of implementation jttcl,,iiients of impact, which can help to guide per- should be characterized by involvement of implementers foriiiance in the course of the activity? in selection of ends-in.-view and strategies for achieving What is the relation of the regional coordinator them; and by a relationship of coordinator or core staff' and his staff to the activity? If it is not their activi- to i@nplenienters ivlzich permits continuing mutual ty, do they have, in relation to it, a continuing modification of strategy and end-in-view and of iiii- monitoring, leariiing-evaluative contact which plenientitig activity. allows mutual modification of the ends-in-view The implementation of strategies toward ends-in-view and the strategies by which the attempt at may take the form of core staff activity, of the conduct implementation is being made? of specific RMP projects, or of the activities of coi-nniit- How compartmentalized is the activity? Is it con- tees or ad hoc groups, under the aegis of RMP. The iiected to analogous activities in the region, or to end-in-view and the strategy may be specific enough to activities which are parts of the same program lend themselves to ordy one of these kinds of activity, strategy, so that both learning and concerted and to a well-defined unit of implementation, or they action may occur, where appropriate? may lend themselves to a widespread cluster of activities. What is the relationship of these processes of For example, implementation to the overall strategies of systems clianc,,c held by the coordinator and/or his col- End-iti-vieiv l@nplet?zentation laborators? Has the coordinator attempted to be To foster collaboration and A coronary care project jointly explicit about these? Is there an effort to relate rationalization of planning granted to the 13 hospitals, them to particular strategies for achieving among 13 community hos- requiring the use of com- particular ends-in-view? For example, to connect a pitals. mon facilities. particular activity as a feature of a "master plan ; To encourage multi-level Brokerage functions by core to identify a particular negotiation as part of an collaboration between two staff, RiNfP support of one overall strategy which seeks to involve key actors hospitals in adjacent rural hospital staff member in a process of negotiation over their interests and communities. charged with working out conflicts in relation to the system of medical care. details of the merger. Is the coordinator able to use the experience of To increase the "power base" particular activities to learn from or to influence of the medical community A series of projects, funded in "on the other side of the that area, linked to major his overall strategies of systems change? mountain." medical institutions. There is one side of the question of impact which Brokerage activities. Use of should be treated separately here, because it involves the RMP committees to estab- impact of the process of implementation, which can lish relationships crossing reflect back both on the formulation of particular ends- the mountains. in-view and on the region's capabilities for carrying out further systei-ns transformation activities. This is the Some of the relevant questions are these: process through which the definition of accepted ends- * Are initiators and leaders of the activity aware of in-view may shift. the ends-in-view, and the processes leading up to 9 The connections established and reinforced in a their formulation, on the basis of which the particular activity may lay the groundwork for activity actually came to be undertaken by RMP? new forms of collaboration, e.g., the joint planning o What are the patterns of access to resources of a coronary care unit which leads to joint plan- required for implementation" Is there a basis for ning of a range of common facilities; the diagnos- judgments to be made, on a continuing basis, as to tic screening project in a County previously cut the adequacy of resources to the task? off from the medical system of the region, which o Is attention given to the possibility of shifting leads to a series of boundary-crossings. Are these definitions of ends-in-view as more of the reality things happening? Are there attempts to make of the discovered system comes to light? Is the them happen? project or activity leader locked into a potentially Learning from an implementation process can lead stultifying view of what constitutes "success"? to changes which facilitate new processes, e.g., the 0 What constitutes progress? Are there operational cumbersomeness of a process of review and tests of performance, short of more nearly final monitoring can lead to simplifications which make 15 it easier and more attractive for others to enter the The evaluative processes adopted by coordinator orbit of RMP activity. and core staff should be conducive to learning Processes of implementation can display or enable across sub-regional boundaries, so that those development of "role models" which influence the engaged in analogous activities (continuing educa- character of new activities undertaken, e.g., the tion for GP's, for example) can learn from one impact of Jim Musser as broker-facilitator on other another's experience, and those whose activities key actors in the North Carolina region, or of Paul are elements of a larger strategy can interact in the Ward in California, e.g., the influence of the few light of that strategy. emerging medical care corporations in California on similar, varying approaches to medical corpora- 5 . The Developmental Cycle. Regional programs tions. develop iteratively, -if at all. Cycle succeeds cycle, each Questions about impact of implementation, then, growing out of, but resembling, its predecessor. A need also to be addressed to the impact of the process of regional program, seen as systems transformation, moves implementation itself. through its cycle: casing the region, planning and im- plementing. Then through another cycle widening and At this point, RMPS criteria for systems transforma- deepening its rings of activity. The evaluative questions tion in the region take the form of meta-criteria for the of any one phase continue to be relevant; only, new sets evaluation processes carried out in the region. of questions are 'also relevant to established activities, Without specifying evaluative criteria to be used in and to other sets of activities. The process of bringing assessing the impact of implementation on any of new elements into RMP, for example, continues even as the levels of change, RMPS should require that the ends-in-view emerging from earlier processes of such criteria be developed and that they be appro- inclusion begin to be carried out. priate to the ends-in-view and strategies adopted. The most relevant new questions help uncover the * These criteria should not be limited to program- directions of change in the scope and purchase of the matic criteria (e.g., how many nurses trained? how whole program as it moves through successive inter- many calls received?) but should attempt to assess actions of, the process. These questions are of several change at one or more of the several levels of kinds: change in substantive health care. Is the process increasing its scope? * In each instance, consideration should have been Is it increasing in the overall volume of activity, given to the choice of level at which change is as measured by actors involved, dollars assessed, aiming at health outcomes, then at access 1-nobilized, number of separate activities under- to delivered care, and so on. There should have taken? been review of the definitions, test-methods, and measures appropriate to the end-in-view and Is there a widening range of parties involved in strategy involved. interaction and negotiation? Is the level of ag- * With respect to the process of evaluation, the gregation of the parties increasing? For evaluative framework should have been developed example, is the interaction beginning to involve collaboratively between the regional center and clusters of community hospitals rather than in- the implementing agency. There should be an dividual community hospitals? Is the level of openness to modification, through the process of aggregation also decreasing? For exw-nple, are evaluation, both of the implementing activity and individual physicians as well as medical society of the original choice of end-in-view and strategy. representatives coming to be actively involved This openness should be evidenced in the demon- in a way that extends the scope of the program@ strated capacity of evaluative activity to influence the planning of the implei-nenting process, and in Is there an increase in the number of hea t the evolution of the concept of end-in-view and issues engaged? Is there an increase in the strategy during the course of implementation; and coverage of the region represented by those the frequency and pattern of contact between core issues and by die ends-in-view and activities staff and implementing agency should be such as generated? Within each phase, the map of the to make that kind of mutual influence feasible. issues confronted and their location in the 16 re ion should reveal changes of the followiiio expression of interest of the parties approached who at- 9 kind: tended the iiieetiii- cl, As the pro-r@iiii has begun to expand, its emphasis has Regional Location shifted away from the categorical approach. The RAG, Issues which began with 30 physicians, has begun to change x x composition to include laymen. In view of the relative x weakness of other institutions, includin(,, the State Health Depirtiiient, KRMP has moved toward a control- ling position for health planning for the State. Concentration ,it the beginning has been on work Phase I with individual physicians and community hospitals, with an emphasis on education, viewed as the easiest and least threatening way in. At the same time, core staff Regional Location became involved in project-writing for individual hos- Issues pitals, KRMP his now withdrawn from CCU proarams, x x X except for continuing education. However, a similar - effort based on the earlier experience (establishina x XI I facilities, loanina equipment to communities who could - not afford to buy it) is now being carried out for respira- x x x tory pro-rar-ns. Dr. P. now realizes that in his region, which is poor in x x physicians and clear in its referral patterns and which has one medical school and not much institutional rivalry, Phase 2 the provision of continuing education to physicians and others is not enouch. What is needed is the provision of a Is the process increasing in depth and intensity? system of care and appropriate facilities within which - Is there an increase over time in the perceived the fruits of education can be realized. importance, urgency, and ambition of the issues Here, since the structure of the program as a whole is engaged and the ends-in-view formulated? built around the coordinator, the development of ends- - Is there an increase in the connectedness and in-view becomes very much the development of his own "clout" brought to bear or, the issues engaged? views of the issues that need to be confronted and the - Is the level of aggregation of the parties de- ends-in-view adopted. Is the process characterized by an creasing? Are individual physicians as well as evolution of issues, ends-in-view and strategies, which medical society representatives coming to be reflects learnincl,9 involved in a way that deepens the program? -ional diagnosis of the coordinator, the issues The re., We can provide an example of the development oi he takes to be important, the ends-in-view and strategies ends-in-view and strategies in a regional program as it to which he is committed - in short, his own systems be-,ins to go through a succession of cycles: rationale - may shift in response to now perceptions of the discovered system of the region, as regional activities The K Region bring that system into focus. Dr. P., the coordinator, came from a program of This learninc., may take the form of an explosion of continuing education in the one large medical school, a "rational" plans for the building of the health care sys- program of continuing education for GPs which, by his tem, by contact with the political interests and powers own present view, was not too successful. He began by of the real-world actors in the system. It may take the seeing the creation of RMP as an opportunity to expand form of a shift in priorities about health issues, as his own educational program, and ob,tained a planning previously "hidden issues" - for example, the depth of grant to create K-RMP. He visited local medical societies inadequacy of health care in ghettos - come to the over the region and with them set up a program around surface. It may take the form of perceiving the extent to tumor re stry, coronary care units, and continuing which the needs of physicians and community hospitals I gi . ed in "have not" areas are inadequately served b diffusion ucation. Boundaries of the region were set up by the y 17 of the technologies and research findings generated at Judgements about a region's progress in systems the major medical center. transformation may be made on the basis of its ability to In each instance, the discrepancies between systems meet critetia within any given stage of'development,- its rationale and discovered system, at the regional level, rate of movement from stage to stage, given the con- may lead to the refon-nulation of regional diagnosis as straitits under which it is operating,- and the level of. well as of ends-in-view and the strategies corresponding scope, depth and learning evidenced by its overall cycle to them. of development. It is not reasonable to set uniform standards for the In point of fact, most of the RMP regions are still periods of time within which regions should have primarily involved in the problems of inclusion of key reached certain levels of maturity in their developmental elements of the medical care system in RMP activity and cycles, just as it is not reasonable to apply uniform on the formulation of Ipreliminary directions of move- standards across regions to the time periods within iiient and strategies. In spite of the number of opera- which the various stages of development should be tional projects, most regions are only beginning the work completed. On both levels, the time intervals will vary of fitting projects into strategies for achieving specific with regional conditions. The key factors here are not so ends-in-view. Most are only now at the stage where the much the size of the region as its complexity, its internal formulation of themes of RMP activity and the con- connectedness or disconnectedness, the nui-nber of froiitation of questions of priority among ends-in-view conflicting or disconnected elements within it, and the become feasible tasks. seriousness of their conflicts or isolation from one another. Elements that affect the speed of motion include: Conditions for the - simplicity of the politics of the medical care Central-Regional Dialogue system. Few elements to be connected; few conflicts to be resolved. Having sketched out a iiational-regioiial dialogue - relative weakness of other elements of the system, aimed at fostering learning in relation to systems trans- permit@- RMP to function from the be-inning in formation, there remain questions about the particular dominant or unusually significant health planning vehicles through which such a dialogue may be brought role. to reality and the conditions under which it can be ef- - relatively high degree of connectedness among fective. elements of the medical care system. 9 The two parties to the dialogue must begin with It may be possible to establish a typology of RMP some commitment to and understanding of the goals and regions in terms of their potential for movement, methods of this kind of evaluative process. The require- similarities in strategy, and characteristic types of iiients here relate both to the theory of the evaluative activities chosen to carry out the RMP program. There process and the role of the dialogue within it, and to the are, for example, many instances of efforts to stimulate particular skills and techniques involved in carrying it collaboration among comi-nunity hospitals through their out. joint involvement in some program of approach to 0 Although we have used simple words like "central" categorical disease; to establish outreach arms of major or "RMPS" and "coordinator," the parties to the dia- medical centers; to reach isolated subregions tlirouoi logues will be complex. On the regional side, the dialogue programs using paraprofessionals, continuing education, will be carried on by groups of varying kinds, depending and the secondary support of specialists, Regions and on the makeup of those involved in carrying initiative at subregions differ as to the constraints they put in the the regional level. In one region, it may be a "strong man way of these kinds of activity, but they, too, can be coordinator," his key assistants, and from time to time others that he may wish to bring along in order to grouped in terms of the seriousness of those constraints. involve or educate them. In another region, it may be The purpose of such a typology would not be so the team the coordinator has been trying to assei-nble much to permit judgements of the effectiveness of one out of core staff, certain RAG members, and certain key region against another as to provide guidelines both for actors in the medical care system of.the region. RMPS and for regional coordinators as to the rates of o On the side of the national staff, there is a key i-novement it is reasonable to expect in a given region requirement for continuity of involvei-nent in the and for a given kind of activity. dialo-Lie with region over long periods of time - ideally, 18 over the life of the region's development under RNIP. region are able to establish with one another, and on the The requirement for continuity becomes particularly extent to N@,liich the dialogue isfound to facilitate learn- critical, given the diversity and open-endedness of ing. regional approaches to systems transformation; it is only The dialogue requires a certain frequency of contact out of intimate knowled e of the content of earlier between central and regional groups. Given the rate of 9 stages of development that central can be effective hi movement in most regions, once a year is not often dialogue with the region. enough. Within the interval of a year, too much happens, But, given the realities of life in both central and and too many decisions are made which lock the region regional bureaucracies, continuity of this kind is to be into patterns of activity. Frequency of contact should be achieved not through one man but through small groups determined by the time required for the coordinator to whose members overlap in the course of time. take significant steps, or for the regional situation to From central's point of view, the small group permits shift in si,,nificant ways that mark important milestones the inclusion of the varieties of competence required to in the stages of systems transformation. Intervals are carry out effective dialogue with the region - corn- likely to var@. over the course of the region's cycle of petence to question and respond on issues of substantive developi-nent. For example, contacts might be estab- medical care and on issues of systems transformation, lished around key events such as the first formulation of and skills in the evaluative process of the dialogue itself. regional dia2nosis, the establishment of themes of RMP There will be no need to distinguish the central-regional activities ard the first effort at establishing priorities for dialogue from funding decisions, and, concurrently, to specific ends-in-view, or the first phase of experience in move away from the usual mode of central-regiona-I implementing a specific strategy. Within the range of contact, in which the region displays its wares for central frequency indicated by "oftener than once a year," and central and the region then engage in a game of there should be provision for flexibility increases if a attack and defense. For the central-regional relation to representative of central and the regional coordinator be solely or primarily in this mode prohibits learning, in can maintain contact during intervals between meetings the senses outlined above, and makes it difficult or of central and regional groups. impossible for central even to gain information about The central-regional dialogue offers another perspec- regional activities. tive on the role and conduct of regional site visits, and On the other hand, the dialogue requires that the on the proposed process of anniversary review. RMPS staff be capable of being tough with the region, The central-regional dialogue could become the main raising issues hard enough to be heard and challenging function of the site visit. The site visit team would then the region in the light of findings and commitments become central's party to the dialogue. Such a concept which emerge from the dialogue over time. would answer some of the problems currently reflected In order to make these things feasible, there is first a in regional and central reactions to the conduct of site need to model the roles involved and to set the tone for visits - for example, the pattern of regional display and such a dialogue, and concurrently to set apart and of attack-and-defense which make it difficult or im- formally distinguish the funding-justification process possible to find out what is really happening in the from the central-regional dialogue. The dialogue will region; lack of continuity in the site visit team; lack of surely feed into RMPS judgments about regional fund- feed-back to the region; inability of the site visit team to ing, but should be formally and operationally separate respond to the region by clarifying or modifying from the funding process. central's "signals." There are also significant potentials Will such a distinction be feasible, given the tendency of the site visit as a vehicle which the central-regional of the region to view central as monolithic and the dialogue may help to tap: the opportunity for on-site region's knowledge that funding decisions will be made contact with regional actors and agencies, and the by central? This problem is comparable to the problem presence in the region of persons regarded as peers by of the regional evaluator in establishing his "helping" many of those undertaking regional activities. role, in spite of the fact that his findings will be influ- There is the further issue of the manpower require- ential for decisions on project funding; indeed, the ments RNIPS would experience if it took seriously the problem is central to any process of good management conduct of central-regional dialogues with all of its in which the manager seeks both to facilitate learning regions. The site visit team concept, in which outsiders and to exercise control. The feasibility of the effort will are mobilized alongside central personnel, would provide' depend ultimately on the good faith that central and the a crucial extension of central staff. But the concept 19 would also require intensive efforts at internal training seen, whatever the intent, as a funding-justificati( and team-buflding for the site visit teams. process. The site visit team would then come to play critical role in the anniversary review process, and ti With respect to Anniversary Review, that event would results of earlier phases of the central-regional dialogi have a very different significance if it were to function as would then provide the basis for the inquiry conduct( the yearly culmination of central-regional dialogue, and the judgments made in the course of anniversai rather than as an isolated contact which will tend to be review. 20 HOW OTHERS SEE REGIONAL MEDICAL PROGRAMS AND EVALUATION ALEXANDER M. SCFMIDT, M.D. favorable. It has occurred to me there are three principal Dean, Abraham Lincoln School reasons for our aversion to the subject of evaluation. of Medicine, First, there is the general feeling, expressed over and University of Illinois over to me, that "seat of the pants flying," if it gets you Having corm in late, I was sitting in the back of the there, can't really be all that bad. Over the past decade, room, rather than here on the platform; and I am very through trial and error, in both education and health pleased to have been able to hear the elegant discussion service, we have evolved methods that we think we know on Regional Medical Programs and systems change by to be both good and effective. It is my belief that we are Dr. Schon, I was late arriving this mornin- because our far too content with this type of reasoning. three upper classes are returning this morning, and I met Secondly, evaluation turns out to be hard work, with them - about 625 strong - for a re-orientation expensive, time-consuming and technically difficult. session. This is something new for us. Change now is so Lastly, it is now apparent that evaluation is a great, and the rate of change is so rapid that we are not discipline all by itself, and not many disciples are only orienting our incoming freshman class of 225 available. It seems also true that the discipline is, to students, but are re-orienting students who have been on some extent, quite backward in its development. Thus, vacation. The need for such sessions was made evident application of the discipline is even more difficult. by their questions, I thought as I was driving to the Regional Medical Programs The great importance to meeting. Among the questions asked were: of evaluation was recognized early by the National "Is Cook County Hospital still alive and well?" Advisory Council and Review Comn-dttee. Many of you "How many medical schools are there in Illinois will recall the numerous early messages from the today?" Division about evaluation, and the resulting anguish, And finally, "How many people have you added to frustration and even outright hostility felt in some of the the university police force?" regions. In retrospect, I don't think anyone concerned And my answers were respectively: fully appreciated the three reasons I have given for the "Not very." initial negative feelings about evaluation. 'Ten. During the early years of the programs, the case for and "Plenty." evaluation was argued. A significant amount of research I was also musing that it was only a couple of years ago that I gave a talk entitled, "Is Evaluation a Dirty in evaluation techniques was supported by the Division Word?" The response from the audience then clearly (wisely, I think) - as well as trai'ning programs, indicated that they thought it was. conferences, seminars and the like - all designed to In the ensuing years, however, it has become.apparent provide needed expertise. As a result, while we are much that the wo' better off today than we were four or five years ago, the rd "evaluation," like some other words we are now hearing almost daily, has had the shock value problem still remains. I'd like to discuss RMP evaluation worn off, as more and more people have used the word as I now see it in 1970, from the perspective of a in open public. member of the Review Committee and a medical school It is really too bad that evaluation got off to a rather dean. shaky start in Regional Medical Programs. From time to To go back for a moment to the first of my three time I have tried to figure out just why it happened. reasons for our aversion to evaluation, it seems obvious Certainly from the viewpoint of the administrator (who to me that the trouble with "seat of the pants" flying i@ hopefully is a good manager) evaluation is a very power- simply that technology has rendered it totally obsolete ful friend. Evaluation ranks along with cost accounting except in bush country. Any.one flying a plane and program budgeting (two other dirty words), as one nowadays, almost anywhere, can pinpoin.t his location of the most powerful management tools we have. We all accurately in seconds. And, if he is approaching O'Hare Probably know this, and believe in at least the theory, Field and wants to survive, he must do so, and know yet our response to the word is too often less than how to use the proper technical devices. 21 In point of fact, the methods we have developed by avoided by this ploy, but such tactics do delay decentra. trial and error over the past 50 years in both the educa- ization of project review and approval. Happily, I thin tional and health service fields simply aren't doing the we are now rapidly overcoming these difficulties an@ job, and we must now very accurately and scientifically using Dr. Schon's analogy, I would agree that th determine our position, and plot a new course. We must metapliase is upon us, and the diagram on the board ti assess our education and health service systems, and plan your right really is applicable now, if the nuclea to make needed changes. I'm absolutely convinced that chroi-natin represents the evaluation and review of mos Regional Medical Programs are, as Dr. Schon has said, activities within Regional Medical Programs. the best mechanism that now is available for doing so. I recently have discovered that most regions realiz Since we are trying to make changes in a lot of that the National Review Committee is only a collection "traffic" @ when surrounded by agencies and organiza- of individuals drawn from regions. Several regions hav, tions and individual citizens (often irate) trying to do begun developing their own specialized review bodie@ similar things, I think the O'Hare Field analogy is quite which often for specific purposes are better than th, appropriate. Anyone trying to get a program off the National Review Committee. On two recent site visits, .ground today had better know precisely and scientifical- was provided with sounder, more detailed reviews an4 ly where he's going, how he's going to get there, and critiques of projects than the National Review Corn very importantly, when to land. "By guess and by gosh" mittee has had the time to develop. Some regions havi isn't good enough anymore. And we should reject the mounted their own project site visits, using both thei argument that intuition tells us we're being good or own experts and consultants from other regions. Severa successful in medicine as in flying airplanes. of these project reviews were so good that the Division The importance of regional capability in evaluation is sponsored site visits added little to the understanding o: made evident by the current efforts of the Division to the project or activity. I'll add parenthetically that . decentralize authority and thus enhance regional autono- have noted a regrettable reluctance by regions t( my. We are moving to the anniversary review systerr., to respond to the criticisms of their own experts and reviev local project review and approval and to greatly in- bodies, so that the same deficiencies existed, both at th( creased overall regional autonomy. In theory, this is time of the Division-sponsored site review and the sub very, very good. In practice, there are definite dangers sequent Review Committee and Council meetings. Bu- and problems. of great importance is the growing realization by region! Early in the program development, the Review Corn- of the value of a sound review process, of good project niittee often found that regions were passing the buck to planning, and of good evaluation (of both program an( the Review Committee when theoretically they projects), demonstrated by the willingness to hire oi shouldn't have been doing so. Two reasons were com- borrow the expertise necessary to do these jobs well. monly given for this avoidance of local responsibility: As for the future, I agree almost completely with Dr First, regions were new, and local expertise simply Schon's estimation of what will be important for us t( wasn't available to allow local determination of the value accomplish. Anyone following developments in th( of the proposed program. The Review Cormnittee early health field today, for example, realizes the probability on saw literally dozens of projects with no stated goals, that private medicine is in danger of pricing itself out ol no hope of evaluation and really no hope of accomplish- existence. As one result, during the next few years , ment. Yet, this was the best the region could do at that great effort will be made to control, however possible time, in that particular field of endeavor. This was very the cost of medical care. This may well involve Regiona understandable, and led to the establishment by the Medical Programs. For example, there is currently , Division of the research and training programs men- great rhubarb, which interestingly enough is pitting th( tione,d earlier. American Hospital Association against the AMA an@ More bothersome, really, was to receive a proposal of others, concerning the idea of creating "Professioni much poorer quality than one might expect from a Standards Review Organizations." They represent particular region. This was often justified by the region expanded, more powerful utilization review committees on the basis of political expendience: it would be better It has been proposed that these organizations be estab for the National Review Committee to turn a poor lished by local medical societies, which would then bc project down than for the local program to run the risk charged with evaluating medical care and makini of alienating some faction. I'm sure that early in the decisions as to reimbursement for this care. The conflict program, many local fights and much hard feeling were arises over who should have this degree of power. But 22 more importantly, the question to be asked is whether without question, be turned to for the process and the or not county medical societies have available to them expertise to do thisjob. the expertise to do this job. If this legislation passes, I An important key to success in all of these 'things is @vould imagine that at least in some areas, Regional good evaluation. Regional Medical Programs are still the Medical Programs will very suddenly be in the business best instrument our society has created to do all these of evaluating not only their own programs, but also the jobs, and we must develop the necessary capabilities. As extent and quality of medical care delivered in their the action moves to the regions, whether we succeed or fail will depend on how well we manage the tasks. If we region. This should be a sobering thought to a good many of us here today. I believe that our traditional know what we want to do, we also have to know how involvement with the providers of medical care will soon well we are doing it. And evaluation in these terms is the be put to very good use, indeed, as we get more and ordy possible way to manage our efforts. I believe that more directly involved in the problems of quality and the climate is now favorable for evaluation. In recent availability of health care. years we have seen significant fractions of Federal If you have also followed the life and hard times of agency budgets earmarked for evaluation. It has become medical education, you know that while we need many accepted practice in Regional Medical Programs to more physicians, simply graduating more of the same budget specifically for the costs of evaluation. Thanks to type of physicians we now have is not thought a solution Regional Medical Programs and other agencies such as to our health care problems. We are told that our current the National Center for Health Services Research and graduates are not able to solve the problems of our Development, growing numbers have been trained in the health care system, that our curricula are too narrow, science of evaluation. If these experts are not locally and the training base, largely the urban specialized teach- available, they usually can be brought in as consultants ing hospital, is irrelevant to much of community for a time. medicine. Thus, there is now general agreement that I suspect that as we mature as a program, national medical education must be geographically distributed, conferences such as this will diminish in number, and we for one thing. Also, medical schools must assume in- will have regional conferences on evaluation, regional creasing responsiblity for graduate and continuing training programs, and the emergence of the word "eval- medical education, and they must train a variety of uation" as a very friendly, commonly used, everyday types of physicians to practice the profession in totally household word - safe even for young children. new ways, Medical schools must engage more and more in health services research. Finally, the new physicians PETER D. FOX, Ph.D. must stay in the state where they were trained, and be Senior Economist, Office of Management paragons of virtue and excellence. What is common to all and Budget these goals is the involvement of what is now called the "private sector" of medicine. Indeed, what we in I would like to begin by discussing some of the trends medical education are looking for is some way to create in Federal health expenditures as background to under- a brand new education/medical care system out of the standing the context in which all health programs, old separate systems of education and care. including Regional Medical Programs, are likely to be In the past, some Regional Medical Programs have evaluated in the next few years. Federal health expend- looked to medical schools to provide expertise for plan- itures are large. They are expected to exceed $20 billion ning and for projects such as training programs for for the first time this fiscal year and represent over 10 coronary care nurses. I'm convinced that medical schools percent of the total Federal budget. should now be looking to Regional Medical Programs for Many of the health programs were started during the help in creating the new education and service mix, in- 1960's and carry with them the potential for tremen- corporating most or all practicing physicians into a new dous demand for increased funding. For example, some system of teaching, learning and service. Our new 80 comprehensive health centers, funded by the Office graduates, like many physicians now, must all assume a of Econon-dc Opportunity and HEW, are now in opera- lifelong responsibility for learning and teaching, for re- tion, and each center receives an annual Federal contri- newing their own talents and slfflls and those of others. bution of roughly $2 nifllion. Few of these centers can If medical societies or the profession as a whole is given be self-supporting without Federal project funds, and or assumes the responsibility for setting and keeping its estimates of the number of centers required to meet own house in order, Regional Medical Programs will, health needs in poverty areas run as high as 800. 23 Similarly, the Federal Government has supported Most of RMP expenditures are for three types c staff of community mental health centers on a seed activities-support of the efforts of core staff, demor money basis. Federally financed centers now in stration projects, and continuing education and trainin operation provide services to less than 20 percent of the programs. Consequently, the questions that the Office o country. Already, the authorizing legislation has been Management and Budget is likely to ask of RMP ii changed to extend the time limit on the grants from 51 future years will largely be directed towards the output months to 8 years because i-nany of the centers have not of those activities. become self-supporting. Whether these centers will be First, with regard to core staff. Are their activities ii self-supporting after eight years is questionable, and in fact promoting new patterns of medical care? Subsidiar, the meantime, increases in budgets are required merely to tl-ds, one can ask whether these activities are success to support existing commitments. ful in rationalizing the relationships among the variou Similarly, pressures exist to expand the Medicare and organizations in the region that deliver health care o Medicaid programs. Many medical schools, rightly or otherwise impact on the local health care system. RMI wrongly, say they face insolvency if they do not receive should prevent wasteful duplication in training program additional Federal support. The pressures for Federal and health care facilities. Core staff should both foste support of health research are strong. Some people argue the acceptance of new technology and promote new ap that health services research is underfunded. And, last proaches to health care delivery. For example, traininj but not least, I see estimates that Regional Medical Pro- programs for physicians assistants and other types o@ grams requires at least twice its current level of funding nonphysician manpower are now multiplying in an un to be fully operative. coordinated fashion. The problems of the location o@ I will not attempt to project the actual size of the training facilities, training content, career mobility, an( Federal health budget in the coming years. However, it is physician acceptance of new forms of manpower shoulc clear that we must do better with the funds that we are be concerns of RMP. Is RMP successful in achieving already spending. This is the environment in which we solutions to these problems? Similarly, is RMP bringinj live, and it is a considerably tighter environment than about proper coordination among health care facilities' the one to which we were accustomed during the last Has it achieved an appropriate level of coordination witt decade. other goverment programs such as Neighborhoo@ What, then, does the Office of Management and Health Centers and Comprehensive Health Planning? Is ii Budget expect RMP to contribute? The goals of Federal providing a vehicle for physician acceptance of nem health programs in general include improving the health forms of i-nedical practice, such as prepaid group practic( status of Americans, increasing the efficiency with which or improved referral patterns, that may lead to highei care is delivered, and fostering equity of access to quality or less expensive care? medical care. RMP is expected to assist in achieving these goals, and in setting budget levels, OMB must We also expect RMP to fund only those demonstra. assess whether the $97 million currently spent on RMP tion projects, continuing education courses, and trainin@ could have higher payoff if spent on other programs programs that are an integral part of a well-conceive@ such as Comprehensive Health Planning or the National strategy to satisfy the health care needs of the regior Center for Health Services Research and Development. rather than their essentially reflecting discrete and un- We also assess the alternative of not spending these funds coordinated proposals that are simply related to the at all. interests of the persons applying for funds. Much ha! Measuring directly the impact of RMP on the achieve- been said about the diversity that exists among RNW's ment of these objectives is difficult, and one must be Such diversity is comniendable if it represents a response content with proximate measures. These include changes to local conditions and factors. It is less commendable ii in decisionmaking procedures, in decision outcomes, and it devolves from confusion over objectives or how tc in attitudes. For example, RMP should be able to carry out these objectives. demonstrate that it has promoted sharing of health re- Core staff should also avoid funding projects or train. sources in a mariner that contributes to better care or ing activities that the market place is likely to undertake increased efficiency. The commonly used argument that without Federal support. Nor should it engage in activi RMP has achieved better communication among those ties that do not result in efficiency increases or medica concerned with the health care system does not in itself care improvements that are of sufficient magnitude tc justify the current level of expenditures. justify the related expenditures. 24 In evaluating demonstration projects, market criteria RICHARD S. WILBUR, M.D. seem very appropriate. Funds for these projects are in- Deputv Executive Vice President, tended as seed money. This implies both that the funds American Nie(lical Association serve to stimulate new activities that- would not have been undertaken Nvithout RMP support and that the Thank you vor,,- much for the kind introduction @iiid activities are sufficiently attractive that the medical the chance to be here before a grotil) of people in whose market place is willing to support them after an initial work the AMA is so deeply concerned. trial period. The extent to which RMP-generated Now, when I s,@ak for the AT\,IA I Should make it clear that I am spe,-,'%-iiia, for an or-aiiiz@itioii of pricticiii- projects are sustained after RMP funds are withdrawn is t, t, an important measure of effectiveness. physicians, and as such we are concerned primarily with Training programs should increase the ability of the problems of pri@ticiiic, physicians - in the maiitiil health professionals to deliver health services by bringing aspects of the deanery of Heart[) care services - tire them up to date on recent technological developments. people who acttiall\r touch the patient. They should also increase the productivity of the Our major pro,,)!@iiis ire those (,eiieral problems you medical sector, Health professionals should be trained to know so well: use new capital-intensive devices. Physicians should be The manpower shortage ... And I'd like to say this is trained to use new forms of medical manpower. Non- mainly a sliorta@e of frontline troops. If any of you physicians should learn new functions so that they can have followed the development of armies over the last substitute for physicians and thereby pern-dt physicians century or so, know that, in years gone by, if a to spend time on activities that only they can perform. general had I 00,000 troops, he could usually expect that As with projects, one might ask why individuals' courses most of them would -et into the fiolit on the day of the require RMP funds and why they are not supported in battle. the private market place. As you no\@, kTiow, if a general has an army of Program evaluation has at least one function other 100,000, he's Itick@. if I 0,000 of these men actually get than simply leading to decisions on whether program into the fightin-. Or maybe they're unlucky. expenditures are justified. In particular, evaluation What we find in the niedic@il care field is very much should result in redesign of the program. Thus, if certain the same sort of thiiio. Everybody wants to be a consult- program activities appear to be successful and others c' not, the successful activities should be emphasized. ant. There's little reward or recognition for the priiiiir@, Similarly, one would hope for infomiation to improve physician, and he sometimes gets a little lonely when be thinks of all those niaht house calls tic has to make and the functioning of even the most successful elements of all the people who are planning on how he should make the program. The Office of Management and Budget is still more of them. interested in the quality of evaluation at all levels. We So the manpower shortage to us is that of the prac- are interested in evaluation of the total program, of in- ticing physician, although there are many other short- dividual RMP'S, and of individual projects and training aaes as well, efforts. We expect evaluation to lead to assisting or t, phasing out weak programs or projects. While an overall Second, we have a concern about quality of care that cost-benefit analysis of RMP may not be feasible at this has been expressed before. today. time, we would like to see a few clear successes perhaps Third, is the much discussed problem of cost which along with quite a few ambiguous ones. We recognize needs no elaboration before this audience. that there will always be some mistakes and failures in this type of program, although one would hope that And, of course, we have the problem of remembering these would be as few in number as possible. that we are dealing with human beings. Problems of cost and human factors are certainly widespread today. At The health care problems of this country will not be solved simply by expanding Federal programs to support least we know the feelings of our college students, who health services or by increasing the supply of existing are well versed as to their educational institutions and manpower and institutions. RMP should be at the fore- the loss of human factors in some of the larger medical front of promoting the changes required at the local teaching institutions. level to make the health care system and its related tech- The problem pertinent to this meeting is how to get nology more efficient, more effective, and more ac- information to those doctors who, so to speak, are in the cessible to the American people. front lines. 25 We find that what the practicing physician needs and other institutions. In times gone by, clinical research most is help in solving the common problems of com- was done by a clinician, who took an afternoon or mon diseases in common people. evening off, or even went on a sabbatical and did re- Sidney Garfield of Kaiser Permanente, in writing for search. He could then 'use this research in his practice. Scienti.fic Ameiican, speaks of the "slightly sick As research became more complex, this evolved until and the worried well." These people make up the there were two people, the clinician and the research volume of patients that these doctors see. man. They got together at lunch time or shared common We need help in knowing how to see them in the meetings to exchange information. office, and possibly even more so, we need help in keep- It's often said now that we need a third man, a trans- ing these patients well so they don't have to come to the lator, who could tell the research man what the clinician office. And what is even more important,. we need help was doing, and who, more particularly in recent years, in keeping them out of institutions, particularly, of was able to explain to the clinician just what it is that course, hospitals. the researcher did and what it means to the clinician in Being in an institution is not only bad for the budget, terms of his practice. as Peter Fox has just stated, but it's bad for a patient, I think we need a fourth person too, and he is in the and he should avoid it if at all possible, Being in a hos- field with which you are concerned - the communica- pital is bad for a patient's morale. And as many of you tion of this information to the clinician after it is trans- know, it's where most of the side effects of treatment lated so that he can understand it and can use it. Just as occur. important is the communication back to the medical The physician needs help in the prevention of disease. school, of the kind of information that the clinician I don't mean by this just immunization, because we really needs, so that it can be translated, at least at the don't see many diseases these days that are preventable clinical level - not at the basic research level - into the by immunization. Maybe it's because we have im- kind of research that is going to help him do a better munized people so well already. job. We don't need help in delivering more physical We need a two-way street. Let me use an example. exams. I won't bore you with the argument of whether a Many of you know John Hogness, a former Dean of the physical examination is worth the money spent or not, Medical School at Washington. He wrote a very good except to say it's a highly debatable subject - and that I article and gave a superb speech about the time he spent intend to go on getting them. As a good internist, I a couple of weeks filling in for a general practitioner in could do no other. the rural areas of Eastern Washington. But we do need help in the real problems that face us, He's not quite sure how i-nedical care in that com- the things that cause people to get sick and to come to munity fared during the period he was there, or how the doctor's office - tobacco, automobile accidents, much he helped it, but he is very sure that he learned a pollution, the lack of exercise, nutrition - in the inner great deal that was of value to him in training more city, too little nutrition, and in groups like this, too physicians. much nutrition - urban crowding, sanitation, alcohol, It's a tN@-o-way street, with which we need your help, drugs,etc. because we need practical planning - planning for And then, of course, what causes us the most trouble, people and not just for census tracts. is the psychic stress of our day which drives the patient We need to avoid overspecialization in planning. We into the doctor's office. need to avoid the problem we run into when we solve This is where we need help. And as we look at and one problem and, as the old saying goes, we cause two evaluate the ability of RMP to plan, it's not just how others. It's all well and good to solve-the problems of many coronary care units are set up, but by working on uremia with renal dialysis and kidney transplants and to the causes of disease, how many people could be make these procedures generally available, but in doing prevented from ever having to use a coronary care unit. so we diminish the budget available for housing and for The value to the provider is in helping him to take pollution and for the other problems of health care better care of people. And, as I said before, he needs which may be more important to more people. help in dealing with the comi-non diseases which As we solve the problems of keeping the elderly and common people develop commonly. the chronically ill alive, we build up the problems in- Now, there is an historical problem that has de- volved in the population explosion. As these people stay veloped with getting this help from the medical schools alive there is less for the rest of us. Or, if you believe in 26 the theory of population zero, the longer you keep an Senators and their personal and committee staffers, top older person alive by now modern techniques, the longer officials in HEW and through the department's health it is before a new life may enter this world. agencies, and many representatives of voluntary and We must have overall planning, not single-problem professional health association and consumer groups. planning. We also go to a select mafling list of medical and The doctor himself must always be concerned with science writers on major newspapers across the country. his own individual patient. He cannot be concerned with Hardly a week goes by that MAN is not quoted in the whether keeping his individual patient alive is a good press, or on radio or television news. So, the public I thing or a bad thing. He is committed to keeping the represent is far wider than the medical community. patient alive. In Washington, my beat is primarily the political and And, therefore, you must be concerned with whether economic side of health. I regard my role as one of his success and your success will actually make this a evaluator. I watch what is going on, attend hearings, read poorer world for all of us to live in. testimony, talk to dozens of policy makers both on the Our evaluation may not be as sophisticated as those Hill and in the Administration, listen to the reaction of which you have heard and which will be discussed other groups, then when the time is ripe attempt to set further at this meeting, but we will certainly ask: Have events into some perspective for my readers. you in your planning helped physicians to deliver better As to Regional Medical Programs? I joined the care and have you helped the people of this country to magazine during the days of the DeBakey Comniission, live a better life? and began to cover Capitol Hill during the House and If you have done this, then our evaluation of your Senate hearings on Regional Medical Complexes. That planning will be that it is a total success. was the time of the mighty 89th Congress when passage of a new federal program was regarded as the answer to all the problems which plague mankind. You know - JOHN M. BLAMPHIN take one RMP, add water and stir. Voila! Instant health Assistant Bureau Chief - Washington Office care for all. I believe that approach, incidentally, did you Medical World News a great deal of harm. But more about that later. For several years now, I've been covering medical So I watched what went into the Congressional mill meetings and listening to speakers get up and give their and I saw what came out. I've been watching and eval- papers. It always seemed so easy. I figured all you had to uating, and reporting your progress ever since. Evalua- do was to step up to the microphone and say with some tion of RMP takes a simple format for me. I merely look degree of confidence - "First slide please." And the to see what progress you're making toward a single goal projectionist would do all the rest. - the delivery of high quality medical care - the latest So here's my big chance and I didn't bring any slides. medical science has to offer - to patients with heart As a result I stand up here this morning with a bit of disease, cancer, stroke and related diseases. I also look to hesitation, knowing full well that of all present at this see in which ways the means developed to deliver that conference, I know the least about the intracacies of specialized care are also used to cope with other more RMP and the science of evaluation. general health problems. But before I jump into the topic of evaluating Over the past five years, I have performed this evalua- Regional Medical Programs from the public side of the tion by reading your annual reports, by hearing your fence, I thought I should tell you something of what I representatives before congressional conunittees, by do, and how I view my own relationship to what you are talking to RMP officials in Washington, by visiting trying to do. regions whenever I can find the time, and by listening As you know, I work for Medical World News, and reading what others say about RMP - the usual McGraw-Hfll's weekly news magazine for physicians. It is routine a reporter goes through covering his beat. my job, simply put, to tell doctors what is going on in In the course of this evaluation, I have formed some Washington that is important to their practices. My opinions about RMP and health care in general which I primary audience is about 200,000 physicians in private believe are shared by a great many people in Washington Practice and on hospital staffs. these days. To me, the quality of care and the way it is In addition, we go to about 5,000 people around the delivered go hand in hand. One is useless without the country who subscribe or are on our "Freebee" list. In other. It does no good to tune an automobile engine Washington, the list includes dozens of Congressmen and with new points and plugs, and add a fancy fuel 27 injection system, if the car's transmission is shot, and the heart attack has dropped better than 60 percent. I cal tires are bald. It's the same in the health biz. Tu@ng the that delivering health care to people. skills of physicians and hospitals to a high degree of In region after region RMP has successfully brough quality and efficiency is no good if the system through the normally fragmented elements of the health corn which those skills are passed on to patients has broken munity together to talk about the state of health care ii down. this region, to admit that gaps and weaknesses exist, t( It is my opinion and the opinion of others in identify them and then to plan ways to improve thi Washington I spoke to about this before coming out situation. here, that a federal program such as yours which is using For the private sector of medicine this is a tremen. the taxpayer's money, cannot stop at providing the dous accomplishment. For the first time ' in man) physician, the hospital, and other providers with quality sections of the country there are evolving systems oi tools. It must also do what is necessary to see that these health care which are more than the sum total of theii tools are applied to patients. Many of us have the un- parts. This advance may well have laid the groundwork comfortable feeling that there are those in Regional for the development of new health care systems such a@ Medical Programs who feel their responsibility has ended the Health Maintenance Organizations now being touted at the conclusion of a continuing education course, or by HEW, which could never come about without the after the technicians have installed the coronary care change in atmosphere which RMP's have created. equipment. But lest you think I have been completely snowed by Nevertheless, I have seen evidence that you are RMP, I must also say that I believe this success has been moving - albeit slowly - toward a patient-centered goal. spotty, and has worked better in rural areas and smaller communities than it has in highly complex metropolitan About a year and a half ago, for example, Dr. Robert areas. As a colleague of mine said to me at lunch the Headly, a Bowman Gray cardiologist, took me on a tour other day, "The real test of the RMPs is not in being of several small hospitals in the State of Franklin in Western North Carolina. During our visit, to the 50-bed able to organize care where it is unorganized, but to C. J. Harris Community Hospital, the doctor showing us organize care where it is disorganized. And tills," he around asked Dr. Headly if he would look at one of his added, "just hasn't happened." patients who was in the hospital's coronary care unit - Since I have followed the RMP for five years, perhaps staffed incidentally by nurses trained with RMP funds. more closely than other reporters in Washington, I un- Dr. Headly readily agreed and a few moments later in derstand the significance of what you have achieved. I the hallway I heard this exchange: "If you can you'd know also how difficult it is to evaluate this type of better send her on in," said Dr. Headly. "You've gotten groundwork in terms of the usudl morbidity and mortal- her out of a failure this time. But if it happens again, ity indices. she'll probably go fast. If we give her a valve, she's got an But let's speak about the public for a moment. By 857o chance." The local doctor pondered a moment, public I mean just about everyone outside of RMP and then asked "When can you take her?" "In a day or so, the health professions. Included might be the present I'm sure," replied the younger man. He pulled a pad of administration in Washington, the Congress and paper from Ms inside pocket and began making a few voluntary and professional health organizations. How do notes. "You talk with her family and I'll let you know they evaluate RMP? tomorrow, maybe tonight, when to bring her down to The present Administration evaluates RMP in terms Winston-Salem." of national goals. And so far as health is concerned, this I don't know the outcome of that case, but I suspect means using federal money first and foremost to in- the exchange between rural physician and medical center fluence changes in the organization, and delivery and specialist saved a life. I do know that it probably would financing of medical care. It also means spending money never have happened were it not for the North Carolina in a more flexible, non-categorical way. One needs only Regional Medical Program. to read the Ad"nistration's Health Services Ii-nprove- I also understand that after the @4P helped put ment Act of 1970 to get the Administration's present coronary care units in several of the small hospitals in evaluation of RMP. Western North Carolina and also established a mobile I must also say in passing that the Adninistration's coronary unit staffed with rescue squad workers and committment to health care so'far hasn't manifested aided by local physicians - that the mortality rate from itself in much more than rhetoric. If the President is 28 serious about solving the so-called health care crisis, he is schools, voluntary health associations - all of whom are going to have to substibute money for talk-. looking for a piece of the action themselves. Now, to the Con-ress: Here the evaluation has been Now why is there so much cynicism surrounding more simplistic. It also demonstrates how RMP got off RMPS? I think my friend Ed Friedlander would boil it to a bad start. It goes like this: "If you spend all the down to a problem of communication, of providing the 1-noney we appropriate, you're doing a good job. If you facts from which others can make evaluations. don't you must be dragging your feet." As Rep. William Certainly, not everyone can spend time looking over Springer of Illinois said during the hearings on extension the operating projects within one or more regions to of RMP two years aco, "The initial legislative testimony learn first hand what is going on. So, I would suggest was presented before this conunittee to justify a pro- that you consider very carefully how you justify your gram for a billion dollars, which turned out three years existence to your publics. In a nutshell they - and I'm later to have spent S85 million." But he also shows a talking about those in Washington - want to know what great deal of insight into the nature of new programs, you are doing for people, for patients, for constituents. and especially of @NIP's when he says, "I think it ought When they hear you talk in your own jargon of regional- to be brought out here that what we get in the way of ization, of cooperative arrangements, of closed-circuit landslide testimony here is a selling job and snow job TV and other gadgetry, they are going to go away claiming that something can be done immediately." shaking their heads. Maybe you can translate those It has been my observation that the promises which matters into improved delivery of patient care, but they were made for health care at the beginning were made in can't. As far as they are concerned, you're off in some terms of the original DeBakey Commission report. They other world. were not significantly modified as the procram itself was t, Let me give you an example of what I mean. About a modified by the Congress. As a result, the evaluation of year ago during the budget crunch for RMP, the Illinois the promises and potential was made before the RMP program, like a lot of others, wrote letters to its Senators even began. And when you try to evaluate what you pleading for reinstitution of RMP funds cut out by the can't even define - and who in 1965 could define a House, and describing what would happen to the pro- region in understandable language - you get into gram if the money is not put back. The letter was well- trouble. written, telling how the region had been organized, Only recently have members of the legislative and about the progress toward achieving regionalization, and appropriations committees begun to understand the how after months of planning, grant applications were subtleties of RMP. But they too, like the administration, pending to put the program into gear. The letter said expect RIMP to pay more attention to problems of that if the money wan't forthcoming that those grants delivering health care. As Sen. Ralph Yarborough said could not be awarded. earlier this year in introducing his bill to extend the %MP program: "Explicitly, the extended legislation But nowhere did it say what the money would be provides that Regional Medical Programs concern itself used for in terms of helping the people of Illinois who with improving the organization and delivery of all just happen to be Senators' Percy and Smith's consti- health services, and strengthening our primary health tuents. care system." You must remember that members of the House and In the meantime, RMP still has to prove itself on the Senate get dozens of letters a day pleading one cause or Hill. One Capital Hill staff member told me the other another. Most of these are handled by aides and only get day that "Regional Medical Programs have failed to take a cursory review by the boss. If you had been sitting in a on broadened responsibility in health care and have hung busy office on Capitol Hill reading that letter, knowing tenaciously to heart disease, cancer and stroke labels.,, nothing about the concept of regionalizatidn and caring True, you have pointed out that without the disease less, what would your evaluation have been? categories, medical center specialists may be less likely To me it boils down to this: If you communicate the to participate. There may be some truth to that. But proper information, and by proper I also mean that it be many on the Hill read it as a cop out and as an attempt honest, and communicate it in the context which your to maintain the status quo. audience can understand, that is patient care, the Individuals within health associations in Washington evaluations you get will more likely approximate the are also skeptical of RMP progress. But these, of course true state of RMP. You will, of course, still have critics represent vest interest groups - hospitals, medical who will say that only total federal control of health 29 services can eliminate the gaps in delivery and quality we program in health care in the eyes of the administration, now face. of the Congress and of the public, those actions in my estimation are going to have to be directed more In the long run, you will be judged by the changes and more toward improvement in the quality of the that occur in the quality and delivery of health care system through which the best medicine known to which result from your activities. And in the future, for science reaches the patient where and when he needs it. Regional Medical Programs to survive as a major federal At least that is going to be my yardstick. 30 LUNCHEON REMARKS HAROLDMARGULIES,M.D. Acting Director, Regional MedicafProolranis and Service When Dr. Glasgow was introducing me, he talked can't see how they could have been on the moon. My about my task of drawina to(,Cther theIthreads of the TV set can't pull in New York. How could it pull in the morning session as a "herculean task." I remembered nioon?" They talked to a man in North Carolina, and he some of the labors of Hercules. One of them was to said, "You know, if you got on an airplane and went to clean the Aegean stables. I guess you do remember, Ashville and then came back and I saw you again, how don't you? would I know you had been in Ashville?" I was also thinking this morning as I listened to the I'd like to also point out the fact that 1,200 of the descriptions of evaluation and of the Regional Medical nation's 1,700 newspapers carry daily horoscopes - and Proaraiiis, of something that I had almost forgotten a few years ago 90 did. And last year there were 2 about - a flashback to my early youth where I once saw million ouija boards sold - which is the greatest bonanza a picture of a man standin- on the deck of a ship. He in the history of the business. Now, those are just casual was the greatest archer in the world, sort of the modern observations, but they are, at the same time, symp- Robin Hood. He was standing on the ship's deck with tomatic of a drift toward the mystic, toward the intui- the deck going up and down. There was an empty keg tive, toward the doubtful, toward the seeking for floatin I in the water with a little cork in the bung, and solutions which , re non-rational, at the same time that his job was to hit that cork with the arrow. we are trying to look very strongly in rational directions. I had the same sensation when someone was talking When I looked in the New York Pmes this morning about evaluating the Regional Medical Programs, and it the present status of important legislation was listed, but gave me the opportunity as I sat there to decide what as always the Regional Medical Programs were not men- the title of may talk should be, because I frequently give tioned. I think this also helps you to appreciate the talks without titles and then somebody wants one. environment in which we are functioning. I have selected one for this one. It is as follows: "By Aside from these general statements, I think we must the Time I Get to Where It's At, It's Always Where it recognize that in looking at the Regional Medical Pro- Was." Which seems highly reasonable. gram or any other health activity from the evaluative Before I comment on the general discussion this point of view, we have to enter into a game to which we morning, I must say it was superb from every point of are generally unaccustomed. The health profession does view. The thanks should go not only to the participants not characteristically evaluate its own practices or its but to Pete and the people who have helped him put the own institutions. It may do so on an individual basis, but program together. It's off to an awfully good start. on a broad basis, little or not at all. If you doubt that, I would also like to say a few things about the general try to look at your own program sometime, or look atmosphere, sort of overall environment in which we are nationally at what you have available if you want to thinking about evaluation, whether in Regional Medical measure the influence of some health event or the corn- Programs or in other areas. I was particularly charmed munity. And look very hard to see if you can find any by the sense of determination to deal rationally with information that will allow you to say, "Here is where systems which have often been dealt vath intuitively, we stand and here, as a consequence of what we are and the expressed preference for the rational over the doing, is where we are going, and here in retrospect is intuitive, where we have been." It s an astounding fact that those At the same time, I had to realize that there is a drift kinds of data bases do not exist. in this nation, a preference for mysticism over thought- We do not, generally speaking, relate our institutions fulness, which expresses itself in interesting ways. The to the processes which we have been discussing today. Knight newspapers did a -survey not long ago of some We do not relate general health problems to the efforts 1,700 readers to see what they thought about people in which we invest. And we allow ourselves little landing on the moon. Some of them had interesting managing room to set up a conceptual basis for future comments which give you a sense of what at least part of planning. To ask Regional Medical Programs, as a the country feels at the present time. One lady said, "I consequence, to enter into this kind of a process is to be 31 asking for perhaps even more than any of you here into it explicitly. If it is true - and I believe it is - th@ realize. RMP has changed its purposes in the very process ( I think at this point it might be useful for me to serving its initial requirements, then to carry out evalu@ recapitulate as best I can what was said this morning, tive activities on what it has been doing runs the risk @vfiicii, I believe has set the tone for this afternoon. being arc@va'I rather than programmatic. We must ver discussions. There were some general themes, which carefully distinguish between what will be of valu were sustained throughout the entire morning. They historically and what will be of value for the purpose were things I have said to you and you have said to me building a new kind of a program. during the past several months. They included the deter- In fact, if I were to make a generalization about RM mination that this health program, like any health - and we all know that evaluation and generalization ai program his tli@- backing of the federal govern- dangerous companions - I would say that certainly on anent - and probably those with other kinds of backing of the great potentialities of RMP is to create an envirot -- must be ij-1vol\,,?d kN,itl-i i3iil)rovii)- the organization and meiit ai-nong health care providers that will allow it to b delivery of health care. as well sensitized as possible to the indeter@nab] Words like heart disease, cancer and stroke were changes that are coming - a kind of preparation for th brought up eitliei- on a pro forina or on a somewhat apocalyptic - which may be a rather tall order. An wistful basis. This by no means suggests that people are since RMP reaches that far, it has to be thought c no longer concerned with the specificities of these perhaps not too loudly, but at least above a murmur. diseases, nor does it suggest that our legislation or our I also heard this morning that we must be careful no legislative mandate has suddenly changed. What it does to ruffle too many feathers. I suspect my emphasi say is that priorities emerge in a society, and the would be to ruffle the feathers but don't smash the biri priorities which have emerged in this society are being in the process, because I think some feathers will inde,ei enunciated from several points of view - from the public have to be ruffled. point of view, from the evaluative point of view, from I heard something else - the need to look at variou the fiscal point of view, from the provider's point of aspects of the evaluative process, the justification aspect view. We were told that if we are to look at Regional for instance, which from my point of view is a ver Medical Programs and evaluate their usefulness, it must urgent one because I have to defend and justify the be with tile prior determination that we have set some Regional Medical Programs at all times in Rockville value structures, that @ve have a clear statement of what Washington, or wherever I may be representing them. it is @ve wish to be in the health care system. I heard about evaluation. It has something to do witl I thought the major speaker of the morning described methods of controlling what happens, of confonnatioi magnificently the problei-ns involved. He told us that we to standards - standards which are established according to the requirements of the program and which ar( must have some gIoals which are clearly stated, that are determined locally - and of a process which is . cafle( determined locpll\ by the re-ioiis, and that these goals learning in which we perceive new ways of doing thos( cannot he controlled with meaning and with purpose on things which we do, and their relationship to where w( some kind of a central basis. He described to us a pattern wish to go. I was especially pleased to hear references t( of Regional Nlcdical Programs which have their own the need to avoid replacing the objectives which at( special knowledge and their own special issues which being sought with the measures utilized to reach a goal must be introduced into their planning flexibility, and It is so easy to establish measures - and we in the Vehicle must desinii well defined goal-related pro- @,r,,ifni-iiatic is that cat) be evaluated. He said, as did bureaucracy are fully capable of doing that - so easy to others, that sot-ne of this i-nay have to be retrospective, establish measures that describe how to get there, and and that makes me t little nervous, because I think very then concentrate on meeting those measures so fully little of what we can evaluate retrospectively is going to that why we established them disappears. have much meaning when we got to the prospective end I used to see that a great deal when I was working ir point, and a reconsideration of what Regional Medical Asia, where the great game was to have a five-year plan Programs are all about. In fact, the issue of which The five-year plan would contribute - whether it was in direction RMP ",ill take is either air evaluative, a political education or in agriculture or in health - what was to be issue, or a social issue which must be looked at very done from point zero to the end of that point. One Attentively,. this was a little bit vague in the could be very sure that at the end of the first year, 20 presentations 'L 1-iis morning or there wasn't time to get percent of the goal would have been reached, at the end 32 of the second year 40 percent, and so fortli. Any am confident that whatever Regional Medical Programs variation from that was easily corrected by replacing the must do, they will have to do it more rapidly than seems fellow who was doing the reporting. This is one way of at all reasonable. getting where you want to go, but it does seem to There are two other aspects of the evaluative process ei-nphasize the measures more than the goals. that I would like to speak about. If RMP, as you have There also appeared during the course of the dis- heard this morning, is to be as diversified as it should be, c,ussion some reference to the need to examine all and if it is to maintain the flexibility which is one of its alternatives instead of simply taking advanta,,e of the great assets, and if it is to mobilize those providers who opportunities. And again I was thinking a little of what are always going to have to be involved with the delivery might be called a kind of Mae West approach to this of health care services, it is going to do it in a variegated thing. A young lady asked Mae West how she could get fashion. And that's fine. But this presents a great out of the particular dilemma she was in. She said, "I'm difficulty for us in Regional Medical Programs Service. sure I can never do what you have done, which is to find Because while this kind of an activity is going on, there a man who loves me and has $10,000 and who would must also be a sense of coherence, which if not main- buy me the beautiful kind of a mink that you're tained, will make the RMP look like another process in @vearing." And Mae said, "Well, honey, you could think fragmentation and in activities going off in a variety of about getting 10,000 men with a dollar each." You see, directions. As a consequence, I think it is essential that she understood the discovery system and she understood we establish more effectively within RMPS and among the ways in which you do develop alternatives. the Regional Medical Programs an understanding in the I suspect that most of us were thinkin- during the process of programmatic development and in the process course of the morning what all of this discussion about of pursuing programmatic goals, a communication net- evaluation meant with reference to the process that we work which lets everybody know what is going on and are going through at the present time in Regional which gives a better understanding of the expectations in Medical Programs. The process is something which is RMPS, with reference to what represent HEW overall called anniversary review,and will obviously place a very goals. different kind of burden and emphasis on the Regional For me to pretend to you that this government or Medical Programs. If I do nothing else in the process, Iany government can support activities without our own would like to say that I am convinced that what you concept of what those goals should be, and without at heard this morning is so highly consistent with what we least a broad kind of framework in which we will anticipate in the process of decentralizing the RMPIS, function, is to be misleading. that it could easily be played back again to you every Now, it is not likely that at any point we will be so morning for several days to make sure that the message foolish as to direct the RMP's to do a specific number of is clear. things. We would fail in that effort. But I think you need There can be no question not only that F.MP's will be to join with us in the interpretation of what really given the prerogative, but it will be demanded that they matters in this country in the health care system. And establish programmatic directions, and within those pro- this you have heard over and over again. You heard that grammatic concepts, establish projects which specifically people are concerned about the costs of medical care. fit those programs. The core activities will all have to You heard that people are concerned about access to move in that direction. There's no question that this will medical care. You heard that they are concerned - and be the way in which we will have to go. There is also no I'm not sure in what way this is true - with the quality question that there will be a need to evaluate the ef- of medical care. In the public mind quality has a lower fectiveness of that whole process and that the way in order of priority. I think access and cost are far and which you evaluate it will have to be based upon your away the greater considerations. understanding of where you wish to go and what your You also heard from the people who are looking at senses of value are. the evaluative system, where we will have to go and what If there is anything to add to w hat was said this will have to be done, on the basis of what we have. morning, it is that there was probably less emphasis on aSimply flooding more into the system is no longer going sense of expediency than I would have liked to have to be the answer. heard. It came out. It was mentioned. It is that part of You heard a very strong inference, which I join in, the evaluative process that had to do with how rapidly from the Office of M.anagement and Budget, that there gs are to be accomplished. But at the present time, Iwill have to be greater selectivity in what is supported 33 and a readiness on your part to abandon what doesn't There are times to use judgment. There are times I really seem to be working very well. This will entail exercise your own knowledge of what is going on an some risks, but careful risks. You will have to eliminate what needs to go on. And what you will have to do what appears to be ambiguous, and give heavy support establish evaluative techniques which anticipate even to what appears to be a strong direction in which to and then be ready to prove, when you arrive at th, i-nove toward the kind of goals which we have embraced. event, that you have done what is necessary in th Now, if we can manage this variety of activities in such a process of projection. Anything that depends entirel way that we can interpret them coherently in our own upon what is here and now is likely to fail. Anythin defense of Regional Medical Programs, I think we can do that is purely retrospective will surely fail. well. If you have difficulty in deciding where you need ti Now, mind you, I'm saying this at a time when our go or what matters, I think a careful scrutiny of th legislation has not yet appeared. We are really living on daily newspaper is very, very helpful. If you need to gi borrowed time - and we're used to that. We still do not beyond that, it helps considerably to go where some o have appropriations. We are living on borrowed money - the problei-ns are, to talk with some of the people wh( and we're used to that. But regardless of how these are not getting the kind of medical care they wish, t( events emerge, and even if the definition of our legisla- consider the fact that the quality of care is not merely , tion is fairly narrow, you as individuals responsible for matter of considering the exchange between the provider RMP's would be most foolish to overlook the elements and the lucky person who enters the system and get! of evaluation that have been discussed today, and there- quality care. You must also consider that the fine fore the elements of purpose in Regional Medical Pro- measurement of quality care is diluted by factors such a@ grams. those who do not get care, or where the quality is so bad that it is a very large minus. If we came out with legislation that says: "Confine These broad considerations can probably be resolved yourself to categorical programs and within those pro- by a sense of societal concern which has been expressed grams to continuing education," - which for the most wherever I have gone in the Regional Medical Programs. part consists of what I now describe as episodic informa- But there is a difference between one RMP and the next tion transfer - if we indeed are to move, are mandated to in the determination that a bold direction is a good move in that direction and we do indeed respond by direction. In fact, at the present time, the bold ones have isolated categorical projects, there will no longer be a been the wise ones, and in a kind of paradoxical way the Regional Medical Program. bold have been cautious. 34 HSNIHA - AN INSTRUMENT FOR IMPROVEMENT IN HEALTH SERVICES VERNON E. WILSON, M.D. Administrator, Health Services and Mental Health Adi-ninistration Thank you, Harold. There are enough of you who If you say HSMHA or Health Services and Mental have heard me speak before, that no one will expect a Health Administration, the usual reaction is, "What is flowing, eloquent speech that is "snake charming" in that?" I understand that reaction because it was my own capability. when they first talked to me about HSMHA last May. I'm delighted to be evaluated by this kind of group. It Let me give just a precis of the Health Services and seems to me if anyone will do this in an objective way, Mental Health Administration for those of you who may you will. not know what it is. As many of you know, the Regional Medical Pro- Regional Medical Programs is one part of HSMHA, as grams have probably been closer to my heart then any of you well know. The Community Health Service is the other new movements of the federal scene in recent another substantial part. It is a program with a budget of years. This is not a maudlin sentiment. It is my evalua- some half billion dollars a year. Incidentally, this is tion of the promise of this program. where Comprehensive Health Planning fits in. The A substantial portion of that promise arose out of the National Center for Health Services Research and De- opportunity to allow the grass-roots mobilization of in- velopment, which some of you have contacted, is novation and the grass-roots decisionmaking process to another component of HSMHA. The National Center for take hold. Health Statistics is another. In that context there were several kinds of problems In the newly established family planning endeavor, with which you've been struggling over the past few Dr. Louis Heilman is setting policy for the Department. days. I'm not going to treat anew the things you have Dr. Frank Beckles, as Director of the National Center for been talking about. But among them, of course, has been Family Planning Services, has most of the adn-dnistrative the continuous struggle between the two polar tugs. One responsibilities in HSMHA. of these is to give clearcut guidelines so that people The Indian Health Service is another HSMHA know specifically what to do in order to assure a "good" program, as is the hospital program providing care to performance. The other is to wait patiently, Rogerian merchant seamen and other beneficiaries. These direct style, until from all of the massed intelligence, the dis- care activities account for a substantial number of our comfort of silence brings forth the new idea. employees. The National Institute of Mental Health, This has been an extraordinarily challenging sort of HSMHA's largest single component, has a wide variety of process following the Rogerian style, because Congress, programs in research, training, and service. The Hill- which votes money on the strength of local support, has Burton program, Maternal and Child Health, and the had some difficulty understanding why there was a Center for Disease Control are other constituents of strong movement. HSMHA. Some of you need to keep this set of complex variables in n-dnd as you look at the way we are trying to To present it in simplistic terms, in the organizational explain to the Congress how extraordinarily important structure there is a director for each of these major we think it is that we let the grass roots make the HSMHA programs who has a direct responsibility for our decisions. legal and fiscal relationship to the Department and to If one characteristic of RMP can be set forth, it is Congress. that RMP has not had a distinct public. There hasn't In addition, included in our programs are some been one particular group, external to the organization guidance responsibilities that we assume for other itself, which has gone to Congress and said, "We must agencies. These include, for instance, the Federal Em- have this." Instead, there have been several publics who ployee Health Service, the medical portion of the Ap- have gone to Congress, each with its own image of palachia programs, the foreign programs under P.L. 480. Regional Medical Programs, and therein lies part of the And more recently we have been asked to have a look at @,iroblem which I hope we are beginning to resolve. the design of the Health Maintenance Organization. 35 In each of the ten regions, which recently have been scope of our agency's role should be as a responsible slightly reoriented, there is a Regional Health Director. Federal agency in health services delivery. And we're Roughly one-third of HSMHA's resources are now being looking at HSMIIA primarily as an agent for working expended at the prerogative and under the administra- with the systems of health delivery. tive authority of the regional health director. The Federal role in health care has been moving I answer for these responsibilities directly to Roger recently from a passive to a more active involvement. As Egeberg who answers to the Secretary. you know, in the past the Federal functions have It's an interesting and complex organization. I'll not emphasized limited direct responsibility and con- go further into this, other than to say that the author- siderable use of various kinds of stimulating mechanisms. ities for all of our programs are vested in the Secretary. It is my impression that even when it is stimulating And most of them, with other than policy impact, are private initiatives, the Federal role in the future will tend then delegated to the Office of the Administrator and, in increasingly toward setting the terms and conditions turn, to the program directors and regional health under wlgch those initiatives will be carried out. directors. Now, we in HSMHA have made some assumptions in I hope this outline of HSMHA's organization will give the early deliberations during my 85 days of this in- you some idea of the perspective from which I will talk carnation; and I should like to share some of them with about RMP this morning. you for your thoughts. These are not dicta but are, I The RMP concept has always attracted stimulating think, bases for departure in the analysis process. and innovative people. Ms conference is simply another The Federal role should always be complementary to manifestation of this fortunate tendency. the private sector insofar as possible. We are at a critical juncture, a decisionmaking point The Federal role must, however, protect the common in the health care field generally. There are a substantial good where inadequacies or inequities appear in the number of evaluations going on at all levels and with all system. degrees of sophistication. Currently, there isn't an effort Maximum effectiveness must be assured when federal in the health care field, public or private, that is escaping dollars are used either as an expenditure or as an invest- scrutiny; and apparently no assumption is going to be ment. taken for granted in the foreseeable future. And Federal leadership must assure coordination of The Executive Branch itself is engaged in a funda- efforts and common communication among health mental reexamination of both the appropriateness and activities in both the public and private sectors. the effectiveness of its health care programs. The None of these is new. You have all thought of them. Congress itself is entertaining proposals that are Perhaps the difference is that we intend to act on this set enormous in their scope and diversity. And all across the of assumptions; and that might make a difference. country groups of health professionals, such as this, and In our opinion, the government should help the individual patients themselves, are weighing the options energy in the health care system to flow where it will do available to them in choosing courses of action that are the most good ' Viewed in terms of energy flow, we have now beginning to determine our health care system of to look at ourselves in proper perspective. For instance, the future. we have 25,000 employees who contribute 25,000 man- Some of these evaluations, like the three-day session years to the health care system; whereas the practicing which you have had, are objective and as thorough as the physicians of this country account for roughly 300,000 state of the art will permit. Others are very subjective man-years. And the 7,000 hospitals across the nation and based only on anecdotes or fragments of evidence. have a combined energy input of several million man- It's important for you, I think, to remember that years. sophistication carries no guarantee of acceptance unless While there is no such thing as a common unit of we make sure that our input is registered. The naive health energy, it is evident that our problem is how to assessments may be the ones that are crucial to our i-nake the contribution we have effective in a very large future. system. In the Health Services and Mental Health Administra- Ideally, the government activity should concentrate, tion, we too are deeply engaged in self-evaluation. as we have already said, in the areas where the free Roughly one percent of our total expenditures, which i-narket is unable, for one reason or another, to fulfill the are in the nature of $I.S billion a year are set aside for public need. Such areas tend to occur when the antici- evaluative purposes, We are trying to find out what the pated private return provides.insufficient motivation or 36 Child Health, the Center for Fa@ly Planning, and the The planning agency had its greatest promise as I others. voice of the people in the political sense, enunciating I'm aware that the relationship among these programs the providers the public determination of needs a and particularly between RMP, Comprehensive Health priorities. Planning, and the R&D Center has been the subject of It has a geopolitical responsibility to assure to endless debate since these programs began. Almost constituents equity of care at the highest possible le, everybody in the health field has had a piece of this of quality through the instrument of planning. action. We have had advice from everybody, but the The R & D Center, the newest member of the tria subject still remains. was' envisioned as an experimental instrument applyi@ I'm sure this is one of your concerns. It is one of my scientific disciplines to the model of the health servic highest priority items; so much so, in fact, that we have delivery system in the community. Hopefully it was initiated an intensive administrative study that is be a generator, tester, and evaluator of innovative a targeted to the specific mission of defining separate, proaches in the system, addressing itself to such things distinct identities for these three major programs. cost containment, equity of access, and efficiency There is an extra special group of consultants who resource utilization. will be functioning in various ways. The work will be These philosophical differences, however satisfactox coordinated by Dr. William Willard of Kentucky. Dr. or unsatisfactory they may be in the intellectual sense Willard will be spending about eight days a month with haven't provided adequate guidelines for practical di us over the next several months. His efforts are going to tinctions in the health delivery system of the real worl( be augmented in various ways by Dr. Monty Duval, Dr. It is imperative that this situation be clarified in suc Ed Pellegrino, Dr. John Hogeness, Mr. Nathan Stark, Dr. a way that we maximize cooperation and minimize th Julius Richmond, and Dr. Ward Darley. I think those of overlap and confusion among these programs. Unnece! you who know some of the stalwarts in the field recog- sary duplication, with its resultant waste of effort ani nize that we really have pulled out the biggest guns we money is intolerable. In fact, it is destructive in the fac, know of to get some administrative discussion of how of a limited budget and an unlimited need for improve we can do this constructively and preserve the tremen- ment in health care. dous promise of each of the programs. The effort at clarification to which I am assigning tol In a generalized way, the shape of the distinctions can priority is not to be construed as competitive. We ar( be deduced from the terms in which these programs not talking about one program versus another. I view ii were originally framed, at least as I understand them. as essential if we are to justify and obtain continuing an@ RMP was originally conceived as a bridge bet%veen productive support for all of the efforts of HSMHA human -need and scientific advance, if you put it in wherever they may be. simplistic terms. It represented in a sense a practical at- If you are concerned about administrative arrange- tenipt to link C.P. Snow's "two worlds," which may be ments at HSMHA headquarters that have an important somewhat out of date now, but, nevertheless is what was bearing on the conduct of your RMP activities across the in mind. country, I am sure that you will make it known to us in The requirements of the individual patient were to be whatever unrestrained or restrained iiianner you have in better served by creating arrangements that would the past. We solicit that kind of interest and input. enhance the flow of greatest expertise to the patient's I have made a fairly fast attempt in a short period of bedside through an effective linkage of the providers- time here to sketch out for you in broad strokes some of and I should like to emphasize the effective linkage of the dimensions of the broader stage on which your in- providers. dividual programs are enacting an important role. It did, as we have already said, give providers an op- portunity to innovate from grass-roots ideas. RMP is an integral and extremely important part of Comprehensive Health Planning approached the same HSMHA. HSMHA, in turn, is the agency charged with ultimate objective from a different angle of attack. Here, exercising appropriate federal stewardship in DHEW for by fostering planning processes at the State and com- health care delivery. It is not simply a collection of pro- munity level, the intent or at least the greatest promise grams; it is a composite. And each of its components is seemed to be to encourage a political consensus, in the to contribute to a comnion mission. broadest sense of a political consensus, as to health goals The test of our perfon-nance, yours and mine, will be and the use of health resources. whether or not we can apply ofir combined leverage so 38 ,vhere the scale of investment required is beyond the When it becomes apparent that a given segment of the capability of the private resources. population-for example, expectant mothers or niigrant I'd like to give you a few illustrations of fields where workers-is not receiving the kind of health care it has a HSMHA believes it can perform a valuable service to the right to expect, someone has to be responsible for set- system as a whole. ting this forth in clear terms and making it a part of As a health care delivery agency of the federal community thought. Someone has to begin a stimulation process to a Government, we can meet one urgent national need for a pond. This does not it-nply central source of valid and creditable information on the point that the system will res delivery of health care. direct action in terms of meeting the need, although we Now, I don't know how many of you have really are involved in some of that, but I think more impor- gone at the business of looking at our performance in tantly involved is getting the selected endeavor into the health care field. But it turns out that any set of data realistic discussion. that you pursue far enough seems to come back to the This function will have in it at least two phases. The National Center for Health Statistics, often somewhat first is a continuing systematic and sophisticated over- mangled in the process, and discouragingly far from view of what the health care system is doing, projected anything that tells us about the health status of the against two grids-what it could do and what the needs nation. are. At this time we don't have really adequate surveil- It tells us a little about the absence of disease, but not lance of performance, capability, or need. a great deal about the health status of the nation. The second phase involves getting something done Somewhere there needs to be a competent group of about it. And certainly from our relatively small fiscal experts to sift through the diverse health care activities base, we will have to look at communication and that are conducted in our many localities under many persuasion rather than direct entry into meeting the different auspices, and to analyze and summarize the need itself. national experience. The tools at our disposal then are going to be com- To the best of our knowledge, no such source really munication, persuasion, selective encouragement of in- exists that is capable of providing validated information novation. And that's the name of RMP as far as I'm on multidimensional and multidisciplinary questions on concerned. health care delivery. We need to use that instrument well. One instrument In my opinion, efficiency dictates a single central of stimulus for improvement can come from RMP's source, and logically this is the role which the federal functioning as a center of expertise; and this is the Government should do as it has already done in agricul- instrument of information display in which we hope you ture, commerce, and to a substantial extent in bio- will join us. medical research through the NIH. I think all of you are aware of the fact that when a It is our intent to become the locus of such an company's stock is performing badly, this is made pretty activity. Interestingly enough, we have the mandate. In a clearly visible in the daily listing on every financial page. substantial number of our programs, including Regional And general knowledge is a powerful spur to self- Medical Programs, Comprehensive Health Planning, examination and. change in those whose stock is not National Center for Health Statistics, and National doing so well. Center for Health Services Research and Development, In health care performance, the criteria may be a we even have the models. We have the instruments, and little harder to define and the comparative information we have the capacity. harder to acquire, but once acquired and displayed, it The role represents one way in which this small could and should have a similar effect. energy input can help direct the flow of a larger system. A second role to which we might aspire will be We have some other instruments for change, programs characterized as a kind of guardian of the nation' that are explicitly designed to stimulate innovation in s standards in health affairs. And I'm not thinking here of health care delivery and effective synthesis of health regulation as much, although this may apply in' a few resources for the benefit of the patient. This, of course, cases, such as in quarantine; rather I am thinking of an again includes RMP. evaluator of performance-which is exactly what you're It also includes the planning and project support doing here today-and an activator of the public con- activities of Community Health Services and the other science. activities we have talked about such as Maternal and 37 the total impact of our efforts is greater than the sum of Finally, in all of our evaluative activities it is the parts. imperative we keep in mind the ultimate objective of our As you go through this evaluation conference and endeavors-that what happens to the patient or pre- your further evaluative efforts in RMP, you need to do patient is really what we are supposed to be concerned so in this broader context. Your evaluation efforts about. That's the hardest evaluation of all. should be an important input for ours. The results of our The one thing we still lack is the measure of health as evaluations, in turn, must be factored into the equations an ultimate yardstick. @vhich involve the total health care system. For this reason, it is important that we make our In the same area, we're dealing with the health care findings and your findings widely and freely available. system which is still a crisis-oriented system. It pays least Just as communication is one of the strongest instru- attention to first things-health maintenance and disease rnents for change within the system as a whole, it is avoidance-the greatest attention to illness after it has almost our only instrument for change. it is going to be occurred. effective in proportion to our use of the evaluation process. If we don't make known what we know, we will We need to be sure, if we are thinking truly about have no cause for complaint if we are not a part of the serving the public both present and future, that we are future. not similarly distracted in the planning process. 39 WORKSHOP SESSIONS WORKSHOP ON DATA Participants @rthur R. Jacobs, M.D., M.P.H. - Moderator David E. Reed, M.D. )irector of Statistical and Evaluation Unit Assistant Director for Evaluation Zochester Regional Medical Program Western Pennsylvania Regional Medical Program @ah J. Peterson - Associate Moderator 3iostatistician @abius, New York Katherine G. Bauer -@les A. Nietzner, Ph.D. Research Associate Professor of Medical Care Organization Harvard Center for Community Health University of Michigan and Medical Care 1. WiWam Gavett, Ph.D. Associate Professor, College of Management and Department of Preventive Medicine and Community John E. Wennberg, M.D. gland Regional Health Coordinator, Northern New En University of Rochester School of Medicine Medical Program CHARLES A. METZNER invention enticed Herman Hollerith to father punched card procedures for mechanical data processing, which A short presentation on this large subject can only now make possible, particularly since electronic pro- sketch the topics and arguments. The attempt to be cedures have been substituted, the derivation of so many short results in more direct and unconditional state- tables that it is hard to find our way around in them. ments than are strictly warranted, but this may be the One additional historical point may be interesting to basis of the discussion to follow, although my aim is not you. It was in 1942, a relatively recent date, that "A to be deliberately argumentative to stimulate contro- New Sample of the Population" was developed, which versy. I shall try to elucidate problems and lead toward embodied the first practicable methods for probability some useful conclusions. Explanations are not complete, samples of human populations. It may be worth recalling either, but questions, if necessary, can elicit more. What that these area sampling methods were a product of the I am trying to do is to stimulate thoughtful considera- WPA, and later incorporated into the Bureau of the tion. Census. Sampling enabled many more data to be generated at much lower cost when estimates are suf- Censuses are not new. In fact, there is Biblical mention of a census and the ideological response to it ficient. then still has repercussions among fundamentalists. One is reminded also that total counts are sufficient - Gideon Sourc s became famous by applying a behavioral test to select a There are some guiies'to data that are useful. The subset with characteristics he wanted. Now many charac- Statistical Abstract of' the United States presents an an- teristics are incorporated into census data. The attempt nual overview of data, with references to sources. It is a even at health data is not utterly recent, however. In good index to availabflity. It is mentioned (on page five) 1870, the United States Census became very ambitious in what should become a basic reference, the National and, among many other data, tried to obtain informa- Center for Health Statistics' short pamphlet, "A State tion on fliness. The procedures were somewhat crude, Center for Health Statistics: An aid in planning com- but the amount of data of all kinds was so voluminous as prehensive health statistics". (Revised October 1969.) It to threaten the decennial census by taking over ten years is available from the Center or the U. S. Government to process. This is the point at which the mother of Printing Office. Among other items, the chart on page 41 11 on input-processing-output relations describes roles of the ordinary beliefs concerning blood transfusion that may conflict and useful advice is given for handling Demoniacal possession is not included in many disea! these. In particular a number of user-designer problems classifications, but you should check with your ow are considered. On page 13 is a discussion of the use of religious leader as to what he thinks your beliefs ought t computers and the necessity for thought that makes all include. It is my understanding from nutritionists th, else commentary. A rich passage, deserving expansion, the usual concept of what constitutes a "bland" diet h, occurs on page 14 with respect to cooperative relations little more to recommend it than the idea of "hot" c between users and suppliers of data. "cold" foods. Some state agencies have been developed, and many The point should now be made that we have vari( health and planning departments generate data that gated subgroups in our population. The discussion nia should be looked into. As mentioned later, the more have seemed farfetched and rather distant from data. local the data the more specific the estimates that may believe I can make multiple use of the ideas, however, i be derived. trying to follow the implications of a pluralistic societ: As an introduction to problems, another publication for a) the-, generality of findings, b) the generality o of NCHS is valuable: The 1970 Census and Vital and concepts used, and c) the necessity, flowing from these Health Statistics. A Studv Group Report of the Public for greater freedom of research, particularly in govern Health Conference on Records and Statistics. Docu- nient. ments and Committee Reports. PHS Publication No. 1000 - Series 4 - No. 10. Government Printing Office, Implications April 1969. This is a planning volume for the 1970 777e Geiierality of Findii?gs Census, still useful on issues. From the fact of high variation within our society, i Problems follows that national data have little specificity and higf National data involve many kinds of problems. In variation. Certainly, the mean outlay per person foi common with other data, becoming knowledgeable health care times the nui-nber of persons equals tli( involves not only the names of variates but definitions, economic load on the private sector for health care. Bul particularly embodied in a questionnaire, the instruc- the mean is not very representative. Not very man) tions, and codes - in short - all processes which shape people get the i-nean income. The standard deviation i@ the final product. so vast as to encompass most of the information and There are some special issues concerning terms and render the mean almost meaningless. If you look at any definitions that arise in a nation like ours. Some of this distributions in the health field, you can see this. This is may be easy to see nationally, but you should not be too one reason why insurance is so important a mechanism certain that this applies only to someone else. Ours is a for achieving a mean value. pluralistic, individualistic society, with plural health care There are several ways in which this is directly systems. A single basis of definition does not encompass important. The variations in national data can and do all. Ordinary classifications, such as the "International spread between Census regions, states, and regions in the List", assume an M.D. etiological base, largely micro- @IP sense. National data do not necessarily represent biological. Because we have not recognized the ways in your area, to the extent that your area is distinct. which other people live and think, we are being tested National data must be sorted to yield data for your area, again concerning some accommodation to multiplicity. which may be done, and cognizance should be taken of How far are we willing (or able) to go? the fact that the Census is willing to do this, and that Would we accept a voodoo health center? The one large study is cheaper than several si-nall studies. question is put in this form to test associations. Since However, for sample data, the results may well be un- the audience is more or less white, and more or less reliable for small areas. And equally important, while Christian (although perhaps not up to the standards of sai-npling distributions for the estimates may be derived, Dr. Martin Luther King), we are inclined to be shocked in general the common statistical tests (t, chi-square) are but accept the racist implication that this would be a not valid for these statistics. black enterprise. We should examine our readiness to A frequent procedure to "adapt" or "derive esti- accept the implication. California does pay faith healers, mates" for a particular area from data for another is to Christian Scientists do not cet the diseases of the Inter- analyze the data by other variables (age, sex, previous national List, and Jehovah's Witnesses do not accept all condition of servitude) and use regression or standardiza- 42 tion techniques to estimate values for our specific dis- does not settle conceptual problems, at least correctly. tribution of the analytic variables. Unfortunately, the To settle an issue of the best, which is to say most residual variation also applies, because the analytic predictive or homogeneous, definition of what is an variables can only transfer as much of the variation as epidemic or what constitutes group practice, we have to they absorb, and what remains is error, in both cases. try them all and find what difficulties ensue or what For health data, our analyses so far do not account for utilities be in each. Frequently, we are forced by much variation, and the estimates are correspondingly circumstance into premature definition which is copied poor. The fact that it is done with mathematical statis- and standardized. Sometimes we just pick tip a handy tical formulae on a computer will not improve it. More classification, as in the case of health studies using the complete statistical analyses or a mathematical presenta- International List, reflecting etiology. This classification tion would verify the locical argument above. Any statis- is no doubt valuable for the practice of medicine, but it tical text dealin- with multivariate techniques will does not resolve (or predict) the use of services which explain this. forms the basis for manpower and cost studies. At least, some concept such as the seriousness of the illness must The Generality of Concepts be added when the fiscal or personal impact is what we really desire. Much more must be done to develop Again, there are several limitations encompassed in concepts stiited to purposes. And this leads to the the generality of concepts. The first involves coni- concluding, issue. iiiunicability and response. Respondents understand and report only those terms they know. And much knowl- Freedom of Reseat-cli edge of what is or was wrong with us comes from t e The argume health care system. This is a feedback process of some nt thus far culminates, I believe, in a plea importance. We learn as we use, both in terms of for greater support for many kinds of data and for re- diagnosis, which must be given us to be at all reliable, search more nearly directed toward well specified prob- and even in terms of recognizing what symptoms are lems. Much of this may best be done in the locality of a important by how health professionals respond. What problem where the distinctive character of the situation this implies is a strong bias against reporting by those may be seen, although without effort and receptivity not habitual or economically enabled users. The prob- there is nothing to warrant a belief that being next to a lems of non-users are not reported by a system problem ensures noticing it. Most people with glasses do assuming use, and the resulting confounding conceals not report any disability, and it is hard to convince problems of the system. When a symptom list is used, it people that they are deaf. will help those who recognize the symptoms, but it will Mainly, I believe, it is necessary for our national not elicit a misery or a devil bothering a respondent policy to incorporate the fact that to encompass the unless it includes these. It is much easier to adapt to this variety and subtlety of our national life entails in- locally, since many terms are regional, although they dependent thought and effort and the development of may be ethnic or status-related also. To check, find out queer and unpopular ideas, and mistakes. Our affluent from your friends from different parts of the country society does have people suffering from hunger. We must when evening begins for them, and you will get some acknowledge that we do not with any certainty know idea of the problem. At any rate, the reliability and how to interrupt the transmission of poverty from validity of data are high only for those using the system generation to generation. Uncovering the hunger implies generating the concepts, and may cause serious under- allowing studies, and particularly analyses. Discovering reporting of illness and the unorthodox treatments procedures for bringing ghetto dwellers fully into the engaged in by those uninvolved with orthodoxy. Of society or organization into the health system neces- course, only a national study can demonstrate all of the sitates evaluated experimentation. But we all too variability, but then only if they are prepared for it. If frequently constrain those with the information from one wants to find out about problems, one must be using it for analytic monographs, and insist that a prob- receptive. lem be fitted with a single agreed-upon solution. Diver- A second issue is the problem of "general purpose" sity in the society must be matched by diversity in ap- data to be used by many. Of course, agreement on what proach, conceptually and operationally. information to get is a political and economic necessity, Our national agencies are producing many good and but however valuable compromise may be in politics, it useful data. If they themselves, who know a number of 43 the weaknesses better than those with second-hand ac- The purpose of this article is to consider the relati( quaintance, were allowed to use them to draw con- ships between patient classification, data collection, a clusions, we might do better. They are willing to meet us facilities and manpower utilization for ambulatory ea more than half way, though. There are many special Before doing so three caveats are offered: analyses that may be obtained, if we ask, and although 1. The design and implementation of a data systi they will not be free, they are less costly than special for ambulatory care should proceed concurrently w purpose studies. The Census Bureau will, for example, the development of hypotheses about the planning a design samples for us using their rich data base. Within organization of such care. The data will provide I the limits of confidentiality, information on special decision maker with the necessary statistical informati groups may be obtained. for evaluation of alternative ai-nbulatory care proposz National data can be exceedingly useful, but they are Unless hypotheses about changes are offered prior to no panacea. They are not universally applicable, they simultaneously with the design of the data system, I are not fully analyzed, and they do not serve id] pur- latter effort may be extremely costly for what inforn poses. We must consider the limitations in the light of tion is required and used. our objectives, and we may thereby help to eliminate 2. There is no single decision maker in the typi, some limitations. cottzmtinity ambulatory care system. The commune system is typically fractioned and consists of a variety Data for Ambulatory Care Planning independent organizations (listed previously). Changes J. WILLIAM GAVETT individual and independent organizations can make set from a community systems point of view only if there American communities are concerned with the in- coordinated and cooperative community planning adequacies of existing primary ambulatory services, but unless severe legal constraints are b-nposed in such do not have quantitative data necessary to plan alter- manner as to force the consideration of conimunity-wi native systems for the delivery of ambulatory care. objectives. The purpose of voluntary "communit, planning is to provide the independent decision mak Studies of primary ambulatory care are relatively new compared to studies of hospital care. New techniques are (administrators, physicians, etc.) with information t@ needed to evaluate existing, as well as proposed facilities, permits a rational evaluation of their decisions releva for the delivery of prii-nary ambulatory health care. The to community objectives. existing facilities include: private practice (solo and 3. Data and information collection must be related various forms of group practice), occupational health both the consumer of health care (the patient) and services, school health services, hospital emergency the processes (methods) of ambulatory health care. T. departments, hospital out-patient departments, neigh- efficacy of a data and information system in aidi borhood health centers, health department clinics, as planning will depend on the manner in which bo well as various state and federal primary ambulatory patients and processes are classified, described, ai measured. It is on this issue that the remainder of t] services. Proposed models include facilities differing in discussion will focus. manpower, financing, and utilization patterns located in - -, different areas under private, voluntary or government It is suggested that a classification schei-ne that w relate health care demand to the manpower, equipmer auspices. and facility requirement is needed. Traditional classific Variables to be studied rnioit include legal, contrac- tion methods include patient characteristics (matern tual, and business arrangements; availability, acces- and child health, pediatrics, veterans services), path sibility; degree of specialization vs. generalization of physiologic processes (tuberculosis, cancer, hemophilia services; consumer payment mechanisms; reimbursement services rendered (radiology, medical, surgical), ar, for services mechanisms; manpower configuration; (neighborhood health center), as well as organizatie equipment; ancillary services; capital, financing arrange- rendering the care (private group practice, hospital ot ments; and characteristics of services rendered. Within patient department, occupational health services). Eai the context of defining the basic characteristics of of these classifications have some use in health care pla primary ambulatory care organizations, consumer at- ning. For primary ambulatory care planning, a classific titudes, outcome of care, and design characteristics such tion system is needed to categorize, compare, ai as: working spaces, procedure and communication project patient utilization for different types of prima systems, etc. are less important. ambulatory care delivery units. A measurement 44 demand in a given ambulatory unit is the first step in the power and equipment) are involved. The other extreme conceptualizing of alternate ways of satisfying demand. is the critical, medically complex case in which extensive The demand or load placed upon a primary care or- resources are used often within a short period of time. It ganization at any given time can be expressed in terms of is suggested that cases might be classified into categories the number of cases (patients), the episodes (specific such as A, B, and C, where A is the urgent, complex, medical problems requiring management), and the visit resource-intensive case; and C is the simple case, in- (interface betiveen the patient and the health care volving minimal resources. The B cases would include sj,stem). A classification system based upon a set of those involving long term episodic illnesses where diag- criteria related to the characteristics of the case and the nostic skills, continuity of care, complex therapeutic visit as they relate to the services rendered is proposed measures, and extended support and observation are tinder the assumption that such a classification scheme required. will facilitate the conception of alternate care organiza- Figure I represents a definition of each class in terms tional designs. This classification method focuses on the of specific attributes. These include manpower and complexity of the case and visit. facility requirements, frequency of visits for the episodic The primary care setting encompasses a continuous illness, diagnostic problems and disposition of the case scale of case complexities. At one extreme is the visit. A fourth class might be developed for psycho- medically simple case in which modest resources (man. somatic cases and minor psychiatric cases. FI G U R E I.-Case Classification Table Manpower Facilities Frequency of Diagnosis Disposition Comments visit for of visit episode MD required ICU Not appli- Extensive Dx Hospital, Acute cable home care, Class Likely team ED Skills required or Life-threatening A effort or not Long term care Case. Hospital facility required facility Totally -nterrupts required normal living MD required MD office or more Revisits re- Diff'lcult Home Non-acute extensive quired Possibly referral Dx or gx Obscure Possible hospital Ongoing comprehen- c7ass or facilities sive and conti- B consultation Chronic Long term care nuity care Case. Possible hospital facility important Interrupts normal living MD or nurse Average MD office One or two Relatively Home Acute or not short simple, Problem does not Dispensary visits obvious, Accessibility and demand a self- availability of specialist limited service important Oass (URI or to patient c minor injury) Interrupts normal living to minor degree Support and reassur- ance may be a ma- jor attribute 45 The ABC classification, and further refinements of it ambulatory care data systems for patient care, institi to include subclassifications, provides a basis for tional management, and for community planning. considering the questions of ambulatory care organiza- 10. What is the role of the hospital in ambulator tion. For example: care? Analysis of the Emergency Department and Ou 1. For a given community, what proportion of case Patient Department by case classification characteristi( visits by A, B, or C type are made to which organiza- may provide quantitative data for reorganization of tt tions; are resources allocated among organizations in an hospital's ambulatory services. intelligent manner, e.g. perhaps demand for type A I 1. Does the measurement of low income urba services should be consolidated at one or more hospitals? ambulatory care demand by the case classification tecl 2. For a given ambulatory care organization, what nique provide insights as to how to organize ambulatoi proportion of case visits are in each of the A, B, and C services for the urban poor? classes; are the unit's resources intelligently related to the given proportions? 3. How can a rural conununity, that cannot attract or Information Systems to Meet Common Data Needs hold an MD, benefit by an A-B-C classification of its of Health Agencies ambulatory patient load? e.g.: KATHARINE G. BAUER a) Could class A cases be serviced by volunteer community-supported emergency units, highly trained It has been observed that information is to tt to provide on-the-scene first aid and transportation decision-making process what oil is to the intern service to the nearest intensive care unit (presumably combustion machine. It does not itself make the proce located centrally)? work, but without it there is considerable wasted effor b) Could class C cases be treated at a private or misdirected motion, and eventual breakdown. Tho., community-supported convenience clinic, manned by who are at the wheel in making health policy decisiot paramedic personnel, and organizationally linked to the usually find themselves in the position of the motori nearest conununity hospital or group practice? with a dry engine in the middle-of a Texas oilfield. Tt c) Could class B patients be provided with long term million barrel output of raw material surrounding him episodic care by nearest physician (patient's choice) but useless to meet his urgent need for a mere two quar with routine and non-complex class C visits serviced by which have been suitably processed to meet his engine the convenience clinic? requirements. We would all agree that health data gush( 4. Where in the management of A, B, and C cases are more freely than oil - and that for the most part A community-sponsored facilities advantageous in the haven't yet found very satisfactory ways of tapping an larger community? For example, what community or- refining it for the particular uses of those who ma@ ganizations should sponsor multiphasic screening, con- health decisions - whether these involve expenditures venience clinics, special preventive medicine clinics, etc. thousands of dollars, or of millions of dollars. 5. Can a clinic for the treatment of C cases be ef- My assignment today is to discuss the organization ( fectively used also for the purposes of triaging non-C a health information system as a means of meeting suc cases to other community health care organizations for important needs, particularly those of RMP evaluator those individuals who do not have access to other and their opposite.numbers in other agencies. Can such primary care organizations? system be designed to supply, link and refine the man 6. What proportion of ambulatory case visits clas- streams of health data that are routinely being generate sified as C cases involve mainly support and reassurance? from diverse independent sources - such as the facilitie 7. How are the concepts of family medicine and manpower, and vital' statistics compiled by Stal comprehensive care relevant to the class A, B, and C agencies, the various utilization and patient origi cases? records from hospitals and other service providers? All 8. How is the question of the use of paramedic can these be i-nore usefully related to the basic denl( personnel specifically related to A-B-C case care? Does graphic and health statistics from the U.S. Census an the C case and C visit load on the community consume the National Center for Health Statistics? I was asked t significant physician resources such that extended use of lead off this discussion by virtue of my association wit paramedic personnel isjustified? a two-year study of this question at the Joint Center f( 9. The case classification technique may reveal which Urban Studies of M.I.T. and Harvard - a study larger variables are important and should be incorporated into inspired by Dr. Osler Peterson, Director of Research fc 46 the Tri-State @IP, who made major inputs to it at every This means that throughout almost all health pro- stage-2 grams, not just RMP, researchers will be trying to What do we mean by the term "health information construct various types of performance indicators - to system?" I suspect this is I one of those in-terms that permit coi-nparisons of past and present experience people have come to use quite widely without benefit of within a program. To measure the impact of their pro- definition. For our purposes today, I'm simply going to gram on specific target populations over time, and to be generic and talk about a systematic approach to compare their program results with those of other pro- producing, storing and gaining access to many kinds of grams which use other techniques or methods. However, health data produced from many sources, for multiple as we all know, the right kinds and quality of data are uses, by multiple users. Also, in an effort to put first rarely available to permit this crucially important re- things first, this paper focuses primarily on the organiza- search to be carried out. One can make a safe guess that tional process for systematizing this access to data - not only throughout our concurrent workshops now, only incidently on computers. As an analogue - if we but in similar health evaluation research meetings every- didn't yet have libraries but many writers were where, the identical complaints are being voiced: "The producing books which many potential readers needed 1960 census data were obviously useless for computing to read, a first step would be to organize some system to 1969 rates - we simply can't tell the trend so far . . ." or identify the books of interest and to decide on policies "Unfortunately the reporting system changed, so it's for their acquisition, storage and circulation. Only as a impossible to compare past and present performance" or second step would one commission a computerized "we can't compare our results with those of prograi-n x index. The health information broker system recom- because they used entirely different age breaks - and mended by our Boston study naturally differs in many besides we have no way to get comparable unit costs@" respects from conventional library organization - yet One concludes that all concerned have an enormous there are underlying similarities of function. stake in iniprovinc, the kind and quality of the data base. -The broker system predicates that it would be To provide the denominators of the rates they need mutually beneficial to a region's major public and for their various pruposes, researchers in all major health private health programs and agencies, such as RMP, programs seem almost universally to require certain Blue Cross, State Health Departments, comprehensive common types of data - the demographic, health status, and area facility planning agencies, to join forces to vital records, facility and manpower and the kinds of obtain and share the kinds of data they need in common utilization data reviewed here earlier. Some of this for their separate research and planning activities. At the simply isn't now available - such as disability rates of same time, the study warned against constructing tin- populations in cities or small geographic areas. Other workable multi-million dollar data banks. Before widely needed data, however, such as about health describing this model, and telling you some of the con- facilities, are being routinely generated for their own siderations that influenced its design, let us briefly operating or management purposes by some one agency review some of the reasons it seems particularly timely which, in turn,, may need management - generated data right now to promote this or some other type of cooper- from other agencies for its own evaluation research. ative organization for improving health statistics. Finally, staffs in a-g-e-nc-ie-s- -q--u-i-te--- often duplicate their research efforts, both in their separate Wliy a Health Information Systeol ? quests for identical source materials, and in time- As budgets in every sector of the health system get consuming activities such as constructing S.M.S.A. tighter in the face of medical price inflation, it seems profiles, or population projections. This costs everyone certain that in every type of program, public or private, money. the big questions of accountability raised to you yester- Given such common needs and problems it would day in the plenary session will be increasingly posed: seem that major health organizations have everything to what benefits are patients actually receiving for the gain by joining forces at least for the limited goals of: money spent? How can the program policies be modified and adapted to improve these cost-benefits? Obviously improving the quality and comparability of exist- the day is almost over when those who pay the bills will ing data commonly shared, be satisfied by simple tallies of patient days and OJI.D. identifying commonly needed data now un- 9 visits juxtaposed with total dollars expended and a available, and finding means to secure them, request for a 15% budget increase next year. eliminating duplications of research effort, 47 arriving at agreements for specific types of data development, sharing and use of information perti. sharing. nent to their common needs. Such an information system should be planned at the outset as the first Although funds were not available last year for a step in a more complex communication network proposed demonstration of our Boston model, it seems should future expansion seem indicated." possible that within the next few weeks Congress will A broker function between independent health in authorize federal support for experimental health in- formation subsystei-ns rather than a centralized datz formation systems of this kind as part of the Health bank was recommended because it would: Services Improvement Act of 1970.* o adapt better to the predictably ever-changiiig dat, Funding is only one aspect of the problei-n of data needs of its users, sharing among independent organizations. Given the o provide better quality information over the lonE realities of the operating environment, can a satisfactory run, means be found to promote inter-agency cooperation? o avoid direct confrontation of the issues of agency When one looks at the activities of the Bureau of the confidentiality and of individual patient privacy, Census and of the N.C.H.S. and other important in- o function better within the present limitations ol formation centers at the federal level one can feel compute .r software-yet peri-nit adaptation to future hopeful, But further down the line at the regional, state technological advances expected there. and sub-state levels where mixes of various public and Finally, a consortium of users was recommended a@ private data sources are sought, issues of agency con- the policy-making body for the broker system, witt, fidentiality and of inter-agency power struggles inject a administration temporarily vested in a university. Thi,; host of complexities. Whether organizational fort-as can structure was put forth in order to avoid threatening the be devised during the next few years to circumvent the existin- power relationships among agencies sufficiently problems while fulfilling the need remains to be seen. to foreclose their participation. Our Boston study's recommendations represent one Before going on to elaborate on some of these points possible approach - Dr. Wennberg will tell you about I'd like briefly to mention some activities proposed in another, and I know that several other people here have the Boston model. been wrestling with these problems in their own regions - from New Mexico to Ohio. Soiiie Possible Functions and Activities The Broker System Model of a Sli,7red Itiforination Systeni 1. Nlakin- data more available for secondary analysis, The Boston study concluded that (and I quote): by: "The needs of health plannin- and research in this t) * inventorying and cataloging data sources and files area at the present time will best be served not by a furnishing detailed descriptions of data files tc new prime data processing coi-nputer system, but by a guide the user - such as dates and methods of data mechanisi-n designed to interface between several collection and up-date; sample size; format ir newly developing hospital, public health, mental Nvliich preserved (file folders, i-nagnetic tape, etc.) health, and social welfare information systems at person responsible for maintaining files; condition regional, state, metropolitan area and municipal of access, etc., levels. Such a mechanism should promote compati- guiding and helping the user select and use corn. bility between the subsystems and thereby maximize puter programs best suited to his needs. the possibilities for mutually beneficial information 2. Improving the utility of available data by actively spin-offs, now and in the future. A consortium of eiicoura-ina data generating agencies to arrive at: health planners, major health agencies and research @ 0 organizations should establish a health information compatibility of key items on report forms - such systei,n to serve this broker function, to facilitate the as age, residencc, condition, service, etc., cojiipatibile definitions of terms used in such reporting. *"'flic Secretary is authorized, directly or by contract, to 3. Identifying common unmet data needs, and undertake research, development, demonstration and evaluation, helping meet them by: relating to the design and implementation of a cooperative promoting addition of new categories of inforitia- system for producing comparable and uniform health informa- tioii in existing data sources - such as. finer age tion and statistics at the Federal, State and local levels." breaks in a State census, 48 developing directly, or contracting to develop, new and promote common efforts to contract for or in other sources of information - such as population ways gain access to commonly needed new data, such as health surveys. from small area Population health surveys. Finally, the 4. Helping users Find the coi-nputer resources most broker system staff would provide direct research appropriate to their needs by: services, such as file-merging operations, and would * organizing local conferences and workshops - furnish regular monitoring reports on health and social such as are now conducted by the census, indicators requested by users. However, it was assumed * inventorying and brokering use of agencies' that the system would not require its own computer partially idle computer hardware, facilities at least in the foreseeable future, but would * evaluating software packages, and purchasing for contract for the use of the necessary resources. joint use, * demonstrating, through case examples, the uses Why a System of Stib-Systems? and bene@its of new advances in computer A coordinating mechanism between independently science. organized information sub-systems rather than a central 5. Developina policies regarding privacy: 0 data bank was dictated by users' requirements for formulating policies governin- agreements for sharing data, D flexibility, quality, and privacy - as well as by the state prornotina codes of ethics; specific legal safe- of computer art. I will touch briefly on these points. FIexibility.-Health researchers need to tap data guards. flowing from many sources. Although much of it comes 6, Furnishing routine monitoring and special status from the operational and management reporting systems reports such as: of institutions and programs, it is important to remem- trends in the locality's death rates, health facilities, ber that despite the overlaps between the specific types manpower, utilization, etc. of information required for good research and good comparisons with other regions, states, etc. 7. Promoting the integration of separate streams of program management, there are usually marked dif- data by: ferences in the characteristics of the data required for 9 negotiating agreements between agencies for data these different purposes. For example, instant on-line 0 inputs and retrieval are hardly necessary to provide data sharing, for studying the effectiveness of appointment systems in o advising on legal matters and computer locks to following, no-show cancer patients, yet can be invaluable safeguard privacy, C, * conducting file merging operations and providing for actual appointment scheduling. Above all, the tables and maps - such as county, city or census particular characteristics of the data a researcher needs tract profiles showing health status, mortality, the usually changes with every new problem he addresses. population's use of hospitals and health resources For each study he may need not only different types of - and utilization profiles according to service, data, but different geographic breaks, frequencies -of patient characteristics, conditions, and proportion data updating, degrees of individual patient identifica- of community served. tion, etc. At an even higher level of generality, maximum flexi- It is assumed that any such information system would bflity is imperative in a system designed to serve the build in its own evaluation process and would con- information needs of policy makers. There can be no tinuously re-cycle on the basis of experience, new health fixed solutions to the problem of providing health in- research and planning needs, and new computer tech- formation since both needs and solutions are dynamic nology. and ever-changing. Many methods of ea re and facilities You will note the heavy emphasis placed on staff for treatment we regard as essential today will be activities. One important thrust of their work would be obsolete or unnecessary ten years from now. New to inventory and catalogue data sources and computer methods of payment will be adopted. New health hardware resources in the region, and to evaluate corn- professions will emerge. Information to serve research Puter software packages. Another set of functions would and policy makers must therefore, above all, be designed relate to improving the quality of the data, by to anticipate and to accommodate to change. A network negotiating format compatibilities, and promoting of sub-systems permits this. adoption of common definitions. Again, staff would Quality.-In view of the massive data base required 'help negotiate inter-agency agreements for data sharing, and the large number of ffles that might need to be 49 tapped for all the various types of health delivery system this model or any other cooperative health infori-nation evaluation that might be desired now and in the future it system. The answer will deteri-nine whether the system would be sheer fantasy to expect that any one central- ever actually gets funded and into operation; whether ized system could incorporate them and manage their those who generate the needed data will, in fact, updating and quality control. Nor would that be pro- contribute to it, and finally, whether it will truly serve gress. It is far more desirable that each organization have the purposes of research and policy guidance for which a genuine and active concern within itself to continually it was designed. improve its own information management, while taking The National Center for Health Statistics, which as due cognizance of the needs and requirements of others. you may know has recently published a description of a Privacy.-The privacy issue was another major factor model for state centers for health statistics, states as a in recommending a broker system where every agency cardinal premise the absolute need for information that would maintain custody of its own files. Clearly, data is completely unbiased and authoritative. I quote: "The sharing is an area fraught with fears and ambiguities - inevitable disagreements on how to deal with health where the power of information can be used on in- problems must not be confounded by controversy over dividuals and institutions alike for good or ill. And the basic facts of the situation ... This also means that no where the conclusions as to what is good and what is ill pertinent facts be suppressed. . In effect, the statistical depend very much on who is making them, and under function must be discharged with high competence and what circumstances. Or, more succinctly, whose ox is cannot be captive to a particular point of view." Thus being gored. Confidentiality of information about insti- the N.C.H.S. model calls for the information system to tutions and organizations relates clearly to issues of the be administratively independent of any one planning confidentiality of their actions and effectiveness. The agency, though with strong working relations with all. Boston study, as the better part of valor, resolved these But how does one identify an administering agency issues by recommending data be limited to that which which will command the trust and respect of all, in an could be used in aggregated form, and by promoting environment where knowledge is indeed power - and specific inter-agency agreements on data sharing where, in almost all programs, worry about loss of power designed with appropriate legal consultation. After the is the name of the game? If there is an answer at all to system had proved itself and appropriate controls this auspices question, I suspect it will be a different one designed, moves mi-ht be made towards more specific in each region or state. Some possibilities to be con- sharing of fine--rained data. sidered are: Computer Limitations.-A huge, centralized data o a generalized state statistical center, system incorporating many files presents problems not o some other state agency (possibly the university), as yet adequately solved by computer science. With long o a regional commission or center, lead times for design, by the time such a system goes o a quasi-public information authority. into operation it is apt already to be behind the state of In addition to auspices, many important questions of the rapidly changing computer art - to become a vastly staffing and function, of cost, and of the cost-benefit of expensive antique. Such disasters have occured regularly such information systems remain to be explored. If in the urban information systems so hopefully installed Congress does now authorize the funding of experii-nents in the late 60s. The M.I.T. computer scientists on the in cooperative health information activities perhaps you Boston study recommended instead, careful develop- can all soon begin learning by doing. Certainly the ment towards a network structure among participating failure to develop satisfactory efforts along these lines programs, where hopefully in the future a variety of can only mean the continued burden of handicap to computers of different types and sizes, with different those who try to measure the successes and failures of hardware and software configurations might be able to our operating prograi-ns and thus to give the public the talk to each other under the control of appropriate most value for its health dollar. permissions. They expect that the next decade may well witness revolutionary software and hardware break- lPeter J. Henriot, "Political Questions about Social Indi- throughs to make this possible. cators," IVestern Political Quarterly, Vol. XXIII, No. 2, June 1970. ftlio Plays the Role of the Broker? 2NIoynihan, Beshers, and Cydel. Problems and Perspectives in the Design of a Community Health Information System (U.S. Undoubtedly this is at once the most sensitive and Public Health Service Contract PH 110-234), Joint Center for the i-nost crucial -question to be faced in implementing Urban Studies of M.I.T. and Harvard, Feb. 1969. 50 The Northern New England Ptepional Medical Program opportunity to accumulate experience with the technical n and management problems of developiiia large data Health Planning Data Base t, JOHN E. WENN BERG, M.D. systems. it also tllows one to evaluate the utility of components of the system. This should be of value to the future development of prospective, population based Those of us involved in Regional Medical Programs health information systems which, I think, clearly will are continually reminded that health plannin-, without C, be given central roles as part of the iiianacyenient strlic- an adequate data base, is more of a visceral than a C, cerebral process. We are often asked to support solutions ttire of a iiitioiiil health insurance system. which clearly are proposed without proper ideiitificatior The immediate purposes of the Northern New England Recioiial Nledical Proorai-n data base are to provide in- of the problems involved and are usually made without C, LI formation for lic@ilth problem identification and pro-raiii reference to other priorities. To a very large extent, our 0 plannin-, evaluation and management. It supports plan- informational base for planning decisions in health is Z, nino efforts at the areawide and state health plannina limited to impressionistic, non-verifiable opinion corn- 0 C, nionly arranged or provided by parties advocating a levels. A primary Customer of the system is therefore the particular solution. Under these circumstances planning Vermont Comprehensive Health Planning Agency. Contractual arrangements have been made with that decisions run a high risk of being - at best-irrelevant. t, A prospective population-based health information acency to supply them with necessary information. The system appears as a particularly attractive solution to data base also supports planning and operating activities our data problems. As you have heard today, the of the Re@ional Medical Program, including primary care necessary technology is not obscure. In fact, it has been activities and disease control and continuing education available for some time. Why, then, does it remain programs. Finally, certain features of the system have generally unimplemented? The reason for this cannot be . been Of use to operating health agencies and in some simply cost; it can be convincingly argued that health instances to planning agencies outside of our area. For information systems could more than pay for themselves example, aspects have been utilized by Vermont Planned by providing the informational base for wise decisions. A Parenthood, The Province of New Brunswick in Canada, The Maine Re-ional Medical Program and the Maine more probable obstacle to establishing prospective in- 0 formation systems derives from the direct lack of utility Facilities Planning Council. of the system to the provider of the data. To provide Basically the data system provides a characterization accurate data is a bother' and the effort to produce it of the health system in terms of: must be either rewarded or required by law. Under our 1. the coi-nniunities being served in demographic, existing pluralistic planning and management systems, socioeconomic environmental terms; 2. the manpower, facility and dollar resources of the good planning is neither strongly rewarded nor required health delivery system; by law. Under these circumstances the establishment and 3. utilization supply and distribution aspects of the maintenance of prospective health infon-nation systems health care system; are expensive - probably intolerably expensive - in terms 4. outcome, as measured by morbidity, mortality and of currently available management and pursuasive patient satisfaction. energies, The major products are planning documents and If the current Regional Medical Program and Com- status reports covering the above mentioned areas. prehensive Health Planning legislation does not contain Examples are available froi-n the Program office on the mandate necessary to promote prospective health request. information systems, is there an alternative approach which can begin to achieve the data base necessary for Establishing the data base has required a major effort planning and management systems? I think the answer is which cannot be systematically reported at this time. a qualified Yes: under certain circumstances, Regional However, I would like to el aborate on five important Medical Programs can establish an ad hoc but systematic features of our approach: (1) choice of the New England data base which minimizes administrative inconvenience town as die geographic base; (2) strategy governing to participating institutions and is at the same time collection of data; (3) resume of the contents of our highly useful to its own planning and evaluation data file; (4) approach to data processing; (5) approach Purposes, to Comprehensive Health Planning and other' to data analysis. health agencies. In addition to the immediate utility of The geographic region covered by the data base in- the data, establishment of ad hoc systems affords the cludes the entire service area of the Northern New 5 1 England Regional Medical Program. However, in records - we have used part-time Regional Medical Pr designing our approach, we wished to use the smallest grant personnel under close core staff supervision. Und feasible geographic unit that was available. The New these circumstances cooperation has been near England town turned out to be nearly ideal - for the universal. following reasons: (1) it appears in the census; (2) most While much of our data base spans more than oi unit records in the region (for example hospitals and year and is updated periodically, the costs involved vital records) contain the individual's town of residence - fielding special utilization surveys led to a decision thus utilization rates can be calculated on a town basis; restrict (at least initially) the complete utilization file (3) there are a total of 356 distinct towns in the region - the calendar year 1969. Informational items correct( 251 towns and gores in the State of Vermont - 5 1 in the through special protocols have been kept to a minimur three counties of upper New York - and 54 in con- These include patient record number, age, se: tiguous portions of New Hampshire. Populations in each diagnoses, procedures performed, length of stay, date ( town vary b?tween 35,000 and IO, with a median value admission, type of insurance, referring and attendir of about 1500. Thus, using the town as a population physicians. base allows for a large number of discrete geographic Resume of Contei7t of Data Files: Currently, our dal units in the system. This in turn provides great analytic files contain the following information: ' flexibility. 1. Utilization review: hospitals, nursing homes an Strategy governing collection of data: all effort has home health agencies. been made to avoid duplication of existing data. When- A complete review - based on unit records - of all are ever possible, we have used existing sources of data, hospitals for the year 1969. 68,000 records were take either published or existing in unit record files collected from PAS and 29,000 collected by staff review of th by cooperating agencies: hospital records. In addition, referral hospitals in Har Existing data includes those collected, processed, over, Albany and Montreal have been reviewed. and published by local, state, federal and national A complete 1969 review of all area nursing homeE agencies: for example, reports of the Bureau of (8S homes, and 4,000 records). Census, National Center for Health Statistics, Amer- A complete review of area home health agencies (4 ican Hospital Association, Blue Cross/Blue Shield, agencies, and 8,000 records). State Health Department and The State Planning 2. Vital records: Agency. Through excellent cooperation with the Vermont Existing unit record files include those collected N.H. and N.Y. health departments, decade files of birt] by operating agencies and i-nade available to the Pro- and death records have been established. Mortality dat grain by special cooperative arrangements: by way of example, a three year file of 200,000 patient dis- is a particularly useful source for defining major healti charge abstracts obtained from the hospitals partici- problem areas for measuring outcome. pating in Professional Activity Survey (PAS), the 3. Manpower file: decade files of the Ven-nont-New Hampshire Vital Hospitals staff listings have been obtained from al Records and the individual tax returns from the State institutions of the region. Health Department anc of Vermont Tax Department for 1967. AM.A. registries are being utilized to classify physician! in the region by locality of practice, specialty training Special collection protocols have been established for age, board certification etc., both on a current and ar missing" data. This includes surveys, conducted by the historical basis. staff, of hospitals, nursing homes and home health 4. Facilities: agency records. It also includes a household survey In cooperation with the Tri-State Regional Medica capability.* Program, special inventories of hospital facihtie@ The avoidance of administrative inconvenience to institutions in providing data is fundamental to success throughout the region have been completed with the of an ad hoc data system. When data collection has following areas being stressed; coronary and intensive required staff time - such as reviewing unit hospital care, emergency care, stroke care, radiotherapy and chronic pulmonary care. In addition, published data encompassing facility *While an integral part of the "data base", this paper does staffing, size and location as well as cost data have been not discuss the NNE/RMP social survey capability. compiled from a variety of secondary sources for hos- 52 pitals, home health agencies and nursin- homes. Sources age to problems in format design and basic character t, include Blue Cross, American Hospital Association and configurations. As an example, sources of data include state agencies. magnetic tape obtained from PAS (Minneapolis Honey- s. Socio-econoinic and environmental: well), Vermont State Government (General Electric) and Arrangements have been made to secure 1970 census New York Health Department (Burroughs). To solve tapes containing available processed tables. This will these problems generalized recoding and formatting pro- establish age-specific population rates on a town basis. grams have been developed. of particular importance is intercemal estimates on a Approaches to Data Analysis.-The usefulness of ttie town and other small area basis. Work has been com- data base relates to 1) the completeness of each file (for pleted in conjunction with the State Health Planning example, one year of hospital experience for the total Agency personnel to construct inter-censal population population) and, 2) the inclusiveness of the system in age-sex structure for towns and counties. terms of the large numbers of separate data files contain- Indicators of economic status are beinc, constructed ing relevant health data. This enables (for example) through the use of income data. In conjunction with the correlations between demographic and environmental State Tax Department individual income tax returns factors in health status. Much of the analysis undertaken have been analyzed by town, occupation and industry to by the RMP has been computer based and allows for the provide an economic profile of the State and its sub- study of complex relationships between "input" and divisions. "output" variables. Examples of correlation analyses 6. Published data: that are possible include relationships between per capita For example, complete set of reports from the income, admission rates, death rates, infant mortality National Center for Health Statistics. rates, expenditures for medical care, procedure rates, Approach to Data Processing. Routine reports etc. prepared by acencies and organizations in the health While a number of general statistical programs have field rarely provide direct answers to specific questions been adapted, we have also developed a series of new as they arise in planning, manacenient and evaluation . novative types of health system analysis. Of C, an in activities within a local or regional context. Processed particular note is a program des.igned to characterize data, organized and tabulated according to external total utilization and allocation of medical resources dictates, is often irrelevant to immediate concerns. The relative to the patient service areas of particular insti- limited utility of reports furnished hospitals by the Com- tutions. This includes resource allocation rates in terms mission of Professional and Hospital Activities (PAS) of admissions, patient days, beds, dollars or skilled man- and of publications of state and federal health depart- power. Because virtually all utilization experience for ments reflect the series of compromises that must be each town in the region is known, these rates describe made in developing multi-purpose reports. From several the total experience of the population, Thus, for the standpoints, the most effective method of information first time, an accurate estimate, based on a small popula- storage is raw data on individual cases. This is particular- tion, is possible: this includes total cost for institutional ly true when efficient storage and retrieval methods are care, procedure rates, bed utilization and beds available available. rates, etc. ,Accordingly, the RMP has devoted a significant effort During the next year, the NNE/RMP will complete a to the development of individual case files. Because number of reports for areawide and state planning accessible data derives from diverse sources, a number of purposes. I hope that the next time I report to you on compatibility problems have been encountered. These the data base we will have much more to say about the range from differences in coding of such items as sex and effect the data has had on the planning process. 53 WORKSHOP ON MEASURING CHANGES IN BEHAVIOR Participants john S. Lloyd, Ph.D. - Moderator Mitchell Schoro@v Assistant Coordinator, Evaluation Assistant Coordinator for Educational California Regional Medical Program - Planning and Evaluation Area V intermountain Regional Medical Program William R. Crawford, Ed.D. Barbara J. Andrew, Ph.D. Associate, Evaluation Studies Section Assistant Professor Medical Education Research Center for Study o@ Medical Education Division of Research in Medical Education University of Illinois University of Southern California Nleasuring Changes in Knowledge WILLIAM R. CRAWFORD Multiple Ozoice Items A. Advantaues 0 Sometimes the measurement of knowledge seems to 1. Some task clearly defined for each exwninee be a straightforward procedure. Perhaps that is true '-). Larce sample of items permissible when one is interested in measurement of simple recall Z, 3. Scorin- keys are standardized of basic information which has been memorized. How- 0 ever, simple recall of basic information is usually not 4. Easy to score sufficient for measuring the achievement of educational B. Disadvantages objectives in areas as conceptually complex as medicine 1. Requires recognition of correct response, not production of it ost cases we are and the allied health professions. In rn 2. Permits guessmg interested in assessing chanues which are related to the 0 3. Difficult to construct ability to apply principles, solve problems, and interpret 4. Task is completely structured data, to name only a few. Clearly, these complex intel- lectual functions cannot be assessed with instruments Nleasuring Changes in Clinical Performance designed to provide an estimate of the number and kind of memorized facts which can be recalled. BARBARA J. ANDREW, Ph.D. How, then, can we approach the greater problem of The health professional's ability to solve clinical measuring the ability to engage in more complex intel- problems has long been regarded as one of the most lectual functions? The obvious first step is to define important dimensions of quality health care delivery. what those functions are, why they are important, and Yet because of its complexity and the challenges which be per- it presents for quantitative measurement, clinical per- how they relate to specific tasks which must formed on the job. Defining these functions is a major formance has not been as widely used as a criteria for operation, and an essential step before specific evaluation as its importance would suggest. measurement instruments can be developed. The second Clinical performance is essentially a problem solving initions and translate them into process which involves: step is to take these def instruments which can validly and reliably measure the 1. knowing what data are relevant; functions, and which will produce meaningful data. Con- 2. gathering the data; current with the development of the instruments one 3 .analyzing the data and evaluating their relative must develop a procedure for scoring and a plan for importance and significance; reporting and interpreting the scores. 4. synthesizing the data into conclusions; 5 .knowing about available health care strategies; Following is a brief outline of the topics covered in 6. selecting and applying the most appropriate this session of the workshop, each of which was con- strategies; sidered in more depth in the working session. 7. evaluating the effectiveness of the strategies; 55 8. making whatever changes in health care strategies and used by different raters (e.g., non-verbal as@ which are needed. well as verbal interaction during history taking)] ; Specitic clinical problem solving activities can be 7. finally, the validity and reliability of the instru- classified as primarily diagnostic or therapeutic in ment are estimated. (In instances where the in- nature. That is, while diagnosis and therapy are inter- strument has been designed for use by raters to dependent components of clinical problem solving, some observe clinical performance, sufficient training to health professionals have primary responsibility for improve inter-rater reliability should be under- diagnosis, while others are concerned with suggesting or taken). administering therapeutic procedures. Still other health The selection of appropriate validity and reliability professionals, such as the physician, are responsible for estimates depends upon the nature of the measuring in- diagnosis as well as therapy. strument itself and upon the purposes for which testing The measurement of clinical performance can focus data are gathered (3). either upon the entire problem solving process employed In estimating the reliability of observation devices by a specific health professional or solely upon the one needs to determine the correlation among the eval. frequency with which certain behaviors within the uations of several raters of the same clinical perform- process are observed. In measuring changes in clinical ance. This procedure necessitates the refinement and performance to determine the effectiveness of particular careful definition of the skills to be measured and cate- experimental treatments, the decision to observe the gories for recording performance, as well as the training entire problem solving process or only some specific of observers so that acceptable inter-rater reliability can behaviors within the process will be a function of the be achieved. purposes of the study and the hypotheses which have When the measuring device consists of a paper and been stated. pencil test of clinical performance or the simulation of a The validity of clinical performance measurement clinical situation, comparability of forms and compar- will, of course, rest upon the quality of the instruments isoiis over time offer the best estimates of reliability. which are devised to record the problem s olving be- Estimates of the test-retest reliability of simulated havior. The following procedures should be followed in clinical performance test are complicated, however, by the development of such instruments: the fact that these simulation tests permit the examinee 1 .the clinical skills to be measured are identified; to receive feedback from his selections and, hence, to 2. criteria for evaluating these skills are developed; some extent constitute a learning situation. Even if the 3. the criteria are stated in terms of specific clinical time interval between test administrations is lengthened behaviors; to enhance forgetting, one cannot control intervening 4. a method of scoring is developed which is logically variables which might improve the subjects' problem appropriate to the skills being measured; solving skills. a. the assignment of differential scores to various Since in i-neasuring changes in clinical perfori-nance levels of performance should be clearly defined one is primarily interested in determining the degree to and require as little subjective judgment of the which the health professional possesses certain clinical rater as possible; problem solving skills, the use of criterion-related valid- b. scoring intervals need to be sufficiently itY is somewhat less pertinent than is construct validity. sensitive to permit the discrimination of dif- The establishment of construct validity can be under- ferent levels of clinical performance; taken by hypothesizing outcoi-nes of performance for 5. prior to establishing the validity and reliability of various groups on the problem solving test, and sub- the instrument, extensive pre-testing is undertaken sequently adi-ninistering the test to determine whether to deten-nine its usability and capacity to measure the hypothesized outcomes occur. In instances where all relevant aspects of the specific clinical skills; other tests of the same clinical performance exist, the 6. if the instrument is to be used by a rater who correlations between the test being developed and these observes an actual or simulated clinical setting, it other measures should be estimated. should not attempt to measure more than can Regardless of the kinds of validity and reliability reasonably be observed and recorded by a single which are considered appropriate for a specific measure individual. [If two or more simultaneous of clinical performance, the subjects on whom validity dimensions of clinical performance are to be ob- and reliability studies are conducted should closely served, additional instruments can be developed resemble the population for whom the test has been 56 ir composition and relevant physical examination, laboratory procedures, and designed, in terms of the therapy. particularly relevant to measurement of this characteristics. asurei-nent of kind is that the evaluation of clinical performance be a Two general approaches to the me specific disease entities and the diagnostic clinical performance may be taken: 1) the direct ob- function of hicli are indicated f L' servation and measurement of actual or Simulated and therapeutic procedures W or each. clinical situations; 2) the indirect measurement of actual Thus, the validity of the conclusions drawn from this or simulated clinical situations. kind of measurement depends not only upon the ap- The advantages of evaluating actual clinical situations propriatciicss and sensitivity of the observation forms, result primarily from the difficulties in simulating some but upon the clinical competence of the observer who in of the complexities and spontaneous aspects of actual an actual situation must not only record physician be- problem solving settings. For example, the clinical per- havior, but must develop his own diagnosis in order to forrnance of some medical technologists requires the use evaluate the appropriateness of that behavior to the of actual specimens, thus rendering observations under particular clinical problem. simulated conditions considerably distorted and of While the direct measurement of a physician's limited value. This sai-ne difficulty is posed by the use of complete iiiani(,,eiiient of a clinical case results in a more simulated patients from whom the physician could take comprehensive evaluation of physician performance, a history and perform, in some instances, a physical soriie studies have focused upon specific components of examination, but on whom it would be impossible to patient iiiatiioeiiient (2, 14). Foster and Lass (14') perform laboratory procedures not only because the 0 obtained data would be inconsistent, but because of the will soon be reporting, procedures for the measurement understandable unwillingness of subjects to underco and evaluation of patient interviewing. The measurement 0 of patient interviewing skills can emphasize content such experiences. Thus, the use of simulated clinical set- (how much and what kinds of information are elicited) tin(ys restricts to some extent the range of skills which LI and/or process (the techniques used to elicit infon-na- can be measured. tion). In order to measure the process of patient inter- However, since the measurement of clinical per- viewing one needs to: 1) identify those dimensions formance is generally for the purpose of assessing the which will account for all possible aspects of interaction; effects of an independent variable upon clinical problem 2) determine whether these dimensions are essentially solving behavior, or to make comparisons among individ- verbal or non-verbal; 3) develop observation forms which uals regarding their clinical competence, the use of provide sufficient scope and flexibility to permit the actual clinical settings may pose difficulties in obtaining recording of relevant aspects of communication and uniform testing conditions and in securing adequate interaction. numbers of subjects. Thus, if one wanted to measure the effects of an instructional film on the management of Barrows and Abrahamson (4) have reported the use hypertensive patients in a hospital clinic one would need of trained actors to simulate patients with neurological a sufficiently large patient population randomly assi-ned disorders in order to measure history taking and physical to clinic physicians in order to permit valid conclusions examination skills. Although the use of the programmed to be drawn. patient imposes limitations upon the kinds of disorders The decision to employ either direct or indirect which can be simulated, the pre-determined nature of measurement of clinical performance in actual or the medical setting permits more accurate evaluation of simulated situations will usually be based upon a number the extent to which pertinent data have been uncovered of considerations such as: 1) the kind Iof clinical skills to by'the examinee. be measured; 2) the availability of subjects and ob- In a somewhat different approach to measuring servers; 3) the number and extensiveness of the clinical competence in data gathering and analysis@ Cline (7), skills to be measured; and 4) the amount of time Langsley (19), and Levit (22) have reported the use of required for observation. motion pictures to assess observation and interpretive Peterson's study of North Carolina general practi. skills. The films which consist of a history and physical tioners (28) represents perhaps the most comprehensive examination show a wide range of signs and symptoi-ns attempt to measure physicians' clinical problem solving which are both pertinent and non-pertinent to the skills by direct observation of an actual situation. The formulation of a correct diagnosis. The data is presented observation forms developed by Peterson and his col- with equal emphasis and in such a manner that the leagues measure the physician's skills in history taking, exan-dnee must analyze all data, make judgments about 57 their relative significance, and draw conclusions con- 5. Beaumont, Graham, et al. "Medical Auditing in a Comp cernitig the nature of the patient's illness. liensive Clinic Program." Journal of Medical Educati4 The medical audit, which in essence is an a posteriori 42:359-367, April, 1967. evaluation of the clinical management of an actual case, 6. Butler, John J., and Quinlan, J. William. "Internal Audit the Department of Medicine of a Community Hospita has been the subject of numerous articles Journal of the A?izerican Medical Association, 16 7:567-5' (5,6,20,21,26,27,29,30).-Such a process requires the May 31, 1958. careful establishment of criteria by which the medical 7. Cline, Marvin. "A Film Test of Clinical Skills in Medi record is evaluated and the training of medical specialists Students." Journal of Medical Education, 36:908-9 who will serve as auditors of the medical record. There is August, 1961. 8. Colton, Theodore, and Peterson, Osler L ' "An Assay the danger, however, that one may be measuring the Medical Students." Abilities by Oral Examination." Jo@ri accuracy and completeness of the medical records them- of Afedical Education, 42:1005-1014, November, 1967. selves, rather than the clinical performance of physicians. 9. Cowles, John T. "A Critical Comments Approach to t Yet another indirect evaluation of clinical problem Rating of Medical Students' Clinical Performance." Jouri solving is the so-called "patient management problem" - ofMedicalEducation, 40:188-198, February, 1965. 10. De Tornyay, Rlieba. "Measuring Problem-Solving Skills a written simulation of a clinical case which measures Means of the Simulated Clinical Nursing Problem Tesi data gathering and interpretive as well as decision- JouriialofNursingEducation, 7:3-8, August, 1968. inal@ing skills (10,16,25,31,33). Although its use has 11. Donabedian, Avedis. "Promoting Quality Through Evaluati been reported primarily with physicians and nurses, its the Process of Patient Care." Medical Care, 6:181-202, M@ applicability to other health professionals appears June, 1968. 1.2. Evans, Lloyd R., Ingersoll, Ralph W., and Smith, Edwin Ja feasible. The problem-solving exercise is initiated by a "The Reliability, Validity, and Taxonomic Structure of t brief description of the patient and consists of "a series Oral Examination." Journal of Medical Educatio of sequential, interdependent decisions representing the 41:651-657, July, 1966. various stages in the management of the patient" (25: 1) 13. Foster, Judilynn T., et ad. "Analysis of an Oral Examinati( in which the results of each decision are given in the Used in Specialty Board Certification." Journal of ilfedic Education, 44:951-954, October, 1969. form in which the health professional would receive 14. -, and Lass, Sandra L. "'ne Identification of Intc them in an actual clinical setting. Moreover, the problem action Patterns in Student-Patient Communication@ not only allows the examinee to make a wide range of Abstract of a Paper to be presented at the 1970 Confereni decisions froi-n very harmful to very helpful, but forces on Medical Education, Association of American Medic Colleges. him to deal with the consequences of his decisions by 15. Hinz, Carl F., Jr., "Direct Observation as a Means of Teachil presenting additional choices through which the andEvaluatingClinica]Skills."JournalofMedicalEducatio, examinee can either correct or further coi-npouiid his 41:150-160, February, 1966. n-dstakes. Allowances are also made, where applicable, 16. Hubbard, John P. "Programnied Testing in the Examinatioi for the use of more than one acceptable diagnostic or of the National Board of Medical Examiners." Education Testing Service, Proceedings of the 1963 Invitatiot7al Co therapeutic procedure. ferelice on Testii7gProblems. . The following selected bibliography has been included 17. ., et.al. "An Objective Evaluation of Clinical Con so that individuals wi@ng to do so may further explore petcncc: New Techniques Used by the National Board ( the literature on clinical performance measurement. Medical Examiners." New England Journal of Medicit2, 272:1321-1328, June 24, 1965. Selected Bibliography 18. Kilpatrick, G.S. "Observer Error in Medicine," Journal ,Ifedical Education, 38:38-43, January, 1963. 1. Abrahamson, Stephen. "Evaluation in Continuing Medical 19. Langsley, Donald G. "Fflmed Interviews for Testing Clinic@ Education." Journal of the Ainericaii Afedical Association, Skills." Journal of Medical Education, 45:52-58, Januarl 206:625-628, October 14, 1968. 1970. 2. Adler, Leta McKinney, and Enclow, Allen J. "An Instrument 20. Lenibcke, Paul A. "Evolution of the Medical Audit." Jotirne to Nleasure Skill in Diagnostic Interviewing: A Teaching and of the Aitiericait MedicalAssociation, 199:111-118, Februar Evaluation Tool." Journal of Aledical Education, 20, 1967. 41:281-288, March, 1966. 21. . "Medical Auditing -by Scientific Methods. 3. American Psychological Association, et al. Standards for Journal of the Ainerican Medical Association; 162:646-65' Educational and Psj,chological Tests and Alatiuals. Washing- October 13, 1956. ton, D.C.: American Psychological Association, 1966. 22. Levit, Edithe, J. "The Use of Motion Pictures in Testing th 4. Barrows, Harold S., and Abrahamson, Stephen. "The Pro- Clinical Competence of Physicians." Annals New Yor graiiimed Patient: A Technique for Appraising Student Per- A cadeiny of Sciences, 142:449-454, March 31, 1967. fonnance in Clinical Neurology." Journal of iVedical Edtiea- 23. NlcGuire, Christine H. "Medical Education, Part 1: A Scien tioii, 39:802-805, August, 1964. tific Approach to Problems of Professional Assessment. 58 Canadian Medical Association Journal, 100: 593-598, April 5, Practice, 1953-1954. Evanston, Illinois: Association of 1969. American i\lcdicai colleges, 1956. 24, _. "The Oral Exai-nination as a Measure of Profes- 29. Phancuf, Maria C. "Analysis of a Nursing Audit." Nursing siotial Competence." Journal of Medical Education, Outlook, 16:57-60, January, 1968. 41:267-274, March, 1966. 30. . "The Nursing Audit for Evaluation of Patient 25. -. "Simulation Technique in the Measurement of Care." Nursing Outlook, 14:5 1-54, June, 1966. Problem-Solving Skills.. Journal of Educational Aleasure.- 31. Rinioldi, H.J.A. "The Test of Diagnostic Skills." Journal oj' metit, 4: 1 -1 0, Spring, 196 7. Medical Education, 36:73-79, January, 196 1. 26, Morehead, Nlfldred A. "fhe @Nledical Audit as an Operational 32. Vigliano, Aldo, and Gaitonde, Mangesh. "Evaluation of Tool." American Journal of hiblic Health, 57:1643-1656, Student Performance in a Clinical Psychiatry Clerkship." September, 1967. Journal of.Ifedical Education. 40:205-213, February, 1965. 27. Payne, Beverly C. "Continued Evaluation of a System of Medical Care Appraisal." Journal of the American lvledica,l 33. Williamson, John W. "Assessing Clinical Judgment." Journal Association, 201:126-130, August 14, 1967. of medical Education, 40:180-187, February, 1965. 28. Peterson, Osler, Andrews, Leon P., Spain, L.P., and Green- 34. Wilson, G.M. et.al. "Examination of Clinical Examiners." berg, R.S. "An Analytical Study of North Carolina General Lancet, 1:37-40, January 4, 1969. 59 WORKSHOP ON THE EVAI,UATION OF CHANGING [IEALTH STATUS Participants Robert R. Carpenter, M.D, - i%loderator Howard R. Keltiiaii, Ph.D. Director, Western Pennsylvania Regional Department of Preventive Medicine Medical Program and Public Health New York Medical College Sam Shapiro Director of Research and Statistics Health Insurance Plan of Greater New York Nlaureen M. Henderson, NI.D. Charles E. Lewis, iNI.D. professor, Preventive lvledicine Prof@sor and Head Department of Preventive Medi@e and Health Administration Division Rehabilitation School of Public Health Uni@-ersit%l of California University of Maryland Transcript of Workshop-Remarks by Nloderator seldom that any individual's results are sufficiently better than those of his colleagues so that he would ROBERT R. CARPENTER, M.D. desire such comparison. Perhaps the results as a whole DR. CARPENTER: Thanks to Mitch Schorow, I would not be good enough to impress the public very favorably. (2) An effort to thus analyze is difficult, time- found an interesting book, published in Boston in 1917 by E. A. Codrnan. It's called "A STUDY IN HOSPITAL consuming, troublesome and would lead, by pointing EFFICIENCY, A DEMONSTRATION BY THE CASE out lines for improvement, to such onerous committee REPORT METHOD OF THE FIRST FIVE YEARS IN work by members of the staff." A PRIVATE HOSPITAL." "Neither trustees of the hospital nor the public are as It says by way of foreward that this hospital has for yet willing to pay for this effort." "Althouc,h the staff would admit that such follow-up sale a product of a standard which is to be described on 0 pages 12 through 63, It aims to be a S I 00 hospital with analysis was a good thing for all, yet each practical man a $100 surgeon. - and the practical men always hold the power - would The volume is dedicated to Richard Cabot because wait for somebody else to do the work." Dr. Codman respected his motives and admired his And he goes on to point out that the superintendent courage and energy though he heartily disapproved of would be the last one to undertake this task because he some of his opinions and methods. "He seems to want to surel@, would lose his job. reform the profession from the bottom whereas I think I enjoyed, that 1917 description of what we are trying the blame belongs at the top," says Dr. Codman. to do in Western Pennsylvania in 1970 and I don't know The case report is subtitled "A Practical Illustration that we have come terribly far in our ability to measure of the Fact that It's Possible to Use the End Result health status and particularly any change in health status System in a Hospital." attributable to any of our efforts to improve what we And the first page I think suggests how little progress are doina.' we have made since 1917: "The trustees of our chari- Yesterday we heard the public and the Bureau of table hospital do not consider it their duty to see that Budcet -- good morning, Dr. Fox - ask for health status good results are obtained in the treatment of patients. outcome measurements. I think RMPS asked for end They see to it that their financial accounts are audited results but not really health status end results. They and they take no inventory of the product for which were asking for lower cost and better distribution of their money is expended." care which is significantly different than outcome "It is against the individual interests of the medical analysis going to end results. and surgical staffs of hospitals to follow up, compare, I was interested in this workshop above all of the analyze and standardize all their results because (1) it is others. Since I am interested in the Regional Program as 61 a way to improve health, I want to know how to The acceptance of the desirability to detern-dne the measure health and its improvement. effect of health programs on the well being of a popula, I think we have an unusually talented group with us tion is quite general not only among researchers, but alsc this morning to help us do this. The speakers who will among planners, administrators, and among those enter into discussions with you off and on during the responsible for allocation of resources. morning have spent a goo d many years measuring health This acceptance moves from a state of passivity to status; I look forward to learning a great deal from them. worried preoccupation when change is contemplated or They are Dr. Henderson, Mr. Shapiro, Dr. Kelman alternatives weighed in circumstances ranging from a and Dr. Lewis. highly specific component of health care to the broad I want to show you just four numbers as an example design of organization and financing of health services. of the problem and promise of end result evaluation. We It usually slips back to an uneasy but quiescent state looked at the hospitals that serve a community of when the complexities of end result measurement, costs, 200,000 and at the mortality from stroke in those and time requirements become apparent. hospitals. We were surprised to find that patients with This is not in the nature of a sharp criticism of the heavy paresis when they were cared for by generalists past. The difficulties of assessing the impact of particular died more frequently whether they were male or female actions on health status were and still are great. Further, than did patients with the same reported neurologic the introduction of changes affecting the availability, signs if they were cared for by internists. delivery and economics of health care often could not I hoped from this la-t we could attract the medical and will not in the future wait for hard evidence from staffs' interest to more careful care of stroke, attract studies of impact. interest in helping to understand these results. Similarly for the introduction of some programs We identified some cases that the medical staffs were aimed at modification of primary and secondary particularly interested in reviewing: The patient who prevention of specific diseases. died with a diagnosis of cerebral vascular disease without However, many of the problems and issues we face any neurologic sicns, for instance. are stubborn, and courses of action are not at all certain. 1 hope as the morning goes on that we can learn the Because selection of an available alternative often value of such measures of outcome as others who have involves commitment of scarce manpower, equipment made them more frequently have seen this value. I hope and financial resources for which there is sharp com- that we can find out when it's worthwhile to make such petition, implementation faces serious obstacles. measures and how to make them. I hope we can learn As we all know, these are the considerations that how to interpret them once we have them. I hope that force i-nany of us to think in terms of demonstration we can learn, particularly from Dr. Kelman, the key data projects or R & D projects in which operational @ffec- bits that help us to measure and talk about outcome. tiveness related to costs and manpower is a central con- And, finally, I hope we can learn how outcome and cern. process analysis interrelate. Now, this is fine, but often the question that will Mr. Sam Shapiro will begin the discussion. He will remain even after a project has been well execut ed is describe some of his studies and discuss why and when it whether any health benefits have resulted. is worth measuring health status. Bypassing the issue can compromise the potential for i-noving from demonstration to general acceptance. In The N'alue of Health Status Nleasures fact, where the effort required for the'extension is great, absence of evidence of impact on health status may well SAM SHAPIRO prevent such extension. Conversely, availability of evidence of a program's I'm not sure that I'm going to be dealing with the health benefit can stimulate widespread consideration of questions precisely the way you have outlined them. early implementation. What I thought might be useful is for me to give you I want to emphasize that many prograi-ns cannot be some general considerations that underlie the concern nor need be tested for health benefits although there are with measurement of health status changes and then use programs under active consideration today that will be a few samples principally from my own experience to plagued by doubts and challenges until the issue of illustrate what's really at stake when you get involved health benefits is dealt with effectively. Just to mention with health status measures. a few: early disease detection through automated multi- 62 breast and mammography. The measure is change in All screening exai-ninations have been completed. An mortality from breast cancer. at every stage of the investigation when findings wer This is the situation. It's generally acknowledged that reviewed it was clear that mammography and clinic, screening will lead to earlier diagnosis of breast cancer, examinations contributed independently to the dete( but there has been no evidence that this results in lower tion of breast cancer. If mammography had beei mortality. Costs for including breast examination with excluded, 31 percent of the cancers would have beei mammography, in particular, are high. And, in fact, in missed during screening. If the clinical examination ha( automated multiphasic health testing programs where been omitted, 44 percent would have been missed. this procedure is used, mammography is the most costly Further, screening did lead to detection of large single test. proportions of breast cancer with no evidence of axillar3 In short, a national effort to screen women for breast nodal involvement - 70 percent - than among the con cancer would require massive expenditures and diversion trol group - 45 percent. of equipment and manpower from other health care Preliminary results on mortality are now beginning t( activities. be collected and will shortly appear in an article ir Clearly, to acquire a high priority, breast cancer JAMA. The findings are highly encouraging. There arc screening should justify its value in the most rigorous 52 deaths due to breast cancer in the control group a@ manner possible. And as many of you know, a compared with 31 breast cancer deaths in the total study randomized clinical trial directed to this issue has been group in the period available for follow-up. underway since 1963 in HIP under a contract with the The case fatality rates for cases with histologically National Cancer Institute. confirmed breast cancers reinforce the impression that, The main objective is to establish whether breast in the short run at least, screening leads to lowered cancer screening using mammography and clinical mortality. examinations results in a reduction in breast cancer These observations are preliminary, and more time is mortality. Other objectives relate to the epidemiology of needed to establish whether the effect of the screening breast cancer and the search for high-risk factors that program is short-term or long-term. might be useful in future screening programs. However, the findings do provide grounds for I don't want to go into the details of methodology. cautious optimism and it would appear prudent to ac- These have appeared elsewhere. But a few key points are celerate efforts to develop and test methods capable of important for me to touch on in this discussion. dealing with the broad demand for periodic breast ex- Thirty-one thousand women aged 40 to 64 enrolled aminations that might emerge within a few years. in HIP have been assigned randomly to a study group What I'm describing is a progression from very and a similar random sample to a control group. Only intensive study involving huge resources, a long period of study group women have been invited for screening time, and dedication of large numbers of personnel to examinations. About 65 percent appeared for the initial achieve a result which if sustained can significantly screenings. affect the approach that medical care might be taking to Three additional screening examinations at annual the whole issue of screening for breast cancer. intervals were scheduled, and large proportions of the Those of you who have been close to this field over the years know how much disappointi-nent there has women with an initial examination have returned for these. been in dealing with the problem of breast cancer, and how widespread is the pessii-nism about the effectiveness Control group women continue to receive their of breast cancer screening. regular medical care. There is a great deal at stake in this study, and as I see Screening examinations have been performed at 23 of it, these preliminary results are placing high on the the HIP medical group centers. The clinician and radio- agenda a new set of concerns, mainly related to the logist record their examination findings and recoiii- question, "what kind of screening prol@prrn would be mendations independently. Later their findings are required to reach effectively @ll.(Te reviewed jointly by a physician for final recommenda- the present findings persist?" tions. Intensive follow-up to identify breast cancers I want to turn now to a much broader type ot'effort diagnosed and mortality is carried out with equal rigor in the field of preventive care, that of automated multi- for women screened, women who refused screening, and phasic health testing. There are many justifications control group women. advanced for introducing AMHT, and I don't want to 64 covering a broad a(,e r,,itige-12 years and older-ftoni an make out a case for or against such programs, However, 0 everyone will agree that AMHT is a costly addition to -,absolute standpoint and relative to a noiipoverty group the spectrum of health services and most will agree that that will also have AMI-IT. hese health it is il-nportant to seek out opportunities to assess Action to modify adverse aspects of t AMHT's effect on health status and on behavior that components aiiiono the poor is to be instituted, and might be expected to have a desirable effect oil health evaluation is in terms of change asIcompared with what and well being. occurs in the iioiipoverty group. Two projects are now directed to this issue. The one An underlying question is whether through the of longer duration is being conducted by the Permanente AMFIT program, and activities generated by it, the Medical Group in California. One phase of that program anticipated gaps between the two groups can be nar- is very well known, probably much better known than rowed. the second phase which deals with the end result issue. A broad spectrum of measures are being developed to This is to demonstrate how automation and computers measure liealtli impact. These include chan-es in itn- can be applied to improve - and I'm now quoting - pairnient of function, immunization status of children, "speed, efficiency, and quality control in multiphasic and complications of disease. screening techniques so that not only more tests, but The last project I want to describe is in the proposal more accurate and quantitative measurements can be stage and is now being reviewed for possible funding. It performed, and at a lower cost." All very important concerns sudden death from coronary heart disease. operational objectives. There is cencral agreement that until effective The other phase of the program includes a set of end primary prevention methods can .be identified and result criteria in the evaluation. Two randomly selected implemented, significant progress in reducing the in- samples of the plan's member have been designated cidence of this cause of death will depend on changes in study and control groups. Efforts are made to have the community practice which bring advances in coronary study group appear for the examination. The control care to patients who under present circui-nstances do not group is not approached, but those who request an survive to reach a hospital. examination are accommodated. It is estii@ated that about 60 percent of deaths due to Morbidity, disability, and medical utilization patterns acute myocardial infarction occur outside the hospital, are to be determined over a long period of follow-up and a great effort is being made to cope with the prob- through periodic questionnaires and medical records. lem of rapid response to requests for medical care when This is an ambitious undertaking. But it has the a heart attack is suspected. Also, increasing, attention is potential of providing decisive information on the value being given to finding out how patients and their of periodic health examinations generally and of selected families behave when faced with prodromal symptoms. components of it particularly. The proposed project is designed to incorporate these Anyone who questions the time requirement for approaches in a comprehensive action program. It repre- reaching an answer really has to look very hard at other sents a combined effort of HIP and two of its Queens issues that have come up in the past which have been medical groups with a population of about 50,000, aged plagued by doubts and questions long after the point in 35 to 74, and HIP's LaGuardia Hospital which serves time when it would have been possible to initiate an end both medical groups. result investigation. The goal is to effect more rapid requests for medical One of the outstanding examples is the Pap smear. It care after the onset of a heart attack or suspected heart is no longer possible to carry out a control study in this attack and to institute a system capable at all times of a country on Pap smear as an effective measure for rapid and appropriate response which fully utilizes reducing mortality from cervical cancer. There are very current medical knowledge. few people on the firing line who really raise any The end result sought is a reduction in the present questions about Pap smear. But if you look at the high rate of sudden dealth from coronary heart disease. scientific literature, there are some very serious ques- Basic changes to be made in the health services sys- tions being raised about the Pap smear. tem consist of the following main elements: The second end result study in multiphasic health testing recently started at HIP. This project is utilizing Patient education. Varied educational approaches repetitive health testing to define the health status, will be made to the entire adult population of the practices and attitudes of a defined poverty population two participating medical groups and., their 65 physicians with the aim of reducing delays geii- Each of the studies I have described contains an er, erated by patients or their families in seeking result criterion. In the breast cancer study we probab'. medical care for possible acute coronary episodes. have the hardest type of evidence. It's a single measure A special target will be individuals at relatively mortality. A randomized clinical trial approach has bec high risk for sudden death (those with prior CHD, used, and it takes a very unusual set of circumstances i hypertension, hypercholesterolemia, etc., as iden- make this type of approach a practical one. tified through the HIP centralized medical record The other investigations shade off in hardness, but a system). effort is made to maximize the opportunity within tf o Centralized telephone screening at LaGuardia medical care setting where the programs are to be carric Hospital by physicians of calls from all possible out, to reach sufficiently hard conclusions about the e coronary suspects in the population will take place fectiveness of the programs, from the standpoint 24 hours a day, 7 days a week. The aim is to health status measures, to serve as a basis for futui reduce communication delays in bringing the action. patient's symptoms to trained medical attention. One question that often comes up is whether all o Operation of a special pre-coronary care area most demonstration programs should attempt to ii (PCA) at LaGuardia Hospital for observation of corporate an end result criterion? I don't believe s( patients in defined categories, one of which Costs are high. Technical requirements are g-reat. Ar consists of persons who do not meet usual current frequently the kinds of issues that are being faced ai criteria for hospitalization, but who may be in an not susceptible to the inclusion of an end result measur early stage of an acute MI not yet recognizable. But in the field of medical care, with all the change The other consists of patients who might be ex- that people hope will take place over the next decade periencing an ischemic episode not destined to maybe they're being optimistic about the next decad lead to MI but capable of inducing a fatal ar- but let's say the next generation - there are very larl rhythmia. issues with very large stakes associated with them. NI For purposes of this meeting I think what is of particular point is that it is essential to seek out those few si importance is that two types of evaluation have been uations where such issues can be investigated effective' planned for. The first is directed at those aspects of the utilizing end result measures and thereby provide tt project that bear on generalizing experience for possible basis for making judgments that have regional i use by other organized providers of medical care in national implications. Queens and the New York area. Information will DR. CARPENTER: Thank you very much. I gath become available regarding the operational effectiveness that hard work makes a man cautious. It seems that yc of the educational program, communication procedures are an enthusiast for health status measurement und for rapid response to patient's call, training of para- proper conditions. medical personnel, and the operation of the pre- I heard you saying that the detailed effort required - coronary care area. carry out a study significant enough to be generaliz( This information would be related to manpower from one hospital to another is very great indeed. Y( requirements and costs. alluded to an indirect method of end result analy! which sounded as though it might be more often a By itself, this would represent an important advance plicable to the problems faced by Regional Medic in knowledge concerning the modification of health care Program evaluators. In just a moment we will have iystems to reach a patient early when a heart attack chance to discuss these and some other issueswitliyo occurs. However, we would still be left with the uii- Before we go on, it's worth noting that the man wi answered question as to whether the effort involved does is evaluating the evaluation conference is with us-GI( have payoff in reducing mortality. Hastings from Nassau-Suffolk. Welcome, Glen; we'll I A second type of evaluation has been included which very careful what we say from here on. is aimed at answering this question. The approach is to Before we discuss Mr. Shapiro's paper, Dr. Heiiders( compare the rate of sudden coronary heart disease will speak to closely related issues. Maureen will discu deaths in the demonstration groups with the rate in some of the problems of end-results anslysis, particular other HIP medical groups, and also provision has been as she experienced these within the framework of t] made to compare the mortality situation in the demon- Maryland Regional Medical Program where she is A stration groups before and after start of the program. sociate Director of the Epidemiology and Statisti 66 Center. She will also discuss the relationship between Let me briefly describe the collection of the informa- evaluation and medical care research. tion I ai-n going to show. We took a random sample of admissions to every short-term general hospital in the region during a I -month period just before the regional MAUREEN M. HENDERSON, M.D. medical program belan. The. data, therefore, describe patients and procedures in every "acute" hospital I propose to review a verv different level of research whether or not it prepares its own statistics or has easily from Sam Shapiro. One major value of this workshop is accessible records. In most other morbidity surveys, in- the way it illustrates the need for many different dis- formation is collected only froi-n hospitals with viable ciplines and approaches in the evaluation of health (for research) record systems. services. Mine is a very limited approach within the total The Maryland Re-ion includes all of Maryland except context of health services research and evaluation. As an Mont(yolnery County and includes York County, Penn- epidemiologist, I am most interested and only corn- sylvania. petent to deal with biological measurements. The end The specific medical records reviewed in our sample results I have been looking at in relation to the Maryland were identified by our own staff and abstracted by Program have therefore been measurements of mor- trained medical abstractors tinder constant quality bidity. control and surveillance. Standardized abstracting forms I believe it is important to talk about ways of i-naking and procedures were used. these particular measurements because non-epidemi- The measurement data collected were specifically ologists are not always aware of the series of con- selected to: foundino issues and problems related to their observa- 1. get estimates of need; tions. 2. look at the secular effects of the total program and I trust those present who are sophisticated in epide- of individual pro-rams miologic techniques will bear with the fundamental LI levels I am going to discuss. 3. insure proper coi-nparisons in assessing needs or effects. Let us first consider biologic outcome measurements in relation to overall evaluation of regional services. The The last purpose is one Sam Shapiro spoke about very two types of measurements consistently used are those briefly and one on which I would like to enlarge. When- of death and morbidity. We have made very little use of ever you examine an effect or an end result in different death records. time periods or between different groups of poeple or In looking at the picture of our total region, we have different geographic areas, you must be sure that you are been using case fatality rates. The latter are of limited comparing like with like. The original numbers that Bob use now for two reasons: (1) there have been great Carpenter presented draw attention to this point, and changes in denon-Linators-the census population from because he mentioned that he was going to show those 1960-1970 and (2) death rates have been at a standstill particular stroke data, I brought some of our own stroke for the past few years. data to illustrate and ai-nphfy this point. In terms of disease or morbidity we are looking at the This slide describes short-term general hospital dis- prevalence and severity-that is, the distribution of the charges in the region of Maryland before the Regional frequency and severity of disease as we see it in the Medical Program began. It shows annual case fatality region. We are also looking at aggregations of disease- rates from four hospitals. The rates are estimated from that is, multiplicity of disease problems in patients who our sample. They vary enormously from 16 percent to are found at points on each disease spectrum. 60 percent between the four hospitals. Just looking at One good example is the presence of cardiovascular the total numbers, you might infer that the hospitals diseases and diabetes mellitus in stroke patients with seeing the most stroke patients give the best care and rnild or serious neurological deficits. have the lowest case fatality rate. But in Maryland there The greatest amount of the data we are currently is a great difference in the patients admitted into dif- studying comes from hospital in-patient records. I think ferent hospitals. The easiest and quickest way to it is appropriate to speak mostly about hospital in- describe patient differences is to look at tire -r -acia-l- dis-- patient records today because I am sure that most tribution. The next slide shows how proportions of ,regional programs use these as their major source of mor- black and white patients differ from one hospital to the ".@bidity data. next. 67 of analvses The next slide shows one of a whole series Pdtiellts had at least one heart Iiseas., t@@ id@-Iltifv truly comparable groups of patients. With you are looking at tire otitcoll, 'of sti ,:c)niparable groups of patients we can begin to look at one place or from one liosiiital to til 11 the outcot-ne of care in different groups of hospitals. ignore the fact that a lot of p@itielits Ili this analysis we divided all "immediately ad- disorders such as heart diseise @k7ili,:Il II iiiitted" stroke patients accordance to the severitv of their lfllood of survival and recovery. Once le@s condition on admission. Classes of severity are in ranking adjust for the presence of' other dismiss., order and are exclusive. The worst class included all say whether outcomes of different tr@it@il [is I patients who were not conscious; the second identified are more Or less successful. All these ex@!-@,@ those who were conscious but had swallowing difficulty why the first qL)CStiOll epidemiologists c i I 11 r 1.@ The third identified those with speech problems who look at anv evaluation c is: Are the pitietit ,jagc a'L were conscious and could swallow. Tire fourth catcoory The second question is: flave tile 1)1)\ includes those with none of the three more severe con- equal pains to make the cli:I,L,llosjs,., Ili t ditions. Looking separately at the data for white males, coiiipariii- the same diagnosis with ti)C -.Iicr tl]211) white females, non-white males, and non-White females, in this example you see tire frcqLICIIC\ you will see that 20 per cent of the white males were diagnostic technique, E.K.G., in patients "k @vented ti@ unconscious when they were admitted-, fifty per cent of' dia-flosis of heart disease. the non-white males were unconscious when they were fit this slide, I want yot, to look Anotlict admitted@ thirty per cent of the white fei-nales were Lifl- frequency with @vliicil tile test Was LISCLI j.Is -.)Ilective di conscious; and fift), per cent of the non-white females. patterns of use in these patients. We li@i@!c tile nIL'il"io If you go to the other extreme and look at patients regional hospitals into four roups accoicii@@ i@: nice, .@iscIll.,ir acL:i with no severe conditions, YOU WU] see 50 per cent of size. We ]lave used the annual ilul-flbers ot- The "IS the white males; 30 per cent of the white females; none our measure of size. These E.K,G. fFc(iuencie, of the Negro males; and 20 per cent of the Negro fe- t'ore. tabulated from the smallest to the big2c,@ This text sli I atietits Ili li males. These data may, of course, mean that blacks and Remember we are oniv talkiii@ , about whites have different diseases; that we are dealing with with a primary diLtaiiosis of heart disease. Ili II ill@t!!@,,@t ilie se@,,rt%' different age groups in the two races or that the two hospitals, onIN, three quarters of the patients li@id diagnoses of races choose to go to a hospital when they have different of art E,K.G. examination. There was evideilc,! Three l@c,- @'ot l@s cli@i@-,]-,osj manifestations of disease. Hospital admission policy is cent in the largest. In all except the largest file another possible explanation. Whatever the explanation proportions of patients with E.K.G. iiiitted Litid 9 of racial differences, you cannot compare admission out- lower in black (mostly service patients) than it) i. IC iiiajoi- degree comes unless you adjust for at least the severity of (a sizeable percentage of private patients). If oil tli disease at the time of admission. III our data, therefore, the decree of certaii)i rhage. N@oL, One other point I mentioned, that of au relations of heart d' were Llil@Olls bg isease diagnosis is correct is -oiilg to disease, is also well illustrated in stroke patients. one kind of hospital to another and from oi),@ kijit] @)l' per ceiit %@@ci In all Baltimore surveys looking for conditions pre- patient to another. Epidciiiiol disposing to strokes, we have observed heart disease and To repeat: tire second question in the epideijiioI()L,.ists diagnoses ,is general vascular disease behind a majority of pre-stroke mind is: ""'fiat was the extent of the effort their together ill t symptoms. This association shows up again it) our lios- into making a dia@ilosi's and were efforts scribed by co Pita] admission survey. The numbers vou see in the slide alike in different hospitals and among different 1)@iticjiL'S We looked are from the reviewed records before total s@iniple that Outcome measurements can be compai-ed'@"' to see estimates were reconstituted. The slide shows data froi-n Nex t wconsider the recording of tile Pli@siciLill's as having die hemorrlia-e approximately 4,000 stroke patient records. Different diagnosis @ifid Helical information. What do Illc il@@()@ I cerebro@'ascul stroke diagnoses are listed across the top of tile table ,ilici librarians do Nvitli them. To get son ic estimate (ii, !II,- One-third down the side are listed other majoi- chronic diseases. Possible V@iiltiol]S WC Should exi) ect front this 1)@il The numbers and per cent of stroke patients with these source. we took two or three troublesome hemorrhage I other diagnoses are shown in the cells of the tabl( . sent them to record librarians ill a majority ol'I-l@)II)i@@! cerebrovascul The heart disease category shows the most obvious and asked tl)ciii to @@ code" them t'oj Lis. tMrd) of addi relationship. For every stroke diagnosis, a Iii-ii pro- This slide shows tire three dta,ail(,),Ses died the diagnoses we portion of discharge records have i secondary diagnosis International Classification Codes ,iveii to flesh i),, noses. of heart disease. More than 50 per cent of stroke of our reL:ord librarians. The fir In the pati St One is oil" tilL@, of those giv 68 I @vill speak about aaain later-cerebrovasctilar accident. number had actually had a diagnosis of cerebrovasculir c Twenty-five out of the 29 librarians coded it as cerebral accident, In a sample of death certificates, again more hemorrhage which is correct procedure by instructions than a third had cerebrovascular accidents. Once more given in the index of the classification manual. Four the decision of the "coder" to put cerebrovascular ac- librarians did not use this code number. The librarians cidents with cerebral hemorrhages and the proportion of were less consistent for the second dia-nosis of transient each in the total group of patients can niale a lot ot' ischemic attacks. The third diagnosis of chronic bron- difference to end-point measurements. In our stroke chitis really gave inconsistent coding results. We were registry, we code all diagnoses separately so our end- particularly worried about this disease because the results for cerebral hei-norrhage will probably differ from average age of admissions with a code number for a majority of others. chronic bronchitis was between 30 and 40 years. I would now like to talk about a different kind of' Obviously the group includes patients with more than bias; one I mentioned earlier and one that I did not fully (other than) chronic bronchitis. In this case we felt that appreciate before we started this survey. Our usual mor- the difficulties inherent in coding chronic lung diseases bidity data come from the records of our best hospitals. prevented us from learning about true distributions of By "best" I mean the biggest hospitals with adequate that disease from our samples of morbidity data. record keepina facilities and the most accessible diag- Another classification problem arises in relation to 0 from which nostic indices. These are the only hospitals collective diagnoses and group outcomes. investigators and planners can easily get the kind of I mentioned above that the diagnosis of cerebro- listings of record numbers and diagnoses needed to vascular accident had concerned us in another context. collect morbidity data. The following slides illustrate The International Statistical Manual suggests that cere- why this is so. This slide shows the status of our record brovascular accident be coded with cerebral hemorrhage. C, rooms in Maryland at the beginning of 1969. Twenty- This next slide shows specific diagnoses given to stroke one hospitals (half) could produce-a computerized list of patients in hospital records. They are cross-classified by their admissions and used the International Classifica- the severty of stroke on admission. If you look at the tion. Thirteen had a card file and used the I.S.C.D. We diagnoses of cerebrovascular accidents, you will see that went through these hospitals card files by hand and three per cent of patients admitted to our hospitals with made lists from which we could prepare samples. Seven this diagnosis were unconscious when they were ad- other hospitals had a card file and used standard @tted and 92 per cent were free from any of our three nomenclature. For these hospitals we had to develop a major degrees of severity. code compatible with our selected I.S.C.D. categories If you then look at the diagnosis of cerebral hemor- and we had to go through the card file by hand to rhage, yot@ will see that 50 per cent of these patients identify all compatible diagnoses in the given time were unconscious when they were admitted and only 18 period. At the time we did the survey two hospitals were per cent were without all three severe degrees of disease. without a filing system., We sampled from all of their Epidemiologically, these are two very different records for one year and read large numbers of records diagnoses as physicians give them but they are lumped to get our balanced sample of patients with stroke, heart together in the descriptions of groups of patients de- disease, cancer, diabetes mellitus, and chronic bronchitis. scribed by code numbers in hospital statistics. The crux of the matter is that the likelihood of getting a We looked at our total morbidity and mortality data list of patient discharge diagnoses varies enormously to see what proportion of the people we were counting from the larger to the smaller hospitals. An even harder as having died or been admitted as a result of a cerebral problem to deal with, and one that limits available data hemorrhage had, in truth, been given the diagnosis of more than the actual mechanization of the index system, cerebrovascular accident. are hospitals that fail to identify which listed or coded One-third of discharge diagnoses coded as cerebral diagnosis was the reason for admission. hemorrhage had, in fact, been a primary diagnosis of cerebrovascular accidents. The same proportion (one- They simply write every listed diagnosis into their card third) of additional non-primary "cerebral hemorrhage" file with no indication which one the physician listed diagnoses were actually cerebrovascular accident diag- iirst. noses. The next slide shows that the proportion of hospitals In the patients' past medical histories, more than half that can identify primary discharge diagnoses increases of those given a cerebrovascular hemorrhage code steadily from the smallest to the largest group. However, 69 not all of the largest hospitals identify primary diag- We have tried to look at the patterns of care and floA noses. This failure is a major barrier to collection of patterns throughout the region. One of our choser evaluation information. You may want to know about measurements was the interval between onset of symp. patients with heart disease. If you go through all the toms and admission to hospital, The next slide shows index cards and count all people admitted in a certain these intervals. I would like you to notice the "not period of time with heart disease, you end up with a recorded" column. About 20 per cent of all records in count of everybody who had heart disease listed in any the sample were without information that would help us ranking order among their discharge diagnoses. This decide the delay between onset of symptoms and hos. specific problem almost doubled the staff work needed pitalization. These incomplete records were con. to abstract information for our survey. To make sure centrated in the smallerhospitals. that my complaints are about systems and not medical records staff let me first show you evidence of the Any assumptions from these data about patterns of magnificent effort and cooperation of our regional medical care have to be made with the knowledge that i-nedical records departments. We asked for about 21,000 one in five pieces of information is missing. It is even medical records. The percentages at the bottom of the harder to find information about the places patients next table describe the few records the record librarians were discharged to from the hospital. could not produce for our review. It was a total of less We wanted to know where patients go when they leave than 2 per cent of 2 1 000 records. The next slide shows the extra work we undertook to acute hospitals. From the next slide we see that in some identify the diagnosis for which each patient was ad- hospitals, 50 per cent of the medical records had no @tted. useful information on this point. Are we going to gener- In this slide the "rejected" records were those pulled alize our findings with the Maryland region-we cannot. and reviewed but unused. The main reason for non-use The data we have apply to only a very small number of was that the disease of interest was not listed first among hospitals and patients. discharge diagnoses. You can see from our "control" We have tried to use other types of morbidity data in sample of all admissions other than heart disease, cancer, our region to get some baseline measurements for stroke, diabetes mellitus, and chronic bronchitis that 12 expected changes over time. They are summarized on per cent of the records were not ii7cluded in the sample. die next slide. We have used death certificates. Some There were excluded for the following reasons: problems in the use of death certificates are mentioned 1. the disease was not coded; on the table. We only use deaths to follow up the out- 2. the record pulled did not match with any record come of individuals who fell into our sample. We have number in our sample; been trying to trace deaths in all of the people that have 3. the admission was either before or after the appeared in all of our samples. This is a large scale opera- defined study period. tioii. All names in our samples have to be matched with With the major RMP disease diagnoses we had to reject names that appear in subsequent mortality data. Once as many records as we accepted. The difference between we get the death certificates, the diagnosis is always in the 12 and 50 per cent was due to the non-primary question and steps should be taken to get validation. nature of the diagnosis, To summarizes available morbidity information is We have tried to get information about out-patient biased towards large hospitals. These hospitals differ visits. Those of you who use out-patient records know from smaller hospitals in their patient populations, their their two major obstacles: There is no way of getting a availability of diagnostic techniques, quality of the in- list of diagnostic problems unless they are listed by a formation in medical records and its method of storage secretary in a log book or clinic file as patients are seen; and retrieval. We should recognize this bias when we the OLIt-patieilt records themselves have no "interval" make generalizations about changing medical care and diagnoses. service programs on the basis of local and national mor. We spent all last summer in outpatient clinics getting bidity information. information about visiting patients. We found that One further problem in using morbidity data from patients attend diabetic clinics for years, and their medical records that I will mention today is that of record contains no definite statement that the patient missing information and the bias it may have on your has diabetes mellitus. The diagnosis has usually to be final interpretation of those data. assumed. 70 other problems met in our surveys are: definitive short-term general hospital bed needs for stroke diagnoses are rarely entered. Further, the information on patients? as stroke patients are con- ,vhich we could make a survey diagnosis is limited. The next question, as far I am not going to talk about functional end-point cerned, is: Do we need acute care beds for admitted ,measurements because I know Dr. Kelman is going to stroke patients; or for how many patients do we need talk about them. acute care beds? We are on the planning road towards in terms of disease measurements, out-patient records getting the answer to that question. Four neurology and physician records have very limited value. centers have funds for acute stroke units. They have all Finally, I would like to show you some of the ways agreed and have already started to set up standard we are using these different kinds of disease measure- criteria for all centers. These standard criteria will allow rnents to get estimates of regional needs. us to describe the patients in the same language so they One of the questions we have asked ourselves about and their outcomes can be compared ' The standard our region in general is: Should every patient seen criteria will also ensure that all patients have at least a (somewhere) with astroke diagnosis be admitted to a rwnimum number of standard diagnostic tests. Each hospital immediately. We are not talking about patients center. will add its own special tests to its protocol but Nvho never appear in any kind of medical care facility, each-has agreed to use a standard basic protocol. only those who appear somewhere in the health care Above and beyond this agreement to develop stand- system. If so, how many bed days would be needed. The ard information in the four centers, we are working on next slide shows an example of the type of construction the design of a randomly allocated therapeutic trial to we are making to get this information. From our in- allocate patients with different degrees of severity into patient survey we have estimates of all the patients ad- our limited number of acute stroke beds and into other @tted to our hospitals in one year. From a surveillance neurological beds. This study will identify the kinds of of the emergency rooms of certain city hospitals we patients for whom acute care makes a difference in out- know how many individuals with stroke diagnoses visit come. the emergency rooms of those hospitals and are sent This is one of the very tight end-points that Sam home. From these two sets of numbers we can get a Shapiro was talking about and one that we believe has total number of people with stroke diagnoses seen some- tremendous implications for the country as a whole. We where in the hospitals in a stated period of time. want to be able to say how many (expensive) acute We have not yet added into our sum of patients the stroke care beds we need. out-patients with new stroke diagnoses we identified Finally, I would like to discuss one figure I borrowed during our out-patient survey. from Dr. Matthew Tayback who is a member of our Now, what else have we done. We have completed a department. It is a beautiful illustration of a point Sam follow-up study of all patients seen in emergency rooms Shapiro mentioned about the need for comparison and not admitted in a defined period of time. We visited groups even when you are looking at changes over time. all living patients two years later to find out what Dr., Tayback has been looking at improvements in the happened to them since the initial emergency room visit. outcome of pregnancy in relation to maternal and infant We also visited all patients admitted to the same city care programs. These outcome measurements show a hospitals two years after they were discharged and asked beautiful downward trend in phase with program de- them the same question. We also know whether and velopment. (Slide) when any patients in both groups died, and whether and My colleague is wise enough to look at trends in cities when they were admitted to other hospitals. We know who chose not to develop maternal and infant care what they say about their experience since the time they programs during the same years. Curves are shown for ,Went to the hospital when some were and some were not prematurity rates and for neonatal mortality rates in admitted. By putting together these various pieces of nonwhites. The hard lines representing cities with information we can look at all "recognized patients with maternal and -infant care projects are mirrored exactly stroke" and see if there is any evidence of a difference in by trends in the cities without programs. The initial Outcome for similar admitted and not admitted patients. assumption that these programs are easy to measure Our outcome measurements for this study are death because they have dramatic changes over time is proven and hospital admissions. wrong. it is very hard to measure the value of these This is obviously a time-consunung and slow study programs because the other cities seem to be doing just but I hope it will give us some basis for estima ting our as well. 71 This one pair of graphs illustrates Sam Shapiro's point were admitted-then we wanted to be able to say there that I want to emphasize-the need for comparisons even has been an improvement. So we wanted baselines from when looking at changes over time. which we could measure improvements both in the I would be glad to answer any questions about other proces s and in the outcomes. aspects of our studies later in the program. 3. We wanted to be able to describe our region in terms of patient. movements through the medical care facilities, consultation, delays etc. We felt this was the Discussion quickest way to get the picture. The best alternative was to take a group of people DR. CARPENTER: That's fine. You noted the with each disease and follow them through the system complexities of analyzing data from existing medical for a number of years. care records, and Mr. Shapiro said it was a complexjob We decided to use a cross-sectional approach. to devise new recordsand get decent information from 4. The last purpose was to identify comparison those. This is one indication of the difficulty of end- groups. We now have a pretty good picture of the people result analysis. seen in all of our health care facilities in the region. If, as Are there any questions for any of the panel members has happened, one area sets up a program for chronic from the floor? respiratory disease, then from our data we can pick out QUESTION: With regard to the stroke patients, when an area that has similar patients that doesn't have a pro. you listed other diagnoses like heart disease, presence or gram and maybe we can make comparisons. absence, what criteria were used in deciding whether a That was our rationalization. stroke patient had heart diseases QUESTION: Point of information. I was wondering DR. HENDERSON: The data I showed you are from how you could define morbidity. hospital records that were already in existence. The DR. HENDERSON: How I define morbidity? diagnoses we used were abstracted from the discharge QUESTION: Yes. It came up in your discussion a diagnosis. In other words, we copied every discharge number of times. diagnosis listed in the medical record onto our data DR. HEN DERSON: I suppose I was just using it form. loosely. In general terms it is a measurement of illness, as Our data say that whoever wrote out the discharge opposed to mortality which is a measurement of death, summary in the medical record listed this disease as There are obviously different kinds of measurements being present. of morbidity. You can describe the disease itself. You QUESTION-. The diagnosis of heart disease in these can describe the use of services by people who have patients may have been based on EKG findings or not? disease or by people who do not have disease as a DR. HENDERSON: May have been based on any- r-heasurement of morbidity. You can use length of hos. thing the physician used to make up his mind that the pital stay. You can use measurements of function. patient should be given the dignosis. I used the term in a generic sense meaning measure- QUESTION: Are these face-sheet diagnoses or iiients of everything related to disease separate from extracted from the discharge summary? mortality. DR. HENDERSON: Discharge summary. Not face QUESTION: How long did it take you to gatlier thin sheets. I had my abstracters copy the discharge sum- data? And in the meantime did you wait to start a pro maries at length. gram? QUESTION: What was your hypothesis in getting DR. HENDERSON: No. Let me explain our situation involved with reviewing these thousands of records prior in Maryland. We have an epidemiology center which ha! to the operation of the regional medical program? been busy collecting these kind of baseline data an( DR. HENDERSON: We had four reasons for this which is now working with the directors of individua survey: 1. To get information for planning. We felt it projects to set up their evaluation schemes. was really unrealistic to plan to set up new programs or The center started after the Regional Medical pro extend programs unless we knew what was already in gram began and projects were funded before any survey existence. were set in motion. 2. The second reason was to get baseline measure- It is supported by RMP funds, but is administratively i-nents for evaluation. If there were improvements over in the department of epidemiology in the Johns Hopkin time, for example, in doing more EKGs when people School of Hygiene and Public Health. 72 It is an advisory and a scientific arm of the program QUESTION: Or whether a facility existed that they administered by the University and it was very iiii- could be moved to that would have been a physician's fortunate that the program began before this particular first or second choice? activity was funded. However, in spite of the delays, the DR. HEN DERSO N: We have not done that. We have result of our surveys (now being cleaned up) seem to be collected a lot of subsidiary data. For example, we did a coming at a good time. survey to identify all of the relationships, both formal In my opinion, (with its limitations) the region has and informal, between and within all our hospitals and seen a lot of activity as a result of initial funding of between our hospitals and all other institutions. So we programs. People became interested and be-an to work. do know with which nursing homes and which other Now I believe we know the active and interested hospitals each individual hospital has relationships. members of our professional society and we are at the QUESTION: Dr. Henderson, would you care to give stage where we really need some overall direction. I an opinion about the necessity to have a program which believe the Center's results are going to be available at would significantly improve the hospital systems for the time when some overall direction needs to be de- data collection and data management in view of the fact veloped. that it's terribly expensive and very difficult to set up a How long did it take us to do the record survey? It modern type of information system? took us a year to collect the in-patient data and another Do you feel that the data that's needed is so essential three months to collect it from the out-patient clinics. that this is one of our major problems? Over this same period of time we have collected data DR. HENDERSON: Well, you've got to separate this from three sets of admissions to a sample of nursing into the data needed for patient care and the data homes so we know about their population and its turn needed for overall planning. The speed at which these over. systems have to run is different for the two purposes. I think it has been a fantastically rapid job in terms of The fast systems are the most expensive. Dr. the amount of information collected. Williamson knows much more about this than I do. We are currently having a lot of problems with I think we need to have our systems improved, there analysis because the sampling frame was really con- is no question about it. The major data problem is founded by all of these problems we met in getting quality. samples of records and finding which ones were usable. Most people improving data systems are really taking We are presently working hard at sample estimates. To no notice of the quality of the data. change our estimates we had to look at all of the re- In my opinion, which is an epidemiologist's opinion, jected records (five thousand of them) and tabulate the a great deal of effort in RMPs across the country has reasons why they were rejected. We have just finished gone into the technical improvement of data systems this exercise. Overall it has taken two and a half years to without taking any notice of information that will come collect, process, and begin to churn out data. out of the system in the long run. DR. CARPENTER: I think at the rate things move in Perhaps we tend to go the other way and place too Western Pennsylvania data collected within three years is much emphasis on the exact meaning of the information bound to approximate baseline data. and its accuracy. QUESTION: Were you able to differentiate between There may have to be some approach between these the care received and the disposition of the patient that two points of view before we reach the best data actually occurred and what the medical person in charge systems. would have wanted for them? But we obviously need an improvement in medical records systems. DR. HENDERSON: Not from past records. We are The biggest holdup in Maryland, if you want to look doing that kind of thing in evaluation of separate at the speed with which information becomes available, projects. These are retrospective data so they are hard to is in making the record summaries, getting them validate. completed and getting them into the record room. No QUESTION: Not even on discharge placement, where system is going to do that. You have to get substitutes their first choice of placement would have been? for the physicians or give the physicians time to write DR. HENDERSON: You mean you go to the patient their summaries. and find out whether they actually went there? We DR. CARPENTER: Dr. Williamson, do you want to haven't done that. It could be done. comment on this? 73 DR. WILLIAMSON: (Johns Hopkins University, extent to which the poverty and nonpoverty groul Baltimore, Maryland): Yes, I agree fully with what differ in what is already known to physicians about the Maureen has said, and my own bias is against trying to health conditions and the health conditions that ai throw a lot of money into developing a fancy record found through this health testing program and what system when nobody has an idea for what purpose it's the medical care behavior of people in light of what going to be used or what kind of decisions are going to known and can be retrieved from the medical records. be influenced by the information you will get out. In every one of the studies with which I have bee I think it's much better to try to aim at developing an associated in the past,. the existing medical record assessment function within the medical care group and systei-n has played a very important role. In some cases i then especially to Yet the physicians involved so that it has been more subsidiary than in instances to th becomes a part of the problem of trying to attack and specialized effort to obtain information, overcomin identify what problems or priorities to aim at first and some of the problems that existing records pose. then what kind of measure they want to mal,-e and then QUESTION: Does the HIP regularly i-nonitor or hav start to work backwards to the system to say, "Now, if a quality control system with regard to how physician we are going to measure this particular disease, this enter their medical records? What do you do abou particular problem, we are going to have to have a much keeping up on a certain minimum level of quality? , better form, and let's standardize it so all the physicians MR. SHAPIRO: There's no continuous inonitorijil will use it, and we can get that and start collecting, system for quality of the information being reported ii standard data on that problem to see if we can arrive at the medical records that cuts across all records. Bu some conclusions as to where we can improve the out- some of you may recall the quality of care studies tha@ come of care for these kinds of patients." were carried out during the 1950's and early 1960's ir So I would strongly stress going after the function of HIP in which the information in the medical record! evaluation and setting up expertise and getting phy- provided a critical source for evaluating physician per. sicians and members of the group involved with that, formance. rather than taldng some part of the process and trying to TMs left its mark on the system and had a ver3 bring this up in a not very sophisticated way without the profound effect on the way in which records have beer balance of the other parts of the system that will organized and maintained. eventually lead to decisionmaking and altering the Also, the payment system within HIP does contain 2 system itself. provision for annual review of a sample of records ir DR. CARPENTER: It's almost what you were saying, each medical group, for new entrants into the system Sam. One needs a special data collection system to and for entrants during the previous two years, and the measure a specific end-result. Existing systems seldom quality of records is judged on the basis of this review, work. and money flows to the ones that meet the criteria. MR. SHAPIRO: Well, by themselves they are almost So while we don't cut across all categories of patients invariably not adequate to serve the purpose of the kind in the system, there is a considerable amount of at. of end-result studies that I was describing. tention given. But at the same time I think that there is a danger I might also point out one of the strengths of the even within the framework of these large studies that I record system in our plan is the ability to retrieve in- was describing to overlook the important role that exist- formation without breaking your back going out into ing record systems may play. the general coi-nmunity. For example, the multiphasic health testing program There are 30 medical group centers, and a centralized that I very briefly described and that's just getting record system provides a very powerful means for ef- underway in HIP is very heavily dependent for some of ficiently obtaining access to the total medical care the evaluation on what will be found in the physicians' received in the system. records. QUESTION: Many dollars have been spent on studies There is being developed a retrieval system to obtain in the name of health planning, and I was interested ill diagnostic and physician and other medical personnel the suggestion that as a result of the work you have one utilization information from the existing records. in Maryland you anticipate some of the course of the One of the questions-it isn't an end-result type of program might be modified from some of the results of question. It's a process type. It's related to process. But your study. I wonder if you could just. enlarge on this 2 one of the important questions that we're raising is the little bit. 74 DR. HENDERSON: Well, I'm speaking here before your own region or health care system. Does anybody ,e have looked at our total data even for one disease. want to claim credit for that? That doesn't mean you @owever, I do believe that it does point out differences claim good data; it just means somebody did something I services, diagnostic services, differences between the because you showed data to them Sam, you must have ,verity of disease at the time people get into hospital, had that experience. rid also follow-up differences. MR. SHAPIRO: Yes. I was waiting for Maureen or I think we have already made a beginning with stroke, Chuck or somebody else. Dr example. We have met with neurologists and pointed Yes, the breast cancer screening program has had a ,ut that there are groups of patients who were not get- very direct effect on what is being included in our multi- irig follow-up care. And I think just looking at our series phasic health testing program. We're going to have ,f descriptions of patients, investigations and modes of mammography there. There's a move within HIP to herapy, the neurologists are going to come up with include as part of the general physical examination not jeas about what is needed to improve care across the orfly palpitation, which ordinarily is included, but also @bard. mammography. That's not a very specific answer I know, but I have Now, this may sound like a trivial affair, but mam- ooked at enough of the data to believe we are seein- mography is a costly procedure. remendous variations. Now, if the information we currently have hardens If the neurologists agree that certain standards of over the next couple of years, there's- Well, I'm going Jiagnostic investigation are necessary, and we show to be the optimist to say there's no question in my mind Variations in frequencies of diagnostic investigations, it is but that there will be major efforts in many parts of the [he responsibility of the region's neurologists to begin to country, including efforts among those groups that are set up a program to see that necessary investigations are concerned with the regional medical program's regional- available to all patients. ization and expansion of services, to include breast We are trying to provide the clinical specialists with cancer screening. data they can use to make decisions about gaps in the In fact, in your area, Abe Lilienfeld has a project that care. I think we see enough variation to predict there ties with RMP to have every woman adn-dtted to hospital will be enough gaps to keep everyone busy. go through mammography. QUESTION: One of the morbidity figures that you This program I am sure he will acknowledge is a showed in the first slide, Maureen, was prevalence. And I direct consequence of the tentative findings that we have wondered if you agree with the viewpoint that preva- made in the breast cancer program. lence should be one of the last measures one would ever DR. CARPENTER: Incidentally, were the mam- use to assess the effectiveness of regional medical pro- mography cases usually curable9 That is, the cases dis- frequency of covered ordy by mammography,7 grams, in that prevalence, which is the disease at any one moment of time, is likely to go up if MR. SH A PI R 0: Well, I'm not going to be able to give regional medical programs are truly effective. you a direct response to that because our numbers are And this is a somewhat embarrassing finding that we still quite small. But the histologic type of breast cancer would probably not want to show, although we may picked up through mammography is more heavily want to know it ourselves. concentrated of the intraductal type where there is DR. HENDERSON: That is very true. We are aware evidence outside of our program that survival rates are of it. We are trying to get some estimates of care needs much more favorable. in terms Of prevalence, not look at the outcorhe in terms of prevalence. At this point, both those cases picked up through The MIC program is, again, a good illustration of your mammography and those cases picked up independently. point. In areas without good facilities before the pro- on clinical examination have very favorable and very gram began we are getting an increase in the stillbirth similar types of survival. rate, and an increase in the mortality rate, only because DR. LOGSDON: I would only add as far as end-result we are finding babies never registered in the past. So the evaluation of a test that the dental examination that rates are going up as the care initially improves. included oral cytology had such a very low yield in DR. CARPENTER: Let me ask the audience and the number of positive cases that were thereby treated that panelist whether anyone has now in their hand end- this was deleted from the process rather than adding to result measurements which have led to new decisions in it. 75 So that end-result evaluation can delete as well as DR. HENDERSON: We have one that was not rea' add. an outcome. That's why I was not speaking. We did DR. CARPENTER: There's another good example follow-up study of patients a year after they had be then of how end-results can change the system - end- discharged from three hospitals in Baltimore with result measurements. diagnosis of stroke. We wanted to find out what medi( Anyone else? care problems they had had in between times. Do you want to give us an example of how some of One of the reasons for doing this study was that t' the end-result measures you have made, John, have Maryland Heart Association wanted to develop stro, motivated either your own institution or your Regional programs and wanted to know the needs of strol Medical Program to undertake health care a little dif- patients living in the community. ferently? We found that many, many patients said that thi DR. WILLIAMSON: I guess the two most dramatic could not get to their usual medical care facilities to g illustrations might be, first, our heart failure study at their blood pressure measurements, their ills, and , p Baltimore City Hospital, where we took a look at a range, the kinds of month-to-month care patients with tt of outcomes from case fatality rates to people who were kind of chronic disease require. still out of work a year after leaving the hospital that They could not get there not because they we should not have been out of work. paralyzed and couldn't be gotten out of bed but beeau@ . And having found that the results did not meet some they could not speak well enough to feel confident I very stringent criteria we set up, the administration of travel or because they were too insecure or unstable I the hospital was impressed but didn't do anything. But go without an escort. then they did okay some more studies. And then one of And as a result of this study the Heart Associatic my graduate students took and replicated the same type was given a van, and it now has a transportatic of study in another area and found the same kind of program-the van driven and staffed by volunteers. It h; rate. For example, the case fatality rate was almost started to offer a free service to needy patients in tt double that which they predicted under the worst of metropolitan area to take them to their medical cai circumstances. facility if they have no other means of getting there. We identified that the problem had been that the care So we did have a particular effect. It wasn't RM given during the time they were at the hospital was great sponsored, but it was a community organization. but it was that year after they left. Then this other study The process of doing evaluations has, in fact, ha found the same thing - they then appropriated money numerous effects on programs. The simplest to describ and hired some new staff and set up what they call a is in our coronary care units. We have been oo 'ng follow-up clinic to follow the patients after they leave coronary care units throughout the region. AU the unil even through the hospital may not have responsibility. have beautiful patient information forms which includ They still wanted to find out what could they do to see all kinds of measurements. Few of them actuall, that these patients get to another physician, to see that measure and record weight. they do fill their prescriptions, to see that they are going We have been abstracting information from one c to be followed. With heart failure there are disastrous these units for some time, and they are now beginning ti results if they don't take certain medications and have make much better attempts to get complete records. W certain medical care. are having a real effect on the recordkeeping systems.o And this has resulted in, I think, quite an innovative the unit. approach to this whole organization of the clinic system, Again we have been going to hospitals looking at th and we are rather pleased with what seems to be happen- performance of nurses who have been through strok ing. education programs. We look for care plans and whethe Now, the payoff will be to-wliich we want to do- care plans changed after the nurses attend the course. repeat the study and now see if we find any different The process of evaluation inevitably affects ordinav results as far as outcomes go or see if we are just prograi-ns. This is not decisionmaking; it is a sort 0 measuring something where there are other factors that infiltration from the bottom. might explain this. DR. CARPENTER: Very good. Does anyone els But this is a definite change that occurred in the have any examples? whole clinic system as a result of these systems. Are there other matters then that came up thi DR. CARPENTER: Good. morning in the discussion of the problems of healtl 76 status ineaSLirenici)" and the need to choose prop-, prob- slii-nmaries of the essential information we want for both lenis for the considerable effort required? patient care and research. We currently have a girl who is DR. MARGULIES: You know, I was inipressolu by a over-edLicated for the position working with us to set up couple of things this niorninc,. One of them is the sense the content of a training program. We are hopefully of reassurance I got that in the years that I lia,.,- left going to add another couple of candidates in the spring. medical practice the problem of medical records has We are coiiil)ariiio the girl's summaries aoaitist medical t, LI remained so stable that I don't have to relearn an%,thin(y. residents' in one special area after the other. A successful it's about where it was. program Would be a great step forward in speeding But the other thin(, that I realty wanted to raise on tliin-s up. Z, LI the basis of the experience of tire people who lia% - been DR. MA R G U L I 1; S: Of course, this still confines you describiii- their efforts in evaluation and nieaSLirii-,2 Out- to what you can do in improving medical records in C, - come is how important they feel this issue of ii-@edical hospitals. And on a continuing basis, as you pointed out, records is. Obviously You feet that it is very ilillik-)rtallt. you have had to confine your observations to isolated This then raises the basic question in my iiiiiiL@ in aii%@ incidents and, in fact, to patient response on the basis of kind of evaluative procedure of liaviiia adequate in- their own experience. fori-nation. And if you are goin- to measure %,oti DR. HENDERSON: Right. are doin- and measure the effects of any kind of DR. LEWIS: I'd like to jump in and comment. activity, whether it's reaionalization or clini,:q-l pro- I think besides Maureen's prograi-n with medical cedL[rC, whatever it may be, as you ver%, co-rectl@@ records librarians your comment raises the issue as to pointed out, you have to have somethin- that %oti can whether or not ambulatory care, traditional or radical, compare with something else. can ever be evaluated without a problem-oriented ap- For Regional Medical Pro@rams that could ver\ easil@@ proach to recordkeeping. be a major goal-to took at the capacities which have DR. MA R G U L I F S: That's really what I'm getting at. for influencing medical records, for introduciii- stability, Can it be? I doubt very much it can. consistency, and so forth. DR. HENDERSON: No, I do not think it can. Now, that's one aspect of it, but you also pointed to DR. LEWIS: The second point-and it's even more another which is of real concern, and that is the Margins subtle than that-is the problem of distinguishing be- perceptions of medical record librarians of how they tween the actuarial content versus the contractual ele- perform in a record system. ments of the medical record. Have you pursued this particular issue further and do Let me put it back. Sam and many of us would be you have some advice for LIS? interested in data that allows us to do a life table kind of DR. HENDERSON: Well, we have been pursuinz it in following of what happens in time on patients from the several ways: One, throu-h setting up meetin2s and actuarial kind of prognostic point of view. instruction. As a matter of fact, it is not really instruc- In fact, however, when one looks at medical records tion. Interestingly enouch it is easier to use the Inter- C, in tracing backwards the history, one is a prisoner of the national Code than a standard nomenclature. We have to 11 unlearn" the record librarians. We have tried to en- kind of medical information which that physician chose courage them to change their use of codes. to write down which really was in part @ fulfillment of We also have a pilot study setting up an educational his contract with the patient. program for a new kind of person that we are callinc, a And one has a highly biased view of the world, much medical records summarizer. I said earlier that medical of which serves to remind everybody who will read that record summary is one of the biggest hold-ups in the record that he was in fact doing a good job as he saw his record system. We have had in our research pro-rams for job with that patient. many years medical record abstractors who are very This sort of contractual, or legal, ethical reason, I competent and can abstract a medical record perfectly think, is one of the more serious problems which has well, getting all the detailed information we need. We are been cited by Garfinkel and others outside of the now trying to see whether we can train an assistant with medical care system, and one which raises the question this capacity who can be used in a service function to as to whether or not professionals can record actuarial Summarize the medical record to the physician's satisfac- information without the kind of super-structure that has tion so that he will sicn it. This would really gj%,- us a been built in special long-term studies to get information b much more rapid flow of records and we will get better that is other than almost a self-fulfillment prophecy. 77 DR. CARPENTER: Let me see if I can take a more have an influence on hospital records that you can positive point of view about the medical records. Our perceives data was obtained from medical records, and we were MR. SHAPIRO: We don't have the experience yet. curious as to what we could find out about the medical We are going to become operational in November. So records, how bad or how good were they. that's a very easy question to answer. We don't know. We tried to find certain expected correlations. One But this issue is one part, one phase of our evaluation. would think if a patient came in comatose that he I want to comment on the quality of records issue. ought to die more frequently than someone who came in What Chuck has referred to as the actuarial approach can alert. And this kind of correlation, in fact, we could find be thought of in terms of prospective studies. There are in the records. But maybe the correlation is so strong enormous difficulties even under the best of circum- that even with a much muddleheaded recording in the stances when you try to use information collected charts it is evident. during a previous period. If you get past the diagnosis sheet and look at what You have the problems of reconstituting a popula- the doctor wrote in the record you can learn some inter- tioii. You have problems related to, again even under the esting things. best of circumstances, absence of information that did For instance, in the county we studied, the Sig- not appear to be terr.ibly relevant initially. nificance of coma with no neurologic signs is really not This in no way detracts from the importance of major adequately recognized. Often if the spinal fluid is efforts to improve medical records, and we too are get- examined and blood is found, the diagnosis of sub- ting involved in new approaches to improve quality of arachnoid hemorrhage is not made. And if spinal fluid is records. not examined, the urine may not be examined either. I want to emphasize that in every research project in It may show unrecognized 4-plus sugar and 4-plus which outcome measures have been used, we have acetone. depended on the HIP medical records in one way or So by getting past that face sheet into the details of another. The record has not supported completely the care, somebody who is adequately trained can learn ainvestigation, but without the record we would have fair amount. been in terrible difficulties. Even in the "purest" types We are now in the process of saying to the people in of outcome studies, existing good records systems can be our study county, "Some of you lose more patients than of invaluable assistance. others. The difference is not related to age, sex, or DR. LEWIS: If I may make a comment, since the issue certain measures of severity." We also can say, "it looks has arisen, I think there is a tendency to confound tech- as though you're not all doing an adequate neurologic nology with validity, or neatness with validity. exam. Generalists lose many patients without definite For example, Maureen's comments about PAS hos- neurologic signs who are diagnosed as stroke. Similar pitals who had a printout, but had no better records patients (without clear signs) who are treated by the than those who didn't support this. I'm sure some of internists die less ftequently." you are aware that lots of people are pushing automated These and other data lead us to conclude that though Wstory-taking and computerized forms so that the hospital records are imperfect, they do contain useful physician gets a very neat printout. The issue of validity data. seems to have been totally overlooked in a good number QUESTION: Was there any evidence that hospitals of these projects, that take part in the PAS program of the Committee on Whether or not it looks neat and comprehensive is Professional Hospital Activities keep any better records one thing, but whether or not it means anything, than those who don't take part in that program? whether or not anything has really been measured that is DR. HENDERSON: No. No, we looked it that. The of any value, is another. only difference was they could supply us with a printed index. That was very helpful. How to Measure Health Status DR. CARPENTER: Are any of your hospitals using HOWARD R. KELMAN, Ph.D. any kind of automated history? DR.HENDERSON:No. What I would like to talk about are some ways ir DR. MARGULIES: Does the utilization of the which - and I think for this audience this will not neces screening program, the automated multiphasic screening, sarily come as anything new or unique - others hav( 78 looked at health status and have tried to measure it you get away from biologic measures of how people focusing principally around the measurement or the function, the more the function of the individual is in- determination of disability and related kinds of measures fluenced by non-biologic factors such as their immediate of discomfort or dissatisfaction. social environment, their aspirations, their past histories I think it was probably Kerr White who first coined and future desires. the five "D's" of measurements of health status - death, So that what we might try to attribute to medical disease, disability, discomfort, and dissatisfaction. care maybe gets less and less influenced by what medical And it seems to me to be as good a way as I can think care can do and more so by what the patient's social .of to define the different kinds of ways in which health situation is like. can be thought about and determined. I wanted to put that out to begin with because I Our speakers this morning have concentrated, or think that sometimes we make the assumption - and I focused I should say, most of their discussion and at- think I'm as guilty as anybody else - that whether a tention on measures of, and utflization of measures of, person can or cannot walk or %vill or will not go to work mortality and disease or sickness, and I'd like to talk a is due solely to whether he feels and actually is healthy little bit about the third D and maybe get a little bit into or appears to be healthy. some of the other D's. For example, there are questions as you know in the Of course, I couldn't help but get the feeling follow- national health survey which ask people whether their ing the discussion early this morning that, why bother activities have been restricted due to illness. Well, this is even to begin, when we have so much ground to cover in a loaded question, it seems to me. It's something that I terms of defining really what the RMPs are supposed to think we need to consider with regard to measures of do, to begin with, and to achieve and develop a kind of disability. apparatus for assessing these largely undefined or global The other thing I think we need to think about are objectives. the data sources for information of this sort which are But I suppose if we waited until objectives were different than those stressed by the previous speakers. clearly delineated and everybody was really sure about There is less dependency here on hospital records and what they wanted to do, we might not even be meeting those kinds of reporting systems with their degrees of here. unreliability and uncertain validity, and more reliance on Let me go on a little bit further and talk about dis- a hard source of information - namely, the patient or ability measures and why and how it might be utilized in somebody who cares for him. RMP programs - which I know very little about because Now, I know that it has been traditional to think of my connections with RMP have been rather peripheral, measures of social functioning as relatively soft and That is, I've been approved but not funded. measures of morbidity and mortality as relatively hard. I suppose it's worth starting out by asking: Why get But I'm convinced by what the first two speakers told concerned with disability or discomfort or dissatisfac- me this morning about how really soft the latter kind of tion? What has that got to do with medical care? information is - and I would say I'll put my bet down I leave the obvious answer to that to you to think on the patient. about for only two seconds, because, for a variety of But, quite seriously, I think the whole question of the reasons, we have become increasingly concerned with reliability of patient in terms of asking him how he feels the social and economic and psychological consequences and what he's able to do or not do for himself, or asking for living of individuals who survive medical care (or his relative or asking somebody who has observed the chronic fllness) and what is done to them or for them or patient either in a treatment program or on a visit, on their behalf. whether it be a visiting nurse or an occupational The increasing concern with chronic or long-term therapist or an interviewer, does pose technical problems illness and the consequences of that for individuals in of reliability and validity which are related to but some- terms of their ability to function physically, socially and what different than the kinds of problems that we have psychologically has led to the desire to regard disability heard about this morning. as a sequela of long-term illness and how this might or Now, in thinking about this subject and in some prior might not be affected by the care that people receive. conversations with Dr. Carpenter, he inquired as to One of the major problems I think we face in trying whether there were some kind of standard measures of to look at disability, to measure, to define it, and to disability, social functioning - you didn't use the word then try to relate it to medical care, is that the further . but I did "happiness" - those kinds of things. 79 There are "standard measures," and each person who Security Administration with regard to disabled persor does a study develops his own "standard" measure. and how they function in the community. There are good and sound reasons for this. Essentially, the kinds of information that they collec One of them is that it is exceedingly difficult to get are geared to basically well or "non-sick" populations, s any real consensus that goes beyond the confines of that how relevant it is to populations of sick people perhaps an advisory group 'about what you mean by dis- something you really -have to decide for yourself. ability and what you mean by social functioning and But with regard to the question you raised earlier whether any of these things have anything to do with Maureen, about denominators, I think this is where th the program that was supposed to influence any of these sort of information may prove to be helpful if you ea states. - use the current information and if you find that th The other problem is that what might be regarded as numbers are adequate for the population you are talkin disability in a person with physical impairments is not about. necessarily going to cover the same kind of ground for Now, one of the problems, of course, is that these ai presumably well people out in the community. usually national surveys, and depending upon the size c So that if you are interested in a small increment of your local community, you may only have a sample c change, let's say, in whether a person can now dress with six or eight people in this national study. or without some kind of assistance because they have In any event, you may be able, with the aid of ver sustained some kind of motor or neurologic impairment, competent people, to relate the local population you ai that would not necessarily be an appropriate measure or concerned with to adjusted rates based on t ese nation, question to ask' of somebody who is out there in the sources. I think it's something that we don't ordinaril community and who is unemployed for one or another think too much about, at least in this area. kind of reason. Now, some of the kinds of things that they try t If you wanted to develop a battery of measures ot coflect information. on in this survey- I'll just ru social, physical and psychological functioning to run through it. very briefly. I'm sure that many of you ar the gamut from patients who may be nearly or com- familiar with it. pletely bedbound to those who are both fully am- They ask questions about days lost from wort bulatory and who work quite effectively as physicians or wholly or partiafly-the extent to which the individu, legislators or RMP coordinators, this is as yet a quite has restricted activity days I guess is the actual term tha formidable task to get anytl-iing beyond the crudest they use-whether there are mobility limitations preser kinds of information. or whether the person in a sense is either confined to th I think the other point that needs to be made is that house or can get about without any kinds of difficulties those of us - and there are many of us in this room that And they also inquire, very interestingly it seems ti I recognize and many who are quite expert in this me, about the person's social role activities - that is, th field-don't view these measures really as replacing the occupational information, if the person is a housewife more traditional and hard-to-get-at and harder kinds of or if a child, whether there have been any activit: information centering on, you know, mortality and restrictions with regard to'those roles. morbidity, but really try to see these measures as I'm not sure what they do about people who are 6: perhaps other kinds of ways in which the benefits or and over, because 'we have no real social role defined fo lack of benefits of programs can be documented or those individuals unless one can call retirement a role. Si I think part of the roblem in talking about people ag tested. p What are some of the ways in which disability has 65 years and. over is that they would probably scor, pretty low on these scales. They don't work, perhaps been thought about and how have some people been They may never get out of the house. (I shouldn't sa, going about it? Perhaps a word or two on that. , 11 never.") I have already referred, I think, to the National Now, ranging from those types of very globa Health Survey, and I think it's important particularly for measures- and I again want to emphasize that the in persons concerned with broad population groups and dividual is asked and his response categorizes him - t( planning for their care and meeting their care, like RMP, say whether limitations or restrictions in activity are du to be aware of the kinds of information that are to illness limitations. Nobody exan-dnes him. Nobod, produced not only out of the National Health Survey, decides a pfiori. The individual categorizes himself ii but also more recent studies conducted by the Social terms of his response. 80 So that you have disabflitv measures of that type on been based on information obtained from professional one end of the continuum, if I can put it that way, and people who know the patient. Sometimes it's a team at the other end of the continuum you have a variety of making a judgment based on their experience with the different kinds of measures of function which center patient, coming up with a group judgment about the around a core of what have come to be defined as ADL patient or individual. In other scales it's a single in- activities - tofleting, dressing, feeding, ambulation, dividual who knows the patient, who may have worked transfer activities. You name" it and you can find with the patient, or who maybe sees the patient or "scales" for it. former patient out in the community - a nurse, perhaps And these kinds of measures have been developed or an occupational therapist. Sometimes the patient essentially to look at rather severely disabled people, or himself or a relative is the source of information. those with potentiality of becon-dng quite severely it may be almost like splitting hairs, but we, some- disabled, who require a great deal of care and who have times seem to take these three rather disparate sources rather profound limitations in the ordinary activities of of information on particular individuals and throw them daily living. all together as though they possessed similar qualities of And you will find on this end of the continuum a reliability and validity - and of course they don't really. variety of different kinds of scales, all of which, or many However, with all of these problems, as I said earlier, of which, have proved to be quite useful in terms of judicious selection and use of these scales has proven evaluating change in patient status over a period of time quite valuable in terms of determining whether a given or over a period of exposure or lack of exposure to one program is having some appreciable effect on raising or another kind of treatment programs. levels of function of disabled individuals or on whether The scales vary in terms of the actual dimension that it has reduced their need for assistance. they are cutting. But what they really are trying to get Particularly with regard to individuals in nursing at, it seems to me, is the extent to whici not only the individual can perform at one or another level but the homes or who require great amounts of nursing care, a extent to which this performance is based on some small increment of gain from dependence to inde- increment of care or assistance, whether this assistance pendence, let's say, in an activity like tofleting can mean may be given by someone in the home, a relative, or by a great deal over a period of time in an institution where somebody in the treatment institution - the extent to many in the population may require a great deal of care which the individual can perform this particular func- and assistance in terms of toileting. tion, dressing, tofleting, transfer, etc., independently or Certainly I don't need to remind this group that a dependently. small increment of gain in tofleting in a patient who has And the scale endpoints usually range from some to be taken care of at home, while it may only reflect a level which is either "unable to" or "completely bed- jump from 3 to 2 on the scale position, may reflect a bound" to, "can do by self" or "requires no kind of great deal more in a home situation if someone has the assistance." responsibility for the care of that individual. I Now, investigators have usually used a battery of I think then that one of the other problems with these scales, and in some of these scales different weights these scales is the fact that a stepwise jump from are assigned to different functions - ADL functions. position 4 to position 3, while it looks mathematically Other scales do not assign different weights but rather neat, may not have the same kind of social meaning as a give equal weight to performance on each of these jump from 4 to 3 on another scale. functions. Still other kinds of scales are concerned with But these are generally problems of scale, and I don't whether the patient needs help or doesn't need help. Some years ago in a study we had underway, we think they are specific really to this kind of problem. found several different ways in which disabled people, When we move from this more or less traditional area could be scaled or the scores manipulated, and we found of definition of disability or disability determination and that where there was change each of these scales revealed its application either to broad populations or to more pretty well who was going to be changed. narrowly defined clinical or patient groups, into the area Where there wasn't any change, it didn't really matter of discomfort, into the area of dissatisfaction, into the which of these scales were used. area of social functioning with regard to let's say the Now, one of the other things I should mention with family or the community, we get into terrain that is not regard to these scales is that they have for the most part nearly as well worked over. 81 rnents Dr. Henderson made about followup studie I guess in large part we don't really think about or try corn to affect family relationships, if I can put it that way, that we ought to be interested in things like whether the., when we think about stroke patients. patient is now better or less able to communicate, to use I suppose the connection between whether the stroke the medical care system, to manipulate it to their own,@ patient will now get along better or worse with his benefit. Maybe this ought to be, if it isn't, one of the spouse, and the application on the other hand of medical kinds of things we ought to be aiming at with sick measures to first see if you can keep the person alive and people. then to make living a little more livable for the person in Their whole knowledge of what is wrong with them biological terms in distant. Life saving does and should and what they might do about it, I think, represents take precedence. But we pay a lot of attention, at least another area that might be thought about with regard to on paper, to social well being, and maybe we ought to looking at some of the kinds of programs that have been, begin to think of broadening some of our concern into or that ought to be, developed. some of these areas. What I'm suggesting is that for a variety of reasons we I shouldn't want to leave you with the impression may not be able to affect very basically the biologic that there aren't studies of social well being of well or functioning or biologic status of many disabled in- sick people and that there aren't studies of family well dividuals. We may be better able to affect some aspects being or cornpatibflity, community.participation and a of the individual social situation, his social or psycho. variety of other kinds of social measurements; e.g. social logical functioning, or the function of those around him. isolation, work satisfaction, work performance. I don't know why, for example, the National Health But what I'm suggesting is that in terms of at least Survey doesn't ask at least for information from family some of the kinds of programs that we are talking about, members, and what their, input is in terms of care of the it may be well to think not only of scales which more sick person. directly seem to be related to biological efforts centering That is, if the individual replies that he is not able to around disability, but also scales which seem - only work because of illness 'oughtn't we to get information seem - less related, a little more remote from our on whether the social role of some other individual has interests - for several reasons. been altered as a result of that? Is that not really part of One is our own bias. That is, it may very well be that the disability picture that we all see pretty often? Does while we may be increasing the person's ability to this now mean that somebody else in the family is now function independently in one or another area of working? For somebody that is disabled and cannot activity, this may have quite deleterious effects, when work, does this now mean somebody else in the family this person gets home, on his family. We don't know situation's work role has been affected? that unless we look at it or think about it. What I'm suggesting then is a kind of broader view The reverse may also be true. We may have very little that we might think about with regard to the plethora of success, for one or another reasons, in terms of basically effects that programs that have been developed ought to affecting the physical level of functioning of an in- be looked at in terms of status or benefit. dividual, but perhaps the application of other aspects of Now, the obvious retort to that is that you can the program has had beneficial consequences in terms of extend the concept and idea of health status to a point how the family may now function or how the person where it begins to be so diluted as to lose its meaning. may function in other kinds of areas. But I don't think that some of these kinds of questions I think part of the problem in moving into these areas that I have raised or some of the areas of inquiry that we is twofold. One is to make, as we all do, some kinds of ought to be undertaking are that far afield for us at least decisions out of the plethora of dimensions of psy- to think about. chosocial functioning, those which have some kind of There are also areas in which there is enough meth- more plausible relationship to medical care programs odologic experience and technique and enough familiar- than others. ity in the health field in terms of sample survey that a And I think here that we do have a wide selection ready transplantation - and I use that word advisedly in of- "scales" is hardly the word I think to use in this this particular context - of these kinds of efforts would regard - but dimensions out of which scales that have appear to be appropriate, with some cautions. been developed or can be developed can be applied. And I want to end up with noting some of these Certainly it seems to me that with regard to sick cautions and then see what you have to say about these people, and particularly with regard to some of the kinds of things. 82 These kinds of studies - you have heard how much it I think you said that it's often worth measuring dis- costs from Dr. Henderson. Well, going out and inter- ability and I was fascinated that you talked about viewing people or sending out people into the corn- measuring, family disability, not just patient disability. rounity to ask and get detailed information about By the time we get that on the front sheet of the activities of daily living is more expensive. It's more medical record, we'll be quite far down the line. expensive to generate, to reduce, process and make DR. KELMAN: Not in your or my lifetime. Well, available information from this source than it is using maybe yours. available hospital records with all of their limitations. DR. CARPENTER: It's interesting you pointed out And I want to mention that because it's a consideration,. that sometimes it's hard to understand the validitv of a we all do think about even today. measure of death unless you know a concomitant The other thing I think one has to think about is the measure of disability. Dr. Stoneman pointed out that practicality of obtaining many of these measures. I recall probably those of our patients whom the internists in one study that I was involved in we were concerned, appear to have saved went home comotose and wet the in addition to getting physical and social measures of family bed for ten years before they finally died. And so functioning, with getting psychological measures of it is necessary to measure both death and disability to functioning. And this involved first obtaining and then understand the value of their treatment. By the way, sending a highly qualified, highly trained clinical psy- disability on discharge was the same for both physician chologist into nursing homes with a suitcase which he groups. opened up and then did his testing in front of the QUESTION: As you were speaking of different ways patients. of measuring health status, one thing struck me. I think This whole apparatus - and I won't even get involved some of. these measurements have to be reproducible if in terms of the development of this procedure, ran we are going to use them in evaluation. anywhere.ftom one to three hours. In evaluation evidently we are going to pick them up Well, I'm not suggesting that these very detailed kinds at one point in time and then later on pick them up in of measures on memory, on judgment and recall are the order to evaluate programs. How can we pick up kind of thing that ought to be done routinely, but there reproducible measurements? may be some programs where this kind of measure DR. KELMAN: I think many, if not all, are re- would be entirely appropriate and may be the most producible. I think the question is whether it's a measure relevant criteria of brenefit for some kinds of patients you want to get and whether it's relevant to your pro- and therefore shouldn't be excluded. But it is expensive. gram. Again one or two comments. The data, of course, For example, it is not difficult to ask one or more considering again the source, is highly- well, I was going times or of one or more points in time, not difficult to to say highly reliable. Relative to other forms of get from the patient the answer to a question, "How do information on mortality or morbidity it is no more or you feel?" I think the prior question is: Do you want no less subject to problems of reliability and validity that piece of information? than these data are, although the problems are different. It is not, I think, difficult. Now, some of the con- And, finally, I would end up with just a reminder siderations you would have to take into consideration that, as I said earlier, the further one moves away from are: How stable is that feeling state? Is this going to be physical and biologic measures of function, the more the something that he is telling me right now and is it going actual functioning of the individual and the patient is to be based on what has happened to him in the past five going to be influenced by things other than what was minutes, or is this a more or less enduring state of being done to him medically or what his biologic status is. And that I am concerned about? this presents a problem in terms of evaluating program- One of the things in the program may have been matic impact. directed towards altering favorably the feeling states, the I think I'll stop there and ask if there are any moods, the emotional status, whatever you want to call questions that I could try to answer or any points that it, of certain kinds of patients. So that may not be the you would like to have me try to elaborate on. most efficacious way of getting information. But you can get reproducible information by asking those kinds of questions. Discussion Now, whether they are the kinds of questions that relate to the kinds of information you are seeking is the DR. CARPENTER: Thank you very much, Howard. prior question. 83 This is true also of measures of social function and and I think that is one of the parameters you have to measures of activities of daily living which are a little watch pretty carefully. more enduring than the example I cited but I really used DR. KELMAN: I agree completely. And it's my that just to make a point. impression that most of the scales you get into with the DR. WILLIAMSON: Howard, if you were to recom- disability measures, whether it's vocational or occupa- mend to us one key reference in the literature on the tional or activities of daily living, are really geared validity and reliability of disability measures, what towards estimating how dependent or how independent reference would you recommend? a person is either occupationally, vocationally, socially DR. KELMAN: On the validity and reliability? )r physically. DR. WILLIAMSON: This question of looking at the DR. RIKLI: I'd just like to add one point there, and reliability and validity or usefulness and general ap- that is about disability. A man may be missing a leg or plicabihty of these measures. What literature could be rrdssing an eye or have many other disabilities which are pulled out that would get us going in studying this more really compensated for and are not really of serious thorougwy? concern to society when a person makes the adjustment. DR. KELMAN: I think one of the places I would But if they are unable to adjust and have a dependency, start is with- I forget the author-but it was a mono- then they become a serious concern. graph put out by the National Center for Health DR. KELMAN: Right. And your comment reminds Statistics. me of something, namely that we have to distinguish I DR. WILLIAMSON: Sullivan? think between an impairment such as this and a dis- DR. KELMAN: Sullivan I think the name is. I think ability. They are separate things. There are many of us that's a good reference to start out with not only who function with a whole variety of impairments quite because of the kinds of questions lie raises and how he well. tries to relate disability to the broader questions of That is, if I were an engraver, with my level of health status, but also because I think he has an excel- impaired vision, I might be quite disabled occupation- lent bibliography. ally. But in terirs of the kind of vocational situation I I think the article by Ellinson in the Hatidbook Of am in now, I'm not at an. Medical Sociology on sociomedical measures or measure- These are other problems, and this is part of what I ment problems is an excellent discussion of method- was trying to get across in terms of the point I made ologic problems. about when you begin to move away from the biologic DR. WILLIAMSON: Levine's? functioning of individuals to estimating how they DR. KELMAN: Right. In that book. I think you function in social terms and in social situations. The would do well, if you haven't already, to write to biologic becomes less influential. Not uninfluential, but Murray Wylie and get some reprints from him. less influential. And I think with that a person would be well armed I'm glad you raised the point of distinguishing and well acquainted with not only the problems of the measurement of disability from measurement of impair- application of these kinds of measures but their poten- ment. They are both important, but they are dif- tial and actual utility. ferent kinds of things. We sometimes tend to think that DR. CARPENTER: You can also look up Kelman in when we measure impairment we are measuring dis- the literature. That will get you a long way down the ability, and vice versa, but'we are not. road. DR. CARPENTER: It's hard to get the diagnosis DR. RIKLI: A small observation. Those five D's that adequately on the front sheet of the chart and a little you attributed to Kerr White - I have heard them on easier I think to get survival indicated on the front sheet many occasions - are most useful in taking a project or of the chart. Are there any obvious measures of dis- a program and running down those five. ability on discharge that could be coded on the front And as you talked about disability, you talked about sheet of a chart? the independence. It seems to me that probably a sixth DR. KELMAN: I think there are a number of things "D" might be dependency - financial dependency, that would be very useful to try to get hi some standard- emotional dependency and physical dependency. And I ized fashion. I think it would be extremely useful to think that's probably the greatest concern of people - know a few pieces of information. (I say that as though when a parent or uncle or aunt becomes dependent on it's so easy and so simple.) The extent to which an in- them in some manner. And dependency is measurable, dividual is able to perform certain limited activities of 84 daily living-and God knows why on any hospital record DR. CHARLES LEWIS: But looking at institutions to you can't find out how much education the person has. I examine the feasibility of recording information, I think- wish we could get that in. And where the person is going the real question is the one you raised: If you have a to, more explicitly than "discharged, improved." If we marvelous language which is somehow or another could routinely know whether the person is returning codable in a series of digits, so what? How will it be home or to some alternate living situation, I think would accepted? How will it be involved in medical records? To be a way of starting out. what extent will it actually influence patient care, or- Again, if you can do that in some kind of standard- ganizational behavior, interorganizational behavior? ized fashion, fine. But to have the ward clerk or some- But I think that's maybe a remote question because body else, you know, just scribble down some things is the real issue is that there is no way of communicatin- not worth the bother. this between institutions, between patients. DR. LEWIS: I was looking for some of the partici- This is an attempt to try to standardize - to deal pants in a four-center contract from the National Center with Howard's original point that everybody starts out now, and I guess there isn't anyone here. by inventin- a new wheel. The Harvard center, Western Reserve, Syracuse and DR. KE LMA N: There was an attempt - some of you Johns Hopkins, have a contract with the National Center may be familiar with it. I haven't heard what happened for Health Services Research to develop a classification to it, but it was called "rehabflitation codes." That system for patients that deals with three levels: effort involved a number of advisory committees who The problem of the patient, the actual management for years tried to develo a common way of codiii- p t, of the patient. relevant information for patients in rehabilitation and 9 Second, the problem of institutional management, related kinds of programs, institutions and facilities. And in terms of length of stay-the usual issues that an or- t ey deve oped reams and reams of material. I don't ganization or institution is concerned about. think it was ever used much by anybody. o And a third level of coding which has to do with I don't really know why it wasn't, because there were interorganizational needs - in reality, what the com. many, many places and many, many people and advisory munity has to furnish patients in the way of extended committees that worked on development of it and care facilities, etc, worked very hard, and a lot of it was good. Maybe all of The 'classification is in the early stages, first de- it was good. veloping a common language, and is designed to work at But I guess there's a different set of problems in- several levels like any taxonomy which progresses to volved in developing these beautiful codes and then deeper levels in which information is going to be trying to take that and translate it operationally in terms obviously less and less available. of some ongoing system like a medical care system. And This is an attempt at the kind of classification which I really don't know what happened to it. would lead to a series of codes some place on a record that would describe a functional disability, to an insti- tutional problem and interinstitutional problems. The.Relation of Process and End-Result. Evaluation And, as I say, I don't see anybody from one of the CHARLES LEWIS, M.D. groups here. DR. CARPENTER: And this language will describe I want to approach this from the stance of an activities of daily living? operator, somebody who has to make decisions about DR. LEWIS: Part of it will. In essence it looks at a lot evaluation data as well as someone who is supposed to of the kinds of things Howard has been talking about. It be providing it. tries to take into consideration all of this. They have And I will assume at the beginning that we evaluate been reviewing the literature trying to develop between things in order to change things, not as some form of four institutions a common language so -that when some- self-amusement, (w@ch it does turn out to be some- body says they are impaired, for example, in mental times), but in order to provide some guidance for those status, they Will now know it is coded, in terms of dis- who would like to really change the way things operate, oriented by place, by time, by person, etc. if they need changing. It poses a real problem in numerical taxonomy. Now, I'd like to restate very, very simply what was MR. SHAPIRO: Well, Chuck, does this effort contem- said more eloquIently this morning. Something - and I plate major changes in the contents of hospital r ecords? have decided to call it a condition, not a problem, not an 85 event, and not an in-put - just a condition at a time The second factor is the state of the art of evaluation zero, whatever time you care to choose that to be - is in general. usually measured in some kind of units. If we present those who are coordinators of programs And the units are hopefully relevant and possible of with end results which say "it worked" or "your pro- being measured, assessable, and hopefully available. And gram is good" - that's all they want. I think we would like these to be valid, replicable, But if one is going to present someone with informa- practical, and sensitive. tion which is other than socially acceptable, it's useful to For operators, the thing we are currently concerned be able to tell them what processes went wrong, because with is that the condition needs altering, or else there is this provides alternatives for strategies in terms of a question of it being altered. After looking at this restructuring programs. condition we do something, not just anything but Donald Campbell and others have talked about the something specific, and that the thing we do is. also problems of reforms as experiments, and the social measurable or assessable. legislation that has been enacted to create social change Having done something, a whole bunch of messy and why evaluation of these programs has been so things happen that are called processes. And I would just difficult. say that one man's process is another man's end result, If you had a million dollars riding on a program in that somewhere in here people may choose to stop and which it was announced a p?iofi there were no alter- say, "That's all I'm interested in." natives to success except through this approach, you And this is particularly true I think in looking at have some idea of why individuals resist evaluation (at continuing education, in which maybe all we want to do the risk of going out of business). is show they were sitting in the room. The failure to specify strategies, alternate strategies, The next thing we may decide we'd like to know is for experimental programs creates a problem. that they sat in the room and learned something. Perhaps one of the few ways we can deal with this Then we'd like to know if they took it home they did type of program is by looking at the nature of the something with it. processes that went on while reaching an end result and Now, as I have just indicated here, most of the times presenting these data to those who have to make policy when we are concerned with process we are concerned decisions. This is particularly important, I think, if one is going about the number of things that are done, the number of to institutionalize experiments 1 programs -. that is, things used, the nature of things done or not done in change the way people do things. The transfer of a terms of quality. Basically, process evaluators count program which seems to produce results into a different heads, or something, or the use of things. People who setting is difficult. Uffless one has some idea of what look at disability, deaths, and so forth, as in the I . went on. morning's discussions, are concerned with end resuits. Maybe this is related to some of the problems in dis- The major point I'm going to make - I hope - is that ability evaluation. it is difficult to affect change without doing both, that I didn't stop and spend as much time as I should have end-result and process evaluation need to be carried out here talking about the measurement of "do somethina." conjointly if one is going to be an applied evaluator and C, attempt to use results to redirect efforts. I think there are probably more "good" programs that have succeeded because the "do something" was, in fact, Let me just point out some of the other things that a phantom treatment that never got done than other by some of these terms I think relevant. kinds, in which something rather dramatically happened. The use of evaluation data depends upon two sets of It's very useful to know what it was you did that factors: made a difference, and I would just suggest as you look One, organizational factors. Organizations need to at the literature (as one moves from clinical trials of maintain themselves. They need to perpetuate the status drugs where we are sure we injected "something", to a quo, their prestige and individual's vested interest. program in which we install a new kind of health man- Evaluation basically questions the reason for being in a power) that we really don't take the same consideration certain business, and doing certain things. Fear of the to standardize the dosage, the blood levels, and other consequences of change, change in rank, or change in the things that we are concerned about. The process of structure of an organization are certairdy sufficient evaluation begins with knowing what the experimental causes to reject evaluation data. treatment was. 86 I have selected out of the literature eleven papers that 1. The first one was a sub-study that came out of ti-ic' are concerned with evaluation, and I'd like to comment national hatothane study of the incidence of hepatic, on them and then talk about the kind of evaluation that necrosis with halotliane. This was a report of iiisti. was done in 'each of them and what it would mean (to tutional differences in post-op death rates. Among 34 ine) in terms of trying to implement these results. hospitals, the end results (death rates) in surgery varied Let me begin, though, by oversimplifying certain by a factor of 27. They were subsequently adjusted for a classes of end, and process, evaluations that these papers few things like age and sex and other things, and that represent. difference is resolved to 10-fold. There were sub- The first one and the simplest type is the reporting of sequently readjusted for severity of procedure, and the end results - end results in a group of patients or insti- difference collapsed to 3-fold. This is the kind of study tutions that describe the fact that different things which says the death rates in hospitals are different - happened. nothing else - and if we age-adjust and do some other The second class or type of paper looks at variations things that we know how to do, they are still different, in end results between groups or amon- -roups or within but we don't really know why. groups, as a function of patient or doctor characteristics. 2. The second paper is by Leon (Gordis) on the Type three is similar to type one but related to evaluation of a program for preventing adolescent process evaluation. These describe what happened - the pregnancy. This is a paper that looked at a program in processes that were carried out and how they varied. wl-lich teenage girls who were sexually active were This ranges all the way from results of chart audits and a treated in a special clinic by social workers, by phy- whole bunch of things that are done to people or thincs sicians, gynecologists, and placed on oral contraceptives. that are used. The design then was to follow these girls to determine A little more complex, and fourth, is the study of how many of them stayed under treatment month after processes of care as a function of certain provider month. About 50 percent dropped out of the program characteristics. This is an attempt to describe the dif- within the year. The characteristics of those young ladies ferences in the way processes were carried out as a who did not stay in the program versus those who did function of the professional's background, training, and were coi-npared. so on. 3. The next paper, an evaluation of community The fifth type, is a look at both process, or treatment nursing services in the care of the mentally ill, was done done to somebody, and the end results of that treat- by Tayback. It looked at what happened when a bunch ment, without any comparison to other similar events. of patients discharged from mental institutions were In the sixth class, there are two processes - one I provided services by visiting nurses in the home, in terms have listed as "C" for control, in which there was no of a criterion called rehospitalization. The result was treatment, and an examination of the end results among that there wasn't any difference among control and two populations or groups with different kinds of treat- experimental patients. ments. The paper raises some interesting questions as to why I'm staying away, in this discussion, from the kind of there wasn't any difference. I think from the descrip- complex experimental designs that many of us would tion, I might point out there wasn't any standardization like to carry out and are very comfortable with in the of treatment. One really didn't know quite what was laboratory, i.e. cross-over, factoral designs. Because they being done and how this might have varied or how don't come along very often in the business we are certain subgroups of women might have had a better involved in. prognosis than others. In terms of looking at the There are some other kinds of quasi-experimental probability of rehospitalization as a function of the designs that are possible such as a time series observation patient, this is another kind that fits in second category that was pointed out this morning, regression dis- also. continuity designs, etc. I refer you to the paper by 4. The fourth papers comments on genetic counsel- Donald Campbell in the American Psychologist, for dis- ing. And if any of you know any other studies of the cussion of these. Iefficacy of genetic counseling, I'd appreciate knowing With this very crude and perhaps debatable them. This is about the only one I have come by. classification I'd like to go over eleven papers. I really Famflies who had had one or more defective children didn't choose these with any bias, except that they il- for whom the genetic inheritance patterns were known, lustrate these types of evaluation. were provided counseling services (not further described) 87 and then followed forward for a period of time. Ap- cedures. It looks at processes as a function of certain proximately 60 percent of the patients went ahead and variables in the structure of medical care. had another child. It would suggest about 40 percent of 7. The next paper is by Thompson and his group at this counseling, however it was done, had some effect on Yale on end result measurements of the quality of further childbearing. obstetrical care in two U.S 'Air Force hospitals. Here again there was no discussion of the effects, no Thompson looked at two Air Force hospitals and discussion of the characteristics of patients. It represents perinatal mortality by race, and found out that in one a straight-forward statement that so many children were hospital, the black perinatal rate was higher, but in the born who had major congenital anomalies or minor next hospital the white prematurity rate was higher. congenital anomalies to families who had been He went back and looked at utilization of care by counseled. trimester of pregnancy and found out that all of these 5. The fifth paper presented is from San Francisco ladies were using prenatal care rather early. It's a very data on the neighborhood clinics for a more effective facinating paper because the more you read it, the more outpatient treatment of tuberculosis. This was prompted you have trouble reconciling some of the results. by some observations that (in San Francisco) about 80 8. The next paper measured the quality of medical percent of alcoholics, (50 percent of blacks and 20 care through vital statistics. This is a comparative study percent of Chinese) broke their appointments to the TB of appendectomy rates in the hospital regions around outpatient clinic. Rochester, New York. There were large variations in The public health department went into each of these rates at which appendectomies were performed. And no neighborhoods, organized clinics with the help of the relationship was found between rates of appendectomy local citizenry. The compliance rate with broken ap- and deaths due to appendicitis - an example of looking pointments, sometimes used as a measure of satisfaction, at a process, and the variations in process as they relate dropped to about 5 to 10 percent. to an end result. The interesting thing about the paper is that nobody 9. The next study of comprehensive outpatient care reported whether or not there were any readmissions or in rheumatoid arthritis is one of the ones that deserves active cases of TB. reading if you're going to read any of these. In this one This is a discussion essentially of processes and Dr. Katz does several things. He defines the condition change in processes related to the structure of a pro- that he's trying to deal with. He measured disability with gram, which, oddly enough, did not look at the payoff - all of the problems that Howard Kelman mentioned which is whether or not 'any of these tuberculous earlier this morning. He describes the processes of care patients complied with their medications, or were for a group that got physical t@erapy, nursing, public readmitted to hospitals. health nursing, comprehensive team approach, and 6. The next paper is a study of variations in the describes it very well. He measures outcome, significant incidence of surgery. This was a study which looked at changes in disability, as a result of applying com- all Blue Cross subscribers in the state of Kansas and prehensive care for ambulatory patients with rheumatoid looked at the incidence of certain common operations, arthristis. T'&A, appendectomy, etc., in various economic I 0. The next one is a study that we did in Kansas on subregions of the state, defined so they'd be fairly continuing medical education. This is a study which homogeneous in nature. basically looked at the tremendously aggressive program The "Glover" effect or variation in rates for tonsil- in continuing education that had been mounted at the lectomy was reconfirmed, as was a 34 fold variation in University of Kansas for over 30 years with circuit rates for appendectomy, cholercystectomy, and a variety riders, with regional courses and with conferences and of other procedures. The rates for surgery were studied seminars held at the medical school. as a function of the availability of sur-eons, beds, and It was an attempt to look at the participation of all general physicians in the area. The percent of the physicians in the state for each year at risk over a ten- variance of these rates that could be explained was year period. rather phenomenal. For appendectomy, 70 percent of We took a look at the predictors of use, as a function the variation could be explained by beds and surgeons. of physician characteristics, and found among other It has some interesting implications, but it doesn't say things that it's related to being near a regional center anything about the consequences of these surgical pro- (having it avaflable); being a specialist, and being a recent 88 graduate, but not at all related to place in class on that was discovered when it was found out that the graduation. deaths occurred after discharge. II. The last is from the nurse clinic study at the Let me tell you about one that I'm willing to talk University of Kansas by Barbara Resnick and myself about, and it's a negative one, about how process in.- which looked at activities, events and the outcome of formation, and perhaps some outcome data influenced ambulatory care in which a population of patients with program planning in the Kansas Regional Medical Pro- defined chronic illness were previously examined, gram. Perhaps we can get a postscript from Bob Brown described in fair detail, were randomized into two who is now in charge of the program. groups. One went back to medicine clinic, the other In 1967, the very start of the program, we like received care by nurse practitioners. everybody else were trying to get people involved and The critical incident technique was used to try to trying to convince everybody it was their program. No measure some of the activities of the nurse clinic, some one believed this. of the things that John Williamson and Paul Sanazaro We were always saying, "If you just bring us projects, piloted. We looked at outcomes; death rates, in which we'll help you get them funded." And they brought us there was no difference; the level of disease, no dif- one from an area in Kansas that has some problems with ference. There were significant differences in disability economic growth, where the population was relatively rates at the end of one year of care under these two aged, the physicians likewise, and no younger physicians systems; the nurses' patients were far less disabled. There were goinal and there were lots of rehabilitation were significant differences in discomfort and satisfac- problems. tion levels. This paper attempts to look at processes and Some of the people in that area said, "We want funds outcomes. to train assistants in occupational therapy and PT assist- If I were presented with the data on institutional ants, because we have a junior college, and we can train post-op death rates I would say, "I don't understand these people, and then they'll provide our rehabilita- why our hospital is either so good or so bad." But I tion." don't have any answers, and if we were good I'd be We said, "Fine. We need some data to support it." happy. We had done a survey and were quite aware this was a I think that regarding the second paper, evaluation of very disabled population. the program for preventing adolescent pregnancies, I We also took a look at the occupational and physical would say, "This looks good, but I really can't tell what therapy facilities in hospitals in this nine county area you're doing to these young ladies, and I really can't tell and found without exception all of them were operating if anything is happening. Therefore, I think you'd better at less than 50 percent capacity. try to measure what you're doing to them a little better We interviewed a sample of about 50 percent of all if you want me to pick up the tab for this kind of a practicing physicians in this nine county area and we program after the grant support wears off." C, For the third paper, an evaluation of community sent our young ladies to them, and they asked: nursing services, I think the comments would be as for "Have you seen anybody who needed occupational or the previous study. physical therapy?" little probe to explain what And then there was a This comment on genetic counseling. I don't know occupational therapy was. what you can say when you're confronted with The next question was, "Did they get care?" information that says patients don't do what doctors tell The final one was, "Do you think we need riiore?" - them to do except begin to deal with their patients in a to which the answer was always yes. little more sophisticated way. When we took this data back, we were able to say to For the TB clinic study, this looks good on the the people, "Look, you have lots of problems, end statistical sheets, but did anybody get TB? Again the results that need to be changed; but you have facilities lack of outcome data creates major problems. that are being underused. There are occupational and This morning when Bob said, "Does anybody here physical therapists who are going to leave their jobs have end results that influenced decisionmaking?", Sam because they don't have any work to do." Shapiro talked about mammography, and someone else If we look at who creates demand for rehabilitation mentioned dental cytology. And John talked about the services (doctors) and talked to them, we found that heart failure study and the creation of a follow-up clinic they (the doctors) were not aware of the need for this 89 service and had identified patients for whom these certainly the end result standing by itself in conjunctior services should be prescribed. with the particular kind of setting in which it was carrie@ We didn't try to make any interpretations. We out has been of, an enormous importance in assessing presented this to influential citizens whose comment the impact by prepaid group practice's impact on health, was, "It looks like we have a job to do with our own So while I want to repeat that, of course, process i,; doctors." terribly important, there are on occasion very important I don't know, Bob, whether there is still pressure for practical considerations that make it extraordinarily this. But I think that in one case we were able to show difficult if not impossible to get at process. that by looking at the processes, that is, why patients The reverse is true too. An advance in understanding who need care do not get it, we were able to avoid process with some implied benefits from process with nc spending some money at least at that time. ability to get at the end result is also worthwhile. I have asked some of the experts around the room to DR. LEWIS: I think that's an excellent example. And give me some feedback on some questions that I have the question has always occurred in my mind: If this raised. I think I'll start by asking Sam Shapiro. It seems sort of care system is related to these kinds of outcomes, to me that one of the reasons you have been so effective, then why have the, let's say, perinatal and infant mortal- Sam, in influencing programs is that you really have ity social gradients in the United Kingdom not been been looking at end results, but also describing to your totally eradicated by the emergence of the national own group the processes that they were pursuing and health system? carrying them right along with you. MR. SHAPIRO: Do you want to get into a discussion of that? DR. LEWIS: No, sir. Discussion MR. SHAPIRO: Look, in a system like HIP, we know that there are very important gradients by social class. I MR. SHAPIRO: Yes. Well, Chuck, I almost have to don't want to get into that issue because I think it openq say "of course." up a new, highly complicated issue. The influence of an end result observation is going to DR. KELMAN: Well, I would like to go a little bit be very heavily affected by the ability to understand the further and reject if I can, just for the sake of a con. process by which you achieve the end result, and as troversy, your emphasis on process evaluation. I'm not much attention has to be paid to the issue of process as against it. Let me say that like everybody else, I'm for the end result. motherhood and all of that. No, these days you're not The only reservation that I would have is that there supposed to be for motherhood. I'm not opposed to are occasions when it becomes incredibly difficult to process evaluation. However, I think, Chuck, that at tease out of the situation anything but very, very global least as I look over much of the evaluation literature, I'm descriptive information about process. But yet the end struck by the fact that we have many more overall r irm one. And I have a very de esult in itself can be a very f' scriptions of program and process and visits than we specific situation in mind. have end-result evaluations. As I look over the process Some time ago we looked at the question of perinatal kinds of things - and this may be strictly personal, but I mortality and prematurity in HIP in contrast to the rates don't think it is - they raise no questions in my mind amon- patients of private physicians in the community about program. However, when I look at outcome eval. Z, uations with or without process, they at least raise a and did all the necessary standardizing. We came up with question and would give me some pause about programs. a finding of lower mortality and prematurity in HIP. And the next question we raised was: What is there in Now, I don't agree with the kind of response that you HIP that produces this type of result; in other words can made to the first study - that if it's good, fine, and if we identify the process of care responsible, as well as it's no good, let's forget about it. I don't think that other factors? would be an appropriate response to outcome result But, it was just not possible for us to examine the where you may not know the process or channel. process by which people received their prenatal care and I can give you an example of a study we're involved the other circumstances in the process of medical care in where the outcome was negative. We had excellent that mic-,ht have influenced this result. I think the whole descriptive material on the process. Nobody paid any cause of reducing infant mortality would have been attention to it because it was a negative finding. So advanced if we had been able to get at the process, but that's one point. 90 The second point I would make has to do with the tant separate question, and I wouldn't treat them as utility of evaluation. I think that obviously there are necessarily intertwined. factors aside from the presence or absence of process The other thing is that my own -hard evaluation information that would make the acceptance or experience - I mean in terms of doing long-term studies rejection of an outcome result affect its acceptance. It' - has been in mental health, which is a little different there is no question to begin with about the program, if from a lot of other studies. everybody is sure that is the only thing that can affect But we did a study where we were looking at re- family planning, if this is the only alternative to patient habilitation of chronic VA patients with control in neglect, then I submit this is not a question to be studied an experimental ward and reached a conclusion on most or to be evaluated and an evaluation is strictly eyewash. of our variables that the experimental ward was a little It would seem to me that what is really wanted is better and on one variable it was worse. And in a sense documentation of the efficacy of what people's faith is that was hard, you know. I mean the data was as good as in something, and I submit this is not an appropriate you ever get in psychiatry, which is a little weak. condition for evaluation of either process or outcome. But then I think the creative part of this in some DR. LEWIS: Let me respond and say that when I was sense came in a bunch of us sitting around the table - making comments about these papers, I hope you didn't by a "bunch" this included some patients too, in- lose the fact that I have been in and out of character in cidentauy - and trying to figure out, "Well, gee whiz, this discussion, one of which is a political animal con- we thought we were going to get big differences." And cerned with getting things done and trying to keep peace yet we were only getting very small differences. and run an organization. What was the process? And, furthermore, what were And mai@f6at's what all this is about - interor- processes that didn't exist in either ward that might have ganizational conflict and the ways one deals with it using been instituted that one might want to carry forward in evaluation information. further experimentation? It seems to me that the majority of people who want That's a very soft set of procedures. I think it's very to evaluate something, Howard, come at it the way you important that this be done. just said: "We have a good thing. Wouldn't it be nice to I agree with one of the comments that was made that show it?" there's a great tendency to get so embroiled in process DR. KELMAN: "For you to prove it." because outcomes tend to b'e more difficult to measure, DR. LEWIS: "For you to prove it." I think some- that you end up patting yourself on the back as the times the most fascinating opportunity for evaluation process looks pretty good. comes serendipitously that way. And you can say "We DR. LEWIS: Let me restate. I have tried to say that I don't do that kind of evaluation," wherever you are think both have to be done whenever possible - but locally enshrined, or you can say, "Okay, buddy, we'll there are circumstances in which only one or the other have a go at it but let's be prepared to take the worst can be done and appropriate circumstances when maybe answer you are prepared to hear." only one or the other should be done. It seems to me that evaluation almost could not be But I don't think there is such a thing as process or separated - just a personal opinion - from the political end results. And this gets to be an ideology, and it really and the ethical context in which it is performed and breaks down between the denominator and numerator without the consequences to those who are involved in people in the world, those who are concerned about it. That may be a little more philosophical than I'm groups and don't give a damn about cases, and those supposed to be. who are only interested in what happens with the case. DR. Fox: Two comments. I agree with what you And these two subcultures have always existed. say. However, I think that one must separate two very MR. SHAPIRO: Present company excepted. important issues. One is the bureaucratic and political DR. LEWIS: I don't want to- I'll run up a flag in pressures to prevent good evaluation. rwnute. But I think, quite honestly, this is one of the Now, that is a very important product. In fact, I tend problems in trying to diffuse this issue of what are you to believe the primary reason why good evaluation going to do, because it really is related to personal orien- doesn't take place is more for that reason than the tations about how you see care. reason that technology doesn't exist. DR CARPENTER: Dr. Brown, there is a luU here. Do The second aspect though, the relation between you want to give us that followup? Are they still trying process and outcome studies, is itself a terribly impor- to train occupational therapists in way-out Kansas? 9 1 DR. BROWN (Coordinator, Kansas Regional Medical It was the most elementary kind of analysis which w( Program): Well, it's a very complicated thing, and there are an used to. has been a great deal of study of the situation. It's And the conclusion of the meeting, which was- essentially where it was at that time. When,you think about it, a year ago we didn't reaII5 Another similar thing, however, Chuck, having to do know as much then as we do now. The conclusion of th( with changing conditions. It's the phenomenon we see meeting was that we had to get a handle on ways tc with the home health care service. If the nurse makes describe Regional Medical Programs, from my point oi rounds in a hospital with the physicians, she builds her view, that would be able to focus in on what kinds ol clientele for the visiting nurse association very rapidly. If transformations were taking place in the health car( she is at headquarters and doesn't go into the hospital system through Regional Medical Programs. and make her own, she doesn't get referrals, which is the You know - big deal - that's the conclusion. same - which has to do with awareness of physicians, Well, we haven't really got any further than that, and you know, of whether everything is really lovely or yet I feel when I go looking at the budget su@missior where it isn't. when it comes in to me and I have to make some recom. The same with the PT. Since they don't know and mendations that I can't really justify Regional Medic@ have personal experience, they really think everything Programs budget just on terms of additional trainees thii must be all right and they really don't need it. year or whatever. That's not really what Region@ It's a complicated problem hooked up with our whole Medical Programs is about any more. educational process in the state. So they haven't really And I don't know what kind of indicators to use, made any progress. This is a tough question. DR. CARPENTER: It was effective evaluation I DR. CHAR LES LEWIS: To drop back and say some. gather. thing here since I'm out of the RMP business, isn't this DR. LEWIS: We didn't spend some of Dr. Brown's the whole problem since 1966, that the RMP was based money anyway'. on a promissory note which could never be delivered, DR. BROWN: They still want it. which was really the elimination of heart disease, cancer, DR. CARPENTER: Well, could we get some dis- and stroke, and some of us had a strong feeling that cussion around the question, "should end-result analysis besides providing "improving the care of the patients" it be undertaken by every region funding a coronary care was really about regionalization, and the establishment program?" of relationships, and the introduction of change within MRS. BLAXALL (Budget Examiner, Office of Man- the system which occurs only under certain conditions. agement and Budget): I don't know if we want to specif- It sure helps to have a little money. It helps to have ically limit it to that. But a year ago we had a session - some doctors who are hurting. with Pete Peterson and Karl Yordy and a couple of I think it's fascinating that we have focused most of people - the Assistant Secretary for Planning and Evalu- our attention on university medical centers, which are ation - and Don Schon was there and a couple others about the last things in the world that are going to from his firm. change because of the density of prestige and popula- tion. Ane, the whole point of the meeting was to try and I think if one really wants to see innovation in the get a handle on the kinds of evaluation criteria, in- medical care system today you go to the small towns dicators, whatever you wanted to call it, that the Bureau any place in the country and you find nurse practi- of the Budget might use not so much in evaluating in a tioners and physicians' assistants and i-nergers of hos- hard sense but perhaps even describing, the process of the pitals and all sorts of interesting things that aren't activities of Regional Medical Programs in the budget making the New York Times. appendix, for example. We were using such things as the process indicators - But I suspect if one were going to invest a little RMP how many participants in the training program, how cash, one could very easily facilitate regionalization out- i-nany regions were operational, just, you know, just side of those sorts of procrustean things that have indicators, nothing that really explained anything related probably already died but the message just hasn't got to to Schon's systems transformation model, not@ing that the brain yet. gave any flavor of Regional Medical Programs in the MRS. BLAXALL: That's right. I agree with' your description. statement. 92 DR@ LEWIS: A lot of people don't. measurable. The diversity - the major strength of the MRS. BLAXALL: But it doesn't help me in the law's pert-nissiveness toward local innovation - makes question I have. This is a tough question we're still for such difficulty of expression that it now becomes the working on. For example, does that mean instead of bane of the evaluator's existence. Having no national using the old indicators that we should focus in on decision that a priority, for example, for coronary care is anecdotal elements? acceptable, he has less clear evidence as to whether his DR. LEWIS: No, there are end results that can be Region has placed significant priority on such care. measured I would assume. DR. LEWIS: I think if your programs had written real If you would like to talk about the availability of care objectives and not statements of vague goals, they might for populations and the provision of care to populations have been evaluatable. And it's like teaching, you know. that don't have any care, as a byproduct of RMP, I think If you just tell them what you want, which we usually that can be measured - providing that's what your do, it's a mess. Writing educational behavioral change objectives were. objectives is a very difficult job. But there have never been any objectives except to DR. HASTINGS: It occurs to me maybe we have got "improve the quality of care for patients with heart a new definition of what RMP is really about. If we disease, cancer, and stroke" - starting at where the care make the assumption that RMP's real business is social was probably the best. change, if we are supposed to be changing things, then .MRS. BLAXALL: Does this get back to the question perhaps we should shift our statement of what our ob- then that I hear when I go around and talk to some of jectives are from disease-related, medically-related the regions, "Who's making the objectives for RMP?" criteria as listed in each of these articles, as enumerated Washington or our local RAG? Is that the kind of in each of these articles that you just discussed, and question that you're getting towards? frankly say that we're in the business - that we're in a Because if there aren't any concrete objectives at the political business, an organizational business instead of national level, which is what I suppose I have to worry being in a task-related business, that we're in the busi- about, then - ness of changing a system. DR. LEWIS: I think RMP when it emerged in And if we define ourselves that way, then it's possible 1966, for some of us that really got seduced into the to state objectives that one can measure, different kinds ocess without knowing what is going on and of objectives that people have tried to measure. planning pr found ourselves operational before we really knew what But if that's what we are about, maybe that's what was going on, we had been at that time fascinated by the we should be doing. fact that this was a program in search of objectives, that DR. LEWIS: It would have been nice if the original there was an enormous amount of money to be spent for law hadn't said in it as long as it doesn't interfere with doing something, but no one ever defined from hierarch- current patterns of practice. ical quarters up there what was expected of regions, and DR. CARPENTER: But interference and change regions grew depending upon, essentially, the philosophy aren't the same. of the coordinator or the parent institution. DR. HENDERSON: That's right. And at that time I think many of us felt the DR. ST 0 N E M A N: I think there is a real gap that has taxonomy of FW. There were hardware-oriented developed in this conference. I think it's been there all regions and software-oriented. There were disease- the time. I think Dr. Lewis alluded to it. It concerns me. oriented and there were people-oriented. They were I'm sure it concerns many other program coordinators. centralized and there were decentralized. They were I think a lot of us were seduced into RMP by the clearly determinable by the nature of the people in- bright hope of local initiative and local decision making volved in the original programs. and system building within.the context of the law as it I do not know whether it has changed or not. This was written, with perhaps a few liberties with the inter- was the equivalent of the identity crisis which over- ference clause. whelms the teachers of preventive medicine annually. . But we did develop regional advisory groups. We did DR. CARPENTER: You know, it's interesting that develop systems. We did spend a couple of years teach- now we are stuck with really so many objectives that ing them what the law says and what it's all about. And there are people who say we don't have any. Each in- we did do this on the thesis that unless we put a system dividual region has a large number of objectives, some of together that could work together we were never going which are immeasurable, some of which, tho.ugh, are to be able to move the system in any effective way. We 93 have begun to make some progress toward doing that, This may be just a time lapse. But I do hot think the but we aren't there yet. I don't know all about all the you should say or anybody should say that we cannc other regions. I know we're not there yet. do it. Given enough money and the proper input it. ea The law is being renewed. It's written by Congress. be done. But it cannot be done except by coflaboratio. It's still virtually the same language except for some between many kinds of experts with background an kidney wording and a few other minor changes. And yet training in the sciences needed for the purpose. the Bureau of the Budget and others in Washington are DR. STONEMAN: I know, but given the fact tha coming through with a clarion call that we're going to be each region is doing its own thing, 1:f'you will, even give@ judged on whether we're agents of social change and the kinds of that you describe-and I listened very closer, whether we can materially, with the dollars we have, this morning, very interestedly-at $200,000 for the firs' affect the health status of the nation very soon. year how long with that kind of a data base.would tha' Now, we spent all day finding out nobody can tell us regional medical program have to go with operation how to measure health status to begin with. So we can't activities directed toward the soft spots and gaps tha evaluate that pursuit except in individual program you identify and develop before you can come bacl activities, and that's out. We're not supposed to measure with a continual status evaluation that will answer th activities as much as we do broad program. The people question that she asked-for one region? back home still think we're. working under Public Law DR. HENDERSON: I can in part answer your que,@ 89-239 and renewals. tion. I cannot give you a time limit. But I can tell you Now, it seems to me that there is an obvious question problem about the whole program that I think extend here that I hope will be addressed before the meeting is this time. Because of the insecurity of funding, fron over. I don't think we can do what we have been asked year to year, our unit has no full-time professions to do until we do what we set out to do - put a system person. No one with enough epidemiological and statis together. And I don't think we can do it by Fiat within tical experience to orga@ze this kind of center can a, the next four months or within the next 12 months, that stage in their career afford to go full-time on @ probably not in less than several years. program without surety of continuity and funding. So i A d this comes back to the question the young lady the program had a more stable base, it could be done ii asked about - what do we put down to justify your much shorter time because you would get people work existence'! I don't think we're going to with $94 million ing at the job full time. The very nature of the progran this year produce enough product in additional health is extending the length of time it takes to do evaluation care delivered to amount to a minuscule fragment of the DR. LEWIS: I think just to reintroduce Buck Rikli'@ total systems production. question as we have come full circle, it's whether or not Maybe we're going to produce a process that can put the kind of data that we are talking about will influence us in a position to do something about that, but I can't planning and operation. give you much more justification than that. MR. SHAPIRO: I don't see how you can answer thal DR. HENDERSON: I want to just try to remove one question-in a kind of global way any more than I could misconception I think I heard. possibly grapple with the global way of stating the issue of changing medical care systems. You can think ir I would not say we cannot get measurements of terms of a change of medical care systems involving a health status. I say we can. I tried to say that it is a total approach. This is a $65 billion-a-year industry. And difficult task and it takes experts in many fields to apply anybody who thinks that RMP is going to change their knowledge and do it efficiently. medical care systems in a very fundamental and decisive I think you have seen that. There are experts in way just doesn't know what's going on. It's unthinkable. several kinds of measurements here today. We have all But you could define medical care systems in clusters, tried to say that it takes a lot of effort, a lot of skill, and in smaller units, in a dimension which you can begin to a lot of skilled personnel focusing on doing the specific grapple with. kinds of evaluation. I do not think the RMPs have had I hate to come back to our own experience and our people with the right kinds of expertise in their pro- own aspirations, but the program that I was describing grams to start off with-for good reason. The majority this morning in coronary care is directing itself at a have been planners and people who had to get programs categorical disease, but to be effective, the way we view implemented and were well versed and became well effectiveness, it means a change in a system. Hopefully, versed in these aspects. through a demonstration of the kind we are projecting 94 there can be an influence on a much broader segment of before we can begin to decide how we are going to eval- the community in developing approaches to a specific uate them. still not clear to me what And I must confess it's disease. going to use for that. I have So I think that there is a danger of stating issues in evaluative methods we are ifig many of such broad terms that it becomes absolutely impossible got some strategic concepts of why I'm do to cope with them. the things I'm doing, but they are steps along the way to DR. KELMAN: Well, again I think if you came away what has been discussed in terms of more profound ftorn the discussion all day with the idea that we can't changes in the system than the reorganization of a given measure health status, then really we failed. subsystem within our coronary care process. I hope that I don't think we here could allow you to slide out clarifies what I said to some extent. from taking a hard look at RMP easily by saying, "We DR. BROWN: It gets back though to this business can't measure health status so therefore RMP can't be about process and end results. If you're going to try to evaluated in those terms." It's not appropriate. define how many people's lives you saved or so on, The discussion we have heard thus far initiated by the that's going to take a very long time and may not be young !ady in the back is very similar to many dis- possible and probably isn't even important. But the cussions I have been in after a program has been process is important, the process by wl-dch subregional- launched and they say, "Well, we'd better get an ization or regionalization occurs. evaluator in here to tell us what we're doing because we Now, that may be hard for people in the Bureau of don't really know." And I think that's pretty sad after the Budget to measure, but that's their problem as well all this time. I cannot for the life of me understand how as ours, because that is where maybe the $96 million can we could get into the sorry state of spending all of these have some influence on what is happening in terms of millions of dollars setting up all of these regional offices the whole. and then come around and say, "Well , I really don't Now, that's about as global as I could make it, and know how to judge whether one or another region or within that there are 55 sub-sets and probably 25 ap- one or another unit should get more or less funds for proaches within those sub-sets of 55 .regions, and then what it wants to do." within that there are a lot of other smaller things that This is an extremely dangerous kind of situation, Dr. Shapiro refers to which are terribly important, but I tying it back into some of Chuck Lewis' comments, for don't know how you measure those in terms of lives you an evaluator to operate in, because he or she can't save. possibly win in such a situation. In other words, you're DR. KELMAN: Could I be antagonistic and ask why putting the evaluator in the position of defining the it's important to have all these subregional clusters and objectives of the program. Do you really want that? I paraphernalia? don't think you do. DR. BROWN: It's a mechanism because someone DR. CARPENTER: Bill, you stimulated a lot of this. feels that there n-dght be a better way or a more DR.STONEMAN-. Yes,l'dlike torespond. economic way or something to deliver health services. I didn't mean to sound like an evaluation nihilist. The DR. KE L M A N: I'm asking an outcome question. thing that bothers me is that we have had for some time DR. BROWN: I was struck with this business here of now some very broad and general aims for RMP out. the neighborhood health clinic where the analysis of the @ned which are extremely vague, and if I overstated their report says that 95 percent of the patients get followup vagueness and the unlikelihood of their immediate ac- contrasted with only - what? - 10 percent or 20 per- complishment, I apologize. But I apparently made the cent. Therefore the neighborhood health center is a good point. thing? If we are going to go to program evaluation at the DR. KELMAN: I don't know if it's good. regional level instead of concerning ourselves with in- DR. BROWN: Well, nobody knows, but that's one of dividual project activities, then I would submit that most the objectives it seems to me we're hearing, one of the of the evaluative techniques that were described this Roals of the regional medical program. Access. Isn't that morning are more appropriate to project evaluation, if I can use that term, than they are to program evaluation. access? It doesn't make any difference whether the out- come was better for the patient. Nobody measured it. Then it's necessary for us to hold our own feet to the But if we could guarantee access, that ,s politically im- fire in terms of setting some precise program objectives portant right now. 95 Now, I'm not saying that's good or bad. I'm just services to migrants all over the country. But because ol saying if you take stability of data you could say, well, the lack of solid evaluation information and because ol here are X number of people who did not get followup. the lack of gras,,-roots support, this program is in real Now they get followup. Therefore, you've improved the jeopardy right now of being lost in the shuffle of another system. bill that was passed. And if I was any kind of prophet I Maybe all you have done is added a component to it would say that the same thing could well happen to the that costs you money. RMP. DR. CARPENTER: I suppose the fear is, Bob, that DR. CARPENTER: Dr. Fox? although that is politically important today, it doesn't DR. Fox: I think Martha and 1,Would like to respond sound as though it's going to stay politically important, to some of the comments. whereas whether or not there is increased access to For those who don't already know, this is Martha improved health care may have a little longer staying Blaxall who is a budget examiner in the Health Branch in power as an argument. the Office of Management and Budget. She also helps riie MR. SHAPIRO: Let me give one example briefly. write speeches for places like here and ropes me into Then I've got to leave. And I'm going to oversimplify the interesting meetings. situation. I think a couple of points have been raised. The During World War 11, there was an EMIC program - problem of insecurity of funds, for example, has some emergency maternal and infant care. Nobody thought in validity. The issue of lack of goals may or may not have terms of an evaluation of that program. There were validity. I think that can be carried too far. millions of women who were delivered through this I wonder, for example, whether you were at. lunch program. After the war that program was abandoned. and listened to Dr. Margulies' speech ' He enumerated There was no supporting evidence that could be used to certain things that were as clear as they are going to be sustain a program that roughly corresponded to the enumerated, and if you people don't understand what EMIC program. There are a lot of people who are they are, then I don't know what else can be done. convinced that some form of EMIC program would have You also heard in the morning that the concept of been maintained after the war and hopefully would have themes versus specific objectives was talked about by resulted in further reductions in infant mortality in this Don Schon, and I haven't heard anybody dispute that as country instead of the long sustained period of small a concept. You know. The messages that you're going to decreases if those responsible had taken the trouble to get will consist of themes. You'd scream if you were think through the importance of evaluation. given specific objectives in terms of numbers of this and There is currently a program in maternal and infant that type of unit that you must engage in. care, and there's a huge amount of money being poured We have heard that you can't measure health status. into that program. I don't believe that that program will Well, you know, I made a big point of this yesterday in continue in the long run unless it can prove itself in one my talk, and presumably you heard that. Not that you way or another. can't measure health status, but that you won't get a I think in the RMP there are very similar types of single measurement of the impact of RMP tied up in one situations. I don't care how carefully you regionalize an cost-benefit measure. We're aware of that. ambulance service to respond to coronary care On the other hand, there are things that can be done. emergency situations. You may have a beautifully I sure learned a heck of a lot today. It (the panel) has operating program. But unless somebody can establish some of the best information of what the state of the art whether or not that program is really accomplishing something in terms of outcome, that program is going to ui measuring health status really is. be chopped. That's the rationale behind outcome. Let me tell you some of the things that I think one DR. LOGSDON: Could I just comment about can expect. I think one can expect movement in another program that had a similar type of outcome in directions. What those directions, the precise directions, the migrant health bill that was passed and which ought to be, that's up to you people again. You know operated on a budget much less than this, about one- the themes. What are some of die sys'L,@i@ ii fifth of the amount, and was passed primarily because of duplication i-ti facilities being eliminated or new duplica- Steinbeck, his writing, and some special interest groups tion being prevented? that were able to get enough support in the Congress. We're on the verge of entering into the kidney field And this program provided health and environmental for big. Are we going to have the same Iliascos there we 96 have had in open heart surgery? If we do, then maybe good things that we think we can do, and these are our the program should be questioned. objectives. This is what we think is reasonable to We have also heard examples here, and I have heard measure us by. Here are some measures that might be them, anecdotally, of important situations where duph- tempting from your point of view but we think they are cation of facilities has been prevented. unrealistic because - We know manpower is important. You know. Is P-NIP And I think the regions have to come forth with doing something to rationalize the introduction of new honest information, not with snow jobs. manpower in the project areas? Now, in a sense, things are bad. There's uncertainty. These are meant to be seed money projects. Are they But the uncertainty isn't, I contend, anywhere near :is engaging in projects that are real projects that are bad as what your statements make us believe. absorbed into the regular system after, say, a two- to DR. JESSE B. ARONSON: I'd like to ask the ques- five-year period? tion as to why in all of these discussions of measure- One can look at core staff and ask whether they are ments we haven't brought in or I have heard really developing a regional strategy that intuitively makes nothing about the measurement of the cost factor. sense or is it a case of just responding to individual We know that we are far from getting cost-benefit requests and interest groups that come in? studies. We certainly can get cost-effectiveness of We have heard statements that the evaluator can't set objectives. Well, this is true in a purist sense. But if the process. And if .we are going to start measuring process evaluator can't help the decisionma-ker set objectives, without measuring costs, I don't think we're measuring can't start to ask questions that assist the decisionmaker process in any realistic sense, in any case that will in any in setting objectives, then the evaluator ought to be fired political sense certainly be realistic. - and I really mean that - because that may be the And I think we ought to put more of our thinking, most important thing that he can fulfill. and we ought to have examples of studies, where the And I know to some extent the regions have to come cost-effectiveness of process becomes an essential forth and say, "Look, within these themes these are the element in our whole measurement system. 97 WORKSHOP ON PROGRAM EVALUATION Participants Harold W. Keairnes, M.D. - Moderator Robert K. Ausman, NI.D. Coordinator for Evaluation Deputy Director Tri-State Regional Medical Program Florida Regional @tedical Program Avery M. Colt chusetts Jack E. Thoi-nson, Ed.D. Field Associate for Eastern Massa Coordinator of Evaluation Tri-State Regional Medical Program California Regional 'Medical Program Claire G. Farrisey Special Projects Coordinator Robert Beckman, Ph.D. Tri-State Regional Medical Program Director of Research Osler L. Peterson, M.D. Nassau-Suffolk Regional Medical Program Associate Director for Data Collection Tri-State Regional Medical Program and William R. Thompson Acting Chairman Deputy Director Department of Preventive Medicine NVashington/Alaska Regional Medical Harvard Medical School Program Dean J. Siebert, M.D. Associate Coordinator for Dartmouth Paul E. White, Ph.D. Medical School Division of Behavioral Sciences Tri-State Regional Medical Program Johns Hopkins School of Hygiene and Public Health Ruth B. Mott Research Associate for Data Collection Tri-State Regional Medical Program Jeannette Forsyth, Ph.D. Project Administrator, Information Support James J. Dunlop, Ph.D. System Arthur D. Little, Inc. Tri-State Regional Medical Program Approaches to Program Evaluation Medical Programs. After talking about the impact of the H. W. KEAIRNES, M.D. anniversary review guideline on local programs, I aid,- "How do you expect the evaluation activities to con- tribute to the development of these program applica- Evaluation is assuming a larger role in the planning tions?" This is the dialogue that followed: and management of Regional Medical Programs. The MR. LAWTON: The program application and program itself new procedures for anniversary review program applica- has to demonstrate that it can manage the process in its tions and in-depth site visits indicate that increased local own region of good health service problem solving. The autonomy in management of activities and funds is (evaluation technique for doing that must not only exist contingent upon a clear understanding by Washington of in the region but must be visible in the application. The yesterday's achievements by the program. Under these region has to know how to apply and use the technique and how to use the results of the evaluation technique. I conditions, past performance is equally as important as think it's a good circle involvement. You have to develop future plans. Evaluation, whether done formally or in- and put down a technique that helps you do your job - formally - if done at all - helps build the bridge from better. the past to the future. DR. KEAIRNES- You talk as if evaluation has something to do with planning. Recently I tape-recorded a brief interview with Mr. MR. LAWTON: I find them hard to separate. I think that the Robert Lawton, Deputy Director of Tri-State Regional credibility factor is extremely important here. I think if 99 You are going to do good things for patients and good But none of these differences negate the value of ti things for patient care through rationalization, then you game movie and the process of planning for tomorro have to demonstrate that what you did yesterday had at happened yesterday. What folloN some merit and improved patient care - so that evalua- on the basis of wh tion is an on-going thing. Today's planning and tomor- is a description of the concepts and methodology f, row's results are pretty dependent on yesterday's evalua- taking an RMP game movie that will allow a clear asses tion. ment of the performance of the teams involved in Wil In the game of improving patient care, that's another ning or losing the game of rationalizing and improvil way of saying that evaluation is part of the process of the process of medical care delivery. winning. Planning and action are, or should be, based on experience. Evaluation involves the systematic de- Concepts of Information Support scription of these experiences and the associated achieve- ments. If done well, evaluation can supplement the gut- Evaluators in Regional Medica Programs play the rol level feelings that play such a prominent role in most of cameraman, not coaches or players. In their role, the decisionmaking about the future. Unfortunately de- must keep the camera focused on the crucial activitie cisionmakers have functioned so long without systematic, on the playing field if the coaches are to have usefu evaluation that many feel that they can win without it, game movies. Evaluators have not been hired as judges or, at least, by paying no more homage to it. ' Only those persons whose decisions influence the fate o The model for winning through the use of evaluation an organization can really be considered as judges Evaluators are hired to provide information to decision has been established by that multi-million dollar in- makers so that their judgments are not made on in dustry - professional football. Each week each team complete, inaccurate or biased information. In thi! records the process of their winning or losing in the sense, they are concerned much more with INFORMA. game movie. The coaches and, to a lesser extent, all TION SUPPORT than with judgmental evaluation. members of the team spend many hours reviewing the game movie. They evaluate every plan and the per- This concept of information support makes sense forinance of every member of the team. Those plays that only when the decisionmakers utilize the information. 11 worked well will be used again. For any play that didn't no one but the cameraman sees the game movie, then work, decisions will be made about the performance of the plans for next week's game will be based on the each player and the appropriateness of the play. On this rather undetafled and unsystematic recollections of the basis plans are made for practice and for the next game. coaches and players. Similarly, taking two weeks to And then they practice. There is little mercy for teams develop the film destroys its usefulness. If the film is that continue to make the same mistakes in decision available and utilized, then it must be of such quality and making and performance that were obvious in the game content that the coaches and players find it useful. If movies. Of course they have to take into account the they feel that. it is useful, they will utilize the informa- limitations of their personnel and their system and the tion in their planning and decisionmaking and they will new challenges presented by the next opponent. In next request that the service be continued. In Regional week's game, if they have successfully evaluated, cor- Medical Programs information support services can be rected and planned, they will win. And they may even justified only when there is utilization of the informa- win over a team that has superior personnel and re- tioii and requests for additional infori-nation by the sources. decisioiimakers. The task of a broad-base social change organization such as Regional Medical Programs appears more Decisionmakers in RMP complex than that of a professional football team, but only superficially. They both have the same over-all Who are these decisionmakers in Regional Medical objective - winning. RMP's goal line, however, is less Programs that correspond to the coaches of the profes- well defined. There are many more ways of scoring sioiial football teams? One of the important differences points. The process of moving down the field involves between the two games is the larger number of players many more players. The opportunity for fumbling is and decisioni-nakers involved in RMP activities. RMP much greater. The rules and the officials are much more dccisioiiniakers.fall into several important groups: difficult to identify. The fans are often not interested in 1. Coordinators or directors - the senior executives paying to see the team win. And there are no time outs %vho are responsible for the implementation of the plan- during the game or between games. iiing and operational activities of the program. 100 2. Planners - the comnlittee i-nembers and core staff to resist the activities directed towards rationalizing the ersonnel who deten-nine the direction - objectives - of health care system. le program and the activities that will move the pro- The evaluator in focusing his infort-nation support rain and the region in that direction. services must first know the location of the goal line and 3. Project directors and officers - the core staff and the rules of the game. Then, if he understands who the' )reject personnel who manage the process of project key players are and how they participate in the game, lie levelopment and operation. able chance of providin- a meaningful stands a reason t, 4. Grantors - the members of local and national service; that is,he will make the appropriate observa- tdvisory groups and the staffs of granting agencies whose tions on the appropriate players during the entire game. recisions determine which activities and programs Being guided by the decisionmakers in this process of )ecome funded. focusing his observations improves his chances of making 5. Consumers - both professional and lay persons a game movie that the decisionmakers will find useful. If whose support determines the success or failure of most the decisionmakers will not provide the assistance or if broad-based social chance programs. their assistance is not sought, making the ai-ne movie 9 If the decisions of all these people about how the becomes an irrelevant exercise. Fortunately for both game should be played are correct, Regional Medical decisionmakers and evaluators there are some general Pro-rams stands a -good chance of winning the battle for guidelines to follow. rationalizing the medical care system through voluntary mechanisms. The thesis of this paper is that meaningful Location of the Goal Line- information based on past experiences and provided in a Problems and Objectives useful manner will improve all crucial decisionrnaking. Each problem in the medical care system defines a Of course, evaluative information becomes only part of different goal line. Setting objectives is the process of the decisionmaking process and, by itself, cannot over- specifying which goal lines should be crossed. Planning come problems in communication, resources or con- specifies the activities which if carried out should lead to straints that also influence the decisionmaking process crossing the goal lines. and its restflts. Analysis of published studies, surveys, reports, and Work of the Evaluator applications gives the first level view of the problems of a health care system in a geographic area. Interviews Meaningful information forms the context for the with all classes of decisionmakers and other key persons Work of the evaluator. He must understand how the are required to understand the relation between game of Regional Medical Programs is played, who the described and perceived problems. The degree of players are, what direction the team is heading, and what concensus or agreement on high priority problems gives the coaches want to see before he knows where to focus some indication of the potential cohesiveness of the the camera. For example, focusing on the wide flanker medical care system for problem solving. while he sits on the bench during a defensive play may Obviously the Regional Medical Programs cannot be the same as focusing on the evaluation of a project cross all possible goal lines or solve all the problems of when the decisionmakers really need to understand how the medical care system simultaneously. Objectives and all the components of the program are working together priorities help direct the team towards those problems to further the task of winning the game. Narrowly that most need to be solved or are most amenable to 'focused observations have limited value to understanding solution. Published objectives may or may not be the the total game process. Indeed, focusing on the wrong true operational objectives. Discrepancies arise when area may prevent the coaches from observing the process operational objectives are perceived as being not socially of scoring. Meaningful information that is useful for planning the next game depends on a description of the acceptable or when there is lack of concensus among entire field including the play of all members of both decisionrnakers about desired objectives. Such dis- teams and the success of both teams in crossing the goal crepancies make it more difficult to mobilize resources line. It includes all the projects - both operational and to accomplish the objectives. Planning - as well as all non-project activities of staff Public objectives can be determined from documents. comniittees. It also includes everything that the Operational objectives can be determined by direct team - the forces for the status quo - is doing interviews with, and by secondary interviews about, key 101 decisionmakers. Following these processes allows de- distribution of key persons in relati on to proble scription of the nature of the objectives and of dis- institutional relationships as characterized crepancies between published and operational objectives. exchange, domain, domain conflict, and jo planning activities Data Source: documents history of previous change efforts direct interviews secon ary interviews. Record of Team Performance - ;knalysis: nature of problems and objectives Results of Previous Resource Allocations consensus on problems Local Regional Medical Programs have up to thi discrepancies between published and operational years of experience as operational programs. Unles! objectives game movie exists, this description of team perfon-naii Rules and Playing Conditions of the Game - will have to be primarily performance statistics that , Resources and Constraints in the generally available, such as the number and types Medical Care system plays, number of yards gained, and the number of fi downs, penalties, and scores. Recollections of t The Regional Medical Program's task lies in a setting players give some clues to the process, but they a created by existing institutions and their services, key subject to bias. Nevertheless this information is part persons both lay and professional, existing legislation planning for tomorrow. and regulations, and financial resources both fixed and The players in the Regional Medical Program gar flexible. General socioeconomic conditions, population can be considered to be staff, committee and adviso ,distribution, transportation patterns, communication group members, and all other persons in the medical ca systems and educational resources are also part of the milieu. Describing these facts makes apparent the system. It is important to identify through interviews playing conditions of the game. the members of the team, their skills and attitudes, th( I The constraints in the system are created by legal assignments in the change process, and their p( forces, institutional relationships and history. Legisla- formance record. Their skills relate to their trainiii their position in their institutions, their concern, ai tion, regulations and guidelines may be found in their commitment. Their assignment as well as their p( published documents, but their impact and their ability to respond to new problems can be learned only from formance vary with the activities. administrators who have had to work within and around Identifying all the activities or plays that are carri( out is perhaps the most difficult task facing the ev; them. Institutional relationships can be characterized by uator. There are so many simultaneous activities wi- patterns of 1) institutional exchange of board members, vague starting points, a paucity of progress report staff, clients, and communications, 2) institutional confusion as to who is participating, and a lack of agre domain for clients and resources, 3) domain conflict ment on when the play is completed and, therefor both actual and perceived, and 4) participation in joint when it is appropriate to measure progress. The easy w, planning activities. Historically the fate of previous out is to restrict one's concern to funded operation change efforts and the general responsiveness of the projects. That is appropriate if operational projec system to new problems and new resources suggests the account for 90 percent of core staff and project sta rules which influence the success of all future change time and budget. Unfortunately that is rarely the cas efforts. These are the rules of the game. The whole spectrum of activities that must be identific This information, although crucial to evaluation, is include operational projects, planning projects, con the keystone of planning. It makes clear the condition of niittee activates, central administration services includir the playing field and the rules of the game. The eval- communication, research, data collection, an uator should watch for ignorance or raisperceptions of evaluation, conferences, developmental negotiations. ,the conditions and rules by persons playing for the local Once an activity has been identified the players, the Regional Medical Program. assignments, their performance, and the effectiveness ( Data Source: documents the activity should, if possible, be identified. Tb interview performance of individuals relates to how well the carried out their assignments. The effectiveness of th Analysis: identification of key persons, institutions, and re- activity asks not only how many yards were gained - sources short term estimate of progress usually based on a 102 chievei,nent of project objectives - but also whether the patterns and should not result in the players becoming Jay or activity resulted in a first down or the crossing of so defensive that they will not participate. goal line - a long term description of the resolution of The ongoing mechanism for recording and reporting ny of the specific problems on which the program evaluative information depends on the philosophy of the bjectives focused. coordinator, the evaluator, and the core staff. But Resource allocation is akin to selection of plays and participation in the evaluative process will probably he assignment of players. Effectiveness in achieving result in more effective utilization of the information. In rogram objectives is obviously related to having the his assigned role, the evaluator should be responsible for ight players and the right play. Resources include surveillance of documents, especially n-dnutes of meet- ersonnel time and funds which are directly accessible to ings, application for planning and operational projects 4e local regional medical program plus all available and reports of projects and studies in order to maintain ersonnel time and dollar resources in the region that some general structure for all evaluative observations. He ould potentially be mobilized towards achievement of may supplement Ws observations by interviews with ,rogram objectives. persons involved in the various activities, by participant observations in committee meetincs, planning activities Restrospectively many details of resource allocations 0 nd player performance are lost. Effectiveness both in and consultations, and systematic reports from core staff noving down the field and in scoring, however, is ap- and project directors. Involvement of many of the staff iarent because significant gains are t, usually obvious. in reporting participant observations and their analysis )etails become more important when progress has not provides an opportunity to train them in evaluation ieen obvious. In this circumstance winning in the future concepts and the use of evaluation information. )bviously depends on developing a more effective Although discrete segments of the program may seem to .Uocation of resources because new players and new require specialized research or project evaluation such [ollars are not usually available. activities are not a substitute for ongoing program evaluation. Program evaluation requires the identifica- Data Source: documents tion of all activities, all the players on the teams or some interviews other major catecory. The performance of the players in each of the activities, the success of the activities in Analysis: descriptions of persons involved in making procress down the field and drawing first downs Regional Medical -Program activities and the effectiveness of the whole mix of plays and identification of activities players and short-term achievements in moving the pro- identification of effectiveness gram across the goal line in scoring gains against the description of resource allocation problems that exist in the region. In this context evaluation itself is one of the major Once the location of the goal line, the rules and activities of a program. Effectiveness of evaluation ?laying conditions of the game, and the record of tewn activities can be judged from its influence on the Performance have been developed by the evaluator, he decisionmaking and the planning processes. Indeed, if ias two o-bvious tasks: first, to report this information to evaluation cannot be demonstrated to contribute to his organization, and secondly, to set up an ongoing winning the game it cannot be justified as an important mechanism for recording and reporting evaluative infor- activity of the Regional Medical Program. Effectiveness mation. Both of these processes depend upon the spe- and relevance must guide the entire process of observa- cific conditions and needs of his program. He must re- tion, analysis and reporting of evaluative information - member that the information should be considered con- that is, effectiveness and relevance to the decisionmaking fidential and that the coordinator of the program should of all classes of decisionmakers from consumers to have complete control over the use of his analyses and Congressmen. reports. The evaluator should work closely enough with The importance of involvement was summarized the coordinator so that the results are made available in a concise, meaningful, useful form, but with enough quite well by Mr. Lawton in our interview when I asked accompanying detail for use by other decisiomnakers if him if he had an opinion about what proportion of RMP effort should be put into evaluation. Let me close with desired by the coordinator. The evaluator in developing response: the information should recognize that the reports should "No, I don't think I have. be constructive and not destructive. The reports 'should I see it working in this way, an evaluation component, allow an opportunity for development of winning such as you and your associates, but in addition I @k out 103 geographic and medical school coordinators have to apply activities of the program and the effects, both observed evaluation in th@ work and, if they need it, get educated and anticipated, of these activities. Program evaluation about evaluation technique. Our program iniplementors, -,niphasizes a comprehensive approach that accounts for whether they are in the RMP orkanization or in outside health institutions have to have an appreciation of evaluation all activities supported by the resources of the program, also, because I find it impossible to separate evaluation from not just discrete operational projects. Ideally program the objectives of programs or projects. evaluation provides a mechanism for monitoring the I would hope that there is a little bit of evaluation in progress and the effectiveness of all activities in everybody - including me." furtlierin- the purposes of the RMP. These purposes include the philosophy, goals, and objectives of the Program Evaluation Workshop - program as a whole, as well as the more limited A Case Study objectives of each discrete project or activity. Program HAROLD W. KEAIRNES, M.D. evaluation information can then lead to better under- standing of the most effective way to allocate financial This paper describes the educational content that was and personnel resources within the program and to main- developed by the faculty, the educational methods that tain quality control of all supported activities. were used in a workshop session sponsored by the When used for learning or planning, program eval- Re-ional Medical Program services, observations on the uation attempts to assist the staff and advisory coiii- educational process, and implications of the workshop mittees through a continuous feedback process to experience for training and evaluation and development improve the quality of their decisions and actions. This of program evaluation methods. is the traditional planning - action - evaluation cycle. Maximum benefit occurs when all the actors participate EDUCATIONAL CONTENT at least partially in the entire process. The evaluation of Regional Medical Programs at the Role of E@,altiator i?i Program Evaluation local level is an open-ended story. Formal guidelines have not been established because they are evolving. The evaluator concerned with program evaluation is Their evolution is based on a process that takes into the keystone for the following processes: account the multiple uses of evaluation, extreme I .Identification of the uses for program evaluation. variability in judgmental criteria and significant dif- 2. Identification of those individuals and groups who ferences in the philosophy and approach of the 55 will use pro.-ram evaluation -information. regions. The open-ended nature of the legislation - PL 3 .Identification of appropriate evaluation criteria as 89-239 - requires this evolutionary approach. described by each user for each use. 4. Identifyin- his own role in obtaining information There are three basic uses for evaluation: 1) jtistifica- tion 2) control and 3) learning or planning. that allowsitidgment on the established criteria for NVhen used for justification, evaluation must deal with each use. 5. Carryinc, out his role. judgmental criteria that have been established and are 0 regularly utilized by various groups, e.g. National In the process he must avoid substituting his criteria for Advisory Council, RMPS, local and national legislators those of the users o f program evaluation information and political figures, organized providers, and consumer and must be sure that the information is accurate, groups. Usually these criteria are unwritten and often reliable, and relevant to the criteria, the uses and the unartictdated, but they do exist. In general these groups users. make judgments based on their private interpretation of The evaluator's role in program evaluation is quite the intent of the legislation, and these interpretations different from that in project evaluation. 'Project eval- form the criteria for their judgments. Their interpreta- uation is concerned with discrete activities based on tions may vary considerably from that of core staff and objectives that are or can be well defined and agreed to RAGs of local programs and often may not allow for by the project staff and the funding agency. Here the problems of feasibility and practicality in carrying out evatuator's role is to apply systems' concepts tempered the legislative mandate. by econon-dc, educational, epidemiologic medical care or When used for control purposes, evaluation helps ad- other technical considerations to the development of an niinistrators and executive conunittees such as RAGs to economically feasible evaluation methodology - and to be aware of progress in implementation of all the various the supervision or performance of the evaluation plan. 104 )rogram evaluation is concerned with the conglomerate temporal flow and sequence of activities - that is, activities that have poorly defined objectives, that often effects (,,n process and organization can be @annot be clarified to the satisfaction of all the users of observed in 1-3 years, but significant effects of !valuation information. Here the evaluator must be transforiiiiii- the medical care system on the !xtremely flexible and understanding in order to deal process or end results of patient care may take vith the complexities of the task. Rigid application of 3-1 0 years. raditional evaluation approaches, such as may be ap- Stated in a different nianner this process calls for: )ropriate for discrete projects, becomes increasingly I .Identification of all activities of the program - rrelevant as programs become larger or broader in their past, present and anticipated. cope. Precise evaluation of one component of the pro- 2. Identification of all possible effects of these ?ram usually gives little insight into the total program activities. nd usually provides little assistance to those who must 3. Developing methods for describing the process and dminister orjustify the financing of such programs. the effect of all activities, not limiting the scope to Access of Program Evaluation funded operational projects. 4. Conducting the evaluation I in a rational time For the purposes of this workshop, program eval- frame. ation was defined as a process. This process definition 5. Reportiti(, the information to decisionmakers in a ook into account the very primitive state of the art of wav that helps them make more rational decisions. @rogram evaluation. Although projects are underway to Understandin- program evaluation as a process rather evelop the methodology of evaluation of broad range than as a procedure is fundamental to evaluators being Dcial change organizations such as Regional Medical successful in their activities. In this context, success in Irograms, there are no generally agreed upon and tested program evaluation is defined as a development in a Methodologies at present. body of information which is perceived as being useful This process of program evaluation follows the by individual and group decisionmakers concerned with Dllowing steps: the operation of the program and that played some part I .The evaluator shall develop a thorough under- in decisions that were made. standing of the philosophy, history, strategy, and activities of the program. In this step, he may infer EDUCATIONAL PROCESS from his observations what the objectives of the The workshop attempted to reproduce this evaluation program are and how these observed objectives process. One particular regional medical program was relate to published or reported goals and selected so that the process and its associated problems objectives. Such inferences should be verified could be illustrated. when possible. Following an introductory lecture on program eval- 2. The evaluator shall determine who wants or should uation, a group of consultants met in a panel discussion want program evaluation information. From each with several members of the staff of the illustrative pro- of these individuals or groups, he shall obtain the gram. This panel had two major objectives: criteria by which they make judgments and their 1. To identify the philosophy, history, strategy, and intended uses of the information: justification, activities of the program. control, or learning. 2. To identify the question's that the staff members 3. Based on these objectives, criteria and uses, the felt needed to be answered by, the evaluation evaluator shall develop a program evaluation process. methodology. This methodology should be The staff members described their regional medical comprehensive, practical, and efficient. Unless he program as being directly concerned with tran sforming has outside financial support for evaluation re- the medical care system through influence and a variety search, the costs of carrying out the evaluation of activities into a system that filled the gaps in care, should probably be less than 10 percent of total made better use of manpower, improved quality of care, program funds. The scientific disciplines incor- and controlled the costs of care. They used the term porated into the methodology should reflect the opportunistic intervention" to describe the fact that needs of the users of the information rather than their activities were guided more by requests for as- the particular scientific discipline of the evaluator. sistance than by comprehensive, objective-oriented The evaluation should take into account the planning. "TiHing the sod" was the term they used to 105 generate such requests in had no previous.experience in evaluatio n, participated,, describe activities designed to re actively in the discussions. Evaluators,. on the othe geographic areas, and among groups not aware ofithe MO availability of.assistancp from the Regional Medical Pro- hand, appeared very uncomfortable with this "process", gram. They, told multiple stories of how this created approach to evaluation. They tended to react as if their change, e.g. a $1500 contract was used to assist seven previous highly technical, project-oriented approach to community hospitals in developing home care programs evaluation was being threatened. As' a result their in less than twelve months; a neighborhood health center participation attempted to force the workshop from la was opened by the city health department in association process orientation towards a structured approach in with the negotiations for a project on the screeningand which, they, the evaluators, defined the questions, treatment of selected chronic diseases that was being criteria, and methodology. One example merits presentation: A staff member submitted from the same geographic area. The staff members developed a series of evaluative who was concerned with the quality of the "linkages" questions that they felt were important. Three major that she helped establish was badgered by an evaluator questions evolved: to provide a precise definition of a, ."linkage.'.' Although 1. How good are my cooperative arrangements9 she gave several examples of what she meant, she could not in ten minutes of interrogation give a precise 2. How balanced are the program activities? 3. What changes in the health care system have the definition. A program person pointed out that such an program activities influenced? approach was not getting anywhere. This led two other The workshop divided into three small groups to program persons to present methods to describing new discuss the priorities of these questions for the contacts and working relationships between individuals evaluation effort and the methodologies that might be that, although not quantitative, did provide a basis foi used to answer them. Each group contained staff understanding the quality of. these "linkages." members of the program and consultants. It was The discussion groups tended to focus on their own intended that each group, in addition to developing questions about program effectiveness and to discard the priorities and methods, deal with the practical problems questions posed by the staff members. As the discussion of implementing the methodologies and reporting the groups followed this path, they became unable in the results. allotted time to probe into the areas of methodology, The closing session of the workshop, a general Ibis reflected the difficulty experienced by many of the session, was designed to demonstrate the range of Participants in understanding the concepts of prograrr priorities, methods, and solutions that were available to evaluation as a process and their inability to play th( solve common problems in program evaluation. This educational game that had been established for th( depended on the developments in each small group workshop. discussion and was intended to reflect the learning that had occurred in this open-ended educational format. IMPLICATIONS Evaluation of programs that have as broad a mandate OBSERVATIONS ON THE EDUCATIONAL PROCESS as the Regional Medical Programs is very difficult Proven methodologies do not exist. The effect of @ The faculty anticipated many problems in this educa- social-change program is often much greater than th, tional endeavor. Approximately 150 man hours of plan- sum of the effects of the discrete operational project ning plus two trial workshops with smaller groups made that they fund. Application of simple concept the faculty aware that most evaluators would require developed in project-oriented evaluation activities i 10-15 hours of training before they would begin to often inappropriate. Evaluators who hold responsibili@ understand the concepts of program evaluation as a for program evaluation are often the victims of thei process. In spite of that, an attempt was made to previous training and experience in project evaluation. compress this learning experience into five hours. The In order for meaningful program evaluatio planning, however, could not compensate for the short methodologies to be developed, traditional evaluatio time allowed for the workshop. As a result the objectives methodologies must become subservient to the broa( of the workshop were only partially attained. range demands of these broad-range programs. The fir: Participants in the workshop immediately sorted step in this process is developing evaluation concep themselves into two categories: evaluation or program that ate similarly broad-tange. It probably requires thi mn@t nf whom evaluators no longer sit on the side-lines of the prograi as judges and that they become actively involved in the SUMMARY entire change process with program responsibilities in The educational content and methods of a workshop addition to their evaluation responsibilities. Only when session on evaluation of Regional Medical Programs has evaluators have a profound understanding of their been described. The objectives for the workshop were program will they know which consultants and which only partially attained. Observations on the complexity methodologies are truly appropriate to the task of of the subject, the time limitations of the workshop, and program evaluation. the previous experiences of the participants were related Training and program evaluation begins with an to the partial success of this particular training method. understanding of the program to be evaluated. It Further developments in the field of program evaluation proceeds to the development of program evaluation depend upon evaluators actively participating in their concepts. Having passed these stages, it can focus on the own program activities and in a continuing educational application of proven methodologies or the development process. of new methodologies. Training in program evaluation is, therefore, just as much of a process as is program eval- uation itself. 107 WORKSHOP ON RESOURCE ALLOCATION/ECONOMICS Participants hn Glasgow, Ph.D. - Nloderator Conrad Seipp, Ph.D. ;sociate Coordinator, Research Associate Professor, Health Services and Evaluation Research Center )nnecticut Regional Medical Program University of North Carolina .A. Florin, M.D. Robert L. Berg, M.D. c)ordinator, New Jersey Regional Professor and Chairman, Department of Medical Program Preventive Medicine and Community Health University of Rochester lichael Zubcoff, Ph.D. Wayne A. Kimmel lead, Health Economics Section Department of Economics Tennessee Mid-South Regional Public Service Laboratory Medical Program Georgetown University Hospital ames R. Jeffers, Ph.D. Charles W. Caldwell, M.P.A. )irector, Medical Economics Research Center Associate Coordinator, Iowa Regional Jniversity of Iowa Medical Program Charles L. Joiner, Ph.D. ohn E. Wennberg, NI.D. Director, Bureau of Research and coordinator, Northern New England Community Service Regional Medical Program Alabama Regional Medical Program Cost-Benef it and Cost-Effectiveness concerns that the user of these techniques needs to keep Analyses in the Health Field clearly in mind if he hopes to use them effectively and if he is to understand what information these techniques JOHN GLASGOW do and do not provide. The purpose is not to present a Rising levels of health care expenditures; the as- step-by-step "how to do it cost-benefit manual" sociated increases in medical Drices and alleged shortages although one might be desirable and desired. Examples of manpower and facilities; the declaration that access to of calculations of both a hypothetical and theoretical medical care is a right, not a privilege; and the growin- nature, in addition to that provided by Crystal and role of the government in the health care field have led Brewster, abound.2-13- Neither is the purpose here one to concern with the effectiveness of alternative delivery of exploring new theoretical frontiers. Indeed, as systems or resource allocations. Concern with the ef- Klarnian has pointed out "so much has been written@ . . fectiveness of delivery emphasizes the importance of about the application of cost-benefit analysis to the using scarce resources (or dollars) in such a way. as to health field that almost every point that n-dght be made ,,I 4 maximize the return per dollar spent. This, in turn, has has been made. Although perhaps something of an led to the search for planning and analytical techniques overstatement reminiscent of Mill's premature claim that WIL-ch might aid in the task of rationalizing the resource everything that was to be known of economics was aflocative process. Two such techniques are cost-benefit known, the observation has sufficient validity to narrow and cost-effectiveness analyses. the present concern. The attempt here will be to ensure The Crystal-Brewster paper' provides an introduction that terms and concepts used in cost studies are clearly' to cost-effectiveness and cost-benefit analysis. The understood as to their definition, the underlying ,Present essay attempts to build upon this introduction assumptions, and the result and implications for the and to suggest certain conceptual and methodological analysis. It should be clear that the objective is not to be 109 critical of previous work. However, an understanding of principle, cost-effectiveness should do the same. the limitations involved in such studies both increases In practice, however, cost-effectiveness analyses their value to the decisionmaker and provides a are often less complete in listing the total cost ren-dnder of the need for constant improvement of the and benefits. For example, external effects are analytical techniques involved. A secondary goal is to often ignored and certain desired results or consolidate into one paper a number of points which are benefits are specified with all others regarded as fairly well-developed in the literature, but widely constants or relatively unimportant. scattered and therefore less accessible to the less special- b. Cost-benefit analysis normally values the costs ized reader. A final objective is to provide to the and benefits in monetary terms. This provides interested reader a bibliographic resource for further the comnion denominator necessary for corn- personal investigation. parisons of alternative types of prograi-ns. In cost-effectiveness analyses the measure of out- THE NATURE OF THE BEASTS put often is not in terms of dollars, but rather Cost-Benefit and Cost-Effectiveness are terms often in some other unit such as man-years saved, used interchangeably. In actual fact, the two are not the 2. These differences in comprehensiveness and tech- same although both concepts do derive from the same niqlic result in cost-effectiveness being used most theoretical fount-capital budget-@ng theory. In essence, often "when various benefits are difficult to capital budgeting theory is concerned with the present measure or when the several benefits that are and future costs, and the associated benefits over time, measured cannot be rendered commensurate."3d of alternative investment strategies. The goal is to 3. Cost-b-nefit analysis allows comparisons among allocate scarce resources to their most productive several programs which have different objectives. (profitable) uses. Thus, the theor,,, is concerned with Cost effectiveness is used to compare differing determining the effects, as well as the costs, of specific ways of obtaining the same objective. alternatives available. 4. The objective of a cost-benefit study is to deter- In cost-benefit analysis, the monetary cost of a pro- mine if an action or program is worth undertaking; gram, or intervention activity, is compared to the the objective of a cost-effectiveness study is to monetary value of the expected benefits. This cost- determine the best way of achieving an already benefit ratio (of total costs to total benefits) might then determined course of action. be used to compare alternative programs to determine which is the best potential investment. For a specific CONCEPTUAL AND METHODOLOGICAL ISSUES activity, the comparison of costs and benefits is for the In this section, differences between cost-benefit and purpose of answering the question: Do the benefits cost-effectiveness studies will be ignored for the most received justify the expenditure (i.e., is the ratio greater part. Here the concern will be with the terms used, the than I or some other arbitrarily set number)? concepts involved, and the implications of the measure- Cost-effectiveness analysis, in contrast, attempts to ment techniques used. In general, the comments will be compare the cost of alternative approaches to the applicable to both types of studies. achievement of a specific set of results. The goal, therefore, is not to determine the feasibility of achieving Ae Measurement of Costs and Benefits a goal (theoretically that has already been decided), but The essence of the cost-benefit approach is the assign- rather to select from among alternative approaches the ment of dollar values to all resources so that the benefits one approach which will result in a given output for the of a specific activity might be compared to the cost of least cost or the maximum output for a given cost. the intervention and to the projected benefits from Although somewhat artificial in nature, the definition alternative investment opportunities. Obviously, it is of the terms does allow us to specify in some detail the vital to include in the dollar valuation all the relevant major characteristics of, and distinctions between, the effects associated with a given action. two concepts. Economic Costs of Disease Defined. The econon-de 1. Cost-benefit analysis is more comprehensive in its cost of disease or injury, as contrasted to expenditures focus than cost-effectiveness analysis. for medical care, reflects both direct and indirect cost a. Cost-benefit includes a consideration of social components. Direct costs include the actual medical care or external effects as a part of the complete expenditures necessary for the treatment of the disease enumeration of the costs and benefits. In or injury. These expenditures would include both 110 )ersonal (i.e., the cost of hospital care, nursing home technology when making a long-term capital invest- @are, physicians' service, drugs, nursing services, and ment' ' Other examples of sigiiif@c-an-t-o-mi-ss'i-o-n--s--c--ould ;i@lar type expenses) and non-personal expenditures be provided, but the point has been made. Many costs 'i.e., the cost of research, training, facilities, equipment, and benefits are excluded because (1) there is no known ind a pro-rated share of the annual cost of health way of measuring the factor, (2) because it is assumed nsuran ce). Indirect costs are those costs to the in- any undesirable side-effects could be corrected if desired lividual or to society in the form of lost productivity through fiscal tax and transfer measures, or (3) because rttributable to the disease or injury. In essence, this the analyst considers them of minor import for his iinounts to imputing a dollar value to the productivity purposes. Valid as the reason for exclusion may be, the ost through premature death or disability. Obviously, fact remains that the end result is for most studies to -,he imputation must take into account varying life ex- concentrate on what is easily measurable. Unfortunately, .)ectancy, labor force participation and earning rates by in many cases, the easily measurable are not the most Different sex and age groups; the "value" of individuals important effects which should be considered. As a )utside the market pricing mechanism (i.e., housewives, result, particularly, in the health field, it is vital to avoid ,Iderly, children, unemployed); and the appropriate undue stress on the importance of.economic measure- liscount rate. ments. In general, this means it is necessary to It is important to emphasize that the economic costs complement economic values with other non-economic )f the disease as defined above are really the projected values in determining the proper resource allocation. Denefits in any cost-benefit, cost-effectiveness analysis. The Quantij7cation Assumption. The most basic as- rhat is, the benefits to be derived from an action are the sumption in any cost study is that it is possible to -Hminated losses in production output, personal well- quantify in monetary terms the benefits and costs as- being, and resource utilization which result from a sociated with a specific activity. In actual fact, even as- successful program. The cost denominator is simply the suming that all benefits and costs will be included, it is projected budget of the program. still not possible to quantify 'with precision even the Enumeration of the types of factors included in the most relevant factors despite major advances in measure- usual cost study makes it clear that a number of costs ment techniques.4,11 The reasons are easily explained. and benefits are typically excluded from the calculation. Th@ implications are somewhatmore subtle. In addition, a number of assumptions, both explicit and implicit, underlie the definition of direct and indirect It was noted that the benefits associated with the costs, the valuation of specific components of each, and success of an activity tend to result from (1) increases in lime use of the technique @f discounting which have economic productivity due to decreased mortality and major implications for the validity of any cost study. morbidity levels; (2) reductions in the need for facility Again, the emphasis on the presence of biases in the and manpower resources given the eradicated or reduced technique and approach is not designed to be overly health problem and (3) the existence of certain critical. Rather, the purpose is to explicitly recognize intangibles (consumer benefits) associated with good what conclusions these studies do and do not allow to be health such as reduced anxiety in the individual and drawn from the data presented. society or an increased sense of well-being. Despite the effort to define the economic cost of It should be clear that (1) and (2 above are more. disease broadly to include both direct and indirect costs, susceptible to precise measurement than is (3). As a it is obvious that not all costs and benefits are included result, most studies tend to ignore the latter effect. 7he in even the most rigorous analysis. For example, it is result is therefore to often significantly understate the i common to ignore the so-called. "spfll-over" effects. potential benefit of any activity and to particularly These are the desirable (or undesirable) secondary underestimate the value of any activity in which impacts of a given action. Illustrative of such a consumer benefits constitute a major portion of the Secondary impact would be the effect on prices and total benefit. That is, since the consumer benefit availability of medical care for the general population component of total cost varies between types of diseases which resulted from the attempt to provide for the or illnesses,' 6 the exclusion of such benefits, or even health cafe needs of the aged through Medicare and their inadequate valuation, wifl.tend to result in a mis- Medicaid. Another cost often not incorporated into the leadingly low cost-benefit ratio for those diseases with a @Wcuiation is the cost of "locking" oneself into a given high consumer benefit element in comparison to and a decision about the desirability of past expend- there is significant unemployment among the resources itures. The second result might be illustrated best by a in question, utilization in this activity not only entails hypothetical illustration ftorn an article by Warren little or no cost, it may provide an additional benefit. Smith." That is, the result may be a pure benefit composed of a assume - an investment of $10 in a device net output gain plus reduced welfare costs. which produces 50 units an hour and we learn of an Third, those who would make use of these techniques invention of two improvements [tolincrease - ef- often desire the specification of a policy which would ficiency. Item A - costinc, S5 - [increases productive simultaneously provide the greatest benefit and the least capacity to]60 units per hour. Item B - costing $7, cost. While theoretically possible, the attainment of this can increase the output to 65 units per hour - which goal is Iii-nited by at least two factors: (a) limits on adapter would be the best choice from a cost- ability to spend and (b) requirements for expenditures effectiveness standpoint? of a given size. To illustrate, it is often possible to obtain The total cost, if we buy item A, will be the a larger benefit from a larger expenditure and the original $10 plus the added $5, or $15, and the total increase in benefit size need not be proportional to the resultant output is 60 units per hour. Dividing output increase in expenditures. As a result, increased expend- by cost gives us a ratio of 60 to 15 or 4. The total itures can often result in a much higher cost-benefit ratio cost, using item B, will be the ori-inal $ 1 0 plus $7, or than would be a lesser expenditure for the same activity. $17 with a resultant output of 65, or a ratio of 65 to But if you do not have more funds to invest, the larger 17 or 3.8. The conclusion using this misleading anal- ratio is immaterial. In the same way it is possible that ysis is that item A is preferred because it seems to unlimited funds properly allocated amonc, a variety of give the largest ratio of effectiveness to cost. alternatives might provide a total benefit greater than The marginal or added cost for item A is $5, and the same amount invested in a single project. Yet if the the added output is 10 units per hour for a ratio of required funds are lin-dted, the use of funds in one area 10 to 5 or 2.0. The marginal cost using item B is $7 effectively precludes simultaneous investment in the or 2.1. Our conclusion using this correct procedure is alternative. That is, given the cost of doincy A, you may that item B is, preferred because of its greater not be able to do any part of B given its minimum cost marginal ratio." requirements. This suggests two factors of import. (1) Additional Issues. Although not technically cost and Cost analysis, in the usual case, will be able only to benefit measurement issues, four other comments need suggest policies which will provide the greatest benefit at to be made concerning this general 'area. First, ma a given cost or a given benefit for the least cost; and (2) ny studies distinguish between the effects of an activity on in order to provide even this direction, there must exist a production (income) and the effect on the distribution clear-cut statement of the objectives desired. In short, of income resulting from the fact that beneficiaries are cost studies are not a substitute for decision making, but not necessarily those who pay for the program, that rather a tool to help rationalize the decision making there can be an impact on relative prices and real process. incomes; that program investment implies foregone alter. Fourth, it is also of some value to emphasize that the natives; and similar forces. Typically, these distributional total dollar cost of a project does not always reflect effects are ignored in most cost studies and for good reasons. Nevertheless, it should be noted that the accurately the allocation of resources which it theoretically summarizes. That is, the relevant market ignoring of these effects can lead to either an over - or prices of resources do not necessarily reflect their true understatement of total benefits derived. For example, if value (ie., actual costs) to the system within which they an activity not only treats a disease but leads to a more are being allocated. Some of the reasons why this is true equitable tax po licy, the ignoring of this latter fact have been previously alluded to (e.g., valuation of "non- seriously understates the value of the program. market resources" or of human life itself'and the use of Second, it was previously noted that one cost often previously unemployed resources). Other reasons include excluded from most cost calculation was the effect of a the fact that prevailing prices reflect a given income dis- program's initation on the price and availability of re- tribution. A different income distribution rWght result in Sources which could have been used in alternative ways a different demand and price structure. Fina , one had they not been used in this activity. This implicitly might note that only if the structure of market prices is 'assumed a state of full employment. However, where that which would occur under perfect competition would the social opportunity cost* equal the net cash the productivity of an investment and society' payments for the project.' ' reluctance to sacrifice current for future consumption 2 3 Ideally, then, as Wennberg has noted, the vigorous Attempts to utilize a private market rate of interes application of these techniques presupposes a detailed assume that the individual's time preference for mone, and accurate analysis of the system and the economic coincides with the collective preference as expressed b 2 4 environment if the cost and benefit implications of the the market rate. This is not necessarily ture. Indeed proposed project are to be fully understood. it is argued that the individual's discount rate for th( distant future will tend always to exceed socie ty,S.21 The Discounting Procedure Fourth, the time preference for money is not constant Previous mention was made of the desirability of with age. That is, it tends to vary inversely with lif( expressing future benefits and cost in terms of their expectancy. Finally, for any discount rate chosen, it i@ present equivalent value (i.e., to determine the present usually assumed that the general price level and value of future dollars). The present value of future productivity will remain constant over time. This is not a expenditures is the sum of money that would have to be valid assumption, but an understandable one given the set aside at present and cumulated at some rate of measurement problem involved. However, Klarman has interest in order to equal the monetary cost of the ex- suggested the desirability of developing an effective net penditure at the time it will be incurred. Reversing the discount rate by combining price and productivity idea, one n-Light discount a sum of future money by the changes that are simultaneously operative into a single interest rate chosen to get its present equivalent. rate.3b For example, one might divide the chosen Obviously, the choice of the discount (interest) rate discount rate by average price change (in percent). This used in the calculation is of vital importance. Some ratio divided into the sum of the present value of output argue that the proper interest rate to use is the pre- in dollars terms multiplied by the increase in vailing market rate. Others argue that this is inappro- productivity expected would give an effective net rate of priate for a number of reasons. No attempt will be made discount. to examine the controversy surrounding the proper rate It is clear from the above summary that the choice of of discount to use since this entails a field in itself. It is the discount rate to be used, no matter how universally of value, however, to briefly summarize some of the accepted, is an exercise in value judgment and quite major issues involved in the controversy leaving to those arbitrary. Under these circumstances, one might wonder interested the task of reading the references previously why the discounting exercise is performed. Blum, for 16 cited. example, suggests abandoning the practice. However, First, even a desire to use a market rate of interest is it seems clear that there is no other effective way to hampered by the fact that there is no single market rate. reduce continuous and unequal dollar streams to Rather the rate varies with the type of loan or obligation comparable values. Consequently, accepting the need for involved, the borrower, and time period, among other and value of discounting, the concern is with the things. Second, in the choice of a proper discount for implications of the process for the results of the study. social benefits and costs associated with public invest- 7he most obvious implication is that relatively small ments, the choice is complicated by the existence of a variations in the discount rate chosen can produce close relationship between investment decisions and the relatively large differences in the cost-benefit ratio. And social discount rate used in investment planning and the greater the time span involved the greater the between investment, the method of financing used, and variance. A second implication is that the higher the fiscal policy. Third, a discount rate is intended to equate discount rate chosen, the less likely programs with long delayed returns are to be given high benefit-cost ratios. *Social Opportunity Cost is the reduction in consumption A third implication, which flows from the second, is that and investment which occurs due to the transfer of funds from service programs will be favored over research programs the private to the public sector. It is the sum of (1) the amount in the usual case. As a result it often is suggested that of foregone direct consumption in the private sector and (2) the studies should provide multiple rate analysis to discounted value over time of the decrease in future consump- demonstrate the range of priority rankin which results tion which would otherwise have resulted from the investment 9 of the portion of after-tax income not presently consumed. For from different rates. an excellent review of the concept and its development, the Miscellaneous Yr6blems @iterested reader might consult the references to Feldstein in the bibliography. Further and more recent works are those by In addition to the biases and weaknesses imposed by Baumol, ARTOW, and Pauly also listed in the bibliography. measurement techniques or the discounting process, two 114 l,nust ,yield unambiguous useful these techniques 12 3 it is true that use in areas might be nen,ioned. First, riteria on the project over another", ves to be proble , valuation of human life Or some of the c techniques does 'Ice pKogram object.! ,pared to th ts, the calculation of amounts of these cified. , difficulties involved in indirect COS s) is conceptually unambigously spe arPue that th edical care (direct cost Finally, One can - lue in itself. Certainly, 'cult te study ha, va ended for in speaking, however, it may be as diff doing an adp-qua aker to practically ts given the these problems should force the decision ni urate estimates of these COS ty of ap- r the technique should even be applied develop acc le, accurate data, the difficul question whethe That is, in many cases k of availab efits when multiple morbid to certain Problems or decisions. analYlis rtionment of total cost-ben and, the existence of free .red, and the sophistication of riditions exist in concert, o the degree this difficulty the time reqtl be greater than equired or affordable,. rvices Or payment in kind.'r ther over-or involved, may tudy itself will involve the use of resource ticular case, beiefits may be ei After all, the I prof jsts in a par icant amount . ,vhich might be more Itably employed elsewhere. riderstated by a signif ost of his ny cost studies subtract thec second, ma alculating the Bibliography lailitenance from future 'earnings ill cit should be ost Benefit and man."' if deducted oyal A. and A. W. Brewster, "C tll Field: An alue of a. an result in a value measure l. crystal, P, ti,eness Analyses in the Heal @conomic v Cost Effec -ealized that the calculation c t is, it may "Inquiry 3(4) 1),c,,ber. Program which might be negative. Tha introduction, I af output loss eakinz the best course would 2. Department of Health, Education aid Welfare ted Disease control Programs appear that econ0mically'sP This is generally AnalY sis Gro up on selec otor Vehicle ff, the population at risk. t-cost Analysis to M be to "kill 0 . In any case, a. Application of Benefi .,itant Secretary for Program red a practical recommendation Accidnt,, office of A not conside - ias program el,,tion toward coordination, August, 1966 Division of the practice Will tend to b , or younger, public Health Programs fr puthrit's those activities aimed at the 'chigh income b. nic Diseases, U.S. Public Health Services, Chro ctive worker ember, 1966 more produ Sept .ng, ilrogallning, and Budgeting C. An Analysis of Planni -troi Branch, Division of c CONCLUSION Cancer Control, Can er Con ices, 1966 re not infallible in ic Diseases, U, S, Public Health Serv It should be clear that cost studies a Chron fact, applied . Herbert F- niversity Press, urce allocation. In 3. Klarman, O,,i,, of Health (Columbia U guides to proper reso fit ratios would a. Econ comparison of cost bene 1965) pp. 162-11 g Benefits of vigorously a rograms serving vt)hillis control Programs, in Measurin fnan, ed. tend to result in a prepondenance of p b.IIs. .ment Expenditure,, Robert Dor adult, white, college niale, Govern ion, 1965) pp. 361-4" the young ally clear that p ent state of the rookings institut pact of Heart Disease," in Second ,iven the pres (B it is equ -e to include all the relevant C. "Socioeconomic In'- Cardio-vascular Disease, The St study can hop those included National conference on ashington, D.C.: art, no co I 0 measure even Heart and Circulation, vol. 2 (W or Experimental costs and benefits 0 tf precision. Indeed, the whole erican societies f with areal degree, 0 e- Federation of Ain ny ization of objectives to measut Biology, 1965)- pp. 69.3-707. osenthal, "Cost Om conceptual rcise ill value j.o.ls Francis, and G. S- P, of Chronic process fr .continuous exe d. ness Applied to the Treatment ment Of benefits is a with events that Effective I Care 6(l) January-February, judgements compounded by a concern Renal Disease," Medka rtain and often unneasurable. he 1968, pp. 48-54, are unce at case, .er with such studies at all? T 4. Rice, Dorothy d why both of Cardiovascular Diseases an In th- . e. if one keeps in mind that these a. ,Economics costs ies No. 5 (U. S. reason is quite Simple of an unwarranted a Dearance on, D.C.) techniques often give P. 'Economic ot a substitute for Health objectivity and that they are nques can-be of real b. 11 Amen .can making, then these techni - Life, decision maker. They can be of valbe by mbe,, 1967) pp. value to the decision itly list the expected C. al lic Health (No, forcing the decision make, to explic ctivity and thereby 1954-1963. -Benefit Analysis-. A bene ts and costs Of a proposed aclaims. It highlights 5. Prest, A. R. and R. TurveY, "Cost 1965) PP- ri ation of these (December, allow critical exainill survey," Economic Journal alue judgements,assumptions and 35. the pre n e of v a method for Sys- 683-7 on chronic Kidney Dieas@s, (U- se C. . . it i' I in short, 6. Report of the Committee ington, D -C.).1967 ry valuations pilatio;,, and tinF arbitra s corn. office,. Wasli development) S. Government Prin ic information be -@temat t is not true that to 115 utilization. Moreover, while i 7. WiIdavsky, A., "fhe Political Economy of Efficiency: 24. Huffschmidt, Maynard M., "Standards and Criteria fo Cost-Benefit Analysis, Systems Analysis, and Program Formulating and Evaluating Federal Water Resource Budgeting." Public Administration Review (December, Developments," (U. S. Bureau of the Budget, 1961 1966) pp. 292-310 especially p. 11 8. Smith, Warren F., "Cost-Effectiveness and Cost-Benefit 25. Baumol, William J. Welfare Economics and the Theory,o@ Analyses for Public Health Programs," Public Healtli the State (Harvard University Press, 1952) pp. 91-92 Reports (November, 1968) pp. 899-906 ' 26. Blum, Henrik L. and Associates, Health Planning 1969 9. Hallan, J. B. and B. S. Harris, "The Economic Cost of (American Public Health Association, Western Regiona End-Stage Uremia," Inquiry (December, 1968) pp'. 20-25 Office, 1969) P. 8-21 10. Levin, A. L., "Cost Effectiveness in Maternal and Child 27. See, for example, D. J. Reynolds, "The Cost of Roa( Health." New England Journal of Medicine, 278(19) May 9, Accidents" Journal of the Royal Statistical Society, 1 19(4 1968, pp. 1011-1047 September, 1956, pp. 393-408 11. Muskin, Selma J. a. "Health as an Investment," Journal of Political Ecotiomy, 70(2) Supplement, October, 1962, pp. Role of Social and Behavioral Scientists 129-57 in RMP Evaluation* b. and Francis d'A. Collings, "Economic Costs of Disease and Injury," Public Health Reports, vol. 74, MICHAEL ZUBKOFF September, 1959, pp. 795-809 First le.t me preface my remarks by stating that 12. Fein, Rashi, Economics of Me?ital Illness, (Basic Books, 1958) believe the social and behavioral scientists' key contri 13. Wiseman; Jack, "Cost-Benefit Analysis and Health Service butions to RMPs are in the areas other than evaluation Policy," Scottish Journal of Political Economy, vol. 10, such as being an initiator of change in the region as wel February, 1963, pp. 128-405. 14. Klar7nan, H. E., "Present Status of Cost-Benefit Analysis in as aiding in the development of program strategy fo achieving RMP specific goals of increased regionalizatioi the Health Field," American Journal of Public Health 57(l 1) November, 1967, p. 1948. and more equitable distribution of health services. 15. Thedi, Jacques and Claude Abraham, "Economic Aspects Before turning to a "definition" of the role for socia of Road Accidents," Traffic Engineering and Control, vol. and behavioral scientists in RMP evaluation, it is neces 2, February, 1961, pp. 589-95. s 16. Weisbod, Burton A., 7he Economics of Public Health ary to spend a few moments revienving: 1) the,variou (University of Pennsylvania Press: Philadelphia, 1961) pp. levels of evaluation that exist and 2) possible strategic 95-98. of evaluation within RMP. 17. YJarman, H. E., "Some Technical Problems in Areawide Planning for Hospital Care," Journal of Chronic Disease Levels of Evaluation 17(9) September, 1964, pp. 735-747 18. See, Eckstein, O., "A Survey of the Theory of Public Basically there are three levels of evaluation: Expenditure Criteria," in James Buchanan (ed.), Public 1 .Monitoring of specific proi ects. Finances (Princeton, 1961) 2. Medical evaluation of specific projects in terms o 19. Feldstein, Martin S. a. "Opportunity Cost Calculations in Cost-Benefit quality of care. Analysis," Public Finance, 1964, No. 2. pp. 11 7-39. 3 .Social, behavioral and economic evaluation o b. "Net Social Benefit Calculation and the Public Invest- RMP specific goals of increased coordination ani ment Decision," Oxford Ecoizo?nic Papers, March, more equitable distribution of health services. 1964 C. "The Social Time Preference Discount Rate in Cost Benefit Analysis," Economic Journal, June, 1964 Strategy For Evaluation 20. Pauly, Mark V., "Risk and the Social Rate of Discount," The following breakdown is suggested as a possible American Economic Review 60 (1) March, 1970, pp. strategy: 195-198. 21. Arrow, Kenneth J. "Discounting and Public Investment Role of RAGS Criteria," in A. V. Kreese and S. C. Smith (eds.) Water 1. The setting of priorities between categories an( I Research (Baltimore, 1966) 22. Baumol, W. J., "On the Social Rate of Discount," American within categories, and the SUPPORT thereof, fo Economic Revie@v, 58 (3) September, 1968, pp. 788-802. complete end results medical evaluation of specific 23. NVennberg, John E., "Cost-Benefit Analyses-Limitations projects throughout the nation which PMS feel, and Uses," pp. 110-1 1 3 in Proceedings: Conference Work- shop on Regional Medical Progra?ns, January 17-19, 1968, 'ne author wishes to express appreciation to Wflliat Washington, D.C. Volume 11, Public Health Services Rushing, Dan Davis and Robert Metcalfe for aiding in the deve Publication No. 1774 (USGPO: Washington, D.C., 1968) opment of these comments. 116 may be worthy of possible replication (i.e., social and economic components) aimed toward as- coronary care units,etc.). sessing RMPs' specific goals of more equitable dis- tribution of health services and better coordination of Role of Local RMPs services. Here projects can often be evaluated on an in- 1. Monitoring of all projects. dividual basis although it is in terms of the TOTAL 2. Assessment of project and the program's ability to program's efforts (results of all projects) that this type effect RMP specific goals of increased regionaliza- evaluation is most relevant. Such program evaluation can tion and more equitable distribution of health only be done in the true sense in the long.run. services throughout the nation. The social scientist's tools for analysis of changes in 3. Aiding in those "medical" evaluations that RMPS the distribution of health services, regionalization and designates as needing such in-depth evaluation. cooperation must be at the heart of ANY and ALL The reasoning behind this type of breakdown is attempts to evaluate local RMP programs. basically that RMPs should "practice what we preach". The methods of measuring RMPs ability to meet its In other words, we preach reduction of duplication of goals will be many. One may study RMPO -role in efforts within our region, while at the same time bringing about: fostering continual duplication of efforts with respect to I .Changes in functions of individual providers. evaluation of projects. Without having access to RMPS 2. Changes in organization of providers. records, it is impossible to tell the extent of this duplica- 3. Changes in the accessibility of care. tion; however, as one meets evaluators from around the 4. Changes in patterns of financing. . nation, it is quite discouraging to discover that the same 5. Changes in behavior following continuing educa- type of project is concurrently being evaluated, often tion courses. without adequate support, in numerous regions. This is In addition to evaluation efforts aimed at judging the using up substantial portions of RMPs limited resources. program's (and/or project's) achievement of its goals, Thus, it would seem wise for RMPS to set priorities there is in evaluation efforts another area in which social where in-depth medical evaluation should be undertaken and behavioral analyses should pay off. to determine whether or riot specific projects should be That is, trying to assess WHY a program (or project) replicated throughout the country with RMPS support- fafls or succeeds (i.e., what are the. behavioral, social, ing said evaluation in terms of dollars and manpower. cultural and economic forces that make for success or With respect to evaluation, a paradox seems to exist. failure). There are a number of advantages to this focus. RMPs are charged with trying to act as catalysts to Foremost among them is the ability to anticipate the initiate change with respect to increasing regionalization outcome of Project A (or Program Strategy #I), that is in many respects quite different from Project B (or Pro- and increasing a more equitable distribution of health services, which as a process is definitely a long-term gram Strategy #2) 'which has received evaluation - (e.g., if there are social and econon-dc forces that are related to phenomenon, while at the same time. the criteria being the failure/success. of a physician's assistant project, the imposed by Washington for evaluation is short run. f same forces may be related to success/failure of projects It is important to understand that the effectiveness o to recruit physicians, or even the success/faflure of RMPs must be measured as a long-term phenomenon and in fact I would suggest that if RMPs do their job as coronary care units). catalysts well, while documentation of change coincident to RMPs' entrance into a situation or setting will be possible, credit for their role will probably not The Application of Economic Analysis ever be acknowledged. This can'in part be explained by to Regional Medical Programs the difficulty and perhaps impossibility of sorting out JAMES JEFFERS the changes that have resulted from the program's activities and those changes which have come from other INTRODUCTION community activities. Economics is the study of the allocation of scarce resources among competing needs for them. It is an Social Scientist's Role in Evaluation economic fact of life that even our rich nation's re- The role of social . and behavioral scientists must sources are not sufficient to produce all the goods and services that we as consumers -want. Therefore priorities primarily be related to those evaluations (the behavioral, 117 have to be established, and choices involving how much because they are plausible. Many propositions are of our limited resources are to be devoted to producing plausible, but not all are true. Such statements rightfully particular goods and services must be made. should be regarded as conjectures or hypotheses and The real cost of producing a quantity of a particular should not be regarded as scientifically. meaningful good or service is the value in consumption of those unless they relate specifically to a body of data that in goods and services not produced which could have been principle could be examined by some means for the produced had resources been used to produce them purpose of adding support or rejecting the existence of instead of other things. Thus economics is the science of the relationships proposed by the theory. determining: (1) what needs exist, (2) how resources can This is the point at which inductive reasonings take be used most efficiently in the production of ' goodsand over. The statements produced by theory are deductive services, and (3) how rational choices can be made generalizations set somewhat unfirmly on a foundation among consumption and production alternatives. of assumption and on some not so certain "knowl- edge." The truth of the theoretical conjectures may be METHODOLOGY OF ECONOMICS presumed to bear no more closeness to reality than that The methodology of economics consists larReIv of of the truth of the assumptions and "facts" on and from abstraction, deduction, and induction. By abstraction I which they are drawn. Thus these theoretical conjectures mean the formulation of models. Models are logical must be tested against data purporting to describe devices erected on a foundation of certain assumption reality. s and empirical knowledge of behavior, custom, and insti- In economics such tests are usually conducted statis- tutions and are welded together by deductive logic tically. While it is not usually possible to effect environ- resulting in one or more statements or hypotheses mental control in sufficient measure to make data capable of empirical confirmation or refutation. conform to the degree of abstraction required of the The trick in model building is to abstract sufficiently theory, advances in the theory of statistical inference from reality in order to avoid the overwheln-dng and econometrics permit a degree of standardization of Complexity posed by the real world. At th@ same time, variables permitting the testing of many, but not all, sufficient specificity with respect to key elements must theoretical conjectures. Multivariate analysis, as be retained in order to provide reliable and relevant e.xemplified by analysis of variance and multiple regres- deductions as to how key variables are likely to be sion- techniques, pern-dts the estimation of the relation- related and how they interact in real world processes. In ship existing between economic variables of greatest a certain sense, abstraction plays the same role as interest while at the same time neutralizing the impact Itcontrol" in the research methodology characteristic of of other variables on these relationships. the natural and biological sciences. Since social scientists Thus the final "proof of the pudding" in economic in general, and economists in particular, seldom have an analysis lies in answering the question: Do the opportunity to "standardize" populations or otherwise hypotheses advanced on the basis of theory square with manipulate social conditions with the exactness of the facts as exhibited by real world data? If the answer is environmental control provided by modern laboratories, no and if the assumption that statistical design used in theoretical abstraction permits, at least, clear thinking testing is appropriate, the hypothesis must be rejected concerning a few highly important elements of a and the theory discarded as not being useful. If the complex system or process answer to the question is yes, the theoretical conjectures The resultant--of the construction of a theoretical should remain in the list of plausible explanations suggesting "how things work" in the real world until iodel is the clear statement of behavior or of a relation @,l@p that logically exists given the assumptions and such a time as subsequent empirical investigation may empirical knowledge on which the model is based. As refute the theory. Meanwhile the estimated values of the such,, these statements purport to say something about parameters of the relationships identified may be used reality and may be useful in the sense that they provide for policy purposes. a logical explanation of how certain things of interest Thus is the methodology of economics. It is sum- work. Very often they are convenient ways of "looking marized in Figure 1. at things" and are suggestive of new relationships and The emphasis in economics is on explaining the new "ways of looking at things" as well. behavior of the economic aspects of a social system, and For the scientific researcher, however, things cannot therefore a premium is paid for a theoretical explanation terminate wit.h accepting such propositions simply that is consistent with reality as opposed to a mere 118 description of reality. Pure induction involving statistical APPLICATION OF ECONOMICS TO PROG,RAN analysis,.Iet us say correlation -techniques, may provide a PLANNING AND EVALUATION good description,of what is "going on" in a social The process of planning involves a continuous context. But statistics by themselves never provide conscious effort involving the following elements: answers concerning "why" things go on the way they 1. The specification of objectives of the course of ,'do. Data do not interpret themselves, but rather they action being considered. must be interpreted @vithin,a context of logic involving. 2. The specification of, alternatives by which cause and effect relationships. Thus interpretation of objectives may be obtained. economic phenomena is facilitated by a clear statement 3. The collection and interpretation of relevant data of. the theoretical relationships that.logically may be and information. expected to exist. This logic is incorporated in what 4. The specification of the potential costs and economists mean by a model which, as explained above, benefits of each alternative means of reaching each is merely 'an abstract prototype of how key variables objective. may be expected to be related in the real world. S. The development of a model that abstracts the relevant features of the situation being considered. In my experiences many medical administrators 6. The specification of a decision-rule or criterion by underempha-size the importance of a clearly specified which it is possible to rank alternative ways of Attai @ing theoretical model prior to the collection of data. In objectives in order of their desirability. many cases g reat haste is made to collect data without a Effective evaluation is also a continuous process and clear conception as to how the data may be analyzed or differs from planning in the following- respects: interpreted to provide answers to questions essential to 1. Alternatives are not considered in the course of program planning and evaluation. evaluation since a course of action for attaining a desired FIGURF, l.-Economics Methodology 7. If Negative Reject Theory 5. Statistical 1. Real World 6: Data Tests and Estimation of Parameters 7. If Positive Accept Theory 4. Formulation of and Use Parameters Statistical 2. Economic Tests Estimate Model Abstraction Assumptions Empirical 3. Hypotheses Knowledge Deductions Concerning Logic How Real World "Works" 119 objective -has already been selected. (However, after interval must be reasonable and initially can only be evaluation has been performed,,it may be decided to determined or estimated on the basis of the experience terminate a particular program in favor of some of other programs conducted elsewhere. or on the basis. alternative); of expert opinion 2. Costs and benefits are measured in. actual rather Economists can be of some assistance in developing than in potential terms; statements of objectives. Economists can point out 3. The model abstracting the relevant features of the objectives that are conflicting and can assist in the situation may be 'modified in light of experience, the development of quantitative statements. However, the accumulation of data, or refinement in its design; and ultimate responsibility for doing so lies with regional' 4. The decision rule or criterion adopted should advisory groups. apply consistently for all implemented programs for Given an appropriate statement ofI objectives, purposes of assessing their relative contributions to the economists can m@e a very significant contribution to overall objectives of the program. the evaluation process in the areas of modeling, data One of the prime requisites of effective evaluation is collection, and the analysis -of data. the statement of the objectives of a given program. COST-BENEFIT AND COST-EFFECTIVENESS Statements of objectives'@hould not be too broad and MODELS imprecise, should not be conflicting, and should be stated in quantitative terms whenever possible so as to There' are many specific models which economists facilitate both planning and evaluation. have developed over the years that would be us eful to A much too broad- statement of an objective for a program evaluation. They are too numerous to describe regional medical program would be: to reduce the pain, in the space allotted. Therefore at the risk of dmittiiig suffering, and mortality of heart patients living within many models that may be of interest for the purposes at the boundaries of the region of consideration.'The state- hand, I will briefly describe the one that, in my opinion, ment is much too broad since any coronary care pro. is particularly useful. This. is the cost-benefit or cost- gram, be it one of continuing education or one involving effectiveness model. It is particularly useful for eval- the use of a mobile coronary intensive care unit, would uation purposes since in principle it permits the conform.to the objective, and it would be impossible to simultaneous evaluation of the performance of several judge the relative efficacy of these two programs. different operational programs. Economists generally regard cost-benefit- analysis as an offspring of welfare An example of a conflicting statement of objectives economics and public finance, although the first. might be: to reduce the morbidity and mortality of practical applications of the technique -were- made by coronary disease in a given region. This statement of engineers in this country around the turn of the century. objectives is conflicting because the reduction of coronary mortality.may well raise the average number of ii.owever, economists had initially developed the tech- heart attacki experienced by many patients, thus raising' ruque in the middle 1850's' and had refined the principles of the methodology by the early 1950's. morbidity in statistical terms. Clearly, reductions in In essence, cost-benefit analysis is a way of evaluating morbidity and mortality are desirable, but it should be the desirability of a project or of a set of projects when recognized that these objectives are conflicting. They it is important to view project activities over a long time should be stated separately, and decision makers must be span where there are likely to be many spfll-over or side- prepared to compromise between the attainment of both effects on people, other programs, and other activities. 'objectives since they are in conflict. In simplest terms the method consists of a careful Nei@e@ of the statement above are sufficiently enumeration of all direct and indirect elements of costs quantitative in that they fail to clearly relate to a body and benefits. It should be noted that when benefits and of data that may be examined in the interest of planning costs are not measured in comparable units (e.g.,. and Fvaluation. A better statement would include a dollars), 'the technique is usually, but not always, labeled specification of the extent to which improvement in the cost-effectiveness analysis-thus explaining the dif- condition of pa'tients is expected. An example of a -ference in. the terms used to title this section of the better'statement would be: to reduce the morbidity of paper. The phrase "cost-benefit analysis" will be used coronary. heart disease by "X" percent over a specified throughout the remainder of this paper time interval. Of course, the specification of the exact Cost-benefit analysis involves a comparison of costs percent of reduction of moribidity or the exact time and benefits associated with a program- or set of pro- 120 an consider the case for a set of projects clusions con- Now we c their relative significance, and draw con . :" 11 . of the patient's illness. which'can be ' rt e varying levels rather than pa icipat din at cerning the nature -grams, where the latter are viewed as alternatives or in an absolute fashion. Attempt, is made to achieve: competitors for overall program funds. The cost side of. the equation consists of estimated or realized program expenditures as itemized in program budgets with due where subscripts 1-n represent_ allowance for the real costs of resources voluntarily different projects. contributed to the project effort. In general, benefits are mb, nib2 nibn Mc is the marginal cost of the viewed as future losses that will be avoided by the - - success of -programs. The major purpose of health pro- MCI mc2 Men ith cure. grams is to save lives and reduce fllness' There are three mbi is the marginal be nefit)of general categories of benefits: (1) gains in economic out- the ith cure. put (usually measured in terms o f income), (2) satisfac- It is profitable to participate in a program until tions from improved health, and (3) savings in the use of mbn > I + i; that is, as long as benefits achieved .are i health resources. m@n - Before going further it should be noted that some of (where i represents the discount rate) times greater than the cost of producing the benefit. With this considera- the differences among authors as to how they measure tion in mind, the optimally sized regional medical pro- benefits are due to differences in the availability of data gram budget is one which allows that all projects a reiibn and do not reflect philosophical differences as to the appropriate use of the methodology. However, some wishes to undertake are participated in to the level that the return-from each project is of philosophy do exist even if the same differences authors had access to identical data. Mention will be made of tms-later in the paper. mb 1 = mb mbn + Once having enumerated all types of benefits and 2 costs, usually some sort of discounting technique must MCI mc2 Men be discounted by an appropriate interest rate to adjust comparable. This is because benefits are likely to be L DIFFICULTIES SONIE CONCEPTUA realized over an appreciable period of time and costs are usually incurred in the present. The first conceptual problem that one encounters is For example, the benefits of a program designed to in developing appropriate measures of benefits. One is save lives may be measured by. the earnings of individuals tempted to measure what appears to be objective and whose lives are saved over the period during which their reproducible at the expense of other benefits not so lives have been extended. Since such earnings extend for easily measured. The economic gains of saving lives is a significant tiyne in the future the income stream must usually measured by taking account of the increased income stream forthcoming to the individual whose life be discounted by an appropriate interest rate to adjust future earnings downward rendering them comparable to was saved. This is tantamount to saying that the value of costs-th@t are incurred in the present. The choice of the a man is what he earns and neglects the affection appropriate interest rate is as yet an unresolved accorded to the aged who have lived a productive life theoretical issue and thus in most applications several and who are retired and who are no longer employed. As interest rates are used resulting in alternative estimates yet a satisfactory measure of the loss of a "non- productive" member of society has not been devised. of the discounted benefits of each program. Similarly no indices of the welfare .gains stemming from Once having measured costs and,benefits for several different projects we can make a comparison between reduced pain and suffering exist. Even if income or earnings are adopted as the them. If we are faced with selecting one project to the appropriate measure of benefits, questions remain con- elimination of all others, -the analysis is simply a matter cerning whether income net of consumption should be of determining which project has the largest benefit to cost ratio and implementing that project. Note that no the measure or whether gross income should be used. provision is made for the project with a benefit to cost ratio' of less than one. Such a project would not be-- CONCLUSIONS undertaken since the returns to such a project would be Mention of these problems serves to underscore the exceeded by the costs of the project. fact that economic models in general and cost-benefit 121 analysis in particular cannot provide easy objective odology for evaluation, we command some,potentiall, answers to all questions involved in program evaluation. powerful techniques for this purpose, but we ham However one of the major contributions of an economic harnessed very little of their pron-dse in a systematic o model is to systematically categorize the key economic organized way.. issues, variables, and relationships that are involved. Like heart disease, cancer, and stroke we seem to lacl Once these have been set out, analysis using objective the ability to relate the various pieces of th e technica data provides guides as to appropriate decisions. Even if competence we command to pursue evaluation irit( complete answers cannot be provided on the basis of meaningful total arrangements. The involvement o objective data and analysis, a systematic specification of social scientists in the evaluation of regional medica the evaluation problem coupled with what objective programs is likely to prove productive only to the exte @ n evidence is available facilitate the consistent application that -there is widespread understanding and conceptua of judgment and expert opinion so vital to correct clarity on the part of program administrators about th( decisions. evaluative process. Social scientists on the basis of th( particular skills they possess are in a position to con Summary of Remarks tribute to the evaluation of on-going.programs. However JOHN E. WENNBERG, M.D. their relevant role is restricted and confined to certair, discrete levels of the process of evaluation. !7urtiler, theii A successful health planning and management entry into the process most often presupposes the capability requires the development of an adequate data exercise of a great deal of prior normative judgment. base. This should be approached through the use of In order to use social'scientists in appropriate ways in multiple disciplines in both the design and analytic the evaluation of social programs, it is necessary to be phases. Relevant disciplines include biostatisticians, clear about the different levels of the evaluative process epidemiologists, economists, sociologists and systems and about the underlying values which assert themselves analysts. in any particular program under review. We must be able The NNE/RMP has developed a planning and eval- to specify the purposes to be served by evaluation and uation base by assembling existing, data sources into a the criteria of judgment that are reflected in the compatible, computer-based system. T ata ase . as formulation of those purposes. been supplemented by ad hoc field studies involving Prograi-n evaluation is predicated on various essential retrospectively collected utilization data and facilities assumptions, however obvious these may appear to be. inventory. In addition, a complementary field social It is necessary, for example, to accept the belief that a -survey capability has been organized. program embraces purposive activity, that socially Details concerning the data system are reported at valued resources are deployed with inte'fit in order to another conference session. Here I would like to report accomplish something. Programs must have goals if they by way of example how socioeconomic analysis, using are to be evaluated. We are also sensitive to the fact that information in the data base, can help clarify, if not resources are limited. At a time when the availability of answer, certain questions of concern to planners. resources appears to be beconu'ng progressively tighter, i The questions chosen for example include those this is another premise which is easy to accept. Programs related to the cost of care and consumer preferences and accordingly reflect the exercise of some form of ration- opinions. The importance of these questions to the plan- ing. The first essential task that we face in the process of n-ing process vAll be emphasized. evaluation is therefore to ascertain t he extent to which our programs are accomplishing the goals which we have Social Scientists and the set for them. Programs consist of a bundle of more or Process of Evaluation less discrete projects. If the planning of a program has CONRAD SEIPP, Ph.D. reached an acceptable degree of precision, each of its constituent projects possesses a clearly defined set of The field of evaluation is like the field of heart targets. A target is a statement of the end results which disease, cancer, and stroke. In both there is a serious gap - are sought through the activity that is called for in a between what we are able to do and what we are in fact project. It identifies the amount oC@-co-mp-lis-h--m-e-nts-, if- today doing. We know a good deal more about the possible in quantitative terms. to be achieved within a process of evaluation than current practice suggests, it is specified period of time. A number of - proj ects 'are my contention. There is a substantial body of meth. collectively the means for achieving the objectives which 122-- efficiency lies in reducing the first level of its efficiency. The quest for , zin the resources ,are set for the total program. Thus, it of output, of mini.mi 9 is in principle at least relatively simple and inpifts per un t or a set of evaluation which must be expended to obtain a: targe clear. It is to- measure the extent to which varions objectives. In the case of effe tiveness, we want to know projects are meeting their targets. 'c how much we are getting as return on the resources we It is an integral part' of the responsibility of the ad- are expending. These must.be seen as separate ptoblei-ns_ ministration of a program to ascertain the extent of the to be dealt with in the process of evaluation. The eval- progress that is made in fulfilling project targets. There is uation of a program entails analysis along both lines. no point in becoming involved in other levels of evalua- How the resulting intelligence is to be Assembled into a tion unless this kind of intelligence is at hand. Perhaps comprehensive assessment of a project or a program the particular tasks and routines upon which reliance has depends upon the assumptions and the suppositions, lye been placed in the planning of some project came to.be bias, if you like, which is incorporated in it. Program viewed as inadequate on their definition, their organiza- tion or their implementation, but it makes little sense to evaluation should conform to the nonns and t@e crite@ia evaluate the adequacy of these unless there is firm- of judgment which are manifest, however co@d@tly, in knowledge of where a project stands in meeting the the planning and the design of a program, even though targets whiclihave been set for it. Similarly, it is point- the evaluation that is done of a particular program by others may be predicated on different ndKmative less to attempt to assess total program accomplishment grounds. until the extent to which the targets of the component I The thrust of these comments is to underscore the projects are being met has been acertained. importance of clarity about the values that inspire evaiu- Buried in the targets of a project, however, are a host of value judgments which need to be made explicit if ative- effort. Evaluation, involving th@ measurement or assessment of program accomplishment, proceeds on the n is to be pursued at a higher and more ertain standards of comparison and particular, evaluatio basis of -c inclusive level of concern. Program people in the field of normative criteria of judgment which are current in a health and medical care still speak of securing -the program and these must be understood and made greatest possible return, however this niay be measured, explicit. The social scientist who is involved in the evalu- for the least expenditure of socially valued resources. portant contri- One knows what they mean when they say that @e@ ation of programmatic endeavor has an im want to get the most for the least. However, this kin(i or bution to make in exposing and laying bare the construct of values which are reflected in a. particular formulation of the economizing intent of a program is program. The need to insure a continuing explication of inimicable to evaluation. It must be challenged if evalua value pren-dses is not only a requisite for meaningful tion is to proceed. For the most is in theory infinitely f of evaluation; it must also be made an inherent attributes great and the least is zero, and. this makes nonsense program planning. This is the point at which planning, their.concern. A. program administrator is motivated evaluation and research, meaning evaluative research, either to maximize output the desired end results of a operations research, administrative research, can it what course of action, with a 'given input or he'wants to vou will, emerge most explicitly as aspects of a single secure some specified accomplishment with the - function. minimum expenditure of socially valued resources. The' ways in which the social scientist is,currently Those responsible for a program are most'o n, in involved in this aspect of program evaluation is at best o accomplish as Much as they can shadowy and uncertain. The relevance of Ws skill at this fact, motivated both t with the resources at their disposal and at the same time level of concern needs to be more.fidly appreciated and to reduce what is required to achieve the objectives the role which he potentially can play requires more which they entertain. The evaluator, however, cannot -definitive delineation. The credentials which the social simultaneously oursue both concerns, for they consti- scientist commands to enhance the sensitivity of the tute separate and discrete analytical tracks. Each must staff of a program to the value implications of their be independently assessed as part of the process of eval- actions are none too solid or convincing. His contri' uation. Further, the evaluator must ascertain the relative bution in this regard is surrounded with difficulty. a pro- Further, the more penetrating and critical he is, and importance to be attached to each in a particul r udgmen't which thereby the more useful the less appreciated he is likely gram. This rests upon a normative j el evaluation. to be. constitutes a given at this second lev the differentiation The task which I am suggesting for the social scientist Correlative to this distinction is between, the effectiveness of a program or. a project and at this level of concern is to ask those administering a 123 program why they are doing what they are doing, what the assistance of a social scientist. There do not appear evidence they possess to validate the assumed worth of to be any compelling reasons to suppose that the social those actions, and how they see the consunnnation of scientist has a unique contribution to make in such tasks particular tasks and activities as related to the attain- as ascertaining whether the development of a coronary ment of the broader objectives of their program.-The care unft in a hospital is on schedule or determining social scientist is hopefully equipped somewhat more where things stand in instituting a tumor registry. The adequately than others to reco nize the 'ways in which same applies if the evaluative concern in regard to the 9 diverse values assert themselves in a program and' to ap- tumor registry is less proximate and centers upon an preciate the various social roots of the normative assessment of its consequences or impact. If the planning judgments that are reflected in the activity he observes. of the program has been adequate, the problem is to His presence first of all may help to make this dimension determine the extent to which the project did in fact fit of a program's endeavor more explicit. He is able to into the larger scheme of the regional endeavor as assist others in identifying and acknowledging the intended. Very possibly the talents of a social scientist normative premises upon which action is based, in might usefully be drawn upon if the issue that emerges recognizing the existence of forces which Militate for in the course of the process of evaluation comes to alternative standards of judgment, and in @ exposing center upon the efficacy or the validity of the technical inconsistencies between the value base of different parts prescriptions that a program has made to achieve a of a program. In this respect the social scientist's role particular end result. Y@t this type of concern, I would within a program is essentially one of education; it argue, should not be included as a primary function of involves increasing the self-consciousness of the staff of a program evaluation. Rather, it should be considered as program about the social forces which impinge upon an assignment for evaluative research which relies upon a them and of which they are a part. different institutional base and set of resources. Regional medical programs will inevitably become involved in The social scientist can obviously make no exclusive such activities but not, as I see it, as the agents who have claims to such a role. Yet he is in a position to deploy a primary responsibility for undertaking such analysis. the special competence he is assumed to command in Rather, they should be a part of a larger consortium of clarifying the normative bias of a program, particularly concern that is involved in the pursuit of such questions. as it is expressed in the functional linkages and 'De social scientist does appropriately come back into relationships which the program generates. In this he the process of program evaluation in the appraisal of the helps to expedite the process of evaluation at the same broader and less specific objectives of a program. Here, time that he contributes to the course of planning. His for example, one encounters the need to evaluate success contribution, if he functions with effect, is to facilitate in promoting the legally mandated obligation to the formulation and appreciation of a clearer, more promote cooperative arrangements as an end in itself but meaningful design of the interrelations between ends and I means. Each project, I have suggested, should have an aiso at the same time to see those arrangements as explicit target, an end result which has been opera- instrumental to improvements in the health care delivery tionalized as a 'i-neasurable accomplishment to be system. However, at 'this level of concern the per- ackieved within a specified period of time. However, formaiice of those social scientists who have been each project must also be seen as the means for attainin involved in the evaluation of programs is far too often 9 disappointing. There is a gross disparity between per- the objectives of a program. Further, the place of the formance and promise. Social scientists tend to be program as a part of the endeavor to realize the 'aims of a ' inclusive health plan must be adequately mesmerized by a conviction in experimental design as more the only road to salvation and they are reluctant to visualized. This is the essential conceptual matrix for the abandon the rigor and the apparent certainty that such conduct of effective evaluation. 1. procedures imply. Only slowly and with great pain are oGiven an adequate spell-out of this kind of a they learning of the tremendous practical difficulties of hierarchy of goals and of the interrelations between hiiposing experimental designs upon on-going social pro- them, the problem of program evaluation is not grams. Yet there are also theoretical grounds-for suggest- especially complex, it seems to me. Assessment of the ing that excessive emphasis has been placed upon the extent to which targets are achieved, even the measure- controlled experiment as i-nethodologically essential in mciit of the accomplishment of program objectives, can the evaluation of programs with broad and ambitious and should proceed without any particular need to enlist aims. These are as compelling to me as the many 124 obstacles which are encountered in practice in trying to of evaluation some methodological alternatives for execute an experimental design. analysis at this level of the process. s fully Upon Let me with desperate brevity identify some of those If the social scientist can't rely a problems. It is extremely difficult for those responsible experimental design as he would like, what methods and procedures can he appropriately pursue in evaluatiti- for a social program to establish and maintain control C, groups, since this entails withholding the assumed social progranis9 Admittedly, he does not have much benefits forthcoming from some intervention for some that is firm to fall back upon, but there may be more individuals or groups at the expense of others. Not only than is currently acknowledged. I would identify a body is such practice often difficult to justify, but problems of effort that has come to be identified as "process are also frequently incurred in trying to enforce such analysis." It is also possible to point to the successful use discrimination. Many courses of action are simply not that has been made of the critical incident technique, an divisible so that the possibility of controls is precluded. ad hoc kind of appraisal of developments as they Another issue is the growing resistance on the part of actually happen to unfold. Much more can be made of various segments of the consumer public to serving as investigations of the communicative network to en- subjects for research. Many are not impressed, to under- compass a systematic appraisal of the volume, the state the point, with what they deem to be the academic quality and the pathways of exchanges and transactions. games which the research worker engages in and they are At this stage of things we must resign ourselves to the unwilling voluntarily to offer themselves as a statistic in necessity of being selective and of concentrating our the promotion of a professional career. Yet even more attention on certain indicator states and conditions, important is the friction which the conduct of ex- rather than the entire range of potential consequences of perimentally designed evaluation studies tends to create a program. One important area to look for help from the between those responsible for the administration of asocial scientist in developing such indices relates to the program and the social scientist who is involved in such assessment of satisfactions and the crossmatch between investigation. measures of provider and consumer perceptions and From the beginning of a program, administrators are aspirations. These are examples of some of the kinds of likely to seek advice and assistance from the evaluator in analytical concerns and techniques which the social refining the desicn of the program and in implementing scientist can bring to the evaluation of the broadest and the plan as it is formulated. However, this inevitably most open-ended objectives of your programs. commands second priority to the evaluator who is com- The hope of enlisting social scientists in such mitted to a protocol which involves experimental design. endeavors depends to an important extent upon the ef- Once he has established his controls, it is necessary for fectiveness with which they are involved in the other him to try to hold a program frozen in order to preserve levels of evaluative effort which I have tried to charac- the continuation of the experiment he has constructed. terize. They must be given the opportunity to learn for Good planning from the viewpoint of the adiwnistrator themselves that the broader the aims of the programs may involve maintaining the greatest possible flexibility they engage in evaluating, the more inadequate is their in order to maximize opportunities to capitalize on penchant for experimental design likely to be. The more unforseen options. But good evaluation in the eyes of rapidly they are able to recognize the necessity of the research workers seeking scientific rigor is certain to relying upon less rigid evaluative' methods and proce- n-fflitate against departures from a predetermined course dures, the sooner they will see their way to relevant of intervention. The exclusiveness of these concerns all endeavors which challenge them not as mere technicians too often eventuates in mutual recrin-dnation, if not out- but as seekers of new knowledge. Two things are neces- right disaster. sary if this is to come about. The administrators of Where the context and the environment of a program programs must become more certain about the impor- are highly fluid and dynamic, where objectives are far- tance of evaluation and they must acquire a greater reacfiing in their consequences and are hard to opera- confidence about their role in that process. The social tionalize or specify with precision, where the unintended scientist, in turn, must become more modest about the results of interventions are felt of major importance, the potential of his claims in regard to evaluative under- program administrator would do well to be critical of takings and seek to develop methods and procedures what the social scientist can accomplish within the which can supplement, at some critical points, substitute framework of experimental design. The social scientist, for the design of experiments upon which he has too in turn, should try more consciously to bring to the task often and too unrealistically come to depend. 125 Cost Benefit-Cost Effectiveness ing ship, or on a desert expedition) or two 65 year old Studies, and Their Application to Other things being equal, the "life-years saved" princip Allocation of Resources should lead you to opt for the alternative involving t] ROBERT L. BERG, M.D. most life-years saved. In this case (using 1967 data) ti maximal number of life-years saved would result fro Making choices among alternate proposals in the RMP saving the 40 year old (34.3 fife-years versus 2 x 14 characterizes something rather new and uncommon in years = 29.6 life-years for the 65 year olds). This leads i the health field: the acceptance that there are limits to an action favoring one life saved over two liv, what can be spent in money or in specific resources such saved,-troubling to all of us committed to doing all v as hospital days, or doctors' or nurses' time. In many can for our patients, but probably comparable wil respects, health care is dispensed as if there-were no public policy in the health field. limits. Except in certain prepaid group health plans, If you have survived this painful moral dilemma, I( there has been little budgeting in ambulatory or hospital us turn to another perplexing issue: that medical cai care. Physicians behave as if they were guided by a may not save many lives but is mostly concerned wit fantasy that they always do everything possible for their improving the condition of life. Instead of simple lif( patients. But none of us has ever done everything pos- years saved, our more usual achievement may be som( sible for his patients. We have always made choices: how thing Eke years of improved f@nction saved. Here w much time we spend with a patient, how long we keep come against the major unresolved issue in the benef him in the hospital, how many tests we order. field (for we are clearly dealing at this point solely wit The following proposal is an attempt to make explicit the benefit half of the cost-benefit problem): namel) the grounds on which medical care choices are made. No how do we weigh years of improved function a new value judgment is suggested. Rather an effort is compared with life-years saved. We intuitively make suc" made to understand what values guide health care judgments in medical care situations. We work harder t, decisions, and in understanding them to make medical save a life than to improve function, but how mucl judgments more consistent with themselves and with harder and under what circumstances? these principles. Furthermore, if we can make the values If we can explicitly articulate the basis for the& sufficiently explicit, a whole new world of decision- judgi-nents, we can be more consistent in futuii making is opened through powerful econometric models judgments. Take the example of the burning house. I which permit rational allocations of resources in circum- contains tluee 40 year old persons. Two single quadra stances where the complexity of data and inter- plegics (totally bed-ridden) are on the bottom floor, an( relationships exceed the capacity of common sense a single working person on the top floor. Assume al decision-making. three have the same life expectancy. In your bes Let us set a goal for any allocations problem: to get judgment, you can either save the two quadraplegics oi the most for our money. But the problem is to define the single working man-what would you choose? ,'most." To most patients and doctors, the "most" is to If you save the working man, then you are assigning 2 save a patient's life. But one patient's life i-nay not have value of something less than one half the value to the the same social significance of another, an evaluation life-year of a bed-ridden patient as a working man. Or, it mostly related to the age of the respective patients. If you save the two quadraplegics, you are assigning them there must be a choice, the 40 year old will usually be something more than one half the value of the working preferred to the 90 year old. This has led to the use of man. "life-years saved" rather than lives saved (cf. Michael, This comprises the basis for a proposed benefits scale Spatafore, et al). Public policy adheres more nearly to as below. The assigned values are arbitrary but not out this notion of life-years saved than the old favorite of of keeping with public opinion. economists, "life earnings saved." 1.0 good health, working What is the practical significance of life-years saved .7 not working, at home when resources are limited? On the basis only of life- .4 not working, institution years saved, would you give the next spot on an artificial 0 dead kidney to a 90 year old or a 40 year old? Most of you If policy makers were able to decide on such a benefit would select the 40 year old. But what would be the scale, and with whatever data is available on the results choice if there were only time.or resources to save either of medical programs, rational explicit decisions could be one 40 year old man from a burning building (or a sink- made with such a scale as a basis. 126 For example, relative benefits of competing programs The example in Appendix C indicates that the rms of life-years saved. But optimal mix would be: could be calculated in te re than saving 750 stroke patients rehabilitated saving the life of a working'man counts mo 6992 persons screened in EDDU the life of a bed-ridden patient: more than twice as 0 patients cared for in ICU much. 430 patients entered in a cancer registry Relative benefits could -also be calculated for Some KMP's may operate as if there is only a money improvement of function. For example, returning two constraint and no limit to the number of available and a half, not-working institutionalized patients to doctors, nurses, or hospital beds. But increasingly plans work would be equivalent to saving the life of a working man. must determine the most efficient way to deploy our These rough and ready calculations have not taken limited resources. Doctors and nurses will be available into account the number of years a patient would gain in for a new RMP program only if they are lifted from other pro,,rams. each functional category,and a proper calculation It must be pointed out that the data are imaginary includ@s these adjustments. A theoretical example with hypothetical numbers and indeed identify a major need for any allocation tech- illustrates how these calculations could assist in choosing nique: better data on the effectiveness of medical pro- grams. The lack of such data is no comfort for the among four proposed projects to be funded from an intuitive planner as compared to the explicit cost-benefit RMP. planner. Both are at a great disadvantage. But the tech- The calculations (Appendix A and B) indicate how to nique forces any planner to specify what expectations he get the most for the money: put it all into the program @ -for his proposed programs an.d what values he where you get the most for the money: the EDDU attaches to the results. (Early Disease Detection Unit) at a cost of $2351 per In summary, cost benefit analyses encourage the benefit unit. planner to specify his value system and behave con- But this solution does not take into account the sistently Nvith it and to be explicit as to the benefits number of stroke patients to be rehabilitated nor the he expects from given programs. It then allows sophis- number of doctors, nurses, or hospital beds available. It ticated solutions for problems too complex to be solved is this quandary that the multiple equation linear pro- intuitively. gramming model helps to resolve. Given 20,000 doctors' hours (on an annual basis), 100,000 nurses' hours, REFERENCE l,fichael, Jerrold M., Spatafore, George, and Williams, Edward 15,000 bed days and 100,000 office visits, what is the R., "A Basi, information System for Health Planning," PUBLIC optimal mix of programs (not the single best program)? HEALTH REPORTS, Vol. 33, No. 1, January, 1968. 127 APPENDIX A Calculation of Benefits 1.0 SI = Working Fl Working Year Saved 0.7 S2 = Not working - at home I (Patient Otherwise Would Diel 0.4 S3 = Not working - institution L= I Benefit Unit 0 S4 = Dead Example Coronary Care Unit Assume benefits of program equal any improvement in functional state due to program. Assume I patient in 20 survives who would otherwise die, and assume his functional status for balance of his life is as follows: IS4 - S3] S3 (Acute Hospital) 0.1 year x 0.4 x 0.05 .002 fS4 - S2] S2 (Home - Not Working) 0.3 years x 0.7 x. 0.05 .0105 IS4 - SI I SI (Working) 5.0 years x 1.0 x 0.05 .2S IS4 7 S2] S2 (Retired-home) 5.0 years x 0.7 x 0.05 .17S .4375 Benefit Units Per Patient By Similar Calculations: Stroke Rehabilitation .9400 Benefit Units Per Patient Early Disease Detection Unit .0265 Benefit Units Per Patient Cancer Registry .1 107 Benefit Units Per Patient 128 APPE@X B comparative Costs Per Benefit Unit For Each Patient in Program Assume MD Hours $ 15 (not counting overherd in institution or office) RN Hours $ 4 Hospital Days $100 Office Visits $ 5 (not counting MD income) Stroke EDDI U c onay unit MD hours 20x$ 15= 2 0.1 x $15 = 1 15 = 300 0 RN hours 80x$ 4= 0.2x$ 4= 4= 600 0 Hospital days 20 x $100 = 2000 0 10 x $100 = 1000 0 Office Visits 10x$ 5= 50 12x$ 5= 60 0 20x$ 5= 100 Total cost per patient $2670 $62.30 $1900 $300 Cost unit benefit 2670= $2840 2@30 $2351 1900= $4343 130-= $2981 - .0265 .4375 .9400 129 APPENDIX C 7he Most Efficient Program 3 Maximize 0.94X, + 0.0265X2 + 0.437SX3 + 0.1107X4 Stroke EDDU ICU Ca. Reg. MD hours 2OX, O-lX2 2OX3 1OX4 20,000 (10 doctors) RN hours 8OX, 0.2X2 15OX3 2OX4 100,000 (50 nurse S) lnst. days 2OX, 0 X2 1OX3 OX4 15,000 (45 beds) Office visits lox, 12 X2 OX3 2OX4 ioo,ooo Solution: 750 6992 0 430 Total Patients MD hours 7@O 6992 0 430 Total Left Over x2O x 0.1 - X10 15,000 699 4,300 20,000 0 RN hours 750 6992 0 430 x8O x 0.2 - x2O 6.0,000 1398 8,600 70,000 30,000 Inst. days 750 6992 0 430 x2O X0 X0 15,000 0 0 15,000 0 Office visits 750 6992 0 430 X10 xl2 x2O 7,500 83,904 8,600 100,000 0 Accountability and Decision-Making Our system cannot be compared with PPBS structures in in the Iowa Regional Medical Program large bureaucratic agencies, but it illustrates how certain PPBS concepts can be applied at any functional level. CHARLES W. CALDWELL I can offer no at formula for evaluating a Program's p My charge is to describe how Planning-Programn-dng- overall impact on a Region, for establishing priorities, Budgeting concepts are being implemented in an even for determining broad program direction. But I can ,accounting/decision-making system in the Iowa Regional tell you of a system that does permit the core structure Medical Program. I will note some of the advantages of to provide certain objective information to the the system over the more traditional accounting systems decision-making process. and relate some of the problems which we face in our constant effort to remain true to the concepts we are It should be emphasized that the system does not incorporating. make decisions. It merely provides objective information The Iowa Regional Medical Program is a small pro- which, in actuality, may be completely ignored by the gram, funded at a level of slightly over $700,000. Our decision-makers in favor of information that is purely core structure consists of ten professional staff members. subjective in nature. 130 Because of our organization's size and due to the Alternative Ways to do a Given Job numberless intangibles which confront all of us in the Rearran-enient of input to an already-existin- pro- health field, we turned to the PPBS approach described 0 t) by Samuel Greenhouse in an article that appeared in grain in order to improve output. Public Administration Review, to guide us in devising Systems Analysis our system.' His approach is simple and clear. He listed the major structural members of PPBS as: (Slide Number Application of cost studies. Objectives Objectives Programs The success of our system stems largely ftoi-n Program Alternatives accurately defining this term. Without doubt, it is the Outputs "apex term" in the PPBS idea-structure. These are Progress Measurements criteria for judging the validity of an objective within Input our system: Alternative Ways To Do A Job I .It Must be directly related to the overall mission of Systems Analysis the IRMP. So that we do not become confused by semantics, 1 2. It must contain a description of an important end- would like to offer a precise definition for each of these service. terms: 3. It MList-at least to the fullest extent possible-be amenable to quantitative measurement. What is SLIDE I-DEFINITION OF TERMS not quantifiable has no valid usefulness within,the Objectives must PPBS context. 1. be directly related to overall mission; 4. It must be honest. In',other words, the stated objective must be identical to the true or real 2. describe an important end service; objective. 3. be amendable to quantitative measurement; 5. When appropriate, it must be broken down into 4. be honest; immediate and long-range expectations. 5. be broken down into immediate and long-range expectations. Programs A program is a package that encompasses each and Program every one of an RMP's efforts to achieve a particular A package which encompasses each and every one of objective or set of allied objectives. A program could an RMP's efforts to achieve a particular objective or set consist of a single comprehensive project or of several of objectives. projects which have allied objectives. It is confusing that in RMP jargon, the overall effort within a region is called Program Alternative a "program." But for the purpo@s:of our system the Other possible programs besides those already term will be used as just defined, - , The whole PPBS idea is to facilitate the coordination decided upon. of all our efforts to meet a particular objective, so the validity of each program may be judged in terms of its Output overall strategy, dimension and costs. This permits it to Tangible outgrowth of a particular program. be compared with other programs, potential or existing. In our system no objectives are acceptable unless they Progress Measurement suggest a program specifically,. designed to fulfill them; Answers the question: How closely does the progress and no entity can be described as a program unless it is planned for match the progress actually realized? designed to accomplish explicit objectives. Input Program Alternatives Total quantity of manpower, facilities, equipment Program alternatives are programs to the sarhe general and materials applied to a program. end other than those alreadk decided upon. Program 131 alternatives suggest a choice between two or more pro- THE IOWA SYSTEM grams designed to advance the same overall mission. Six major steps have been identified by the IRMP a Output essential to significant progress toward its overa. mission. In aspect, each step is continuous and oper An output is a product or a service. As produced by ended, and its influence changes as now information i the RMP, it is a tangible outgrowth of a particular gathered and updated. (Slide Number 2) program. It must be a kind of service that can be singled The first of these six steps involves the gathering o out as an indicator of program results. It must be an morbidity and mortality data and related infon-natiot important end-service and must satisfy an important that pern-dt us to evaluate the effectiveness of th( objective. existing health care system in Iowa. The second step is the assessment of all existini Progress Measureiiient health resources within the region that fall within the If output means only those pragmatic end-services parameters of RMP legislation. that satisfy explicit RMP objectives, then program ful- The third step is the identification of needs. From the flument demands an output that was planned and has information provided by steps one and two, an Iowa been produced. Therefore, progress measurement must Regional Health Profile is being developed. It should be satisfy one question: Does the progress achieved match ei-nphasized again that this profile is open-ended and will the progress anticipated? continually change as new information becomes available. On the basis of the existing profile, we Input endeavor to identify where existing services need to be Input is the total quantity of manponver, facilities, expanded, coordinated or reinforced to meet the needs equipment and materials applied to the program. Like identified. We determine where new services need to be most, we summarize this input in units of dollars. initiated and supported. The fourth step is the establishment of priorities. Alternative TVays To Do A Given Job Conventionally, the criteria considered in the establish- ment of priorities should include evaluation of need, This concerns the rearrangement of input invested in scientific feasibility, practicality, effectiveness, timing, an already-existing program to expedite production or amount of resources available and community ac- upgrade services. In other words, one would rearrange ceptance. the manpower, facilities, equipment and materials going The fifth step is the planning and implementation of into a program in order to improve the quality of service programs to meet these priorities. or arrive at the stated objective in a shorter period of The sixth step is the continuous evaluation of those time. Do not confuse "alternative ways to do a given programs accompanied by modification based upon how job" with "program alternatives." Program alternatives are output oriented. Utilization of a program alternative Nvell they meet their planned objectives and-insofar as it can be determined-the ii-npact that meeting these changes the output, because it is a substitute for a whole program and has different specific objectives. Alternative oojectives is having an achievement of the overall ways to do a given job are input oriented and deal with n-dssion of the organization. the best way to achieve an already chosen output or Our accountabflity/decision-niaking system involves objective. only steps five and six. Not until we reach step five can we measure precisely how well an objective is being met Systems Analysis and consider-if indicated-an alternative program to meet those objectives. Systems analysis within the IRMP system is primarily Other than intuitively, we have no way of evaluating the application of cost studies. These studies are of the overall impact of the IRMP on the health systei-n in special usefulness in tavo areas of the system: (1) the Iowa. A principal reason for this is that presently we determination and evaluation of alternatives and (2) the have no way of obtaining accurate morbidity data. We measurement of costs versus 'progress within a given hope to solve this problem soon. program. Good data on morbidity will certainly aid us in These might be called "pure" definitions. As Iselecting priorities. Such data will not, however, better proceed, you will see how we bend and abuse these enable us to evaluate the overall impact of the IRMP- definitions within our system. because we will still have no basis for relating changes in 132 SLIDE 2 STEP 1 dity and Mort tistics and Ot statistics on the Order to Determine the Effective- ness of lowals Existing Health Ca System. STEP 2 STEP 6 The Assessment of Program Evaluation all Health Resources STEP 3 STEP 5 Identification of The Design and implementa- Needs tion of a Program to Meet Those Needs Establishment of P STEP 4 morbidity to the existence of our organization. Too hierarchy. Actually, the system can be applied at any ptions would have to be made, due to un- level-so long as we remember that, viewed fron, the top many assum of the hierarchy, all these levels are means to anetid. and controllable variables. can be used no ends in themselves. Now let us examine how the Iowa system In Iowa, all staff members contribute to our syst@th- in the development and selection of programs to meet king program particularly in gathering information, ma priority objectives by looking at the decision-rnaking ing cost studies. They also .process from another perspective. (Slide Number 3) evaluations and undertak Visualize a hierarchy of objectives that relate to dissen-dnate .the resulting information to the decision- different levels of this process. makers. 1. At the top is the organization's overall mission. The nuts and bolts of the system can be best 2. At the second level we put program objectives recognized by breaking it down into four broad areas of designed to meet priority needs. activity: 3. At the third level are groups of project objectives 1. Establishing the costs of program alternatives. Ithat make-up a program package. 2. Establishing the costs of Alternative ways to do a 4. At the fourth level we find the objectives of given job. specific activities within a project. 3. Accounting and costing of existing programs on a 5. The final level is occupied by the day-to-day monthly basis. 4. Accounting and evaluation of core activities,oii a objectives that are to be met within a project activity. monthly basis. Our, system is applicable at the second level and First, cost estimates are made on all program alter- downward, since it supplies data that grow in objectivity natives. Most of our program alternatives come to us in and preciseness as we travel toward the bottom of the the form of new project proposals. The, cost estimates 1:33 w o-C c)- rri CL 4A n CD C 4 co o 3 :to@ c- O' 0 - ', 0 3 3 M G) M < '0-0 =, 0 -..R CL M 0 m M ID 5@3 -a o ;u rn 0- -C mm O@ 'D CL@ 0 cn T:o 0 =3,D m 0 cn =,a. 0 c: 0 1 0 m :1 B 2; OM co , m r L- 0 Iq M"Y. 3 O' rn o 0 0 n o m wLI1 o- :i 011' - < CD a, , - 0- @. --- ror, m cu =a) Flo cn 0 0 cx rz c o- :,o, 4x c = 5-0 mm ID 0 O' Lc MO < 7;. cw R s-@ @. C-1 = 0) Er ID au 0) goZ -3 cr CL 0 0 U, c- r; 05 m 0 0CX m 0 w 0 rr Z-.o a:E 0. o (A 3 m Di, o o CL 0 M 0 0 SD 0 M co m 0 0 I 0 0) M r 0 CL cn MO q o =r t4 m 0 M5 g-. o< 17 m 0 > rr-- !SF 0) -o o U) (D - 0 CD 0 Ci) CD =r 0) m 0 =1 =r =r 0) 0) m m 0 o =r M 0) OQ (D cr 2 1 2 > 0 a 2' -Z i I . I 1; . .. 4 0 - . C) I I e broken down to determine what it will cost to Our monthly accounting of programs by objectives is :hieve each precise objective of a program alternative. based primarily on time studies that are completed daily Each program alternative must compete with other by project staff. In each project, these time studies are -ogram alternatives and with all existing programs for broken down by the project's tangible output. (Slide Le limited resources available. In summation, we are Number 4) Each output can be easily measured and king two actions with program alternatives: (1) we are relates to a precisely stated sub-objective. The per- .termining cost factors and (2) we are providing a centages of time are converted to dollars. Since salaries amework for comparisons by the decision-makers, usually make up more than 75 percent of a project's Where arriving at the costs of alternatives are budget, the unassignable remainder of the budget )ncerned, we have a lot to learn-not only about the expenditures are arbitrarily broken down according to chniques involved but in making the figures under- personnel expenditure percentages. Large equi pment, andable. We need to improve in the presentation of consultation or travel expenditures that can be easily .formation to our decision-makers so that they will be assigned to a given output are assigned separately. )le to use it more readily to make informed decisions. Alternative ways to do a given job are usually in the The monthly report is similar'to this one. (Slide )rm of new, single project proposals that fit within an Number 5) Monthly expenditures for programs, projects ready-existing program package. They may also and project activities are reported by traditional budget -nanate from an existing project as a request to alter or categories. As you can see, projects are grouped together place a certain project activity. The latter source is in program packages when that is appropriate. This way, ;ually staff-generated. Except that they fall at a lower we are able to know more precisely what we are vel of the hierarchy of objectives, they are treated achieving within a program area and what that luch the same as program alternatives. achievement is costing us. Name: SLIDE 4-COMPREHENSIVE STROKE MANAGEMENT PROJECT Position: Time Study by Tangible Output Month: Oate Nursing Nursing Home Stroke Public Physician Total Notes Workshops Education Service Unit Education Education 135 SLIDE 5-MONTHLY AccoUNTING BY PROGRAM,PROJECT AND OBJECTIVE Personnel Consultants Equipment Supplies Travel Publications Other Total Direct Cost * Program A * Project AI * Objective Ala * Objective Alb * * Project A2 * Objective A2a * ,Objective A2b * Objective A2c * Objeptive A2d * * Project A3 * Objective A3a * * Program B * Project BI * Objective Bla * * Program C * Project Cl * Objective Cla * Objective Clb * Objective Cld * * Project C2 * Objective C2a * * TOTAL At the same time costs are recorded, accurate records One important evaluation factor that isn't portraye( of tangible output are maintained, which makes cost here is the quality of the output. Output is evaluated fo analysis an easy task at any time it is needed. Here are quality in much the same manner that all RMPs earn examples of how these outputs are reported. (Slide out evaluation, which includes pre-testing and post Number 6) (Slide Number 7) There is no uniform testing, attitudinal questionnaries and other techniques method of reporting and these outputs are reduced to Like all RMPS, we, are constantly endeavoring to improvl different types of units for costing. We probably need our evaluation methodology. more tiniforn-dty, but due to the constant changes in Of course, it is easy to see that this system isn' many of our programs, any standard form would be comprehensive. Many intangible benefits are unac obsolete before it was off the press. Each of these counted for. In the presentation of our objective infor reports usually involves several telephone calls to clarify niation we attempt to. qualify the information, carefull) information. spelling out those probable benefits which are nol This is an example of the type of cost-analysis report reflected by tangible output. We cannot ignore thal that can be made at any interval and presented to the benefits, whether tangible or intangible, form an im- decision-makers. (Slide Number 8) (Slide Number 9) portant part of the analysis. This particular example includes costs other than those In the third broad area of activity, we are trull being met by the IRMP and therefore required informa- bending-if not breaking-the conceptual rules of PPBS tion not available on a month-to-month basis. because we are accounting for core activities that it 136 most cases cannot be related to an end-service. Here is SLIDE 6-NURSING WORKSHOPS how it works: (Slide Number 10) (July, 1, 1 969 - January 31, 19 70) Daily time studies are made based on a breakdown of Capital Division core activities into the functional activities shown. Each of these activities produces a measureable output that Type of Number of does not relate to any precise objective in many cases. ivorkshop workshops Attendance The time contributed to operational projects as depicted in the first nine columns can be related to project output I............. 15 295 if ............ 15 301 and figured into the costs of operating the projects. III ............ 11 214 The column entitled "other" is for those core staff IV ............ 9 161 functions that can be related to an end-service other TOTAL ...... 50 971 than that of an operational project. For example, we North Central Division have a central medical library network that receives a ............. 16 371 limited amount of attention from core staff members. ............ 14 395 The last four columns,'entitled "Project Planning," III . . . . ....... . 10 199 "Data Collection," "Public Information'. and "Staff IV ............ 9 151 Education," are strictly functions and do not relate to TOTAL ...... 49 1,116 an end-service. However, we have arbitrarily identified NorthwestDivision tangible out'put as a gauge to evaluate our core activity.. For example, we can compare the amount of time I............. 33 810 invested in new project development, -,vhich would fall 11 ............ 26 617 under "Project Planning," with' the number of new III ............ 35 725 proposals submitted to our decision-makers. We can IV ............ 35 643 TOTAL ...... 129 T7 9-5 compare the amount of time we are spending on a given operational project with that project's output. Each workshop is three hours in duration. SLIDE 7 -NURSING EDUCATION CONFERENCES (July 1, 1969 - January 31, 1970) Location Number of Days A ttendance Des Moines .................. 2 110 Mason City ................. 25 TOTAL 135 HOME SERVICE CONSULTATION (July 1, 1969 -January 31, 19 70) July -August, 1969 ................. 139 visits 49 patients September ....................... 42 visits 23 patients October ........................... 28 visist 25 patients November .......................... 23 visits 19 patients December .......................... 41 visits 35 patients January, 1970 ....................... 58 visits 37 patients TOTAL ....................... 331 visits 188 patients Average patient load: 72 STROKE UNIT July 1, 1969 -January 31, 19 70) Total number of patients admitted: 81 Average patient stay in stroke unit: 12.5 days 137 SLIDE 7-(Continued) PUBLIC EDUCATION Conferences Attendance Northwest Division ............. 7 122 Capital Division ............... I 0 347 North Central Division ............. 26 576 TOTAL T4 1,045 Conferences averaged one hour each. SLIDE 8-CoST ANALYSIS FINDINGS Nursit?g IVorkshops 1. The cost to the RMP for each nurse who was a student in the workshop was $1.88 per hour (Student Hours). 2. The combined cost to the RMP and the Heart Association (I 6 cents per student hour) was $2.04. 3. The total cost to the RMP, the Heart Association and Heart volunteers (an added 39 cents per hour) was $2.43. 4. The cost of instruction to the RMP for the workshops was $40.31 per hour (Instructor Hours). S. The cost to the RMP and the Heart Association per instructor-hour was $43.60. 6. The cost to the RMP, the Heart Association and Heart volunteers per instructor-hour was $52.07. Nursing Education Conferences I .The cost to the RMP per student-hour was $3.77 2. The cost to the RMP and the Heart Association per student-hour was $4.08 3. The cost to the RMP, the Heart Association and Heart volunteers was $6.15 4. The cost to the RMP per instructor-hour was $307.70 S. The cost to the RMP and the Heart Association per instructor-hour was $ 3 33.30 6. The cost to the RMP, the Heart Association and Heart volunteers was $502.66 Home Service Consultation 1 .RM-P cost per visit made to a patient was $20.86 2. RMP and Heart Association cost per visit was $22.56 3. RMII, Heart Association and Volunteer cost per visit was $24.1 5 4. RMP cost per patient in the program was $83.44 S. RM? and Heart Association cost per patient was $90.34 6. RM?, Heart Association and Volunteer cost per patient was $96.60 SLIDE 9 Stroke Unit 1. RMP cost per patient admitted to the stroke unit was $197.24 2. RMP and Heart Association cost per patient was $213.07 3. RMY, Heart Association and Volunteer cost per patient was $306.04 4. RNIP cost per patient day in the stroke unit was $15.77 5. RMI? and Heart Association cost per patient day was $16.24 6. RNIP, Heart Association and Volunteer cost per patient day %vas $24.48 Public Education 1. RMP cost for each individual attending conferences was $3.71 per hour (Student hour). 2. RMP and Heart Association cost per student-hour was $4.06 3. RMP, Heart Association and Volunteer cost per student-hour was $4.90 4. RMP cost per instructor was $91.64 5. RMP and Heart Association cost per instructor-hout was $98.79 6. RMP, Heart Association and Volunteer cost per instructor-hour was $119.1 1. 138 There has been no attempt to make a judgment as to whether or not these costs are reasonable. 'Me value of this type of analysis is -atly enhanced when the unit costs can be compared to the sanw unit costs in similar projects. This information should be shared ilar projects in the hope that they, in turn, will th other Regional Medical Programs and Heart Associations that are conducting sim are similar information with us. the sub-regions of the Stroke Management Proje t Nursing The one classification that can be compared internally between c is orkshops, -since all three sub-regions of the project have identical programs. The results of these internal comparisons, based on RMP ,sts only, are depicted below. Northwest Division Capital Division North Central Division ,-r student-Hour $ 1.18 $ 2.70 $ 2.24 ,r Instructor-Hour $25.49 $52.40 $50.97 139 ts z 13 1-3 bo - - - - - - - - - - - - - - - - - - - - 00 tn - - - - - - - - - - - - - ---- ----------------- en - - - - - - - - - - - - - - - tn 140 This information places us in a better position to Our cost studies on projects would be more valuable letermine how and in what areas we should be spending if we had cost studies from other regions with which to )ur time. Except as it relates to operational projects, this compare them. Because not everyone is willing to play reformation is not reported regularly to our decision- under our rules, we sometimes feel -like the only honest liakers. It is presented at regular staff meetings, where guy in a crooked crap game. . ve jointly evaluate how usefully our time is spent and We need a national review and evaluation system that @stablish work priorities. is more consistent in both scheduling and methodology. Following a true PPBS structure, the entire expense For example, we have had four fiscal years assigned to us )f the core activity would be assigned to project output. in three years' time. [n truth, within the. core structure we are not evaluating We need to be permitted to set our own priorities. )ur success on the basis of end-product. We are evalu- Presently, while we are setting our own priorities we ating means, not ends. However, we are supported by must try to second-guess what is currently popular in )nc school of thought which believes that indirect Washington. activities should be allocated to a program only when Finally, in my opinion PPBS is not a set of techniques such an allocation would contribute to a better decision. so much as it is a set of attitudes. Unless one is really In summation, the system permits us to, interested in getting the most for the.tax dollar, it will 1. Undertake better cost-accounting for individual not work. Old concepts such as the "budget is a political projects. tool," "the harboring of privileged information," or the 2. Obtain more efficient use of scarce manpower, "measure of an organization's success by the size of its including staff time. budget" are concepts which are not corn atible with 3. Provide more accurate cost estimates to our p decision-makers. PPBS concepts. What I have described here is only a start on the The purpose of PPBS is to bring together the construction of a system designed to support our budgeting process with the decision-making process, evaluating both processes on the basis of tangible out- decision-making process with objective information. I put. Its intent is to make and keep us mission oriented believe the system has influenced decisions as those since we will be ultimately judged on how well we ac- decisions are concerned with alternative ways to, do a complish our mission. Oven job. In all honesty, I can see very little influence on decisions that relate to program alternatives, possibly FOOTNOTES because we haven't considered that many program alter- 1. Samuel N. Greenhouse, "Ile Planning-Programming-Bud- natives since the IRMP became operational. I think it geting System: Rationale, Language, and Idea-Relationships," may have more influence at the end of the current Public Administration Review, XXVI, No. 4 (December, 1966), three-year funding period when political influences and p. 273. obligations will be greatly lessened. The system is faced with many problems. We need Resource Allocation and th.e.Evaluation Process. in-depth cost-benefit studies which will carry all the way CHARLES L. JOINER down to the consumer and will take into account the alth care. We ECONOMICS, SOCIAL PRODUCTION FUNCTIONS, many econorrdc variables that affect he need to develop better ways to present our information AND RESOURCE ALLOCATION to the volunteer decision-makers. Economics Presently, we have neither the resources nor the Economics is the science of allocating scarce re- expertise to deal with social costs. Comprehensive sources among alternative uses so as to attain the costing should also include estimates of cost that are greatest or maximum fulfillment of society's unlimited related to changes in other human systems as a result of wants, i.e., "doing the best with what we have." decisions we make. Optimum Allocation ofresourees We must continue to search for better and more comprehensive ways to quantify services. It is to be Classical econorrgcs assumes. the "rati,onal. man" remembered, however, that we are primarily a service . concept. Therefore, if the decision maker then wishes'to organization and therefore must be conscious that there combine resources to n-dnimize the,co@t@ of producing a is a point of diminishing returns. given level of output; if heknowsthe. resources (inputs) 141 that can be used in producing the output, and if he also this paper now turns to the question of social produc. knows the prices for increasing each input (and the tion functions in relation to the political decision increase in output that will result from each input process and such problem-solving approaches as PPBS. entry), then the way to achieve minimum costs is as Social Production Functions and the Decision Process follows: the decision maker should use those resources Before one is completely enthralled with the idea of in such a combination that the additional increment in the determination of social production functions and the output per dollar spent on each input is equal. role of benefit-cost analysis in the allocation of scarce The allocation of resources under the assumptions of resources, some reflections on the realistic political classical economics is assumed to be optimized because decision process are necessary. Charles Lindblom' has of the competitive nature of the system itself. Unlike the quite adequately described the real political decision classical model, many social action programs, including process which in some ways appears to be distinctly health, involve the allocation of relatively scarce public different from the problem solving approach of PPB. resources. In addition, there is the need of properly Lindblom states as a first rule of the successful political meshing these public funds with private- resources for process, "don't force a specification of goals or ends." maximum effectiveness for improving or maintaining The reasoning here is that not only is the specification of health. Needless to say, any model constructed for the objectives intellectually difficult, but also pragmatically allocation of resources for better health will have its harmful. In fact, it could mean that agreement among shortcomings, e.g., the allocation of resources for health diverse interests on specific measures may be completely means fewer resources available for non-health purposes. blocked. If one considers the health sector as a system of itself, For example, the Elementary and Secondary uca- optimum resource allocation requires that the additional tion Act of 1965, which is considered a landmark piece benefit rising from the allocation of an additional of legislation in terms of federal aid to education, expenditure (cost) for a particular health problem must needed the support of at least three divergent interest be equal to ratios of benefits to costs for other health aroups. The pa rochial schools saw it as a step in pro- problems. For a theoretical explanation, additional viding financial assistance for parochial school children. benefits and costs may be referred to as marginal A second group saw it as an anti-poverty measure, since benefits and costs. Therefore, the optimum allocation of the distribution of funds for Title I of the bill was based resources toward the solution of various health problems on the number of poor children in each school district. is accomplished when: A third group saw it as a broad beginning of a large program of federal aid to public education. It does seem .MB uite possible that the bill would have been defeated had a @ MBb @ MB... - MBn q mca MCb MC, Mcn any attempt been made to secure strong agreement on long-run objectives. where: MB equals marginal benefits accruing from the iinple- A second major feature of a desirable decision process mentation of a particular technique or approach for solving the health problem within a series of health problems, a, b, c, n. as seen by Lindblom is its incremental characteristic. The process toward objective attainment should proceed MC equals the marginal costs resulting from the implementation of a particular technique or approach for solving the health prob- in very small steps because of our inability to foresee the lem within a series of health problems, a, b, c, n. full social consequencies of any program and the fact that political decision costs tend to increase as the This marginal benefit-cost approach for optimum decisions conflict with values held by interest groups. allocation of resources for the solution of various health The third major element in the Lindblom approach is problems may also be applied to the allocation of referred to as the "advocacy'. process of reaching resources among alternative strategies or approaches for decisions. To the extent that advocates of every related the solution of any given health problem. In fact, this interest have a voice in policy making, the self interest benefit-cbst approach should be an inherent part of any motivation will insure that each advocate takes the normative decision making process. However, the ap- responsibility for researching the consequences of any pfication of such a theoretical approach becomes action for the value he represents. Obviously, this ap- extremely difficult when the decision maker does not proach is not idealistic. Instead, it is pragmatic, stresses know or can not determine precisely the benefits or outputs of a particular technique or approach to the Charles Lindblom, "The Science of Muddling Through," Public solution of a health problem. It is for this reason that Administration Review, Vol. 19, No. 2 (spring 1959), pp. 79-88. 142 process rather than substantive criteria. Therefore, by instances, evaluation must involve in-depth studies using definition, a "good" decision is one which obtains sophisticated statistical techniques-particularly when consensus rather than one which meets the requirement the 'impact of one program is only a part of a much of efficiency or effectiveness. larger program. Feedback of results from operating pro- In order to properly relate political values to grams is an absolute essential to program planning, and analytical program decisions involving the allocation of systematic analysis provides the necessary feedback for resources, the decision process must include some deter- decision making and planning. rnination of the social production functions that translate program specifications (input) into program INTRODUCTION OF CONCEPTUAL AND ACTUAL consequences (output). An analogy may be drawn here PROJECT EVALUATION PROCESS to consumer preference theory. Econon-dc factors of IN RELATION TO PPBS production-land, labor, capital, and management-are It is commonplace to wade through an article on not directly evaluated in terms of consumer preference evaluation and find it is like the last ten you read. The functions, but only through a process which translates mass of articles on evaluation emphasize the necessity these inputs into outputs. It is the output, or final for evaluation and they generally state that a conceptual product, that enters directly into consumer preferences. evaluation model should be designed. These evaluation The process of translating inputs into outputs, of course, articles stop at this point. I plan to go beyond where assumes knowledge of the production functions in- others stop and speak to you on a conceptual model volved. designed and tested at ARMP. If the analogy is applicable, we need to know the In June, ARMP instituted a systematic and in-depth social production functions of health programs. It is at evaluation of all approved projects. This was a first step this point that the task of the social scientist becomes in total program evaluation and an experiment with more difficult because many of the social action pro- PPBS. grams of the federal government do not deal with the The majority of core staff at ARMP were skeptical simple translation of factors of production into com- about PPBS. It was decided that the first two aims of the modities, but the production functions are determined PPB system should be used to evaluate ARMP projects. 2 largely by institutional or behavioral characteristics. These aims are: Determination of social production functions involves l.'the careful evaluation and examination of goals complicated systems in which institutional, technical and objectives in each major area of activity, and and economic factors interact with each other. There- then to fore, we cannot expect the technical expert to define all 2. analyze the output of a given program (project) in of the input-output relationships, i.e., relying totally on terms of its objectives. physicians to evaluate all health programs or engineers to An evaluation model was designed and used with four implement the design of pollution control systems. projects this past summer. The model was found to be It seems imperative that the analysis of production adequate for use on these projects. The experience functions in most public programs must take a acquired by developing an evaluation model along with systematic approach rather than being confined to tech- the actual evaluation process has led to a more knowl- nical considerations. Many times it is extremely difficult edgeable understanding of problems associated with to predict with any real degree of certainty the speci c analytic investigation as well as giving an indication of performance of new or proposed social progrwns. Some problems linked with PPBS. of this uncertainty concerning the relationship between inputs and outputs can be reduced via either ex-post evaluation of operating programs or the implementation Divisions of the Model and Experiences Gained and evaluation of demonstration projects. Although the Project Development process of decision making described by Lindblom of Assumptions: 1. When projects are developed, the incremental changes has been recognized as an effective alternatives, if known, are brought out and discussed. means of proceeding under uncertainty, this does not reduce the need for systematic analysis. In some 2. The project goals and objectives meet program goals and objectives. 2 Cbarles L. Schultze, The Politics and Economics of Public Step 1. Determination of the project goals-This first Spending, (Washington, D.C.: 'Me Brookings Institution), pp. step consists of determining in rather broad and long- 55-76. range terms what is to be achieved by the project. A 143 statement of project goals is. necessarily broad and Actual Evaluation Begins frequently long-range, and, for these reasons, a project's Step S. Collection of performance data-once the goals may not be capable of direct measurement in the desired level of action is decided, the relevant data which sliort-run. One problem encountered in the evaluation process was that seIveral of the projects did not have will permit the determination of the actual level of per. formance must be collected. Collection of evaluation realistic goals. data should be an integral part of the on-going project Step 2. Determination and statement of project implementation. If steps I through 4 are complied with objectives-Project objectives, as used in this evaluation, as described above, then actual evaluation can easily be are narrow and short-range statements of what the accoi-nplished. It is a matter of inserting data into the project is to accomplish. Project objectives are derived proper place. Output studies are important and the type from and must be compatible and consistent with the study (e.g., cost-benefit analysis) should be determined project goals. The difficulty encountered here was that when the project begins so that adequate data are often the project objectives were vague (e.g., increase available. patient care) and had to be rewritten in measurable Step 6. Comparison of actual performance with terms. standards previously set-This is considered the program Comment: These problem areas have been corrected. (project) effectiveness step. Programs may differ in their Realistic goals and measurable objectives are' a part of all effectiveness depending on the extent to which pre- new projects. The evaluation process actually begins established objectives are attained as a result of activity. during this stage of project development. All goals and Based upon a comparison of actual performance with objectives are being challenged by the evaluation co- the standards, the performance will be concluded to ordinator to make sure they are feasible and applicable have been satisfactory or unsatisfactory. After a deter- to total program goals and objectives. mination of satisfactory or unsatisfactory performance has been made, the project administrator has a number Pre-Evaluation Process of alternatives available to him. If the performance is concluded to be satisfactory, the pr 'ect may be Oi Step 3. Determination of measures of objective continued unaltered, or, if the goals and objectives have attainment-these measures would include, for example, been met, the project can be satisfactorily concluded. If such things as: days, hours, dollars, ratings, ratios, per- the performance is determined to be unsatisfactory, the centages, attitude changes, and patient behavior. administrator may modify his project objectives and/or Repeatedly, it was found that project directors of standards (objectives or standards are unrealistic), funded projects did not know what data to keep and attempt to improve efficiency (inefficient use of re- how to record collected data so as to justify the project. sources), or recommend discontinuance. There were several reasons for this, one being poorly Comment: It is felt that a seventh step is required written project objectives. between step 6 and the final recomniendation. This Step 4. Establishment of standards-sta-@idards, as would be a step for feedback between the evaluator(s) used in this evaluation, refer to desired levels of attain- and members of the program (project). Honest com- inent. Only through the use of implicit or explicit state- munications should take place between the evaluator(s) inents of acceptable and/or unacceptable standards can and the project staff so that apparent results can be the administrator decide whether to continue, adjust, or discussed. If discrepancies are discovered during these discontinue a particular project. Standards frequently discussions, further study can be made. The evaluator(s) were not written into the projects. This has led to a poor and project members should agree on the results, percentage of approved projects for ARMP at the whether satisfactory or unsatisfactory. national level (27%). The lack of standards has also made projects difficult to evaluate. Sunitizary a)zd Coyiclusions Comment: The problems in steps 3 and 4 are being Economics is the science of a ocating scarce re- corrected by a pre-evaluation process. Before any project sou rces among alternative uses so as to attain the is written, measures for objectives are agreed upon by all greatest or maximum fulfillment of society's unlimited people concerned. During the pre-evaluation process, wants, i.e., "doing the best with what we have." standards are established. Alternatives to the project are If one considers the health sector as a system of itself, further discussed. optimum resource allocation requires that the additional 144 benefit rising from the allocation of an additional ex- Step 4. Establishment of standards penditure (cost) for a particular health problem must be equal to ratios of benefits to costs for other health prob- Beginning of Actual Evaluation lems. Step 5. Collection of performance data. This marginal benefit-cost approach for optimum Step 6. Comparison of actual performance with allocation of resources for the solution of various health standards previously set. problems may also be applied to the allocation of resources among alternative strategies or approaches for the solution of any given health problem. However, the After a small-scale testing of the first two aims of application of such a theoretical approach becomes PPBS, ARMP reported the following benefits: extremely difficult when the decision maker does not I . Improved project development. know or can not determine precisely the benefits or out- 2. Increased control of funded projects. puts of a particular technique or approach to the 3. A better appreciation and understanding of the solution of a health problem. value of evaluation. In order to properly relate political values to 4. An acceptance by the staff that the total program analytical program decisions involving the allocation of should be evaluated, probably using the PPBS resources, the decision process must include some deter- method rnination of the social production functions that 5. Development of a more sophisticated decision- translate program specifications (input) into program making mechanism. consequences (output). In November, ARMP will continue to experiment Determination of social production functions involves with PPBS and will further evaluate its effectiveness. At complicated systems in which institutional, technical the present time, however, ARMP is working on other and economic factors interact with each other. priorities-some of which were determined by the eval- The second part of this paper speaks to a conceptual uation process described in this paper. model designed and tested at the Alabama Regional I EDITORS NOTE.- Tv,,o Appendices to Dr. Joiner@ Medical Program. The model was foLmd to be adequate paper are not repiin ted in the Proceedings. ney are: after it was used to evaluate four projects during the summer of 1970. 1. Medical Information System via Telephone (M.L S. T) Evaluation Report. Divisions of the model are: 2. Reality Orientation Technique Evaluation Report. Project Development Step 1. Deterryu'nation of the project goals. Both were prepared for the Alabanm Regional Step 2. Deterrfflnation and statement of project Medical Program by Edlwrd M. Smith, Ph.D., Research objectives. Associate, Bureau of Research and Conununity Service, School of Health Services Administration, University of Pre-Evaluation Process Alabama in Binningham and Douglas Patterson, MHA, Step 3. Determinatio of measures of objective at- Evaluation Coordinator, Alabama Regional Medical n ta'mment. Progrant 145 SPECIAL -INTEREST MEETINGS STATISTICAL NIODELS AND OPERATIONS RESEARCH Participants Francis C. Ichniowski-Moderator Vernon E. Weckwerth, Ph.D. Acting Chief, Systems Management Director, Systems Development Project Regional Medical Programs Service University of Minnesota Member, Regional Advisory Group David H. Gustafson, Ph.D. Northlands Regional Medical Program Assistant Professor Industrial Engineering Division University of Wisconsin A "Weighted Aggregate" Approach strategies. A parallel strategy involves simultaneously To R&D Project Selection taking two or more approaches to solving the same DAVID H. GUSTAFSON, GOPINATH K. PAI, problem. In a sequential strategy the best approach is pursued; other possibilities being considered only if the GARYC.KRAMER first approach proves unsuccesful. The authors have in- corporated the incremental cost of, adopting a parallel Introduction strategy, the probability of success of each strategy and There appear to be few formal decision theory proce- the cost of failure in a normative mathematical model which selects which stratep-v to use. dures for optimally allocating funds among potential projects. One reason for this is the lack of effective This paper will (1) describe a general project evalua- methods for assigning a value to each alternate project. tion model, (2) discuss problems with current ap- With a fe@ notable exception @, 3 @ previous project proaches to implementing the model, (3) propose evaluation systems have been either theoretical efforts methodologies to solve these problems, (4) report on the requiring many modifications before being practical or evaluation of some of these methodologies, and (5) methodologies lacking the scientific rigor to assure suggest areas for further research. reliability or validity. Two excellent articles5, 6 have reviewed the research 7he General Model up to 1967 so their efforts Will not be duplicated here. 3 Complex evaluation problems generally possess five Since then, J.R. Miller has suggested some interesting ..but relatively untested procedures for evaluating alter- characteristics. First, there are several criteria which are native projecti usin additive model where the criteria important in evaluating the merits of the projects. are weighted according to importance. Second, the relative importances of these criteria vary from one judge to another. Third, the extents to which L.P. Hellman' has evaluated a value measure for these criteria are satisfied are not always directly selecting proposals for research grant support. The measurable on an interval scale. Fifth, the criteria are model he used is based on the Churchman-A@koff8 ap- ometimes interdependent. proximate measure of value, modified to satisfy the s needs of the National Institutes of Health. The evaluation Recognizing that the overall evaluation is some ag- of each proposal was based on the relative values of the gregate of the valuations of individual criteria, we write objectives of the funding agencies, the relevance of the n n+m proposat's objectives and the probability of success of E Wig (@.) + 2 WjRj the proposal's objectives. Proposals with high overall j=n+l expected values were selected for fundiig; this model appeared to be superior to the previous method of proposal selection. The i subscripts are associated with quantitative variables Abernathy and Rosenbloom9 have discussed the pros and the i subscripts are with qualitative variables. Wi and cons of parallel and sequential pr 'ect selection represents the relative weight of the ith criterion and Oi 147 (X he utility function associated with the Second, the regression approach will not improve corn- itd)crrietperrieosenntsxit represents the extent to which the ith n-dttee decisions, only predict them, because this method variable is present and Ri represents the extent to which is based on decisions that were made by the committee, the ith criterion is satisfied. All criteria as well as rather than decisions that should have been made. Man is projects are assumed to be independent. progressively less accurate in evaluating complex In order to implement such a model we must (1) problems as the number of criteria influencing I-ds select project evaluation criteria, (2) assure inde- decision increases' 1 " 2. Hence, the regression app roach, pendence, (3) establish the relative importance of the as a normative model, breaks down when the number of criteria, (4) develop scales with which to quantify or criteria are large. The decision makers become "cogni- categorize the variable, (5) determine for quantitative tively overloaded" and the decisions made may not be variables the utility function associated with each the ones they would like to make. criterion, and (6) aggregate the evaluations of all judges. We evaluated a third set of criteria weighting methods Such a model has two uses. First, it can be used as an where weights are estimated by the committee members. aid in the proposal evaluation process. Technicians can 'Mere is evidence' 1 l 2, 1 3to indicate that under certain use the model to estimate the relative value of each conditions, men do this quite effectively. Miller3 proposal and report the results to the committee as ad- suggests a hierarchical approach to criteria weighting. ditional information for their decision making process. Second, it could be used as a guide to proposal modifica- Example tion. The model could predict what decisions would be made by the committee. The proposer could then Assume that a list of criteria have been developed improve the proposal where necessary. By knowing Wi, in a hierarchical form (Figure 1). AR criteria in'one 0 (Xi R,, and the cost of increasingXiorRibyone unit, )@e could select the criteria to give the 'greatest column that are connected by lines are related in that increase in value for the least cost. they are components of one larger criterion in'the left, adjacent column. We will refer to each column as a "level". Decision makers are asked to: (1) rank, in Oiteria Weighting order of importance, the related criterion in a I given A criterion's relative importance (weight) should be level, (2) assign a value of 100 to the most important directly proportional to its impact on the decision and values between 0 and 100 to the others so as to making process. Because weights define organizational reflect relative criteria importance. These weights are needs, a set of concisely defined and properly weighted normalized and then successively multiplied by criteria can guide proposers to develop programs to meet weights of related criteria at each higher level. In those needs. Those who lack this guidance may propose Figure 2, vertical lines represent criteria and programs of little interest, become discouraged with the horizontal lines connect related criteria. Suppose the process, and be lost as a resource to the organization. first level criteria were -tanked II, III, and I and From the proposal evaluator's point of view, criteria weights of 100, 60, and 40 were assigned..Weights weights permit him to more accurately and consistently assigned within criteria sets (BI,92B3B4), (CD), model the committee's project evaluation philosophy. (D,,'D2) are shown in Figure 2a. Next, weights were Proposals are frequently too detailed or numerous to be normalized by dividing each weight by the sum of all evaluated by the whole decision making committee so weights within a set. The final weight of each lowest they are normally reviewed by a subset of members and level criterion is the product of the normalized staff. Unless each evaluator knows the relative impor- weights of itself and the connected criterion at each tance of each criterion, their evaluations will lack of the higher levels. Thus, the final weight of criteria DI' is the product of weights assigned to criteria DI", consistency. $ Some project selection techniques assume that all Do , and C:', in Figure 2b. criteria have equal weight in the decision making While this approach reduces the number of criteria process. The success of this approach is directly propor- being considered at once, it replaces one bias (assessment tional to the degree to which this assumption is true. error due to cognitive limitations) with another (ag, Other models estimate weights by using an empirical gregation error due to multiplication of errors occurring technique such as multiple regression.' 0. The committee at each level of the hierarchy). As the number of levels rates hypothetical projects described in terms of !h@ increases, the second type of error becomes important. criteria. Coefficients are estimated, using the method or least squares, so as to best predict committee decisions. We compared this approach with a modification (the There are two problems with this approach. First, it is "ratio method") that appears to reduce both aggregation difficult to obtain enough data (and therefore degrees of and assessment errors: freedom) to yield valid, reliable coefficient estimates. 1. Rank the criteria in order of importance. 148 FIGURE i.-Project Evaluation Criteria Dispia)?ed in a Hierarchical Fashion. Incidence Mortality Duration Impact on Health,4 - Distress --Quality of Whole Life .Different Allocation of Resources Benefit Potential Benefit Income From Using Another Projects Resources Improves Subregional Groups Ancillary Strengthens Other Programs Other --Within Now the Region After Demonstration Regional Cooperation outside Region is Complete Considerations -Geographic Needs meets Standards of Medical Profession PROJECT Use of Personnel VALUE Data Data Collection Method Methodology s Project Design -Steps in Process -Time Required Chance of Per Step Accomplishing Method of Objectives Starting Project Time Before Benefits Reach Population Measure of Effectiveness -Types -Uses of -Technical Proposer -Administrative Capability Personnel -Needs > Capability -Capability > Needs Total Cost - Cost - Percent to Be Assumed 149 FIGURE 2.-Demonstration of HierarchicalMethodfor Diteria Weighting 2a-Criteria pyramid including criteria weights. A!40 B;: 1 00 5 0 B 00 B@50 B,-25 B, =25 D@E100 D, 1 0 0 2b-Criteria weights normalized within subsets. A'1.2 B'::.5 C, lu .667 ".333 B, 1.5 B, 1.25 125 B4='125 D, E75 D' 1.5 2 2c-Criteria weights for lowest level criteria. Criteria Products Weight A (.2) .2000 B (.5) (-5) .2500 B'2 . . . . . . . . . . . . . . . . . . . .(.5) (.25) .1250 B (.5) (.125) .0625 (.5) (.125) .0625 B 4 . . . . . . . . . . . . . . . . . . . . C' ......................... (.3) (.667) .2000 D .......................... (.5) (.333) (.3) .0500 D'2 ......................... (.5) (.333) (.3) .0500 Total 1.0000 2. Compare the most important criteria with every Two factors may have caused the superior per- other related criteria. Estimate how many times formance of the ratio method. First, the hierarchical more important the top ranked criterion is than method may yield higher errors because the errors are each of the other criterion. multiplied rather than added. Second, the ratio method 3 . Repeat steps I and 2 for a new set of criteria uses an odds estimation methodology while the hierar- composed of the most important criteria from chical method uses ratings on a 0 to I 00 scale. Previous each set. research' ' indicates that odds estimation leads to more 4. Multiply the weights assigned to criteria in step 2 accurate estimates of subjective probabilities. Possibly by those assigned, in step 3, to the top ranked the results extend to criteria weighting. criteria from its set. Example Criteria Independeiice Suppose for the criteria in 2a ratio weights are assigned to each set as shown in Figure 3a. The new Two criteria are dependent when (1) the extent to criteria set (A, B2 C, D2) are ranked (3,1,2,4) and as which one criterion is satisfied is influenced by the signed weights of (1: 1.5,1:1,1:1.25,1:3). The weights extent to which another criterion is satisfied and (2) the in Figure 3c are obtained by multiplying the weights utility associated with a given level of satisfaction on one in sets A, B, C, and D by values of I/ 1.5, 1, 1/1.25, criterion is influenced by the degree to which another and 1/3 respectively. The final normalized weights are criterion is satisfied. When the assumption of criteria obtained obtained by dividing each weight in Figure independence, postulated in equation 1, does not hold, 3c by. the sum of all the.weights in Figure 3c. The total project value is no longer equal to the sum of the normalized weights are given in Figure 3d. values associated with the individual criteria. 150 FIGURE 3. Ratio Method of C)Itetia Weightitig. 3a. Ratio values assigned to criteria in Figure 2a. A 1:1 B, = 1:1 C = I DI = 1:1 B-2 = 1:2 D2 = 1:1 B3 = 1:4 B4 = 1:4 3b. Ratio weights assigned to new criteria set. A = 1: 1.5 B, = 1:1 C = 1: 1.25 DI = 1:3 3c. Ratio weights of all criteria. A = 1: 1.5 B, = 1:1 C = 1: 1.25 Di 1:3 B2 = 1:2 D2 1:3 B3 = 1:4 B4 = 1:4 3d. Normalized ratio weights. A 0. 16 B, = 0.24 C = .20 DI 0.08 B2 = 0.12 D2 0.08 B3 = 0.06 B4 = 0.06 An additive model with interaction terms may Criteria interdependence ha s been treated in several compensate for criteria dependence if enough degrees of ways in project evaluation models. Some ap- 3 7,1 5 , 1 6 freedom can be obtained to accurately estimate co- proaches assume that all criteria are in- efficients empirically. However, if coefficients must be dependent. This biases the evaluations in direct propor- subjectively estimated, the multi-dimensionality of the tion to the magnitude of the interdependencies. 1 7 Other interaction term would increase both the number and evaluation models4 eliminate criteria causing difficulty of the estimates. While we have very little dependencies. Fishbum" has suggested a method for information about the performance characteristics of identifying such dependencies but there has apparently judges in weighting multidimensional criteria, we may been no experimental validation of the technique. His draw some insights from research into subjective method, which uses the concept of indifference between probability estimation.' ' Several researchers' ' I 2 have pairs of gambles, is suitable when each criteria has shown that men are conservative probability* estimators discrete levels and when the number of criteria is small. and that this conservatism increased with the number of -Unfortunately, bias reduction may be more than offset data to be simultaneously considered..Future research by the information loss resulting when dependent should determine (1) if the same problem exists in criteria are discarded. This loss can be reduced by (1) utility assessment and criteria weighting and (2) the best discarding criteria only when there is a high degree of methods for obtaining these estimates. Until then, the interdependency and (2) discarding those criteria having criteria independence problem will have to be treated in the smallest influence on project evaluation. , some other way. We propose the following untested procedure for discarding criteria: *Conservative eIstimators overestimate the importance of 1. Select and estimate the relative importance of a set diagnostic data and underestimate the importance of non of criteria using the procedures suggested earlier; a diagnostic data. subset of those criteria will be independent. 151 2. Select pairs of criteria having a major de- C, has the largest number of dependencies (four) so pendencies. This can be accomplished empirically it is the first to be considered (step 5). WI -< W2 + ta f- s if da are available. If no , experts can W,7 + Wg + WI 0 so C, i discarded (step 6). subjectively select those pairs. 1 9 C2 is the next criteria to be considered (step 7). 3. Remove from consideration those criteria that do W2 > W4 + WI o so C4 and C, 0 are discarded. not have at least one major pairwise dependency. Since C2 no longer has pairwise dependencies it These criteria can be considered independent. forms a new subset leaving only the dependency 4. Divide the remaining criteria into subsets having between C,7 and Cg to be rectified. W,7 > Wg so Cg is high intradependence but low interdependence by discarded. The new group of criteria subsets is C2, having experts sort 3x5 cards, each containing the [C3,C6 1, CS,C7,C8- W3 < W6 so C3 is discarded. The name of one criterion, into groups such that final set of independent criteria is C2,Cs,C6,C,7, and a. the extent to which one criterion is satisfied Cs strongly implies or is implied by the extent to Criteria Measurement which another criterion in that group is satisfied.* Measures of the degree to which criteria have been b. the utility function of each criterion in the satisfied must be reliable, valid, and easy to obtain. subset is influenced by the degree to which Some evaluation models' 5,16 use ordinal values as Xi another criterion in the subset is satisfied.' entries in some variation of equation 1. These are S. Select the criterion, C, with the largest number of obtained by ranking projects according to extent to major pairwise dependencies. We will either w@ich they satisfy each criterion. Unfortunately, ordinal discard this criterion or all the criteria with scale values should not be added21 because the resulting which it has major dependencies. project scores will be biased in proportion to the degree 6. If its weight, as determined in step 1, is less than to which the intervals between project ranks are the sum of the weights of all dependent criteria, unequal. IS3 Se discard criteria C. If not, discard all those criteria Other evaluation mode lect only criteria whose having major dependencies with it. values can be added. The important but qualitative 7. Repeat steps 5 and 6 for the criterion having the criteria are replaced by less appropriate but more easily next largest number of dependencies. measurable criteria. In such an exchange, important 8. Repeat step 7 until all dependencies are information may be lost. eliminated. As- an alternative, we suggest that criteria should be measured on an interval scale whenever possible and Example otherwise, ordinal scale values should be transformed onto an interval scale using the method proposed by Suppose we have a set of IO criteria, Cl,. - CIO) 12 'A set of statements (verbal descriptors) Eckenrode. with weights WI,. . ., WI 0 assigned in step 1. Step 2 are assigned values on an interval scale which indicate yielded subsets [C,,C2,C4,C,7,Cg,Cl 0 1 , [C3,C6 ], [Cs j , the degree to which a project possessing that descriptor and [Cs]. Step 2 yielded major pairwise dependencies satisfies the criterion. Sensitivity can be increased by for the first subset as shown below: increasing the number of descriptive phrases as long as c this number does not exceed the evaluator's ability to ci 2 C4 C7 C8 C9 CIO discrin-dnate. Previous research 23, indicates that men C, x x x x x may have difficulty discriminating beyond approximate. C2 x x x ly seven criteria. C4 x x In order to test the effectiveness of these two C7 x x methods, nine of thirteen members of a committee C8 evaluating medical research proposals used the hierar. C9 x x chical and ratio methods to estimate weights for the 40 CIO x x evaluation criteria in Figure 1. They also rank ordered each of the 40 criteria. This rank ordering was a good approximation of their true feelings because their *This method for detecting criteria dependencies was cognitive limitations were not exceeded. They compared evaluated by GusWson. He attempted to predict patient lengt two criteria at a time until the ordering was complete. of stay by a Bayesian model that assumed data were conditional- These rankings were compared, via Spearman Coffela- ly independent. in one case, he acted as if all data were in- tion Coefficient, with those derived by the subjective dependent. In the other, he used procedure 4a to form weighting methods. conditionally independent subsets of data. The second method predicted length of stay better than the first. This would indicate The results indicate (Figure 4) that the ratio" that the proposed approach may be effective for identifying method does predict rankings more effectively than the major dependencies. "hierarcmcal" method. The average Spearman coefficient 152 was 0.676 for the "ratio" method versus 0.309 for the from them to an interval scale. For 13 of these criteria "hierarchical" method. The standard deviations of the the scale went from 0 to 100: for 11 of them, it went coefficients indicates that the ratio method has less from -100 to +100. variation between subjects (0.021) than does the hierar- chical method (.295). This implies that the ratio method The results (Figure 5) indicate that: (1) The 0 to 100 may more consistently model the decision maker's true scale has less overall variability than the -100 to +100 feelings about criteria weights. scale. (2) On the 0 to 100 scale, the end point Inter-rator variability was examined for twenty four descriptors have less variability than the intermediate qualitative criteria in Figure I using a diverse group of descriptors. (3) It would appear that in each case, twelve health related professionals including engineers, subjects perceive the descriptors to be approximately economists, physiciars, planners, and hospital adiwnis- equally spaced in importance. This finding is somewhat trators. Verbal descriptors were established for the 24 discouraging because it indicates that subjects may not qualitative criteria. Each committee member estimated accurately perceive differences between these descriptive the importance of these descriptors by drawing lines phrases. Group discussion between decision makers may FIGURE 4.--Evaluation, via Spearman Correlation Coeffcient of the Degree to Which Criteria Rankings Were Approximated by Methods for Criteria PVei,@hting. l'O .80- 00 00 00 00 100 .60 - r=.676 Ratio 0 Method .40 - CD !7 0 E Multi-Level a) .20- Method a 1 2 3 4 5 6 7 8 9 Subjects -.20 - -.40 J 153 be one way to improve their perception of the values. the criteria being -measured have clearly defmed- end (4) The variation between subjects appears to be quite points, there are several utility estimation techniques large. This wide variability between subjects may be at- that may be used.' 0,20'When the range of values is not tributed to individual differences in utility functions. clearly specified, the end point can be approximated by This may be especially pronounced in a group as diverse asking experts to estimate the value of the, criterion for in background as the one tested. Much more investiga- which they would be very surprised to find a project tion is needed into performance of subjects using exceed. descriptive phrases. However, these initial data indicate that subjects can give more than a simple preference Model Modification ordering to the phrases. If men are conservative estimators of criteria weights, FiGuRE 5.-LRelation between Value of the Descriptors they will not attribute enough importance to diagnostic and Variation between Subj:ects.- criteria* and will attribute too much importance to non- 100 - diagnostic critera. By raising the weight of each criteria to a constant power greater than one 'we are in effect, increasing the value of diagnostic criteria and decreasing the value of non-diagnostic criteria ' Equation 2 represents such a modification of the weighted aggregate model: 50 n n+in E 'E wag(x,) + @aRj (2) n a. Standard If the value of "a" that maxin-dzes model effectiveness 0 Deviation x were constant between decision makers, it would be 10 0 @O 4'0 so practical to estimate its value and thereby improve the model's evaluation capability. The value of "a" that bD opti@zed the performance of equation 2 was calculated in order to investigate this question. Ten members of the proposal evaluation committee rated on a 0 to 100 scale twelve hypothetical projects, -50 - each described by five of the criteria in Figure 1. These ratings were compared to estimates made by the x subject's weighted aggregate model (equation 1) where (1) criteria interdependence was not investigated, (2) utilities were . assessed by method of order 10, and (3) criteria weights were established using the ratio method. -ioo The results are ind'icated in Figure 6. A "comrrdttee model" was then developed by averaging the individually determined weights and utility curves. The resulting evaluations of the twelve hypo- thetical projects were compared, via the Pearson product moment correlation coefficient with the average ratings Utility Assessment Techniques assigned to each project by the ten members. The Before an additive model can be employed, all criteria measures must be transformed to have the same units of value. One such transformation would be to relate *Diagnostic criteria are those having a major influence on the extent of criteria satisfaction to a utility scale .3,2 0 When rating given a project. 154 Pearson product moment correlation betw&en model and We next investigated the variation between com- ratio estimations was 0.9.* mittee members in the optimum values of "a". A significant variation would require separate estimates of We next calculated Spearman Correlation Coefficients "a" for each comn-iittee member. This would be a time indicating association between averaged subject ratings consun-dng task for both the conurdttee and the and committee model evaluations developed using experimenter. Individual evaluation model performance several values of "a" in Equation 2. Results, Figure 7, was measured at several values of "a". The results, indicate that model performance first improves with Figure 8, indicate that there is substantial variation in the additional weight being given to more important criteria optimum value of "a" between individual subjects. and then drops off as values of "a" exceed 1.50. This Conservatism does not appear in all subjects. In fact, data would lead us to believe that subjects are conserva- some subjects appear to be radical in their criteria tive in weighting criteria and that equation 2 is a useful weightings. At the very least, this would indicate.that modification of the weighted aggregate model. values of "a" for equation 2 must be developed for each FIGuim 6.-Correlation Between Experimentally Derived Project Ratings and Ratings Computed via the Weighted Aggregate Model. 1.00- .2 0 .80- E 0 .60- p CL 0 EL .40- .20- -T T 1 2 3 4 5 .6 7 8 9 10 SUBJECT NUMBER 2 comrrdttee member if they are to be used at all. This *These correlation coefficients are useful only as standards finding is not too surprising when viewed with the against which to compare evaluation models with "a" 0 l.O..@e results of similar research on subjective probability results cannot, for instance, be used to imply that the committee estimations. The optimum value of a modifier of model is an effective predictor of committee decisions because comniitte-es do not necessarily operate on a majority rule basis. subjective probability estimates was influenced by the 155 importance of the criterion under consideration. They Fortunately, it does not appear that there i also found substantial variation between subjects. improvement to be obtained by using equation last column of Figure 8 indicates for each su percentage improvement that could be obtained Y Usi FIGURE 7. A Compafison of Committee Model Per- fornzance Using Various Values of "a" equation 2 rather than equation 1. In 9 of the subjects the improvement is 5'Yo or less. This relatively Value of "a" Spearman Correlation meager improvement in performance indicates that the Coefficient additional work required to improve the basic model may be justified only when the projects under consider. o.6 .908 ation will require a large investment. 0.8 .923 0.9 .923 FurtherModelModi .fications 1.0 .922 1.1 .935 Equation I assumes that all potential benefits will be 1.2 .937 achieved and that the time required to achieve each of 1.5 .95@ them is the same. Neither of these assumptions is true. 2.0 .915 Model performance might be improved by considering 2.5 .886 expected benefits modified by a present worth factor as 5.0 .702 in equation 3: n -rti n+,n -rti E=z P(Silyil,---,Yin)@Vig(Xj)e +E P (Si yil Yin) WjRje j=n+l (3) C(t) where Si = success of project in achieving benefit i r = the exponent of the present worth factor re- Yik = degree of satisfaction of the kth factor lating benefit utility to time for achieve- influencing the success of project in ment achieving benefit i C(t) = present worth of project costs' AU other ti = time required before benefit is achieved symbols the same as in equation 1. FIGURE 8. Performance of Individual Evaluation Models at Va?ious Values of "a " Circled Value is Best Spearman Correlation for Each Subject. VALUE OF "a" Deviation Subject from opt. for No. .2 .4 .6 .8 1.0 1.2 1.5 2.0 2.5 3.0 a=]. 0 1 - - .955 .955 .955 .910 .815 2 2 .771 .764 .764 .585 .705 .705 .655 24.5 3 .809 .832 .770 .760 .707 .5 80 .222 12.5 4 - - .895 .910 .910 .910 .910 .924 .937 4 5 - - .820 .820 .856 .856 .856 .8LO .820 .781 0 6 - - .960 .960 .960 .9@7 .948 .926 5 7 - - .895 .895 .895 .895 .916 .938 .9@41 .912 5 8 .944 .930 .930 .930 .950 .965 5 1 0 - - .810 .810 .868 .853 .810 .783 0 1 1 .925 (aD .925 .925 .93 .935 .920 .850 0 156 The basic concept behind equation 3 is not new but 11. Philips, L.D., Hays, W.L., and Edwards, W., "Conservatism much work is still needed to validate its potential and to in Complex Probabilistic Inference," IEEE Transactions on Human Factors in Electronics, HFE-7, 1966. develop methods for estimating its parameters. At the 12. Philips, L.D., Some Components of Probabilistic Inference, same time, there is evidence to indicate that the model is Doctoral Dissertation, The University of Michigan, 1965. 1 1 4 practical.' , " The research reported here has 13. Schum, D.A., "Behavioral Decision Theory and Man- evaluated methods of weighting and measuring benefit Machine Systems," Technical Report, Systems#46-4,Rice criteria. We have suggested but not evaluated methods University, Program in Applied Mathematics and Systems Theory, July, 1968. for establishing independent criteria. The results of 14. Smith, Lee H., "Ranking Procedures and Subjective experiments conducted at the research laboratories of Probability Distributions," Management Science, Vol, 14, Monsanto Company 2 4 tend to support the hypothesis No. 4, December 1967. that R&D planning and control models that are based on 15. Mottley, C.Nt. and Newton, R.D., 'The Selection of subjective probability estimates may reliably be used as Projects for Industrial Research," OR, 7:6 Nov.-Dec. 1959, an aid in project selection and funding. Other behavioral pp. 740-751. (Propose Method: Decision Theory, Rating). 16. Hertz and Cerlson, Phillip G., "Selection, Evaluation, and research9 @l 0 @l 1 indicates that the posterior probability Control of Research and Development Projects," in Burton of project success, P(Sil, - - -, Yin) can be effectively V. Dean (ed.), Operations Research in Research and estimated by combining, subjectively estimated likeli- Development, Wiley, 1963. C, 17. MostcHer, R. and Wallace, E.J., Inference and Disputed hoods through Bayes' Theorem as follows: Auti2orship: The Federalist, Reading: Addison-Wesley, 1964. 18. Fishbum,Peter, "Independence in Utility Theory with P(Silyil,.-.1yin) @ P(Yalsi) ... P(yilsi) P(Si) (4) Whole Product Sets," Operations Research, Vol. 13, No. 1, P(Silyil, - - - Yin', P(YitlSi) P(YinISi) P(Si) 1965. 19. Gustafson, D.H., Unpublished Doctoral Dissertation, The University of Michigan, Ann Arbor, 1967. Bibliography 20. Fishbum, P.C., "Methods of Estimating Additive Utilities," Management Science, Vol. 13, No. 7, March, 1967. 1. Friedman, M. and Savage, L.J., "The Expected Utility 21. Nadler, G., IVork Design: A System Design Strategy, to be Hypothesis and the Measurability of Utility," Journal Of published soon, G.P. Huber, Chap. 7, General Models- Political Economics, Vol. 60 (1962), pp. 463-74. Decision Making, in the above book. 2. Cramer, R.H. and Smith, B.E., "Decision Models for the 22. E@ienrode, R.T., "Weighting Multiple Criteria," Manage- Selection of Research Projects," Engineering Economist, ment Science, Vol. 12, No. 3, November, 1965. Vol. 91 2, 1964. 23. Miller, G.A., "Magical No. 7 + or -2: Some Limits on Our 3. Miller, J.R. 111, "A Systematic Procedure for Assessing the Capacity for Processing Information," in Alexis and Wilson, Worth of Complex Alternatives," 1967, Defense Docu- OrganizationalDecision-making, Prentice-HaU, 1967. 24. Souder, W.E., "The Validity of Subjective Probability of mentation Centre, AD 662001. Success Forecasts by R&D Project Managers", IEEE Trans- 4. Schoner, Bertram, The Selection of R&D Projects, un- actions on Engineering Management, Vol. 16, No. 1, 1969. published doctoral dissertation, Stanford University 1965. 5. Baker, N.R. and Pound, W.H., "R&D Project Selection, 25. Edwards, Ward, "Conservatism in Human Information Where We Stand," IEEE Transactions on Engineering Processing," in Formal Representation of Human Management, EM I 1, pp. 124-134, December 1964. Judgment, Benjamin Kleinmuntz, ed., Wiley, 1968. 6. Cetron, M.J., Martino, J., and Roepcke, L., "The Selection of R&D Program Content-Survey of Quantitative Methods," IEEE Transactions on EngineeringManagement, EM-14, pp. 4-14, March 1967. Comments on an Evaluation Model for the 7. Heilman, L.P., "A Value Measure for Selecting Proposals Regional Medical Program for Research Grant Support," Operations Research Division, The Johns Hopkins Hospital, Baltimore. VERNON E. WECKWERTH, Ph.D. B. Churchman, Ackoff, and Arnoff, Introduction to Opera- tions Research, John Wiley and Sons, Wiley, 1957, Chapter How generic one wishes to make a model depends on 6. 9. Abernathy, W.J. and Rosenbloom, R.S., "Parallel and how far one is displaced from the reality of application. Sequential R&D Strategies: Application of a Simple The creator of a model in the Ivory Tower can easily Model," IEEE Transactions on Engineering Management, assume away the inconsistencies of the world. To the Vol. 15, 1, 1968. 10. Huber, G.P., Sahney, V.K. and Ford, D.L., "A Study of day-to-day doer of what could be called evaluation, Subjective Evaluation Models," Social Systems Resear there is no way to assume away the problems in the ch Institute University of Wisconsin, Madison, Firm and world. Judgment is totally pragmatic. The applied model I Market Workshop Paper; 6817. either represents what is or it is rejected. 157 As a -group, you have been subjected to some high process is dynamic, that outcomes (and benefits) are level forms of abstraction in terms of starting points, what we seek. Change is a means or an observation of preliminary strategies, ends-in-view, and implementation means, not an end. with stated intents of transformation of the system. The generic nature and benefits of the model for This introduction will, by virtue of that type of evaluation proposed.here are one of a system possessing presentation, try to be as abstract and obtuse. six ordered elements: You have been told, and by report most of you have I .Context - That piece of the world under considera- acquiesced at least, to the proposition that the RMPs do tioii as it is found at a given point in time. This is not form a closed, but an open system. That open the "where" for the RMPS. system is a seductive proposition. It is as seductive an 2. Content - The inputs of men, money, and material alternative as many propositions are when the ends-in- in whatever extant form they are possessed, view are mundane or repetitive. If the system is one of a whether or not they are identified, ordered, or static nature - closed, just input, throughout, and out- measured. This is the "who" and the "what" for put - which is routine, reproducible, repetitive, stand- the RiNIPs. ardized like a ball-bearing production system, then it is 3. Process - The way the content is put tocether in even easier to be seduced. some functional, or-anized way, both in terms of I propose, however, that the open-ended system the static, i.e., repetitive closed system meaning embrace is as deceptive in the argument for it as the like a production process, as well as in terms of the argument that any living, on-going process like life itself dynamic system of self-modification and directed is better than a dead-end. Even the old truism sum- chance. This is the "how and when" of the F-LNIPS. marized that belief from antiquity - you only have one These three elements are in fact the indepe?ident life to live - you can't live it over a-ain - you are all variables for any RMP. Each RMP, by its existence, different. Each RMP is unique and dynamic. For our structure, and function, delimits and encompasses at any own mental health, could we believe otherwise? point in time the dependent elements which are: There are two points to be made: 4. Output - This is the product produced from 1. A model is only a model. It can be made suf- content in the process in use within the context of ficiently complex so that it fits within a predetermined the operation. These are the observable, record- degree of closeness to perceived reality so that you able, reproducible, measurable "why's" of the choose to believe it and use it, i.e., you choose to believe RMP's using the classical definition of evaluation, that the model fits your perception of things rather than i.e., cornparing accomplishment with stated concluding that life is a haphazard sequence of chaotic objective. These typically form the evidential basis happenstances. It depends on your view of the meaning of hard fact observation, on which "output only" of change - from what to what in what direction at evaluation is based. what rate. In fact, one could play with the words and 5. Outcome - These are the time-delayed impacts that redefine status quo to be a constant rate of change. What demonstrate whether the outputs were any more then happens to the obligation to transform the system? than just outputs at the points in time. Outcomes It is merely the difference between evolution and (over time) show the time-delayed impacts of out- revolution. Orderly change with a built-in planning put on health states, disease incidence, updated sequence is a necessary part of any dynamic organiza- practice, altered organization, complete and con- tion. I am concerned that what the "change" model tinLIOUS care delivery, equalized access, cost ef- implies is best described as the "rocking chair model" - fectiveness, etc. These Should be the "why's" for giving the health field a sense of movement but no sense the RMP, but these kinds of "why's" are either of direction. Restated, "evaluation of transformation of too soft in the data sense or take so long in the the system" requires an articulate statement of change time sense that they are onlv rarely used. The out- from where we are to where we intend to go by a series put "why's" are accepted as the basis for funding of defined steps. perpetuation and classical evaluation. 2. If the end-in-view is looked at only as input, 6. Benefit - This is the ultimate "why." It is also the throughout, output, rather than in the structure of vaguest and "softest" element in evaluation. It gets input, content in a context, then I propose that it's the at the associated, serendipitous, as well as intended wrong model. I propose that the definer of a closed effects that are evident in an altered context. system has forgotten: the context of uniqueness, that Benefits can be represented in imputed cost 158 benefit terms or in a gestalt sense of total changes The framework of the conceptual model represented or differences in the context at a subsequent po-int here which has as its basis a markovian process is a in time. model which may not be explicit enough for day-to-day Obviously each RMP is unique if one considers a suf- doing in RMP but the sequence - context, content, ficiently large number of items of context. The process, output, outcome and benefit - is, however, 14 wheres" are unique by combinatorial reduction to applicable at all levels - be it to projects, to the local absurdity. advisory or regional advisory groups, to the core staff, to Each RMP is also unique if one considers the specific the board, to separate RMP'S, or to the RMP as a combination of the process (how's) chosen. The how for program. each RMP given merely combinatorial structure makes It should be clear that I believe that evaluation is each obviously different from any other. merely a means of responding to the question of the Given these unique, independent variables in 46 social good" of the RMP. It can be answered relatively combination the outputs will be by definition unique, or absolutely. It is simply a judgment or opinion of the depending on how crude or fine one chooses to make person with the right to decide. This point is made very the output units. clearly in the pa per, A Tool or a Tyranny. The outcomes will also be obviously unique, depend- One last comment before the paper: Evaluation is ing, of course, on what time frame is used. distinct from assessment. Assessment means to produce The benefits must of necessity be unique since the the evidential base by which statements such as more, context was. less, or equal. can be made. Evaluation means to attach Obviously one can chose or not to be seduced by the such words as good or bad to those assessment findings. age-old proposition that each is different from everyone It is necessary to be clear on the value judgment meaning else, i.e., each RMP is an open, not a closed system. The of evaluation versus the quantitative meaning of assess- issue is not that RMP's are open or closed and therefore ment. For example, it is possible that the same level of different but how different. How different must they be assessment data could be judged to be "bad" in one so that being different makes a difference? 'Me burden context-content-process combination and for the same of an evaluation is to categorize, order, measure, and level judged to be "good" for a different context- interpret the differences - either relatively or absolute- content-process combination. Obviously, an evaluation, IY. in my opinion, can be good or bad, better or worse, The evaluation issue at hand is answering the simple whether the assessment data is identical in measured question, "What social good has the RMP produced?", quantity or order. This ends the introduction and leads where in fact the evaluators have the right and the to the delivery of the formal paper which Mr. Ichniowski obligation to defme "good. asked me to discuss with you, weaving into it your Or restated, what are the outcomes (or benefits) upon questions and comments. which RMP is to be judged? On what basis are they to be held accountable? On Evaluation: A Tool or a Tyrannyl If it is to be on the basis of a change or the fancier VERNONE.WECKWERTH,Ph. euphemism, "transformation of the system," there must be a clear statement of what "good" means in terms of changed to what, from what, at what cost/unit of change Evaluation is a ten letter word - in English. Beyond in what time frame - not just a nondirectional rockiniz that statement the only consensus about evaluation is a chair model. lack of consensus. This paper is a series of loosely related topics which attempts to give some limited perspective If it is to be on the basis of process, then the rate of change and the time horizon must be defined. into what evaluation means, how and why it is done and . One would have to conclude that the goals and purposes of RMP were intentionally stated in the vague way they were because there was no desire to be held iThis paper is distributed for general interest. Reproduction accountable or there was no clear raison d'etre for them. @'Apparently, it is now becoming necessary to define in whole or part is permitted if proper credit is given. 'Ibis dis- tribuflon neither expresses nor implies approvAl of its contents "good" in terms of the process of change without saying by the Project, the University of Minnesota or the Granting from what to what at which rate in what time. Agencies. 159 in what ways the vernacular use of the term in manage- value in the value system of the one with the right ment relates to the discipline use in research. It high- decide what is to be measured. lights four points: All of us know the single most common a I .There is no one way to do evaluation. of evaluation is to the evaluation of the qualit) 2. There is no generic logical structure which will care. Quality of care, we know, serves to expl, assure a unique "right method of choice." are high, productivity low or demands too great. It will 3. Evaluation ultimately becomes judgment and will serve here as the example to trace the development of remain so, so long as there is no ultimate criterion how we arrived at where we are in the Art and Science for monotonic ordering of priorities, and: of evaluation. 4. the crucial element in evaluation is simply: who has the right. i.e., the power, the influence, the EVALUATION OF THE QUALITY OF CARE. authority, to decide. Consider the word quality.. It has the same root as qualities. Originally, qualities were selected as the basis INTRODUCTION for the first quality of care studies. The first question A discussion of evaluation will lead to no useful result asked upon beginning a quality of care study is, "what is unless one states at the beginning what evaluation to be included to be measured?" That's where the means; why evaluation is being done; to, by, with, and laundry lists began., Out of that long list, a set was for whom; what is the intended outcome of evaluation; chosen by whosoever had the right to decide. Typically, how does one "evaluate evaluation" and who has the the qualities were chosen because they either had to be right to decide the what, why, where, when, how, and present or were desirable. Thus: who involved in evaluation. Development one: A list of qualities was presented (a Evaluation includes within it consideration of ap- value system value decision) in which merely presence or proaches, methods, techniques, and uses; a process absence of each quality was recorded. versus a goal approach; program veitus individual An array was generated with a laundry list on the left objectives; needs, demands, desires, and their inter- and two columns to check either absent, score it 0, or relationships. It includes objectives versus goals; ac- present, score it 1. tivities versus accomplishments: inputs versus outputs; The measure of quality was therefore simply the outputs versus outcomes, outcomes versus benefits; number of qualities present divided by the total number effectiveness versus efficiency; structure versus qualifi- of qualities. Low quality meant: a proportionately small cation; and so forth. It includes the context, the con- number of qualities present; High quality meant: a tent, and the process; the served and the server; the proportionately large number of qualities present. individual and the group; the quantity and quality; and The first use of evaluation of quality was to make others. It includes when and where, wi'th or without present the qualities that were absent. feedback, and how often. It includes a research versus an As time passed, it became obvious that some qualities administrative meaning. It includes vernacular versus were more important than others. discipline definition. It includes much more than this. Development two: A weight was attached to the qualities reflecting the importance of each quality. Obviously, these weights were attached based on who Dictionary Definition of Evaluation had the right to decide. The array was modified by The dictionary says that to evaluate means "to deter- adding a column of weights. mine or fix a value of" or "to examine and judge". The measure of quality thus became the sum of These two meanings give the first insight into evaluation. weighted presence of qualities. As time passed, these The term, "evaluation" has value as its root. weights became somewhat "standardized" and there de- Using the dictionary definition, one can separate veloped what we now know as the setting of standards papers and practice into those to whom value means: 1) of quality of care. It was a way of sayin2 what qualities a number value, or 2) a value system value. These two had to be present. High value on a quali-, was reP@.,ctej groups can each be divided into those who are process in a large weight. Sometimes qualities wel,-c ;udged and i versus goal oriented. What is commonly n-dssed is that weighted so highly that absence was identical to a veto. any element (variable, quality, attribute) that one selects Development three: Place a sufficiently large weight to be included for number value measurement is the on any one quality so that if it were absent the "quality result of someone's priority in its selection, i.e, it is of of care" would assuredly be "low". 160 Quite soon the simple dichotomy, absent or present, two completely separate endeavors with the practi- ias as unacceptable as was the equal weighting. It was tioners worse o@@f than before, since "evaluation" must [atural to expand the measure of presence from 0 or 1 now mean something detached from day to day practise, inly to 0, 1, 2, 3 ... to as many "units of more-so-ness" and in use most likely plinative in addition. s was useful. These degrees of more-so-ness did not have o be whole units or integers. These "measures" tied WHAT CAN THE PRACTITIONER DO? asily into "standards" since some standards were in fact Every practitioner has taken at least the first steps in level of the degree of presence rather than merely evaluation. Each practitioner must determine how iresence or absence. sophisticated he wants to got and be prepared to defend Development four: Specify a measure of the degree where he stops, if he stops short of research design. The @f presence for qualities. steps are simple: Such a development was conceptually easy to come I .Choose the qualities. )y, but operationally very difficult to achieve. However, 2. Attach weights reflecting priorities. @hat mechanical difficulty didn't deter the doina of t, 3. Specify measures of degrees of presence. Evaluation of quality of care. The procedure merely 4. Combine the created array in some functional )ecame a listing of included qualities; the listing of as- form(s). ;ociated weights, and an associated measure of the S. Generate distribution(s) of those function(s). legree of more-so-ness but combined in some "arith- 6. Set the cut off points to, determine where the netical or number value way". quantitative representation concurs with his Once that "arithmetical way" was determined, one judgment of desired quality. ' nerely proceded to specify the distribution of the values He can call in help at any step; develop any number Lnd define low and high quality on the scale of that of experimental designs and number value functions, but 'neasure. ultimately that evaluation will bofl down to who has the There were, however, in the 40's and 50's many other right to decide and who renders the judgment. 'orces operating; new knowledge of statistics, proba- )flity theory, experimental design, and other measure- ACCEPTED OPERATING DEFINITION nent technology. People were increasingly dissatisfied OF EVALUATION with simple arithmetic ways, including the implicit assumptions of independence among qualities in the list. Dictionary definitions help to give insight into the Those faced with evaluation were soon developing "whats" of concepts. Operational definitions help to Sophisticated research designs with fancy mathematical give insight into "how's" of concepts. -nodels, formulae, and techniques. The limit functions, The most commonly accepted operational definition Interdependence of qualities handled by multivariate -of evaluation, the "how", is: Compare accomplishment correlations, covariance, factorial designs with inter- with stated objectives. This is itself a goal oriented ictions, simple and main effects plus factor analysis all definition. The objectives are analogous to the qualities Decame involved. In fact, these developments became the fife blood of the biostatisticians and the death or elements chosen in the quality of care example. potions of most of those involved as delivering practi- Since the operational definition is so simple - why is tioners - both clinical and administrative. evaluation so tough? Let's look first at that operational Development five: Only qualities with experimentally definition. In it five assumptions are made: 1) objectives determined measurability, validity, and reliability were are stated; 2) in measurable terms; 3) accomplishments permitted to enter quality of care evaluations. are documentable; 4) in the same measurable terms as As a result, the evaluation of quality of care the objectives; and 5) one knows what compare means, developed to such a mathematically sophisticated extent i.e., what is to be done? that those who first desired it and created it were WHAT USUALLY LEADS To DIFFICULTY? bypassed and found that it couldn't be applied on a day to day basis. Hence, evaluation became so detached that First: Objectives aren't stated. Goals versus objectives now it is not recognized as a part of the ongoing process are rarely differentiated. Purposes, goals, salutes to of clinical management, or program administration, i.e., mother and country - and lots of other things are usually planning, orgaiiizing, assembling resources, directing, stated - but not objectives. An analogy may be helpful to controlling, replanning, reorganizing, etc. It is seen as distinguish objectives from goals. Consider the sequence, 161 1/4, 1/9, 1/16, ... I/n2, .... In this case, that sequence of SOME COMMENTS ON MEASURES terms will approach a limit. That limit is analogous to a Frequently, a quality selected in evaluation has no goal. The individual terms in the sequence are like direct measure or has one which is too costly or tedious objectives. to obtain. There frequently, however, are associated or Second: Even if objectives are stated, most of them indirect measures which can be used in lieu of a direct are not independent. In fact, they frequently are in measure. conflict with each other and rarely would their sum- Some measures which are indirect are called proxy mation add up to the program goals. Additionally, the measures. This obviously means that they stand in lieu state of the art (or science) of evaluation has not of what is desired to be measured. Frequently. proxy developed means of measuring most value system measures in evaluation are used to predict or monitor objectives. Thus, our measurement ineptness reflects activities, and are useful because of their high associated both our ignorance'and our errors. though not causal relationship. Third: Even given stated objectives and appropriate For example, the number of individuals usinc, an measures, we likely can't enable the documentation of emergency room in hospitals is associated withcthe accomplishment. Frequently, the measures are too phases of the moon. For administrative purposes of complex or the day to day documentation is either too staffing and the provision of service, it is not necessary tedious, or not visibly relevant to thejob being done on to know the direct or causal elements. However, if one an ongoing basis. As a result, we substitute approximate were to change the pattern of service "demand" it would measures or frequently just get lost in the data acquisi- be necessary to know cause - and the relationship, and tion problems and consume so much time and resources not operate purely with proxy measures. Commonly, that we judge that documentation isn't worth it - unless "Comparison" in evaluation highlights differentials in it is an experiment in which service is only a necessary such proxy measures. Actions are then frequently taken evil or a necessary context. on forces putatively "causal" but to the dismay of the Fourth: In the rare event that evaluation has action taker, produce no change because - in medical measurable objectives and documented accomplishment, jargon - he treated the symptom and not the disease. commonly nobody knows what to do with it! Or if, in - These experiences further alienate the practitioner fact, someone knows, the comparison will still depend and result in Ws questioning even more, "NNhy eval- entirely on the judgment of whoever has the right to uate?" decide what to do with it. A facetious and trivial example may help: suppose USES OF EVALUATION that an MCH Program has an objective that 75% of all mothers-to-be are to be seen by an O.B. physician before No attempt is made to provide a laundry list of uses. the third trimester. We find that 73% do in project A, An attempt is made, however, to fit "evaluation", in the and 77% do in project B. Now what? If n is big enough, non-experimental design meaning, into day to day the difference may be statistically significant. So what? operations. Is the project with 77% awarded a gold star or more First, we must answer, "For what purpose is the money? Does the project with 73% get a budget cut? In evaluation done?" Regrettably, the answer that would fact, is it not true that since both missed the objective, now be given (if honestly ascertainable) is, "The that both are bad? Why is doing more an ultimate good? law requires it." That is regrettable. In a After all, the 77%er allocated more resources than sense, the requirements in the law reflect a failure on should have been to that objective and that project behalf of those responsible for programs to document, could be "penalized" for n-tisahocation while the 73t7oer accomplishment in an orderly, measurable, and ar- should be given more resources because it was under- ticulate manner that met the desiderata of those with allocated. right to make laws. The overriding question being asked is, is the classical With the legal emphasis on evaluation and the mean- operating definition of evaluation: Compare accomplish- ing of the term to be the rigid mathematical, numer- ment with stated objective the end of e valuation? Is ological, hard fact one, the day to day intuitive or soft evaluation to be only descriptive? Is it merely to tell data meaning and use, has been both lost and rendered how it was? If not, is it to include ground rules for unacceptable. translating description into prescription, i.e., admin- Evaluation has always been - in the dictionary mean- istrative action? ing of the word - present in anyone who was responsible 162 in his work, and had a personal accountability for his you stated it would. You have evaluated at the program acts. Evaluation is inherent in the process of adniinistra- level. tion - be it clinical or program management. Although there i.s a reasonable basis for saying there is Anyone who manaaes successfully either a program a single generic process in doing evaluation, the qualities or a patient goes through some orderly staces, beginning chosen for patient management are so different from with planning: that is deciding what is to be done; by, those chosen for program management that the with, for, and to whom, with what materiel, at what singleness of the process is lost. In fact, because,the time, in what sequence, at what places, for what priorities assigned to the qualities in patient versus pro- intended outcome. gram evaluation are so discrepant, conflict has resulted Thus the planning is the what step in the admin- in the whole health care delivery system. istrative or management process with the how steps being organizing the what, asseniblinc, the resources, THE QUALITIES C, HOW TO CUIOOSE directing the delivery and controlling (or supervising, or Since all of us come from rigorous scientific fields, we monitoring) the operation (or performance). almost without thought believe we choose qualities An inherent part of the management process is its based on the facts. What one means by "based on the evaluation. That examination and judgment in delivery facts" necessitates some expansion. of care is used as feedback to alter the process or treat- For this paper, consider four groupings of facts: ment of choice in order to replan, reorganize, reas- First: Theoretical facts. Starting withgivens and a set semble, redirect, control, etc., ad infinitum. of known theoretical relationships, one by deductive Clearly, evaluation has been, is, and always will be, a logic can arrive at some qualities which are to be part of such a management process - be it for a program included in evaluation. 'or a patient. Second: Dogmatic facts. Dr. Lebon (that spells nobel The similarity in the process can be seen if we move backwards - and he has one of those prizes and don't from the individual care of a patient, through a cohort you forget it) says this is a fact - and it is. In general, of patients to a program. Consider yourself first as a these are the qualities which those in positions of power, physician beginning with a work-up. You first chose influence, or authority include in evaluation. input facts, i.e., qualities, such as lab tests, signs, asked Third: Pragmatic facts. Those which are based on symptoms, soundings, touchings, etc., plus using the astute observations, with data acquired from day to day history to assess the patient, derived mentally a set of practise which every intelligent practitioner gathers. weights of what's important, arrived at what's relevant (by degrees of presence plus weighted priority), deter- These form the basis for selecting another set of mined a most probable "value" or judgment (or evalua- qualities. In general, they derive from "experience and tion) and rendered a care plan. You subsequently demonstrated use ..." compared this to what happened to the patient and, Fourth: Experimental Research Facts: These are the depending on the outcome, either altered the care plan facts derived from research studies which meet the most or reinforced your confidence in your own medical rigid of experimental design requirements. The resulting judgment, i.e., you evaluated on a one case basis. qualities are chosen by approaches and methods such as Consider next a cohort of patients. You look at them factorial designs, controlled probability selection, or any as a group. You select another set of qualities (some of of the research statistical methods that strikes fear into which are different from the case specific qualities most day to day practitioners. chosen in the one patient sense) and look at the cohort From these four fact bases, one can get the qualities from a view of those qualities being a set of intertwining to be used in any evaluation schema. It is here also that degrees of presence and priority. You mentally and standards with which we are so obsessed in health care numerically measure and then compare the results of the delivery are included. cohort to what is "good medical practise". At the program director level, you'd look at more W14Y MUST ONE USE EVALUATION? than only physician case management for either individ- If one is the perfect clinical practitioner or the uals or for his cohort and include the other health care perfect program director, his intuitive ongoing soft data functional services, living conditions, or what have you, system would be "evaluating" without need for a hard that are qualities of the "program" and go through the fact base. But, since perfection is not a human reality, same process to deternii.ne whether it accomplished what one must set up a hard fact data system to document 163 accomplishment. The less prestitious one is, the more Consider the old fashioned grade school report card.., subject one is to the "tyranny of hard fact evaluation". The "qualities" are analogous to items like courses Since one cannot get continuous evaluation, some math, others like art, others like deportment. choices of time intervals must be made - hourly, daily, By analogy, three groupings of qualities of repo rt weekly, monthly, quarterly, yearly, or what have you card items are apparent: for ongoing programs. Evaluation of single shot pro- I .Those that have an inherent measurement ab- grams are relatively easy if only a "final" evaluation is to soluteness in them (even though the measure may be made. One must determine if feedback is to be used - be arbitrarily defined) like feet, inches, etc. The of what kind, and how often. If so, how does feedback units have a meaningful metric on the scale. The fit into a subsequent round of. evaluation? Is it now mathematical formulas work beautifully. another quality or element? If one does feedback 2. Those qualities that have an inherent relative or "evaluation" with the intent to alter the program, how inore-so-ness meaning to them but lack absolute. does one now evaluate the effect of evaluation? ness such as strongly agree, agree, indifferent, disagree, strongly disagree. Again, the mathematics is reasonably easy to apply. APPROACHES To EVALUATION 3. Those qualities that are named or categorizable No attempt to be either scholarly or complete is only. These are those qualities that either have no intended here. Only three commonly used approaches inherent measure of absoluteness or relativeness or are included: that as yet aren't understood well enough to be First: Very commonly, programs are subjected to measured. It is with these, where real difficulties in periodic review. These "evaluations" are made by a the mathematics are found because the weighting squadron of outsiders. Let us call this the J.D.A. - the is not inherent nor is there a logical way to attach judgment day approach. priority values. The big brother squadron, usually called a site visit Since every program or practise includes all three team, comes in on judgment day. The concern is kinds of qualities, we must, in our wisdom choose, obvious, "Will one be judged for sins or virtues using the weigh, scale, combine, and then compare to the same qualities that one has used to live by?" objectives, i.e.,judge the result. Second: Another commonly used approach is one of We render an evaluation. So what? We have gone being reviewed by a hand picked panel called peers. Let through a magnificently structured and logically jus. us call this the B.R.A. - the bunny rabbit approach. It's tifiable process with bewildering numerological finesse title comes from the setting in which Johnny brought a to arrive at the end point - a judgment or opinion of rabbit to kindergarten for drag and brag (Show and what to do with it. Tell). Mary asked if it were a boy rabbit or a girl rabbit. WHAT DIFFERENCE DOES IT MAKE? Johnny said, "I don't know". The precocious Mary said, "Since this is a participatory democracy, let's vote." It makes a difference only if the person in the Although both these approaches have been practised position with the right to decide agrees. successfully (at least in the evaluation of those with the This formalized ritualistic numerological game called right to decide), the invalidity is obvious: for the first, evaluation, is a series of decisions of those with the right one only needs to have the right to choose the qualities to decide and ultimately rests on the judgment of the and the measures and the weights and the cut off points; person who can determine the outcome by: and for the second, one merely needs the right to choose 1. Choosing the doers of evaluation. the majority of the panel. 2. Choosing the elements for inclusion, and/or Certainly no one could object to those simple 3. having the right to decide what comparison means. requests if the "right to decide" is not the crucial issue Such evaluative manipulation can occur whenever in evaluation, and if evaluation does not ultimately there is no ultimate criterion which assures a unique become judgment, i.e., the opinion of the person with ordering of priorities, and the resulting correct method the right to decide. of choice. Third: The third commonly used approach is the R.C.A. - the report card approach. It is essentially the WHAT DOES IT MEAN To HEALTH PROGRAMS? approach used for evaluation of the quality of care 1. If one doesn't play the game, or even worse realize example at the beginning of the paper. what the ground rules are, one may lose the, 164 funding or have funding reduced since it's easy to By analogy, we are merely counting how many times relate dollars to points scored on a hard fact the bird flaps his wings, without asking, did the bird fly - evaluation index. let alone how far and how high. 2. It's an effective way for funders to mold or shape Clearly, outcomes as the measures of effectiveness the program, i.e., dictate the hea Ith care delivery must be the starting point for evaluation before any of system. They need only specify the proxy , the measures of input or output analysis of the ef- indicators or elements, their weights and their ficiency kind have any meaning or usefulness. measures, and attach adequate punishment and rewards so that grantees desiring continued TWO AIDS To ASSIST IN EVALUATING support will allocate the resources to maximize the HEALTH DELIVERY evaluation index. It's +,he health care version of In the face of such a bewildering maze of considera- "shape up or ship out". tions, two simple lists of elements are helpful in retain- 3. Quite clearly, those elements that are easily ing ones sanity: The first are the five A's. measurable will get the attention and be assured of In the evaluation of any health care delivery, inclusion in such an evaluation. I am personally questions of appropriateness, availability, accessibility, concerned that what is really important in life is and acceptability to both seeker and server must be inversely related to what is easily measurable. answered. These are a dependent sequence. For services OTHER CONSIDERATIONS can be deemed appropriate yet be unavailable. Or they can be defined to be available, yet not accessible. Or Currently in vogue also is efficiency of health care they can be defined to be appropriate, and available and delivery. One of the chosen qualities is efficiency in accessible, but still not acceptable to either or both the every evaluation. The usual operational definition is one serving staff or the seeking cli-ent. However, overriding lifted bodily from engineering - the ratio of output to these four A's is the one called accountability. It is the input. essence of the moral contractual agreement made The hazard of this measure is clearly from whose between the seeker when he seeks and server when perspective output is viewed. From the doer, Ms serves. activities are always viewed as output. From the receiver, The second list is the generic structure of evaluation those outputs are always viewed as inputs. implicit in this paper and necessarily a part of the The classical data which allegedly measures output of process of evaluation. There riiust be six interdependent laboratories, groups of personnel, institutions, etc. such elements to any evaluation undertaking: as visits, lab tests, encounters, and the rest are really First: Context (what, where, when, and who). only inputs to the health of the seeker of service. Even Second: Content (program elements being or more interestingly, within the sequence of doers, the intended to be provided and why). prior doers, output is also viewed as an input to the next Third: Process (how care is organized and delivered). one in sequence. Thus, the lab technician believes he is Fourth: Output (how many times did the bird flap its highly productive because of outputting many lab tests, wings). and the engineering definition gives him a very high Fifth: Outcome (did the bird fly). efficiency rating. The physician or nurse, however, lool-,s Sixth: Benefit (how high and far, with what re- at the lab tests merely as inputs and they in turn, value sources). their visits and activities as the real outputs upon which Clearly, context, content, and process combine in efficiency should be based. many ways to produce the output, the outcome, and the What is incredible is that none of these measures of benefits. efficiency really get at the question to be answered - namely, are any of these inputs or outputs effective in SUMMARY maintaining or altering the health status of the recipient of service. This paper was intended to give some limited perspec- Clearly, effectiveness must first be defined before tive into the what, why, and how of evaluation. It high- efficiency has any useful meaning. It appears that we are lighted the reasons for misunderstanding between the producing a health delivery system which is unit by unit hard fact approaches to evaluation and the day to day approaching 100% efficiency while simultaneously uses. It is not easy to describe a program even in terms 'narching toward the other extreme in effectiveness. of telling how it was. For ongoing programs it is even 165 more difficult to tell how it is. It is virtually impossible between the SCIENCE (retrospective description) ail( to tell how it will be, or as some glibly say, how it the ART (prescriptive action) of adn-dnistration.,Ho should be. fully, program evaluation will continue to serve an( It is necessary for each of you as accountable and develop as the management tool it first was, and stii@i responsible health devotees to DESCRIBE how it was, intended to be and will not become the program tyranfii but it is more important to structure ways to of the 1970's. PRESCRIBE how it will be. This may be the difference 166 EVALUATION OF CORONARY CARE TRAINING Participants .Zodger M. Shepherd, M.D. - Moderator MarieHa Larter, M.S. Assistant Program Director, Continuing Education Coordinator, Washington/Alaska Coronary ,Presbyterian Medical Center Care Unit Nurse Training Program Carol Larson Allied Health Specialist Daniele Deverin Continuing Education and Training Branch Cybern Education, Inc. Regional Medical Programs Service Evaluafion of Coronary Care Training: technique has a disruptive effect and reduces the Some Direct Observations of tendency of other physicians to implement the Performance in Hospital Practice advocated procedure at all. RODGER SHEPHERD, M. D. 2. Availability of Equipment: Standardized tech- nique depends on standard materials. Instructors The objective of our Intensive Care Training Program have observed the lack of certain critical materials is to enable physicians in cadres from small general hos- or instruments during introduction of a new tech- pitals to perform certain intensive care skills in their own nique. The attendant frustration during this hospital settings. These skills include: use of central critical phase may abort or seriously retard the venous catheter, use of intra-arterial monitoring adoption of the new practice in spite of adequate catheter, interpretation of blood gas data, continuous ly trained personnel. EKG monitoring, airway care, controlled ventilation, 3. Supporting Services: Interpretation of blood gas cardloversion, and others. The staff of our ICU had data depends on complete confidence in the data. visited small hospitals and identified these skills as We have encountered one hospital setting where feasible but underused in smaller hospital ICU'S. the student's training in interpretation of blood The training program is conducted in three phases. gas data was not implemented until we had During the first phase, the cadre and project clarify rectified certain analytical problems in the clinical mutual objectives. During a second phase, each physician laboratory. from the cadre undergoes a week4ong program of one- to-one instruction at a metropolitan medical center. During a third phase, an instructor-in-residence is main- Report on Xerox Study of Eleven tained in the cadre's own intensive care unit around the National Coronary Care Training Centers clock for 10-12 days. DANIELE DEVERIN The direct observations of these instructors have provided valuable anecdotal data on both the project and the resulting student performance: In 196.7, Xerox Education Division was contracted by 1. Standardization of Technique: The same single Public Health Service to conduct a 2!year evaluation standard technique for insertion of central venous study of eleven national coronary care training centers. catheter is advocated during each individualized OBJECTIVES instruction. The mastery of this technique is certified by the instructor. However, the student The study was designed to fulfill the following may not implement this technique in his own objectives: hospital setting. It has been observed that the 1. To determine the effectiveness of the training pro- failure of some physicians to support standardized grams in imparting the knowledge, attitudes, and 167 skills needed for a nurse to perform in a CCU at an ments constitute the basic evaluative data bank of th( acceptable level. study. 2. To determine the effectiveness of the training pro- The follow-up portion of the study was conducted ir grams in developing a high quality of performance two ways: mailed questionnaires were sent out to al in the training graduates. graduates of the programs, except as noted below. Jr. 3. To determine the most effective training program addition, other questionnaires, including the per, for achieving these aim@. formance checklist, were sent to their hospital super. 4. To determine the distinguishing qualities and visors. A systematic mail and telephone procedure characteristics of a successful CCU nurse. assured a return rate of at least 90%. In order to monitor 5. To determine the most effective and reliable the reliability of the mailed returns, and to assess the methods for the selection of the "best" training effect of non-respondent. bias, a 10% random sample of applicants. the graduates was selected for personal, on-site follow-up visits. The results of these visits confirmed the high degree of reliability in the mailed returns. MET HODOLOGY Data-Collection. The survey period extended from A systematic model was designed to analyze the three August 1968 to September 1969. In the eleven centers interrelated primary spheres of concern: under study, a total of 57 sessions were monitored, for a 1. Input variables: Trainees' demographic data, total of 862 trainees. The 456 sponsor hospitals were all education, personality, expectations and attitudes included in the survey. towards CCU nursing, etc. Data-Processing. Standardized procedures were estab. 2. Process va?iables: Training Centers' facilities, ap- lished for handling and coding of raw data. Data- proach, curriculum, etc. processing was completed at the end of October 1969. A 3. Output variables:. Knowledge gained, post-training correlation matrix-was designed and run on 85 variables. expectations and attitudes, clinical performance both in-training and on-the-job, etc. FINDINGS In addition, Environmental variables were studied. Trainees. The "typical" trainee was female (989o), the They consist of the sponsor hospitals' facilities, ap- mean age for the group was 34 years, and the median proach to nursing, etc. that influence both input and was 28 years. About half of the trainees were, or had output. been married; of these, 60% had families, half of Nvhich The project staff then prepared, piloted, and revised consisted of 2 or more children. 83% had obtained a nine data-gathering instruments. A standard personality hospital diploma, 5% had an associate degree, 18% had a test, the 16 PF, was also selected. This process involved baccalaureate degree and 2%, a masters. Previous discussions with PHS contract officers and with various coronary care experience was as follows: 17% had consultants, visits to CCU'S, a review of pertinent litera- worked in a CCU for an average of 8 months, and 36% ture and existing research information, and an analysis had worked in ICU/CCU's for an average of 14 months. of the content to be covered. Most values of the 16 f.F. were close to the normal In general, data were collected on the trainees before mean, except on the general intelligence scale when their and after training, and at follow-up, between three and mean was substantially higher than the mean of any oc- four and a half mon'ths after training. Data were also cupational sub-group reported. collected on the training programs, and on the sponsor Sponsor Hospitals. Of the 456 hospitals surveyed, hospitals to which the trainees were returning after 55% had sent one nurse to training, 27% had sent 2 completion of the program. In terms of the specific nurses, and the remainder 3 or more nurses. Hospital size problems addressed in this survey, two instruments are varied considerably: 16% had less than 100 beds, 26% of special interest. between 100 and 199, 21% between 200 and 299, and The knowledge test was especially designed and the remaining 37% had 300 or more beds. Results standardized. The test contained 12 weight-ed sub-tests, obtained before training showed that 27% of the hos- with each sub-test containing a number of weighted pitals had a separate CCU and 41% a combined items. It was used both before and after training. The CCU/ICU. These figures increased slightly at follow-up. performance checklist was designed to tap the degree to The most surprising finding of the survey was the which the training graduates performed specific CCU number of training graduates still not working in coro- nursing functions at follow-up. Together, these instru- nary care at follow-up. With a 90% response (N=779) 168 only 5 1 0 nurses were found to work in coronary care, or would seei-n, then, that the sponsor hospitals view the 65% of the follow-up population. The reason given in major function of traiiiin- as developing tec nica 65% of the cases was absence of CCU. competence, while general nursing qualities are viewed as Trainees' Preparedizess upon CCU entry. Evaluation bihereiit in the potential trainee. This will be discussed of objective one was done on the basis of knowledge later tinder objective four. tests, trainees' attitudes, and supervisors' general ratings. A fair comparison between nurses trained at different Out of a possible 220 points, the 862 nurses averaged centers require that some allowance be made for skills a pre-score of 127.4 points, or 57.9,7o, wlgch increased to the trainees brought with them on entering the program. 74% on the post-test. The variability of test scores at pre In all cases, it was found that the centers rated highest in and post for the group indicates that instruction had nursing performance had trained the most experienced brought about a leveling of test performance. population, while the lowest ratings were obtained by Trainees' expectations of CCU activities changed those centers having trained the least experienced group. markedly between pre and post-training, with a general Model program. The study failed in providing an shift towards a "middle-of-the-road" attitude, indicating analysis of the model program, objective three of the a tendency, over the training cycle, for attitudes to study. Both dependent and independent variables dis- become more realistic, which is felt to be a positive and played inadequate variance characteristics. Further, the desirable training output. 63% of the trainees found the training centers were quite similar, at least on the program excellent and 35% rated it as "good". When variables tapped by the instruments. This result was, of asked, after training, whether they would select CCU course, disappointing, but it should be noted that the training again, 63% answered "definitely yes", and 25% basic reason for this failure is the success of the pro- "probably". grams in fulfilling the overall objectives. The following curriculum areas were mentioned at Optimal characteristics of a CCU nurse. The fourth post-training as needing more preparation: Fluid and objective was examined from the standpoint of high per- Electrolyte Balance (53%), Interpretation of ECGs formance, satisfaction with CCU nursing, and motivation (49,7o), Recognition and Treatment of Arrhythmias to continue work in coronary care. (35,7o). While these figures may have represented the trainees' anxiety at assuming new responsibilities, similar Since performance ratings were typically either good results were obtained among nurses working in CCU's at or excellent, a detailed study was made of those per- follow-up. Fluid and Electrolyte Balance was still on the formance items rated as "deficient" by the supervisors, top of the list, non-coronary complications were men- yielding a picture of what a successful CCU nurse should tioned by 38% of the working graduates, basic elec- not be, and inversely what characteristics she should tronics and interpretation of ECG by 37%. possess. The largest number of deficient ratings were Suggestions by supervisors regarding possible im- found in the broad area of "Communication and Inter- provements in the programs agree in general with those action with Staff"; next in line was 'Terformance of of the trainees, the main one being for more stress on day-to-day assignments" with stress on general nursing the technical aspects of CCU nursing. Rating the nurses' competence, and skill in handling and verifying the tech- preparation on a seven-point scale, 77.5% of the super- nical equipment; finally "Communication with the visors selected the two top categories. patient and his relatives". Thus, a successful CCU nurse Trainees'Perfonnance in CCUS. Performance check- would appear to need excellent nursing skills, an ability to relate well with the members of the CCU team, with fists were received by each hospital, one for each trainee. A total of 487 checklists were completed and ret . urned the patient and his relatives, as well as technical (56% of those mailed). By far the main reason given for competence. non-completion of the form was "CCU not open". 73% of the training graduates working in CCU's Overall mean performance was rated from "good" to stated that they definitely wanted to pursue coronary "excellent"; however, while the mean ratings do not care as a specialty. A number of problems were ex- deviate siiznificantly from one another, there was a pressed, however, the great majority stressing staffing general tendency for nurses to be rated higher for tech- difficulties, and lack of support and communication nical, CCU-specific activities than for non-technical, within the hospital in general, and the Unit in particular. jeneral nursing activities. There is an apparent contr.a- A smaller number of nurses also expressed frustration at between these high "technical" ratings, and the occasional "dullness" of Unit work. Successful ions for more program depth in the same areas. It trainees derived great satisfaction from bedside nursing, 169 and from the challenge and diversity offered by relating to a set of regional objectives A pool of ol coronary care work. 800 test items were edited for content and format; th( Since the follow-up period extended from 3 to 4 1/2 revised items. were then rated by their authors as bel months, long-range tenure could not be ascertained. either "essential", "desirable", or "supplementar When asked about their plans for a two-year period, 61% knowledge for a CCU nurse. Only those items rated of the nurses stated that they wanted to continue "essential" or "Mghly desirable" were retained, leavini coronary care nursing. pool of 2SO questions weighted in the following fasMo Selection criteria. The main criteria used by the sponsor hospitals when selecting potential trainees were: CCU Concepts 1) motivation and interest in CCU nursing; stability in Anatomy and Physiology 3) Summarized as CCU Concep present position, and demonstrated excellence in general The Classic MI 8) Diagnostic Tests 5) nursing skills. Rehabilitation 8) The results of the study point to the necessity of providing the potential trainee with a clear perception of Complications of an MI 13) what her role will be, prior to selection. They also (excluding arrhythmias) Summarized as Complicatio ri suggest a need for closer communication between train- Electrocardiography 8) ing centers and sponsor hospitals, before, during, and Equipment and Safety 3) Summarized as Arrhythmias after training. Arrhythmias 19) Chemical Therapy 13) Swnmarized as Chemical Evaluation of CCU Nurse Education in Therapy Washington and Alaska Other Therapy (i.e. MARIELLA LARTER pacing, resuscitation) 19) Summarized as Other Therap! In July 1969, the Subregional CCU Nurse Education Evaluation of CCUNurse Education Project of W/ARMP became operational. The goal of the Test items were randomly assigned to version A or I project was to train 873 nurses per year in basic CCU, of the exam. Each exam@ is equally weighted by conten and to train them in sixteen (16) subregional centers but the individual questions remain different. After fiel( rather than in a "core" or "Seattle based" setting. In the testing on student nurses with no CCU background an( last year, all but one of the sixteen centers has become on graduates of a USPHS five-week CCU nursing course jational and an additional three communities have fourteen items from each version were eliminated. Aftel become subregional education centers. Each center plans item analysis of the test results of 200 nurses involved ir its own objectives, curriculum, elegibility requirements, subregional courses was completed, eleven additional course length, and teaching methodology. items were deleted from each version. The plan as outlined in the following pages was Test A and Test B now contain 100 items each, and developed by the Subregional Project staff in con- are of equal difficulty according to standard statistical junction with the Office of Research in Medical Educa- measures. In addition to answering each question with tion of the University of Washington, Charles Dohner, what she supposes to be the correct answer, each nurse is Ph.D., director. From its inception the evaluation was to asked to rate her certainly about that answer on a scale meet two goals: 1) to evaluate the impact of the project from one to three, or absolute certainty to guessing. The on regional CCU nurse training; and 2) to provide feed- computer summary of her scores then computes not back to course instructors on the strengths and ordy how many questions she answers correctly, but also weaknesses of their courses and of individuals in them. how many questions she was certain about, and how The evaluation design at present involves measures of many which she says she was certain about that she knowledge, attitude and skill. A patient care assessment actually answers correctly. too] is presently under development. We can thus measure with our instruments three areas KNOWLEDGE TESTING of potential change from pre to post course: 1) change in knowledge (right-wrong score); 2) change in expressed Practicing physicians and nurses from throughout the certitude and guessing; and 3) change in ability to region were asked to submit multiple choice questions evaluate her knowledge about CCU nursing. These three 170 factors are recorded for the overall exam as well as for SKILL TESTING five content areas within the exam for each student and A skill test was designed to evaluate the functioning each class. of nurses when presented with simulations of clinical The instructor can go over with each student her emergencies. The testing involves an evaluation of areas of greatest knowledge or growth in knowledge, her psychomotor abilities as well as the rationale for initia- expressed confidence, and whether that confidence is tion of certain therapeutic measures. The skill test is well founded or not, She has considerable data about the designed to be administered in a mock-up setting using a continuing education needs of her students at the standard hospital bedside area, an arrhythmia anne completion of her course as well. resuscitation doll, a bedside monitor, a defibrillator, and The regional mean for the pre test (Test A) is 32%. standard emergency equipment (i.e. suction, medica- tions). The regional mean for the post test (Test B) is 53%. The reliability coefficients of these exams, when admin- In initial field testing, nurses suggested the following: istered to a group of ten or more individuals, range from 1. that they be in uniform when tested, and 0.89 to 0.95. 2. that the evaluator "role play" as a new orientee to a coronary care unit, rather than assume a strictly ATTITUDE TESTING observational andjudgemental pose. A standard semantic differential scale is used to Taking these suggestions, a group of 28 nurses from evaluate the attitudes of nurses on ten concepts in CCU both metropolitan and rural hospitals were evaluated nursing. Those concepts include: coronary care nursing; using this tool in their own clinical setting. The range of coronary heart disease; cardiac monitoring; cardio- scores was 8 to 30, out of a possible 32 points. The pulmonary resuscitation; doctor-nurse relationships; mean score was 22 pohits. change in nursing; independent nursing decisions; patient The evaluator summarized her conclusions regarding individuality; patient teaching; and emergency sit- initial use of the tool as follows: uations. One other concept, "death", was eliminated because of possible disagreement about what a desirable 1, Greater consideration needs to be given to the change in attitude would be at the completion of a standing orders under which a nurse functions in a course. given agency; accepted therapy for nurses to The attitude scales are filled out on an anonymous initiate varies greatly from agency to agency. basis at pre and post tests; results are scored by sub- 2. No more than one agency can be evaluated on a region. Analysis of results in the first formal evaluation given day in view of unit pressures, staffing, and period (September 1, 1969 - February 1, 1970), in patient census; the cost of sending an evaluator which time over 500 nurses participated in Subregional any distance is considerable unless other duties can courses, revealed that overall there was not a significant be performed concurrently. change in attitude from pre to post course. Individual 3. It is very difficult to remain neutral even when courses showed significant attitude changes in one or involved in role playing; there is a constant more "concept" areas; in discussing some of the puzzling temptation to correct errors and teach during the attitude shifts with course instructors, it was learned testing. that the majority of these shifts reflected an attitude 4. The nurses tested need to be thoroughly familiar consciously conveyed to the students by the instructor. with all equipment used in the testing situation; thus, hospital equipment or a like brand must be brought to each testing site. Evaluation of CCUNurse Education 5. Many nurses responded appropriately to situations Future planning calls for use of attitude measure at but for the wrong reasons. pre course and at six months after course completion. 6. A weigliting scale needs to be further refined so Knowledge and attitude scores have thus far showed no that there is a greater spread in scores and dif- correlation; if this lack of either significant attitude ferentiation between levels of performance. change or correlation between attitude score and knowl- 7. The skill test is an excellent teaching tool but edge continues, a new strategy to measure this area will needs further revisions to increase its effectiveness be devised. as an evaluation instrument. 171 Evaluation of CCU Nurse Education strate changes in patient care as a result of postgraduate Future plans call for the random skill testing of learning experiences for nurses. nurses at the completion of basic courses utilizing equip- Additional Matetials Available on Reqtiest: ment they have used in mock-up drill sessions. I . Objectives for nurse training upon which knowl- OTHER EVALUATION TOOLS edge tests are based. 2. Sample computer printout and explanation of data A. A personal profile sheet revealing 17 pieces of contained in it. demographic information about each nurse is filled out 3. Copies of Test A and Test B for review. (Copy- at the completion of courses. Correlations are being run righted material - not to be retained or duplicated between these 17 variables and combinations of variables in any fasl-don) compared with pre test score, post test score, expressed 4. Answer sheet incorporating certitude measure. certitudes, accuracies, and changes in score. Data will be 5 . Attitude test. available soon. 6. Sample of results of attitude testing returned to course instructor. B. Use of chi square measure in conjunction with 7. Skill test. certitude score has been employed by the School of 8. Personal profile sheet. Medicine. A high and significant correlation was found between a low chi square and the overall knowledge of C, fvrite.- MARIELLA 1ARTER, R.N. the students tested. This measure is being incorporated Subregional CCU Project W/ARMP into the CCU nursing data analysis. 180 "U" District Building C. A patient care assessment too] is under develop- 1 107 N. E. 45th Street ment. It is hoped that this tool can be used to demon- Seattle, Washington 98105 CORONARY CARE NURSING EXAM...CONTENT AREAS RELATIVE WEIGHTING, OBJECTIVES CCU Concepts .... relative weight I b: Evaluates the techniques used in the physical and psy- a: Synthesizes a concept of intensive coronary care in chological preparation of the patient for diagnostic tests. relation to its implications for the professional nurse (content areas include: history and physical; serum enzymes, b: Values the necessity for assuming responsibility and self- ESR, NVBC, temperature elevation; circulation time; chest direction for continued learning in CCU nursing. X-ray; serial EKG'S; heart and breath sounds; vital signs, CVP, Anatomy and Physiology .... relative weight III jugular veins, urine sp. gravity, 1&0; nursing care plans a: Comprehends basic anatomy and physiology of the related to scheduling of tests; teaching plans to minimize cardiovascular system fear, discomfort, emergencies) b: interprets significant interrelationships between the Complications of an Acute Myocardial Infaiction (excluding cardiovascular, pubnonary, renal, and nervous systems arrhythnijas) relative weight XIII c: Interprets significant concepts of stress. Uncomplicated Acute Myocardial Infarction. . relative weight a: Applies the problem solving method to the identification Vill and treatment of the complications of coronary heart a: Synthesizes a concept of coronary artery disease in disease: relation to its implications for professional nursing care. I . congestive heart failure b: Develops a systematic approach lo the assessment of the 2. cardiogeriie shock individual patient's status upon admission and in sub- 3. acute pulmonary edema sequent days of hospitalization. (content areas include: epidemiology; pathophysiology of 4. pulmonary-systemic emboli coronary heart disease; physiologic stress responses; psy- 5. pericarditis chology of rife-threatening diseases; history and classic signs 6. cardiac rupture and symptoms of acute MI; cardiac ischemia as it relates to 7. cardiac arrest relief of pain, anxiety, and administration of oxygen; dietary 8. extreme emotional reactions modifications, activity restriction, fluid balance; planning Electrocardiography .... relative weight VIII individualized care) a: Synthesizes basis principles of electrocardiography to Diagnostic Tests .... relative weight V serve as a basis for the evaluation of cardiac status of the a: Analyzes the major diagnostic tools used in the diagnosis individual patient of coronary heart disease in terms of their implications (content areas include: electrophysiology; hemodynamic vs for planning nursing care. electrical properties of the heart; depolarization and 172 repolrization of the myoca-rdium; correlation of the electro- d: Develops a systematic approach to the identification and physiology of the heart with the electrocardiographic tracing; treatment of cardiac emergencies. basic principles of polarity, amplitude, and configuration of e: Differentiates the nurses' responsibilities in elective the PQRST in terms of lead axis and cardiac vector) cardioversion and the preventive use of pacemakers, as :quipment and Safety .... relative %vei.-ht III opposed to the emergency situations involved with these a: Applies fundamental principles of electrocardiographic therapies. techniques to achieve maximurh effectiveness and safety f: Utilizes the problem solving method to determine of electrocardiographic monitoring and twelve lead equip- priorities in nursing care in the post-resuscitative period. ment. Rehabilitation. . . relative weight VIII (content areas include: grounding; monitoring capabilities a: Develops and communicates a nursing care plan that in- and limitations as opposed to the standard EKG; essential corporates preventative, therapeutic, and rehabilitative features; purposes and standards of electrocardiographic aspects. I equipment; interference and means of eliminating it; how to b: Evaluates the patient's CCU experience in relation to his use standard monitoring equipment; safety for staff and total fife situation. patients with monitoring equipment) c: Determines implications for the planning of comprehen- ,rrhythmias ... relative weight XIX sive nursing care. a: Applies the problem solving method to the identification d: Values the role of the professional nurse in the health and treatment of the complications of coronary heart team, especially in relation to her potential contributions disease, specifically cardiac arrhythmias. regarding the individual needs of the patient and family b: Evaluates alterations in the electrocardiographic rhythm and continuity of care into the post hospitalization phase. strips and rhythms displayed on the oscilloscope accord- e: Reviews select basic nursing knowledge and skills in the ing to their significance to the patient's total condition light of their implications for the patient with coronary and th@ implications for medical and nursing therapies. heart disease. c: Develops a systematic approach to the interpretation of arrhythmias. (content examples-vital sips, pulses, tracheal auctioning, d: Utilizes the problem solving method in the treatment of oxygen administration, respirators, patient positioning, arrhythmias. venipuncture, IV therapy and administration, rotating (content areas include: arrbythmias by site of origin, effect, tourniquet, skin care, passive exercising) treatment, and implications for nursing care) hemical Therapy .... relative weight XIII Summary of Content Areas and Relative Weighting onboth pre a: Develops a systematic approach to the classification, and post Tests: analysis of, rationale for, and the nursing implications involved with chemical therapies in the treatment of CCU Concepts: 1/100 coronary heart disease and the frequently encountered A - P: 3/100 complications. Classic MI: 8/100 ther Therapy .... relative weight XIX Diagnostic Tests 5/100 a: Appreciates the nurses role in the early recognition and Complications: 13/100 treatment of conditions that may precede life threatening Electrocardiography 8/100 conditions. Equipment & Safety 3/100 b: Appreciates the importance of effective habit patterns in Arrhythmias: 19/100 the handling of emergency situations. Chemical Therapy 13/tOO c: Appreciates the importance of frequent review and Other Therapy: 19/100 continued refinement of emergency procedures. Rehabilitation: 8/100 173 A SYSTEMS APPROACH TO CORONARY CARE EVALUATION Participants Morton Robins - Moderator M. A. Rockwell, Ni. D. Acting Chief, Study Design and Analysis Director, Rand Health Program Staff Rand Corporation Regional Medical Programs Service A Study of Coronary Care Unit Effectiveness The CCRNIP, aware both of the importance of col- M.A. ROCKWELL lecting performance data in CCUs and the difficulties experienced by many units in collecting such data, embarked upon development of a standardized data This report describes a continuing project conducted collection and reporting system for CCUS. In December by The Rand Corporation for the California Committees 1969, a contract was given to The Rand Corporation to on Regional Medical Programs (CCRMP) to measure the develop a prototype system and test its feasibility. operational effectiveness of coronary care units. During Medical guidance of the project was provided by the the past two years the project, which began as a feast- CCU Steering Committee of the CCRMP. bflity study, has become a community action project During the past two years, a prototype data col- involving more than I 00 hospitals. This report traces the lection form has been designed, tested and revised. On evolution of the study from its initiation up to the January 1, 1970, a prototype data collection system present, describes what has been accomplished, and out- became operational and participation in the study was lines future objectives. opened to any California CCU that wished to partici- Our study is based on the belief that every CCU pate. should continually monitor its performance. Data should The current system requires that about 100 items of be collected describing patients admitted to the unit, information be reported on each acute myocardial how rapidly they reached the CCU following their onset infarction patient admitted to the CCU (only 10 items of symptoms, their clinical -course and treatment during of information are collected on non-Ml patients). The their CCU stay, and their clinical course and treatment data forms are mailed to The Rand Corporation where during their CCU stay, and their discharge status. Col- they are keypunched. Every three months the key- lection and analysis of such data is necessary to ensure punched data are processed by computer to produce that the unit is performing effectively. summary reports. Each hospital receives a 15-page report In 1968, the CCRMP found that most CCUs were describing the patients admitted to the unit and the out- trying to collect and analyze such data but many of the come of their hospitalization. Each unit can compare its units were having problems in their data collection. experience with that of the participating group as a First, development of the necessary data collection whole. forms and procedures proved to be too difficult for Preliminary indications are that the data collection many units. Second, many CCUs soon collected such a large volume of data that it could not be analyzed by system has become an important part of the CCU manual techniques but required computer methods. operation in many hospitals. Although participation in Most units did not have access to the necessary equip- the study is voluntary, the number of participating hos- ment and expertise. Third, once the data was collected pitals reached 120 by June 1970. Thus, about two-thirds and analyzed, it was often difficult to interpret because of California's CCUs are now involved in the study. In there was no standard against which to compare the addition, units from the WasWngton-Alaska RMP, the results. It seemed desirable to allow each CCU to Northern New England RMP and Missouri either have, or compare its results with those of similar hospitals, Such are soon expected to join the study. comparisons, however, required data collection and We believe that the study has had an important and analysis procedures to be standardized, a task obviously beneficial effect on CCU effectiveness. First, it has beyond the capability of an individual CCU. helped some CCU directors improve the operation of 175 their units by, for example, finding ways of speeding the ways of reducing the cost of CCU care without com. patient admissions. Second, periodic summary reports promising its quality. These include: (1) using specially have served as a focus for teaching conferences for CCU trained CCU technicians to supplement nurses in the physicians and nurses. Third, data collected by the sys- units, and (2) using automated monitoring equipment to tem have helped the CCRMP assess the effectiveness of eliminate the requirement for continuous surveillance of their nurse training program. Fourth, data collected by ECG monitors. the system should make it possible to investigate several 176 EVALUATION OF INSTRTJCTIONAL TECHNOLOGY PROJECTS Participants James E. Dyson, Ph.D. - Moderator James Barrett, Ph.D. Director, Continuing Education'Division Continuing Education Division Colorado-Wyoniing Regional Medical Program Colorado-NVyoming Regional Medical Program M. Gene Aldridge Continuing Education Division Cecilia C. Conrath Colorado-Wyorfflng Regional Medical Program Chief, Continuing Education and Training Branch William Engbretson, Ph.D. Regional Medical Programs Service President, Govemor's State University Swnmary of Session While Dr. Barrett interviewed participants a written CECELIA CONRATH recording of the answers was projected on an overhead projector by Gene Aldridge also of the Colorado/ The workshop session on Instructional Technology Wyoming RMP staff. This enabled a running inventory was developed by the Colorado/Wyo@ng RMIP. The to be kept in front of the participants as the session objectives of the session are given below in order of progressed. priority. At the conclusion of the problem census a long 1. To learn interests and needs of workshop partici- distance telephone conference was held with the follow- pants for help in evaluation using instructional ing Consultants: technology. William J. Paisley, Ph.D. 2. To help participants learn functions of various Director, ERIC Clearinghouse on types of instructional technology, approaches to Educational Media and Technology at the Inst. for Comm. Research evaluation of such technology and relative effects Stanford University of various approaches. Stanford, California 94305 3. To present information on effective evaluation procedures. E@abeth Norman, Ph.D. 4. To develop an awareness of consultation/referral Associate Professor of Nursing resources nearby within region and on an inter- College of Nursing regional basis. Northeastern University The whole idea was to show how questions and Boston, Massachusetts 02115 concerns can be quickly identified, how resources can be Rick Breitenfeld, Ph.D. located and used effectively, and to demonstrate that Executive Director the basic strategy of evaluation grows out of the needs of the participants. Maryland Center for Public Broadcasting The session opened with a brief statement of the Owings Mills, Maryland 21117 statu s of instructional technology within RMPS by the Gerald W. Gaston, t).D.S. Chief of the Continuing Education and Training Branch OSRMP-CAI Project Supervisor followed by an outline of the session by Dr. James The Ohio State University Dyson, Associate Director of Continuing Education, Columbus, Ohio 43210 Colorado/Wyoming Regional Medical Program. A prob- lem census of interests and needs of participants was David L. Bell conducted by Dr. James Barrett of the Colorado/ Box 488 Altadena, California 91001 Wyoniing staff. 177 The results were not entirely successful because of lems, i.e. temporarily losing California participants a the small attendance at the session. This technique is poor voice transmission interfered with the reception. productive with a minimum of 10 and upward in an almost unlimited number. There were only 5 partici- Gene Aldridge assembled kits of material on eva pants and two left early. uation of instructional technology and learning theory Issues concerned with cost effectiveness of different Bibliographies on the general field of learning, teaching media, adaptability and conversion from one modality with films, guides for TV teachers and considerations for to another, and status of evaluation research were judging audiovisual presentation standards were among brought up during the conference call. Technical prob- materials distributed. 178 EVALUATION OF PHYSICIAN EDUCATION Participants Daniel Fleisher, Ni.D. - Moderator William B. Munier, M.D. Director of Health Professions Staff Assistant, Continuing Education Temple University and Training Branch Regional Medical Programs Service Summary of Session the projects was very well constructed and contributed WILLIAM B. MUNIER the majority of desirable aspects. Following critique of all three, a fairly complete list had been developed of what constituted an effective project. It had been The objective of the Special Interest session on developed by the participants themselves following Evaluation of Physician Education was to increase the careful analysis of three projects representative of actual knowledge of the participants about the essentials of RMP grant requests. sound educational projects. The methodology employed It was felt that the active involvement of the people was that of active involvement of participants in attending the session was more likely to increase their deciding on what constituted a sound project. No evalua- knowledge than would a didactic presentation. The tion of learning was planned. actual proceedings at the session involved active debate Specifically, three surrogate projects were presented, concerning which aspects were good and which were one at a time. In each case, desirable and undesirable not. Errors in judgement by a given participant - from aspects were listed, as volunteered by the participants the moderator's point of view - were quickly following review of the projects. Explanation and lampooned by others. The resulting list at the conclusion analysis of the projects was led by the moderator, Dr. of the conference was educationally quite sound. Insofar Daniel Fleisher of Temple University, Philadelphia, as- as.no evaluation of learning was planned, the product of sisted by Dr. William Munier. the session was good, and all present participated Two of the projects were poorly designed, and actively, the conference was subjectively judged a contributed the bulk of the undesirable aspects. One of success. 179 EVALUATION OF MULTIPHASIC SCREENING Participants Donald N. Logsdon, M.D. Frank R. Mark, M.D. Associate Director, Multiphasic Health Chief, Operations Research and Systems Screening Center Analysis Department of Community Health Regional Medical Programs Service Brookdale Hospital Center Evaluation of Multiphasic Health Testing Tayback considered establishment of a multiphasic screening (testing service to be based upon the follow- DONALD N. LOGSDON ing operational model. Therefore, evaluation of MHT projects funded In the chapter entitled Evaluating the Quality of through NCHSR&D should proceed on three levels. Medical Care by Avedis Donabedian from the recent ACHIEVEMENT OF TECHNOLOGICAL OBJ.ECTIVES book Presymptomatic Detection and Early Diagnosis by PHS was at that time proceeding on the assumption Shark and Keen, the conclusion is reached that that MHT is basic to the attainment of a national health "alth.ough the assessment of the quality of medical care objective - periodic assessment of the health status of remains difficult and imprecise, there are several ways in each adult, 35-69 years of age. The system must undergo which one may arrive at judgment sufficiently valid for a continuous improvement resulting in added validity of variety of administrative decisions". Among the ways the health testing and in improved cost effectiveness. suggested were "studies of the effect of greater precision Specifically, it was recommended that the achievement and detail in standards on the reliability and validity of of technological advances in MHT be measured by the judgments (measurement)". As applied to MHT the current operating programs have attempted several eval_ completion of defined tasks and with time specified end- uation studies which I will briefly describe and comment points. During the twelve month period, January 1, on. 1969 - December 31, 1969, the following tasks were to be initiated and progress reports submitted by the end of Dr. Matthew Tayback, in several meetings sponsored the period- These tasks are not a complete description of by the U.S. Public Health Service in 1967-68, set forth the technical problems which need solution. criteria which he suggested for determining the value of Glucose Tolerance Test Multiphasic Health Testing. He restated the proposition It is im that evaluation should rest on the success of attainment perative to detern-dne the relationship which of project objectives, namely, (1) per cent of target exists between the result of the abbreviated glucose sample reached (2) precision and accuracy of individual tolerance test as employed in MHT and standard oral measurements (Quality Control) (3) yield of screened glucose tolerance test as performed in conventional positives per major procedure (4) per cent of screened hospital or private laboratory centers. positives who make contact with personal physician, and The effect of time of day on the abbreviated tests (5) per cent' of screened sample with minimum sig- must be clarified. nificant benefit in health knowledge due to MHT. Although it is highly pertinent and eventually critical to Standardization of Norms consider cost-benefit characteristics or end results of Interpretation of results from clinical lab tests, in- MHT, such data will not be forthcoming for several cluding blood chemistry, hematology, and non4ab tests, years. In the meanwhile MHT technology needs to be such as spirometry, by the practicing physician is diffi- advanced on the principle of its cost-effectiveness and its cult when the normal population ranges for a specified capability to efficiently process large populations. measurement is not given in the report. 181 A. Service inputs provided to a B. Target Population produces of of Specified Quantity Specified Composition and Quality Behavior C. which when D. Personal Physicians produces Individualized interpreted by Utilizing the Health Information in Information A Specified Manner E. Desirable Benefits Which Can Be Specified A standard procedure for reporting MHT results Quality Control should be adopted. The exact measurement obtained Each major component of MHT requires a protocol should be reported and the normal range for the age, sex for establishment and control of quality of measurement and ethnic group category given. and test information generated. Each project should Since the distribution of deflned measurement by develop a manual of procedures in respect to quality age, sex, and ethnic groups has not been determined, this control. PHS should then produce a standard manual on should be developed as soon as possible. quality control and annually update this document. Standardization and Documentation of ACHIEVEMENT OF PROGRAM OBJECTIVES Computer Programs Pending demonstration of benefits relating to reduc- Inefficiency (excessive cost) is generated by failure to tion in disease, disability and age specific death rates, develop systems which can be replicated with minor MHT must receive process evaluation on the basis of the adjustments. attainment of program objectives and the cost-effective- Existing computer monitoring of SMAIZ and VCG ness of services. interpretation needs to be validated with a view towards Such process evaluation will be possible by the selection of a standard program so that widespread use following strategy: can be made of the standard @rograms with minimum 1. Each project should set forth the target population further investment for software development in these it seeks to reach with its screening program and should specified areas. specify the fraction of the target population which it Cost Analysis proposes to reach. 2. Periodically (quarterly) demographic characteris- Major components of MHT needed to be defined. tics of the screened population were to be reported to Each component must then serve as a unit for the deter- PHS. Comparison of 2 with I will indicate the extent to niination of cost. which the target population is reached. The minimun, Cost analysis data should be generated within the set of variables for which information is sought should next 12 months. include age, sex, race, income, occupation, source of In view of the limited staff, this task should be regular medical care, utilization of medical care within accomplished through a contract negotiated with an past six months, date of last general physical examina- interested and competent cost analysis service. tion, and follow-up results. 182 3. The yield of significant positive findings per exami- 3. Documentation and evaluation of the experiences nation procedure for age, sex and ethnic groups will with this type of health service as compared with the permit assessment of the cost per abnormality detected. existing health services of the Hospital Ambulatory Care 4. The patient-physician contact ratio by major Program. A central record system would enable monitor- screening classification type is a measure of the extent to ing of the two types of care. which significant screening findings receive follow-up 4. Further utilization of paraprofessional personnel exai,ns. and instrumentation in health care. The use of physician 5. A survey of selected classes of the screened popula- assistants, nurses, technicians plus hardware can be tion, prior to and following the date of visit to the tested. screening center could provide suggestive clues relative The above factors are considered important in assess- to the health attitude and knowledge of consumers and - ing the difference MHT makes in the delivery of health could provide information of consumer reaction from a services. It is recognized that the addition of a Follow-up single exposure to MHT (a consumer study). Clinic would not alone provide an answer to the ques- 6. A questionnaire survey of physicians, who have tion of benefits in terms of biologic outcome or end received MHT reports, with a view towards determining results. However, as the methodology for this type of their attitudes and knowledge of the usefulness of MHT evaluation is adequately developed, a prospective longi- would complement information obtained through 5. tudinal study of morbidity, mortality, and disability could be attempted. ACHIEVEMENT OF LONG RANGE OBJECTIVES We began to evaluate MHT at BHC as part of a Control of clinically significant chronic disease and primary health care system at intermediate points and to prolongation of life are end results of MHT which can be determine feasibility of assessing end results. demonstrated only by ambitious research involving In December 1969 a subcontract was signed and work careful prospective follow-up of large samples of adults begun for biostatistical retrieval and analysis of the data over long periods of time. Such a project is under way at on the 14,000 screenees processed at the Brookdale the Kaiser-Permanente medical service in and around San Hospital Center MHS program from the beginning in Francisco. February 1968 through October 1969. Initially, the data it was recommended that aside from Kaiser-Perma- were examined in terms of frequency distribution for nente, no extensive investment of funds should be made continuously distributed quantitative variables by age, at this time to demonstrate long term effects of MHT. sex, and ethnic background. Dichotomous qualitative variables were tabulated an(f percentage positives calcu- PROCESS EVALUATION lated also by age, sex, and ethnic background. Measure- The Brookdale Hospital, Multiphasic Health Screen- ments of central tendency and variation were performed ing program has been a successful demonstration of on continuously distributed measurements. This included AMHT. The questions now to be answered are: What mean, standard deviations, median, 5 and 95 percentiles. difference does MHT make in the delivery of health The number and percent of screenees with clinically services in an urban environment? Can MHT become an significant overt and occult abnormalities based on effective component in' a primary health care system? it currently acceptable critera was detem-dned also by the was established by PHS support to test the feasibility of variables of age, sex, and ethnic background. Investi- operating a MHT program in an "open" medical care gations will also be made for correlation analysis, i.e., system with an adjacent poverty population. history vs. test results, and screening results vs. physician In order to answer these questions it would be neces- diagnosis. This effort has been successful for Brookdale sary to "close the infomwtion loop" by establishing a MHS and should have application to other demonstra- Follow-up Clinic which would have enabled the program tion programs in MHT. to accomplish the following. Problems of data retrieval and analysis include: 1. Quality of input-measurement and keypunching 1. Provision of the necessary follow-up medical evalu- ation and management for the screened poverty popula- errors. tion. 2. Storage on historical tapes, i.e., completeness and 2. Validation of the screening results by comparison documentation with results of diagnostic studies for the poverty popula- 3. Retrieval group intervals, criteria of normal, physicians. abnormal. tion by Follow-up Clinic 183 4. Analysis - Mean or median, standard deviation or 2. In regard to the latter, the input for a time-effort percentfle, test of significance. study has been built into the computerized module for sui)vort of follow-up activities. These components must An econon-dcal evaluation can be approached by cost be costed out in a AMHT in addition to those items per test as indicated in the cost finding protocol for the included in the recent reports on costs of the Kaiser- past project year. The SRI method is being tested. Permanente MSP in the New England Journal of Medi- Effectiveness is being evaluated in terms of the yield cine. These additional items will obviously increase the of unknown and uncontrolled occult and overt condi- cost of AMHT programs involved in motivation and tions detected at the MHT Center. This, of course, is follow-up and the question to be answered is how much. related to the prevalence of disease in the target popula- In addition, a cost effectiveness report can be pre- tion. High yields are expected for certain conditions in pared for the follow-up activities wherein comparison of poverty populations, groups, due to prevalence and the costs for furnishing the follow-up services using alter- lack of adequate medical care, e.g., rates for hyper- native methods will be used with the objective of being tension and hypertensive heart disease. The methodo- able to minin-dze the resources expended and maximize logical problem of determining "unknowness" can be the number of individuals receiving medical follow up. solved by the use of questionnaire information from Investigate Consumer. and Physician Reaction to patients rather than from M.D.'s. Efficiency is being AMHT.- calculated on the basis of the cost per positive screening 1. During the 1969 project year 50% of the individ- test and cost per valid diagnosis. Of course the latter is uals screened resided in the Hospital's core area, and dependent on adequate follow-up reporting. These eval- 25% were Black or Puerto Rican. A number of tech- uation efforts should be performed as the program niques for increasing registration from the hard core high activities are carried out. Simultaneously, the end-result priority areas were tested. Good progress has been made, evaluation is being explored for feasibility in an environ- but it is apparent that "hard" data on the behavior ment which prohibits randomization into study and factors are necessary to improve performances beyond control groups for longitudinal study. Present plans this point. could include labehn- a sample of the screened poverty Similar considerations are involved in improving the population for monitoring over time and comparing 70% figure for successful follow up. their experience in morbidity, mortality, and disability The data generated by the screenee process must be with non poverty population and/or national statistics evaluated by a physician in the context of his examina- for the same age, sex, and ethnic group. tion of a specific individual. The physician's knowledge For preliminary results for total population see of and attitude toward AMHT therefore becomes of Appendix I. central importance. Detennine the cost of MHT in affinwry Health Care The staff of the Brookdale MHSC is actively engaged in assisting the research group at Columbia University System: School of Public Health and Administrative Medicine in 1. The Brookdale MHSP, as a result of the SRI Cost the development and implementation of two relevant Finding Study of ANIHT has begun to examine the cost studies: Consumer Reaction to AMHT #HSM of the totalprogram.Informationisbeingcollectedon 110-69-212, and Physician's Attitude Toward and the total expenditures for this program through the Acceptance of AMHT #HSM I 10-HSRD-57 (9). Brookdale Hospital Center business office. However, it is 2. The Physician Attitude Study is designed to apparent that a true cost analysis of this program will deternune: require the establishment of bookkeeping - cost account- a. What are the social and psychological factors ing procedures separate from the Hospital System in which affect the physician's cooperation with, his order to identify the various costs involved. In addition acceptance of, and his behavior concerning the to the usual items that are included under direct cost, it MHT at Brookdale, including those factors which will be necessary to itemize those costs involved in facilitate his utilization and acceptance of the recruiting the target population to utilize the facility and service, as well as those factors which are barriers the follow-up activities. to effective utilization? 184 b. What factors differentiate those physicians who In your opinion, did the summary contain more information actively participate and accept the screening pro- than was necessary, just about the right amount of information, gram from those who do not, and what factors or not enough information? Ntore information than necessary .......... 123 39% differentiate those physicians who change their About right amount of information . . . . . . . . .133 42% attitude and behavior concerning ANLHS? Not enough information . . . . . . . . . . . . . . . .59 19% c. How can an automated system such as multiphasic screening be made more useful and acceptable to 315 100% the practicing private physician? How easy was it for you to follow the general layout of the d. How do physicians adapt their practices to an summary? Was it very easy, fairly easy, somewhat difficult, or automated health testing program? very difficult? e. What inferences can be drawn from this specific Very easy .......................... 111 34% study to the more general area of physician's Fairly easy ......................... 112 34% response to automation in medical practice and Somewhat difficult .................... 70 22% Very difficult ........................ 32 10% what impact does it have on medical practice? - - In order to perform this type of study the project 325 100% must actively engage the support of the local Medical Society. This can present a difficulty. In your opinion, should the normal range of results be indi- cated on the sumniary? The study of physicians' reactions to automated Yes .............................. 290 90% multiphasic health screening presently provides for an No ................................ 31 10% initial and a follow-up survey 10 months later of 1200 physicians in Kings County, New York. The two inter- 321 100% views will determine their attitudes, knowledge, and Was the blood glucose test and result clear to you? utflization of the Automated Multiphasic Health Screen- Yes .............................. 280 90% ing Center at Brookdale Hospital in Brooklyn, New No ............................... 30 10% York. The re-interviews were intended to concentrate on 310 100% changes in attitudes, behavior, and perception of auto- Was the histogram arrangement of the hearing test results mated screening resulting from exposure to the program. clear to you? When the study was planned it was anticipated that at Yes .............................. 186 63% the time of the first interview, at least half of the No .............................. 111 37% doctors would in the interim become exposed to the 297 100% Brookdale program and, as a consequence, alter their image of it. How useful was the medical history questionnaire? Was it very useful, somewhat useful, not very useful, or worthless? However, results of the first wave of interviews indi- Very useful .......................... 34 12% cate that diffusion of the screening program has Somewhat useful ....................... 92 34% occurred more rapidly than anticipated. This fact has Not very useful ....................... 92 34% bearing on the timing of the re-interviews and in part Worthless ............................ 55 20% motivates this suggested modification. 273 100% Of the first 712 completed interviews, only 101 What did you think of the fraction arrangement of positive physicians have not been exposed to the program (86% responses by body system? Did you think this was a good way of were exposed). There is no reason to expect that the presenting the medical history information or not a very good remaining interviews will show much departure from this way? 7:1 ratio. Therefore, we cannot expect dramatic changes Good way ......................... 127 50% between the first and second interviews as a result of Not a very good way .................. 127 50% contact with the program. Some early results from inter- views of doctors follows: 254 100% 185 How helpful did you find this reference manual in reading Pretest in local laboratory. Example: cholesterol the patient summary? Did you find it very helpful, or somewhat studv. helpful, or not at all helpful? d. Design general outlines of AMfiT internal quality Very helpful ......................... 75 44% control system: calibration, serum monitor, Somewhat helpful ..................... 74 43% Not at all helpful ....................... 23 13% laboratory responses; design a system of external evaluation. 172 100% OTHER STUDIES OF MULTIPHASIC Now about what you think should be done by screening HEALTH TESTING programs like Brookdale's. Do you think that a screening pro- gram like Brookdale should be free of cost to examinees or 1. HIP - Utilizing MHT to define the health status, should there be a charge? practices, and attitudes of a defined poverty population Should be free ....................... 152 50% covering a broad age range (12 yrs. +) from an absolute Should be a charge ..................... 152 50% standpoint and relative to a nonpoverty group in the 304 1007@ same medical care environment. Action to modify - adverse aspects of the health co mponents is to be insti- Do you think that a screening program like Brookdale should tuted and evaluation is in terms of change as compared refer both normal and abnormal patients for follow-up by a physician or only patients with some positive condition? with what occurs in the non-poverty group. An under- Both normal and abnormal .............. 241 67% lying question is whether through the MHT program and Positive condition only ................. 103 33% activities generated by it, the anticipated gaps between the two groups can be narrowed. 344 100% The program expects to begin processing patients in November 1970. C7inical Laboratory Qua-lity Control Studies., 2. Meharry Medical College MHT Project - Evaluation After several attempts over two years, there has been of this project will be performed as part of the study on relatively little success in providing assistance to the comprehensive health services by Dr. Sam Wolfe. Clinical Labs in AMHT for developing a sufficient pro- 3. North Florida ., RMP, Gainesville, Florida, Dr. gram of quality control. The Clinical Chen-dstry Section, Richard Gordon and Co-workers. NCDC, has repeatedly demonstrated their interest in In summary MHT is a complex, relatively expensive, providing this support, but various bureaucratic delays experimental system of health services. Evaluation in have prevented any progress. The problem of assisting terms of program effectiveness and efficiency is feasible these labs remains, and a modest beginning is proposed but the methodology for successful end result or out- for the next project year. come evaluation has yet to be demonstrated for the total This effort would initially consist of a six-month system. MHT is advettely affected by two circum- study and evaluation of AMHT interlaboratory standard- stances: ization utilizing the Brookdale Hospital Clinical Lab as a 1. It appears to o easy and glamorous which is starting point, and then extending the protocol to probably the result of over-selling the technological include the other AMHT labs. The brief outline that developments, when in fact there are multiple techno- follows describes the activities and resources required: logical problems still to be solved. The major program Study and Evaluation of AMHT Interlaboratory problems involve the recruitment of the target popula- Standardization: tion and providing adequate follow-up for the individ- Preparatory efforts - Brookdale AMHT and NCDC uals tested. through individual and group interaction. 2. The latter relates to the major uncontrollable vari- a. Develop recommendations for reference method- able in assessing the value or benefit of MHT and that ology, enzyme units. being the lack of proven therapy for most of the chronic b. Anticipate problems in SMA technology and conditions detected. calibration. After struggling with evaluating MHT for several years c. Design and prepare Multiphasic Text Panel for the I usually caution people about trying to implement this eucidation of methodologic, technical, and cah- system of health services and especially to think through bration problems. Check stability of materials. the planned evaluation. 186 References 4. Metropolitan Life Statistical Bulletin, Dec. 1969 Bio- chemical Profiles 1. Sharp CLE & Keen H. Presymptomatic Detection and Early S. Use of "Normal Range" in Multiphasic Testing Files, et al Diagnosis Williams & Wilkins, Baltimore, MD. 1968 JAMA. Sept. 2, 1968. Vol 205 No 10, p. 684 2. Collen et al, NEJM, 283:459-463 1970.and Vol. 280; 6. Clinical and Biological Observations on Working Men, 1043-1045 1969 Clark, T.W. et al Arch Environ. Health, Vol 19. Nov 1969, 3. Neuffield Foundation Report Screening in Medical Care p. 700 APPENDIX 1 NUMBER AND PERCENT PREVALENCE OF CLINICALLY SIGNIFICANT ABNORMALITIES ON 13,000 SCREENEES THE BROOKDALE HOSPITAL CENTER MULTIPHASIC HEALTH SCREENING CENT FEBRUARY 1968 - NOVEMBER 1969 Test Brookdale Hospital Center KaiserPermanente* Total all ages Total all ages No. 170 170 Cost Blood pressure > 160/95 ................ 4058 31.5 4.1 Electrocardiogram .................... 3203 25.0 17.-3 $S.90 Chest X-ray ........................ 1053 8.8 7.4 6.20 Cervical cytology, III .................. 8 .13 Visual Distant >20/40 .................. 1917 14.9 15.8 1.85 acuity: Near >iO/50 .......... ... .... 817 6.8 Tonometry: > 21.9 mmHg: OD .................. 545 4.7 Os .................. 659 5.7 > 23.8 mmHg: OD .................. 194 1.7 0.3 183.00 Os .................. 25S 2.2 Spirometry: Pred. FVC <80,7o .................... 2505 24.1 2.2 14.10 Pred. MW< 90% .................... 2327 22.4 Audiometry > 30 db .................. 3050 28.3 16.2 Dental: Teeth, poor or bad . .............. 1454 11.7 X-Ray Edentulous .................... 1333 18.0 Alveolar bone loss severe ............. 1439 19.5 Other X-ray abn . ................ 8785 37.8 Cytology II-IV ...................... 31 .25 *NEJM Vol 280 No. 9 p. 4S9463 187 CLINICAL LABORATORY TESTS Tests Abnormal limits Clinically significant abnormalities Brookdale Hosp. Center KaiserPermaneizte Total all ages Total all ages No. % Cost Hemoglobin: Females .................... < 12 gms. % 737 9.5 10.3 Males ..................... < 13 gms. % 249 5.3 3.1 Total ..................... 986 7.9 Hematocrit: Females ................... @ 38% 614 7.9 Males ..................... < 40% 122 2.6 WBC ......................... < 4 & > 12,000/ 3.4 2.2 cu mm RBC: Females ................... < 4.2 ni Males ..................... <4.5 m Cholesterol: > 95 percentile for age ....................... 622 5.0 Males ................................. 235 5'0 Females ............................... 387 4.99 VDRL ....................... Positive 81 .7 I.S Urine: Culture: Females ................... > 16 col. 679 8.8 3.3 Males ..................... > 16 col. 6 3 1.5 0.4 Glucose ................... 1+ to 4+ 780 6.3 8.2 Protein ................... 1+ to 4+ 550 4.4 6.4 Acetone ................... 1+ to 3+ 155 1.2 188 EVALUATION OF STROKE - REHABILITATION Participants Jerome Tobis, M.D. - Moderator B. Lionel Truscott, NI.D. chairman Department of Physician Medicine Director, Stroke Program and Rehabilitation North Carolina Regional Medical Program University of California at Irvine Philip A. Klieger, M.D. Charles M. Wylie, M.D. Assistant Director for Organizational professor, Public Health Administration Liaison for Stroke and Rehabilitation University of Michigan Regional Medical Programs Service Bertram L. Tesman, M.D. Assistant Coordinator, Stroke Program California Regional lvfedical Program - Area VIII Evaluating Stroke and Rehabilitation Programs: moderate stress, and use that stress to galvanize us into An Overview improvements rather than into fits of depression. This CHARLES M. WYLIE, M.D. i-night be regarded as the power of positive thinking about evaluation. At this late stage of the conference, evaluation is no longer an attractive and novel word. The discouraged or WHO SHOULD DO THE EVALUATION? bored may suspect the reality of the Turkish proverb: If First, a brief word about the site of evaluation. To a stone falls on an egg, alas for the egg; if an egg falls increase the likelihood of acting on the findings, it seems on a stone, alas for the egg. If we fail to evaluate our essential that the effect of RMP's on the national health program, alas for the program; if we do evaluate our program, alas for the program. levels be assessed by those working in the'federat office, To evaluate or not to evaluate - that is not the ques- the effectiveness of regional efforts be evaluated by tion for those of us who wish to continue working in those in regional offices, while the evaluation of local RMP'S. Society has always advised us to be critical of program, be carried out by local personnel. Too often in what we do.' The saying, afl's well that ends well, their health activities federal personnel evaluate state reminds us that even centuries ago activities were activities, states evaluate the local picture, locals don't considered good primarily when their outcomes were evaluate, and little change occurs. Fortunately, RMP's good. Thus the salient question is: how can evaluation have learned from raistakes made elsewhere. be a constructive force which improves programs rather The evaluation findings are more likely to be acted than a destructive force for the eradicatioti of programs'? on when program personnel evaluate the effectiveness of It will destroy, for example, if it uses criteria which are ,their- own activities. -Examining evaluation realistically, so strict that we cannot meet them. It will also destroy if however, we must admit that the first priority of the it uses so much of our resources that we have little left agency staff is to continue the program; program im- to run good programs. provement is only a secondary goal, and destroying the Must evaluation affect us adversely, however 9 It will program is their great fear. They may often feel that if we insist that it be completely free from stress. It will "conventional wisdom" from which the program arose is if we expect it to resemble the French view of love, a more important than negative evaluation data. They will pleasant diversion between meals, or even more the correctly add that some decisions must be political and Swedish view, a pleasant diversion during meals. But humanitarian, neither of which viewpoints is considered evaluation won't harm us if we expect and accept in evaluation. 189 'fhe likelihood of corrective action may be lower ADEQUACY of what we do (how much of the entire with an outside evaluator, who may have other biases. problem we are likely to overcome). He may view evaluation as a chance to test theories or Documenting a change in resources is a step which methods which interest researchers. He may suspect the can be swift and cheap; in our concern to "get on with evaluation effort, perhaps from bitter past experience, as the job," it is only too easy to stop evaluation at this designed to give the program a legitimacy which it does point. To ensure the long term survival of RMP'S, how- no t deserve. He may suspect further that a critical evalu- ever, and to gain information on how our programs may ation will be ignored, or that negative outcomes will be be improved, we must regard this as only the first step in quietly forgotten so as to ensure the growth of future providing more convincing information on the value of funds. Such events, we may hope, will be rare in RMP'S. stroke and rehabilitation programs. In evaluating stroke and rehabilitation programs, our efforts are likely to aim at three levels of information: ACTIVITY CHANGES 1. Changes in resources, including the number or Many activities are held to be desirable when they quality of trained personnel. seem likely to delay the onset of stroke or improve the 2. Changes in the activities produced or the work function and speed the recovery of stroke patients. An performed by these resources. effective change in resources, as described above, will 3. Changes in the end results of these activities. result in more of these desirable activities; we should @t us consider the strengths and weaknesses of each show that this has truly happened. The process of evalu- level of evaluation. ation becomes more complete and impressive when it REsoURCE CHANGES shows clearly that the new or improved resources have truly raised the output or quality of activities as well. RMP funds may improve the quality, quantity, or 'Me steps to collect these data must be planned before both, of facilities, personnel, knowledge, or other re- the resources are changed. This advance planning makes sources involved in producing stroke and rehabilitation it possible to contrast the activities before and after the activities. A new hypertension clinic may be supported change occurs. to prevent stroke, another clinic established for the early Let us take the situation where an educational pro- detection and treatment of transient ischemic attacks. gram has been shown effective in improving the knowl- Health personnel may attend new courses which review, edge of the participants. We wish to show that this for example, the optimum care of stroke patients. More change in resource produces a change in subsequent rehabilitation personnel may be recruited to consult activities. One goal of an educational program may be, with personnel in home care programs or extended care for example, to encourage physicians to make better facilities. diagnoses on their hospitalized stroke patients. A If RMP personnel document that such resources have regional committee of experts or of peers, let us say, as been changed, but go no further in the evaluation effort determined the content of the optimal diagnostic (like some annual reports in the past), they imply that examination. The purpose of evaluation will then be to these changes will inevitably improve patient care. show that physicians taking part in the educational pro- However, there are too many skeptics among politicians, gram perform an examination which is closer to the ideal the general public, and the health professions to expect after than before the program. Is such a step feasible? that such a primitive evaluation, with its possible but When physicians may frown on taking a test of knowl- still unproved assumption, will go unchallenged. Too edge and attitudes before and after the educational many clinics improve the care of small numbers of course, they will not rush to welcome an effort to assess patients who are already under care, but have no impact their methods of diagnosis. Compromises may be on the large burden of neglected disease in the surround- needed, and we may have to monitor changes in groups ing community. Too many health personnel may fail to of health professionals rather than changes in the ac- act on new information, obtained in courses, or may tivities of individuals. return to environments in which they cannot apply their new knowledge. Too many rehabilitation personnel must provide minute doses of advice or care to their large CHANGES IN END RESULTS caseload of personnel and patients. All of these relate Expert cominittes have been known to err in the both to the EFFECTIVENESS of what we do (the past, and a change towards "optimal" care may not in- extent to which we attain our objectives), and to the evitably improve the health of the recipients of care. It is 190 therefore, that some RMP's try to show that cases prevented are mainly those which will develop essential, symptomatic stroke some five, ten, or twenty years health status is raised when a change in resources is fol ct an immediate and measurable fall in lowed by more optimal activities. The evaluation of later. To expe changing health status has been reviewed in an earlier hospitalization rates or mortality for stroke is to expect workshop, and it is only too clear that this effort is too much of primary prevention. In its first few years, primitive and difficult. It seems likely, for example, that this program must be evaluated in terms of its inter- most outpatient care must be evaluated in terms of ac- mediate activities and short range goals, the early detec- tivities, since few tangible end results exist. tion and effective treatment of patients with hyper- Iease-fatality ratios are high in the acute stage of tension. Primary prevention is liable to be wrongly classi- stroke. An improvement in diagnosis and treatment fied as ineffective if we evaluate it by an immediate fall should be reflected in lower death rates among patients in incidence. in general hospitals. This will not be the only change in The benefits of primary prevention must be balanced end results, but it is the change which is most readily against the costs involved in this process. What must we monitored. Moreover, it is a change which should occur include amon- the costs, in addition to the more obvious at the same time as the change in activities, and will not steps? Certainly we should include the costs involved in be delayed for years after the onset, for example, of diagnosing, the false positives, the referrals who are effective educational programs. We must have a different diagnosed as normal by their physicians. Probably we time perspective for programs of primary prevention, should include the costs involved in diagnosing and however, and I shall discuss this in the next section. treating hypertensives who do not respond to care, or In the field of rehabilitation, many measures exist to who respond adversely to it. And if we wish to be strict reflect changes in the physical status of patients. Most with ourselves, we should also count against the program indices are based primarily on activities of daily living; the cost of diagnosing those who are confirmed to be they range from those which describe a functional hypertensive, but who are given no active treatment; profile of each patient to those which give one overall reassurance, supervision, and periodic office visits have score to reflect the degree of impairment. Most scoring no magical ability to control the adverse effects of an procedures seem to be repeatable, but little attention has elevated blood pressure. been paid to their validity. The fact that no single method has been used widely may suggest that each has COMPARISON GROUPS serious inadequacies. Nevertheless, we cannot wait for If evaluation were partly a research activity, pro- perfection to occur; it is probably true that any one of a ducing new knowledge that can be applied to many number of indices is better than none at all, and can similar situations, evaluators would have to insist on contribute much to evaluating the end results of rehabgi- tative care. strict control groups with whom study groups could be compared. Evaluation efforts have the more practical PRIMARY PREVENTION OF STROKE aim, however, of showing whether or not a specific endeavor is reaching the goals which have been set for it. Primary prevention of stroke involves those measures Its generic value has secondary importance; the evalua- taken to prevent the onset of cerebrovascular disease. tive study does not have to show that other similar From the more distant viewpoint, however, cerebro- endeavors are likely to be effective. Thus evaluators do vascular disease is merely a part of the natural course of not feel compelled to use the rigorous methods and hypertensive and atherosclerotic cardiovascular disease. strict controls of those involved in experimental re- Since these conditions begin at a young age, preventive search. measures before onset are difficult to institute. In Nevertheless, evaluators must show that activities practice, therefore, what we label primary prevention is change and end results improve because of the program the taking of preventive measures before symptoms being evaluated, and not because of an artifact occurring begin of cerebrovascular disease. throughout the region. The evaluation effort must Probably the technic with the strongest scientific usually involve, therefore, a facility or group of patients support is the early detection and active treatment of which have not received the service being evaluated. hypertensive disease. How should we proceed to evaluate Such a comparison group need not resemble the treat- this effort? We must first form the realistic perspective ment group so closely as it must in an experiment.@It that primary prevention is a long-term investment. The must be similar enough, however, to be exposed to the 191 same extraneous factors which could produce the physician takes an intensive two-day course; the nurse' changes under study. "Before and after" studies become has three weeks of training; and the physical therapist' much more successful evaluation efforts when they @how has two. weeks. Af of September 1970, seventeen teams that the change occurred only in the group under study plus selected guests, have been trained in Memorial and did not occur in a somewhat. similar group, perhaps Hospital of Long Beach. located in a different institution or community. The medical faculty to train these stroke teams in- cludes specialists in all aspects of the stroke problem. CONCLUSION 'he paramedical faculty includes all standard rehabflita. To seek a graceful end, perhaps I should tell you that tion disciplines, i.e., physical therapist, occupational around 160 A.D. the Roman emperor Marcus Aurelius therapist, nurses, speech therapist and social service gave this advice: "Thou hast embarked, thou has made workers. the voyage, thou are come to shore; get out." At that After completion of the training program the core time, sailors feared to test the effectiveness of their returns to its own institution to utilize the team navigational efforts by jumping ashore promptly. They approach and to train fellow workers in the method- knew ordy too well the uncertainties and errors involved ology. As a result of this experience, the team members in sailing in those early years, and feared the unpre- have improved not only their own expertise but also dictable welcome that might greet them on foreign their awareness of the techniques of the other disciplines shores. in dealing with stroke problems. In the 1970's, we may still expect some voyagers in The stroke team training divides stroke care and the ships of stroke programs and rehabilitation to be rehabilitation into three phases. The first phase, Phase 1, slow to leave their vessels for fear that they may have provides the supportive care to the patient until his vital reached wrong and hostile shores; even more reluctance signs have become stabilized. This includes passive range to evaluate the situation may stem from doubts that the of motion exercises, proper positioning and meticulous vessel has actually left the port of embarcation; and skin care. The second phase, Phase 11, consists of a multi- perhaps most reluctance to assess progress will stem phasic patient evaluation and implementation of an from realizing that it takes more than a brisk jump active rehabilitation regimen designed to meet the ashore to determine whether we have or have not individual's specific needs. The last level of care, Phase reached our goals. 111, essentially is a continuation of the second phase, but emphasis is placed on the post-hospital needs of the stroke victim. An Evaluation of a Stroke Program The nurse-coordinator is the catalytic agent among in California the various modalities in the stroke team. She visits and BERTRAM L. TESMAN, M.D. assesses each new patient in her facility, initiates Phase I at the physician's request, assists in developing the patient care plan with the attending staff and demon- Area VIII of the California Regional Medical Pro- strates proper care techniques when indicated. In addi- grams consists of Orange County and, for this specific tion she is prepared to complete forms which are program, Long Beach. This area incorporates approxi- intended to elicit data for the stroke registry in Area mately two million people and includes 35 acute hospi- Vill. tals and approximately 7S extended care facflities. To The physician is the medical coordinator of the promote effective treatment of patients with stroke, a stroke team who is responsible for leading the patient training program has beeriset up at Memorial Hospital of care conferences. He serves as moderator at staff meet- Long Beach. Although all disciplines of rehabilitation ings when stroke data at his particular hospital are re- ideally are involved in stroke, the basic core of the viewed and analyzed. He will be available for consulta- stroke team concept as implemented in Area VIII con- tion about the team approach to care of stroke patients sists of physician niirse-coordinator and physical thera- for other members of the medical staff at his facility if it pist. Each hospital in the Area is invited to send these is requested. three members of the health team to Memorial Hospital The physical therapist is responsible for a continuing of Long Beach to take special stroke training; back-up assessment of all the stroke patients in the hospital and teams also can be trained. Hospital administrators also he helps establish their active rehabilitation programs. are encouraged to attend the training session. The He also is available to all staff members for consultation. 192 Presently, one year after the team trainin" was monthly basis, the personnel from the surrounding initiated, an assessment of the stroke teams in Area Vill extended care facilities will be invited. It is hoped that reveals that only one hospital has an active proaram. We personnel in the facilities will become more aware of would like to discuss some of the difficulties and obsta- complete stroke rehabilitation and also that the physi- cles we have identified as a result of the evaluation and cians on the staff of the acute hospital will become coc,- C, we also would like to discuss our resultant plans for nizant of those extended care facilities which are willing increasing the number of effective stroke teams in this to cooperate in giving better care to their patients on Area. discharge from the acute hospital. The problems we confronted when attempting to We also hope to develop a mobile van unit which will initiate the program were numerous. One year elapsed transport a stroke teat'n to the various extended care between the time Area VIII submitted the grant and facilities in our community in an attempt to introduce funds finally were available. This posed a recruitment the phases of rehabilitation that we have been teaching. problem for us. Although I had visited every acute We hope that this demonstration pilot project may seIrve hospital in this Area and discussed the prouram with as a model for other communities to augment rehabflita- administrators, by the time the project was funded many tion care where it is not available. changes had occurred in all levels of personnel. There- In addition, we have instituted a stroke volunteer fore, most of the commitments for placing staff in tile training prooram. Ten volunteers have begun a two- training program were no longer valid when the course month intensive training program utilizing a carefully actually began; so, again, we have to begin a recruitment selected faculty representing all disciplines of stroke program. Moreover, as a result of the change in fiscal rehabilitation. These volunteers will function in a capa- policy in Medical and Medicare fundinc,, there was acity to aid in the resocialization of the stroke patient marked curtailment of available monies to extended care and, whenever pos sible, will assist him in his rehabflita- facilities in our Area. This not only makes it econon-d- tion program under the guidance of the special therapist cally impossible for them to send staff for an extensive following the patient's discharge from the hospital. training program, but also limits their ability to provide In 1969 the Collaborative Community Stroke Survey optimum rehabilitation in their own facility. was begun in seven counties throughout the United Although the team concept in rehabilitation is not States in an attempt to gather pertinent epidemiological totally new to the field of medicine, it is a new approach data concerning stroke throughout our country and in many of the hospitals in this Area. Because of the compare various separate areas. Orange County became emphasis placed on the active involvement of all team involved with this study and we hope to use this data to members some of the physicians reacted to the program help us evaluate our stroke program concepts. The with diffidence. Also, many of the nurses felt uncom- mobile van team also will be recording their efforts with fortable about suggesting the proper level of care to the patients and comparing them with a control group to see doctor as the patient's physical needs changed. if a coordinated team can aid and improve rehabilitation Analyzing all of these difficulties, we believe we now care in extended care facilities. have some practical solutions. First, a follow-up faculty We shall begin a follow-up study on stroke patients is being organized to aid and supervise the already this Fall utilizing a form which was developed by a trained stroke teams in their own institutions. This committee of members of all health disciplines involved follow-up team will consist of a nurse and appropriate in the delivery of comprehensive stroke care. It was therapists to aid and help organize the individual stroke designed to extract the following kinds of information: teams within the hospitals. They will remain in an acute the patient's functional condition, types of medical care hospital for approximately two to three months until and rehabilitation being rendered, social and economic the training of all personnel has been accomplished, conditions, special needs of the patient and his family. team conferences and other aspects of the team Follow-up visits will be made by public health nurses approach are underway and the total team feels from the Visiting Nurse Association of Orange County confident in their activities. They also will discuss the from a random sampling of stroke patients six months entire program at staff meetings to orient the physicians after their episodes, then again at twelve and eighteen in the new rehabilitative techniques. In this way we hope months. to stimulate the physicians as well as the hospital per- It is our feeling that the level of acute care to the sonnel to institute the team approach to stroke care. At stroke patient has improved in our Area as a result of the the conferences, which will be on a weekly or bi- stroke team training. However, we have also made many 193 mistakes in attempting the introduction of the stroke cation, Discharge Planning and Follow-up, Area team as we have designed it. An analysis of our work has Resources Development, and Public Education.) given us approaches to solving problems relating to the 2 .Development of a Basic Training Course for Stroke stroke team. The assessment also has helped us seek new Teams and of an In-Service Education Program for and innovative methods of meeting the health and other professional health personnel of the conunu- rehabilitation needs of the stroke patient beyond the nity. wafls of the acute care facility. 3. Development of an Ih-Service Training Program for paramedical personnel to make them knowl. edgeable in rehabilitative techniques. North Carolina Comprehensive Stroke Program 4. Development of guidelines (organizational, medi- B. LIONEL TRUSCOTT, M.D. cal, nursing, and rehabilitative) 5 .Coordination with State Board of Health Physical OBJECTIVE Therapy Consultants and with Medical Centers for consultative support to the community- To offer the stroke patient increased opportunities 6 .Development of a system to identify the accom. for early diagnosis and treatment, early hospital dis- plishments, problems, and breakdowns. (hospitali- charge, and continued follow-up through a community zation forms, discharge planning forms, follow-up stroke program. reports, etc.) 7. Computeri zation of appropriate data and retrieval Development of the Program for feed-back to community health personnel. Identi .fication of Subobjectives. The objective must 8 .Determination of qualifications and procedures for be reached as a result of accomplishing subobjectives, obtaining a local, part-time secretary. and these must be (a) realistic within the limitations of personnel and time of the average community hospital Program Design It was not considered feasible to and the area it serves, and (b) subject to measurement. involve each community in the planning process of such The major subobjectives thus identified were: a complex program. In consultation with practicing I .A community health team for comprehensive physicians and resource personnel from the three medi- management of the stroke patient: from diagnosis cal. centers and the State Board of Health, the Project through follow-up. Staff accomplished the above activities. To ensure that 2. Professional health personnel knowledgeable in the all necessary steps were completed in correct sequence most advanced methods of diagnosis and treat- for maximum efficiency, a time-sequential work plan ment of stroke. was developed according to the Program Evaluation 3. Increased availability of manpower trained in Review Technique (PERT). rehabilitative techniques. 4. Guidelines for high quality, uniform, total manage- Establishment of a Community Stroke Program ment of the patient. S. Consultative support for communities lacking in I .Community Approval. (a) The aims and proce- specialized personnel. dures of the Program are explained to a few in- 6. An evaluation mechanism to determine the extent terested physicians. (b) The interested physician or to which the subobjectives and activities had been physicians appoint an ad hoc Steering Committee achieved. representing all deliverers of health care; Project 7. Feedback of data to community, for measuring Staff describes details and responsibilities in the impact of program and identifying needs. local program. (c) A permanent Local Stroke Pro- 8. Part-time Executive Secretary to administer all gram Committee is formed, and chairmen of Sub- activities. committees appointed. (d) Members of In-Service Activities. The activities to accomplish each of the Education Subcommittee ("Stroke Team") are above subobjectives were: selected by the Program Committee. 1. Development of an organizational framework for a 2. Educatioti and Training. (a) Stroke Team attends a community stroke program, with clearly defined 4-day Basic Training. Course. (b) Project Staff and areas of responsibility: Local Stroke Program Consultants conduct two In-Service Education Committee with Subcommittees (In-Ser-vice Edu- sessions (2 hours each) for community physicians, 194 and nurses. (c) Project Staff and Consultants con- Computerization and Retrieval of Data duct 5-6 practical sessions in rehabilitative tech- niques (positioning, transfer, ambulation) for Feed-Back to Community nurses and physical therapists. (d) Community 1. Periodic visits hospital nursing staff, with aid of training aids 2. Annual Workshop loaned by Program and help of Project Staff as needed, conduct 34 practical sessions for licensed Summary of Results practical nurses, aides and orderlies. (e) Project Improvement of, and accessibility to the health de- Staff helps plan periodic continuing education sessions. livery system is apparent in the following brief sum- mary: Implementation of Community Stroke Program 1. Community Stroke Progiams presently involve 22 Admission of Patient hospitals and 8 nursing homes, with follow-up con- 1. Nurse notifies Secretary ducted by 19 county health departments. Over 2. Secretary notifies: Project Staff and State Board 915,000 people reside in the counties with local of Health Physical Therapy Consultant. stroke programs. Evaluation and initial orders 2. Education, training, and more effective use of 1. Nurse and physician record admission clinical data, manpower participating in local programs: consultation and laboratory requests on form lb. M. D....................... 125 R. N...................... 390 2. Physician writes Stroke Admission Orders. P. H. N. ................... 103 Treatment ofPatient P. T. ..................... 18 1. Guidelines of Management followed. L. P. N.s and Aides ............. 314 Discharge Planning Conference Others ................... 55 1. Secretary notifies Project Staff and Conference Total ................... 1,005 members of date. 2. Conference held. 3. Altered and improved patterns of care are indi- 3. Copy of Discharge Plan sent to Project Staff. cated by gradually increasing precision and com- PatientDischarged pleteness of clinical and laboratory evaluation, 1. Secretary notifies Project Staff of date of dis- institution of early rehabilitation, more organized charge and of first follow-up. discharge planning, and systematic post-hospital 2. Forms la and lb completed and sent to Project follow-up. Some pertinent facts, from the hospital- Staff. ization forms used in this program, illustrate Follow-Up changes after the start of a local program: (These 1. Project Staff and physician receive follow-up eval- figures are based on 122 pre-stroke program and uation reports. 145 post-stroke program patients.) 'ome Features of Evaluation Pre-stroke Post-stroke -pro,ram program Basic Training Course: Evaluation by participants Patient evaluation cohort cohort In-Serpice Education 1. Evaluation by participants. 1. Blood pressure ...... 71% 96% 2. Pre-and post-session testing. 2. Type and speed of onset .......... 7 6T,, 88% Hospitalization Data 3. Side, severity of weakness ........ 59% 72% 1 . Date of admission 4. Functional ability .... 46% 63% 2. Clinical and administrative data (Hospitalization Forms I a and I b) Use of Multitests 3. Date of Discharge Planning 4. Discharge Plan 1. Electrocardiogram .... 27% 51% 5. Date of Discharge 2. F.B.S./2 hr. p.p. sugar .......... 39% 63% 6. Date of first follow-up 3. Other (skull x-ray, Follow-up Date.' Periodic follow-up reports etc.) .......... 18% 27% 195 Measurement of Health Status (side and'se ty Pre-stroke Post-stroke veri 0 program program weakness, functional abilities, etc.) at admission, di'$ Patient evaluation cohort cohort charge, and at 3-month intervals thereafter is prese available on approximately 200 patients treated acc ing to the Guidelines of Management. These data art now being retrieved for evaluation. Treatment Reduction of hospitalization costs. Comparison of 1. Stroke admission pre-stroke program cohorts with post-stroke program orders .......... 7% 71% cohorts indicate that the latter have a reduced hospital 2. Rehabflitation begun stay of over 4 days (approximately $200 less per pa. within 48 hrs. after tient). admission ........ 0% 22% Mortality within 48 hours 24% 16% FuturePlans Discharge planning done . . . 49% 61% 1. Consolidating gains of participating communities. 2. Stroke Prevention and Surveillance. Scheduled, follow-up care 3. Training additional manpower through new pro. to date .......... None loopts. grams. 196 EVALUATION OF CANCER REGISTRIES Participants Charles R. Key, M.D. - iNf oderator George Linden Assistant Director for Cancer Chief, California Tumor Registry New Mexico Regional Medical Program State Department of Health Abraham Ringel Public Health Analyst, Operations Charles R. Smart, M.D. Research and Systems Analysis Director, Intermountain Tumor Registry Regional Medical Programs Service Use and Evaluation of Cancer Registries value. Most physicians and other users of registry data ABRAHAM RINGEL do not have the time or background to evaluate statis- tical data. A subsidiary value of a cancer registry is its effect in Service-oriented cancer registries are organized and the preparation of complete and accurate medical charts. operated primarily to assist physicians and patients in One way to measure this would be to evaluate the com- the care of the latter. This is accomplished most directly pleteness and accuracy of various items in medical charts with periodic letters to physicians, and sometimes also prior to the initiation of the registry, with medical charts to patients (with the physician's consent) to ensure rou- completed after the registry was organized. Comparisons tine surveillance of the disease. Thus, one measure of the of the information recorded concerning diagnosis, extent effectiveness of a registry is the increasing percentage of of disease, pathology, and therapy for the same sites in successful medical follow-up of patients over time. The the two periods might show significant changes for the advantages of medical follow-up are also reflected in the better. increased diagnosis of additional primary malignancies Examples of measures to detern-dne the effectiveness and recurrent cancers in the early stages of the disease. of cancer registry programs are: Additional services may take the form of periodic I .Improvements in the medical follow-up of patients comparative reports to physicians to evaluate the diag- in each of the participating hospitals; nosis and management of cancer in the community and 2. Improvements in the proportion of cases micro- in the separate hospitals. Patient information by age, scopically confirmed in the participating hospitals; race, and sex by cancer -site and histologic type, by 3. Improvements reflected in the earlier diagnosis of extent of disease (stage), methods of diagnosis, treat- cases by anatomic site; ment modalities, and survival may lead to improved 4. Changes in the length and/or quality of survival, understanding and management of the disease in the by age, sex, race, and socioeconomic group for community. For example, 'the data collected by the each type of cancer; registry may be used to determine the trend in the diag- S. Improvements in the completeness of reporting by nosis and survival of patients with various sites of,cancer. participating hospitals; This information may also be helpful to hospital adn-dn- 6. Improvements in the completeness and accuracy istrators in the development of strategies for optimum of abstracted cancer cases (quality control); operation of their institutions, as well as to community 7. The schedule of participation and compliance with planners to determine priorities and the allocation of agreed upon procedures and definitions by partici- resources for facilities, equipment, and manpower. pating hospitals; 8. The utility and value of the central registry in However, it must. be emphasized that statistical re- intramural and community programs of profes- ,,Ports without analysis and interpretation have little sional and public education. 197 Alternative Methodologies for Evaluation of cancer patients and that this increased survival would' of Registries not have occurred without the activities of the central GEORGE LINDEN cancer registry? I can assure you that this is a very diffi. cult hypothesis to prove directly and conclusively. Let me first express my appreciation for being invited This does not mean,, however, that evaluation is to participate in this Regional Medical Program special ii-npossible; cancer registries can be evaluated by session on evaluation of cancer registries. Let me also dropping down to a lower order of evaluation. There are make clear that I am not in any way an expert in pro- many areas in our work and personal lives where com. gram evaluation. I am here today because of my back- plete scientific proof of a given hypothesis cannot be ground and experience in the organization, operation obtained but where the preponderance of evidence leads and use of a central cancer registry. My first impulse, us to what we regard as a reasonable'conclusion and when Mr. Ringel invited me to participate, was to back action can be taken. on the basis of that conclusion. For off as fast as I could; all I could think of was "I don't example-one of the goals of most central cancer regis. know how to evaluate cancer registries." But Mr. Ringel tries is to provide data and information that is useful to is very persuasive. He accepted my statement and then the medical community in its cancer educational activi- went on from there to convince me to participate in this ties. These data provide a resource for the physician in session. describing and analyzing his experience or his hospital's Evaluation itself is not new to me. My training as a experience and can also be used as a basis for clinical and statistician and my position as Chief of California Tumor other studies. I would not want to take on the task of Registry for more than fifteen years have forced me to proving conclusively and scientifically that the use of the be continuously aware of the problem of evaluating data for medical education did in fact assist the cancer what I was doing or attempting to do. Most of it has patient. Conversly, however, there would be little dis. been informal-the one formal evaluation having agreement with the assumption that the continuing occurred when I first joined the Registry staff. The education of physicians who are diagnosing and treating Registry, which had been operating for seven years, cancer patients will help the patients with cancer. If we underwent a thorough evaluation of its activities. This can accept this as a reasonable assumption, we can then resulted in the deletion of many items which were say that one of the goals of a cancer registry is being met originally thought to be "nice to know about" and some when we provide physicians with these data. The next which were important but not obtainable and also in- step is to decide whether the registry is in fact providing volved some basic changes in procedures which made the such data, and here we are on much flrmer ground. We Registry more efficient and better able to meet its goals. can review the activities of the registry and determine Our purpose here is to discuss means of evaluating whether the registry has or has not provided such infor- cancer registries which have been developed as part of mation for the use of the medical community. We can go the Regional Medical Program activities. Any such eval- one step further and try to determine whether in fact these data are being used and how they are being used uation must, of course, go back to the purposes and the goals for which the registries were established. These will by the medical, hospital, and public health community. differ among the various operations and each registry Another example of the evaluation of a cancer will have to be evaluated in terms of its own precise registry on what I call a secondary level has to do with purposes and goals. There is, however, a common goal the following: Can we prove conclusively that medical that underlies the activities of all cancer registries and all follow-up increases the survival of the cancer patient? cancer programs and that is the benefit to the cancer Obtaining such proof may be possible (it certainly would patient. The primary question therefore becomes: "What be difficult to do) and I've heard the statement chal- effect does the registry have on the cancer patients?" lenged. I think I am a reasonable person. I think it rea- Since the survival of cancer patients is usually the focus sonable that medical follow-up of cancer patients will of our measurements, the question can be narrowed to: result in longer survival than the survival of patients who "What has the registry done to improve the survival of receive no medical follow-up after their first course of cancer patients?" treatment. I am willing to accept this assumption and It is precisely this question which led to my initial therefore would accept an increase in the level of reaction of pulling back and saying I didn't know how to medical follow-up of cancer patients as evidence that the evaluate cancer registries. Can we prove that the activi- registry activity had'been beneficial (being also hard ties of the registry actually led to the increased survival headed, I would want to see evidence showing that it 198 was the activity of the registry which had increased the Is the codiiic, and classification of the vario C, us pieces level of medical follow-up). of information entered into the registry system accu- We are on much firmer ground when we consider an rate? Is there any check on the quality of coding? Is the item of evaluation which the American College of Sur- data processing system working as it should? Are the geons is planning to introduce as a requirement in their data being processed accurately and on time? Are the cancer program-that is, the quality of survival. I think it computer proalraiiis for processing and retrieval of data would be easier to prove and certainly easier to accept functioning properly? Can data be obtained quickly and the hypothesis that continued medical follow-up is bene- at minimum cost? ficial to the well-being of the cancer patient. How current are the data? Are the hospitals reporting The point of these remarks then is that while we may cases early enough or are they lagging behind in their not be able to prove directly and conclusively that a abstractiti,-? How good is the follow-up system? Is it cancer registry increases the survival rates of cancer workmen, as originally planned or are there difficulties in patients, we can on reasonable grounds show that the carrying it out? What proportion of patients are actually successful operation of a cancer registry does benefit the followed? What proportion of patients are followed cancer patient. medically? The Regional Medical Programs are a comparatively These are sonic of the questions which you will be young operation and the cancer registries organized as concerned with in evaluating the effectiveness of a part of this program even younger. Most of the registries registry operation during the first organizational years. were organized during the last couple of years. It may Although the accumulation of information on sur. therefore be a little premature to attempt to evaluate vival may take a number of years, it is still possible for them in terms of the final goals for which they were the registry to feed information back to the hospitals, established. It may instead be necessary to look at the the medical comnilinity, and the individual physician (if registries in terms of their developmental goals during this is part of the reporting process) during the early their early organizational years. It is obviously not years of the operation. It can fairly early provide basic possible to judge a registry in terms of publication of information on the demographic characteristics of the five-year end-results if it has only been in existence for cancer patients, their cancers (site, histologic type, stage, two years. The factor of time therefore becomes im- etc.) and treatment. Information on stage of disease can portant. How long has the registry been established? If it provide an estimate of the level of early diagnosis of has been in existence for only one year or two years, cancer. This can be used to support a program to what goals were specified for completion within that one improve the level of early diagnosis and bring patients to or two year period? treatment earlier. Evaluation will therefore probably be in terms of I don't want to exaggerate the output that a registry technical goals. Is the registry system itself organized can produce in its early years. A registry's usefulness and operating? Does the registry have properly trained increases with time and the early years are a time of staff? Have the details of the operating system been limited output. What is most important of course is that worked out? Have they been documented? Is the system the community and especially the physicians be in- actually working? Are the various parts of the system, formed of the progress of the registry. and be the recipi- hospitals, physicians, etc., cooperating fully? ents of early output of information. This is an important Has the registry developed suitable forms for obtain- point for evaluation. Has the registry produced any ing the original data plus a handbook of instructions for data? If so, has it been disseminated to the medical, those who are charged with obtaining the data? How hospital and lay communities? How has it been used? good are the data being entered into the system? Does I would like to take a few moments to stress the the abstract or other form on which the data are entered documentation of activities carried out by a central accurately reflect the patient, his cancer and his treat- cancer registry. At the beginning of my talk I mentioned ment? What educational means are being used to insure the informal evaluation which occurs almost continu. that the personnel in the hospitals abstracting the data ously. On occasion it becomes very immediate and are trained and knowledgeable? Are workshops being important. We were asked, several weeks ago, to provide conducted to assist these people? Has a program of documentation regarding the value of our activity to the quality control of data been instituted? Is there review Department's program. I was told, at 3:30:@P.M. @on a of incoming records and independent abstracting of a Thursday, that the documentation was sample of cases to insure a high quality of data? before noon on Friday. We have, during C of operation, developed material which can be readily A few words about what one writer of the material I used for documentation. I wrote a very short statement received in preparation for this session called "dynamic regarding our program and attached to it two of the evaluation". I agree wholeheartedly that the evaluation documents which we had developed. One of these was a process should not be static. The placing of a value on any part (or all) of the registry system should be fol- Progress Report which is compiled every six months. We originally started this to keep track of and to evaluate lowed by the inquiry, "Does the evaluation indicate that our own activities. We have since found it useful in many changes are necessary to improve the situation?" If so, other situations. A copy of this report is available for the evaluation should at least indicate the necessary observation on the table. The other document I attached changes and possibly initiate action to make the changes. was a list of publications which the Registry has pro- Maybe we should propose that a future evaluation be dliced. I believe this kind of documentation is extremely made of the effects of the present evaluation. Was it important in evaluation of a cancer registry. worth the time and effort? Did it really result in an The Progress Report for January-June 1970 shows improved registry programs the number of cases received during the first half of What I have said today is certainly not exhaustive in 1970 and the total as of June 30. It also shows the status of current follow-up efforts, including the number of terms of evaluation of cancer registries, but I hope that cases in active follow-up, how many died, the number the combination of your own discussions on program actually followed and how many were medically ex- evaluation and our presentations and discussions here aniined. The report also contains a detailed description will make it possible for you to evaluate the activities of of the requests for data which were completed during your own cancer registry program. this six month period. There were a total of 57 such requests and I believe that the listing of the individual requests constitutes evidence useful in evaluation of the Methodologies for Evaluating Effectiveness activities of the registry. The annual reports which we and Value of Registries provided to hospitals during this period are also de- CHARLES R. SMART, M.D. scribed. There is a section on the Hospital Data Books which we developed this year for each of the partici- Incidence and Epidemiological Registries study the patiiig hospitals. The Data Books provide a compre- differences in geographic, racial, religious, environ- hensive and clearly presented account of the cancer mental, social and economic groups seeking etiological experience of each hospital and are a solid example of factors leading to prevention. the usefulness of a central cancer registry. A copy of the End Results Registries study survival to determine Data Book is also on the table. The ProIress Report also national baselines and to monitor change in survival 9 includes a description of a number of studies in which rates. we were involved during this period, a listing of two new The Clinical Cancer Control Registry has in the past publications, a description of the future plans of the been hospital based and patient oriented, attempting to Registry, and an account of our activities with the control cancer through encouraging life-time interval Regional Medical Programs in California. It also covers a follow-through examinations on all patients having had proposed central cancer registry in Los Angeles County; cancer. Through the regular follow-up examination it is hospital consultation, training and lectures carried out hoped that recurrences and second primary cancers will by the Registry staff; the activities of the Alameda be discovered at a time when they can still be cured. County population based Cancer Registry-, work per- This type of registry also seeks to serve as a self-evalu- formed under contract with the National Cancer Insti- atory and educational mechanism for both the hospital tute's End Results Group; the Third National Cancer staff and individual physicians. Survey; and a list of visitors to the Registry. The Pro- While these various registries are emphasizing one gress Report has developed from very modest beginnings phase of the problem or another, their functions greatly to a very useful tool for orientation and for documen- overlap and their goals can be summarized under the tation of the activities of the California Tumor Registry. headings of service, education or research. Other evidence and documentation of the Registry In October 1965, Congress passed Public Law 89-239, activity is available for your review on the table; there known as Title IX - Education, Research, .Training, and are also sign-up sheets if you want copies of any of the demonstrations in the fields of Heart Disease, Cancer, material. Stroke, and Related Diseases, encouraging through 200 @rants the development of cooperative arrangements Cancer Control depends upon both Physicians and @mong medical groups and institutions in making avail- Patients. The physician is busy and must find the inter- Lble to their patients the latest advances in the diagnosis action with the Cancer Control helpful and satisfying - ind treatment of these diseases. This bill has given rise to the re-enforcenient must be meaningful. Knowledge, -nany new clinical cancer control types of registries to skills and attitudes developed on the part of the physi- assist in clarifying the local problem with solid facts, . cian will be transferred to better patient care. Evaluation :hus allowing logical planning of needed programs and feedback from the physician and from patient care assuring a greater continuity of re-examination of cancer should be utflized in adjusting the prograi-n's methods or ,)atients. We shall concentrate upon and describe objectives. -nethods of evaluation of this type of registry. By modifying the evaluation measurements used in OBJECTIVES AND EVALUATION educational programs (attendance, opinion, gain in knowledge, change in behavior), one can develop the Analysis of goals and objectives must precede eval- following parameters for the evaluation of cancer Liation. registries: I . Decide upon the goals you intend to reach at the I .Participation end of the program 2. Opinion or Attitude of the physician 2. Select the procedure, content, and methods which 3. Improvement in life-time interval follow-through are relevant to the objectives examinations 3. Carry out the program 4. Improvement in patient management 4. Measure or evaluate the performance according to 5 .Improvement in patient survival the objectives or goals originally selected. EXAMPLE OF PROGRAM EVALUATION PROGRAM DEVELOPMENT Utah Tumor Registry In October 1966, the tumor registry of the,lnter- mountain Regional Medical Program was formulated on paper. The registry was an integral part of a compre- hensive cancer control program involving clinics, semi- nars, telephone - radio - TV programs, etc. The general concepts are depicted in the diagrams on the following page: The objectives of the Cancer Control Registry were: 1 .To survey and to establish local baselines. 2. To provide local practicing physicians with accu- rate, meaningful feedback. 3. To save lives through the systematic follow-up of all cancer patients. 4. To identify deficiencies and design operational DYNAMICS OF PROGRAM EVALUATION projects accordingly. 5. To evaluate operational projects. At first the importance of No. 3 was not completely appreciated. It now heads the list. Methods 1. Gain the support of the medical profession, hos- pitals, health department, cancer society and other in- terested health agencies. 2. Enhance presently existing hospital tumor regis- tries by providing: a. Meaningful listings of their patients' data Accentuate the positive Eliminate the negative Eliminate the negative but alone leads to stereotyping b. Survival reports by site and stage (PROGRESSION) (STATIC) 201 Cancer New Approach Cancer Education Patient Survival in Communication Cancer Education IONAL valuati MEDICAL PROGRAMS Resi rigin Ca.Pts Hospital T,eatment Centers Physician Load & O.ality PATIENT SURVIVAL EDUCATION blk Delay/ Education