01 * 'I CONTENTS - VOLUME I TOPICS PAGE Operational Grants Robert Q. Marston, M.D. 3 Emetging Issues in the Review of Operational Grants Karl D. Yordy 15 Data Collection: Measuring Progress in Reqional Medical Programs Maurice E. Odoroff 31 Data Collection Margaret H. Sloan, M.D. 37 Continuing Education William D. Mayer, M.D. 45 Some Aspects of Communications and Public Informat or Regional Medical Programs Edward M. Friedlander 57 Grants Management James A. Beattie 71 Planning, Evaluation, and the Report to the President and the Conqrbss Stephen B. Ackerman 77 OPERATIONAL GRANTS Delivered By: Robert Q. Marston, M.D. Associate Director, NIH and Director, Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's ouse Bethesda, Maryland June 16-17, 1967 -3- This morning, as you see from your agenda, members of the staff of the Division of Regional Medical Programs have iden- tified some selected items for discussion with you. These vary from items that are very large and represent global problems down to small, almost- housekeeping, items. The presentations are not meant to be comprehensive or to represent full reports. Rather, they are items that have been selected out for discussion in the attempt to give some selected messages. The first area that we would like to focus on has to do with operational grants. This is the most important thing that is happening, not only as far as the Division is concerned but in Regional Medical Programs throughout the nation as we move into the next fiscal year. Robert Oppenheimer has a pertinent quotation. He says, "Both the man of science and the man of art live always at the edge of mystery. Surrounded by it, both always, as the measure of their creation have had to do with the harmoni- zation of what is new and what is familiar, with the balance between novelty and synthesis, with the struggle to make partial order in total chaos. They can, in their work and in their lives, help themselves, help one another, and help all men." Similarly, as we move more fully into the problems of the operational grants and of this not uncomplicated program, we will be at the edge of mystery much of the time and will experi- ence the type of excitment reserved for explorations in importan areas. Before moving to a specific discussion of operational grants, I would like to comment on the background, the national perspective, and the activities related to the implementation of this program during the next year. A number of recent and perspective events reflect the grow- ing awareness within the Federal extablishment and in academia of the problems on hand in the further development of the relation- ship between the Federal government and the scientific community. This is not a new subject but I think it is plain there have been a number of activities that give particular emphasis to this discussion. One of the points is the President's April 6 report to Congress and his 16th Annual Report of the National Science Foundation. In his covering letter, the President was at pains to emphasize the support of basic research. He described scien- tific research as "the key with which we unlock the doors of the future." He referred to the need for "a large and constantly replenished pool of basic knowledge and understanding" and asserted that such knowledge is not a laboratory curiosity but a critical tool for our national health and our national progress. -5- science (may 5, 1967 issue), in commenting on this strong statement of support for research, pointed out that there seemed to be some discrepancy between the President's praise and the budgetary decision, but it went on to point out that the view of the White House however is that, in a time of tight budget, re- search has fared relatively well in comparison to other Federally supported programs. This indeed is the case. In virtual every field of research, cience says, the Johnson administration as annually increased the amount of Federal support. The problem is that the pace of increase has not kept up with appetites of old-time recipients and the hopes of new competitors who share the money that Federal government provides for research. In the same issue of Science, Dr. Donald F. Hornig, the President's science advisor, in an address to the American Physical Society on April 26, talked about the breath-taking pace with which the United States filled the post-war vacuum in ts support of science and observed, "What has changed now is not that there are restraints to be imposed on science either by the Congress or by the Executive, but that the initial vacuum has largely been filled and a new situation has arisen which requires new thought. "The question is not whether we should have basic research or even whether it shall continue to grow--but rather in what ways and for what purposes it should be expanded." He goes on to say that "we are determined to make use of every bit of -6- availabl@ knowledge whose application is feasible, economic, and useful" but "it does not follow in the slightest that this implies A decreased interest in basic research." The two activities are separate, both important, and both will go forward. Now there are many examples within the scientific community, within Government, within the health field, of ferment and one can choose-from a number of them. There have been meetings to discuss problems that will arise from the anticipation of the increased needs for graduate students. It has been estimated that this will amount to a tripling of enrollment of these students within the next ten years. There have been meetings with the Department of Defense on this subject; there have been meetings of the Presidentis Science Advisory Committee. The Federation of American Societies for Experimental Biology at its meeting in Chicago this spring asked Dr. Shannon to speak on the relationship between science and Federal programs,, and at that time.Dr. Shannon suggested a change in circumstances would require not, only differ- ent Federal support devices but new academic organizations and the possibility of new institutional plans. In Regional Medical Programs, new organizational structures are being devised to meet new needs. These discussions have not only been concerned with space and dollar support but have explored questions of basic institutional support. The point is that this is a time of ferment in a whole variety of areas in the health field. -7- We have had our hearings before both the House and Senate. The House has made its recommendations. The Senate has not yet made its report. We do not know what the budget will be for fiscal year 1968. The House Report of the Appropriations Committee is important for several reasons: "The bill includes $54,314,000, a reduction of $10 million below the amount requested, and an increase of $9,310,000 over the amount appropriated for 1967. "The regional medical programs represent a new and innovative effort to make the best practices of modern medicine more readily available to all the American people, wherever they live. This goal is being achieved by linking medical research, medical education, and medical practice and service in a voluntary partnership for speeding up the transfer of advances in diagnosis and treatment from the research centers of the Nation to local practicing physicians and community hospitals. "During the past year grants totaling nearly $25 million have been awarded to'establish and support' planning activities for 44 regional medical programs." And then it goes on to describe some of the things being supported under planning and operational grants. "The committed is concerned about the degree to which the regional advisory groups, that oversee the programs, provide an opportunity for local physicians, local hospital administrators, and other local health, medical, and civic and public groups outside the large metropolitan areas, where most of the programs have their coordinating headquarters, to play an active role in advising on the development of the programs. The committee has been assured that all applications for planning grants are studied closely for evidence of broad geographical, professional, and civic and public representation on the regional advisory groups and that the role of the regional advisory group is explored thoroughly before any grant is awarded. However, the committee intends to continue to keep a close watch on this matter of local representation and local involvement in the programs for it has reason to believe that it is not always adequate to assure the best possible program. he introductory section on "As explained in t ittee has the National Institutes of Health, the comm reduced the budget request by $10 million. In addition to the appropriation of $54.3 million, the Division of Regional Medical Programs will have available at least $21 million in an unobligated balance carried forward from fiscal year 1967, because of the program running behind schedule. The total of $75.3 million that will thus be.available for fiscal year 1968 is more than double the amount available during the current year. The committee is convinced that this amount will be sufficient to cover the actual needs of the program. The committee wishes.to emphasize that the reduction is based on its assessment of the probable rate at which new commitments will be made and does not reflect any lack of enthusiasm for the program. On the contrary, the committee is thoroughly convinced of the great importance of this innovative program to the health and welfare of every American. The concept of regional medical programs must be made to work, and no effort should be spared to insure that it does.11 There are several things in this House Report that are very important. The emphasis on the expectations of Congress that this be a program that not only reflects local initiative and local control but has the mechanisms whereby the needs outside of both the metropolitan areas and the large medical centers can be expressed came through very clearly. The pace of development was one that was expressed in the words "the program is behind schedule.'t This raised a fair amount of discussion in the National Advisory Council. There has been a strong feeling in the Council and indeed a strong feeling from many of you that the emphasis be placed on a sound start rather than on a rapid start. A public witness appearing before the Senate Appropriations Committee made a very strong -9- statement that the program has moved not only rapidly, but more rapidly than he had thoug)-it possible, and as rapidly as it should have moved during this initial stage. I think pace is something that we probably do want to spend some time talking about because this is one of those areas in which it is very easy to get messages badly confused. This is a program which must emerge over a period of many years. But there is the need for definition of goals, the ability to describe more clearly where one is heading, and examples of the various ways in which different parts of the country plan to achieve the goals of this program. I think the House Report gives emphasis to this need over the next few years: to demonstrate clearly that the goals are clear, the concepts workable, and the commitments at the regional level firm. (After Mr. Yordy's presentation, which appears on page 15 of this volume, Dr. Marston made the following additional remarks re- lated to operational grants. Since there is a clear continuity, between these and the preceding, they are included in this section.) -lo- ADDITIONAL REMARYS May I add that at times in the review process it may be determined that some types of projects are just outside the scope of Regional Medical Programs and can not be supported. In addition, some may be judged to be below minimal national standards. Martha Phillips and Karl Yordy have outlined the peer judgment dependent review process we use. Indeed the Surgeon General cannot make an award unless it is recommended by the National Advisory Council. There is another responsibility which must be assumed by the "program operators.11 That is the administrative responsibility for final action on Council recommendations for program direction, for implementation of policy, program management, etc. We would share this responsibility with those of you in this room. In the year ahead the group assembled in this room will be determining the degree to which the concept of regional medical programs can be achieved. I don't mean that the work will be done in a year, or by this group, but rather that existing pressures are such that if the goal is not clearly defined as obtainable at some time by the processes of Regional Medical Programs there will be an irresistible need to seek alternative ways of achieving some of these goals. The basic needs expressed by the passage of the law establishing Regional Medical Programs and the strength of the language in the -11- House Appropriations Committee Report are clear evidence of p-res- sures of great magnitude. A very serious charge has been placed on the Regional Medical Programs. Next year will be a critical year. One of the things that we will be concerned with is the protection of the flexibility under this law, because the pressures always Are to make rules to decrease flexibility. Mr. Yordy has indicated some areas in which we will have to resolve policy issues. We all should be aware that policy statements, although necessary, may substitute for good judgment. It is in this sense that I want to turn to the 1965-66 Annual Report of the John and Mary R. Markle Foundation. John Russell writes here about the prob ems of bureaucracy, a subject particularly important to this program and particularly important during the next year: "'You should have a rule about it,"a chairman of a department advised us this spring. 'He should not be allowed to have that much money left over at the close of the grant.' our response was to suggest that his medical school make the rule, if they wanted one--that rules are an abomination to a foundation and the quickest way to turn it into a bureaucracy. "We hear a lot of,talk about the government and bureaucracy. An independent foundation can become just as rigid and stuffy as any government bureau ever thought of being. Even our Foundation with a staff of only five tan become as autocratic, fussy and thoughtless as any large bureaucracy if we allow ourselves to be wound up in red tape, demanding reports (ten copies) at every turn and imposing our views on our grantees through rules and regulations. -12- "To keep free of rules, and hence bureaucratic management, is no easy matter. The Directors, when setting up a program (such as Grants for Scholars in Academic Medicine) and voting grants to implement it, have a definite objective in mind. In the case of the Scholar program, the objective was, and still is, to advance academic medicine by encouraging bright, broad-gauged young men to become permanent members of medical faculties. The Directors know these men personally and thus are interested in their progress and welfare. As we said earlier, to make the program flexible and as useful as possible, the Directors have placed the responsibility for the Scholar's develop- ment and the details of the administration of the grant in the hands of the dean of each medical school. Local responsibility, the Directors believe, is pre- ferable to management from an office in New York or Washington. Men familiar with the local situation should be in.charge. "This policy gives effect to a noble idea, but it is not an easy one for a granting agency to live with. Freedom of management can lead to abuses, or at least to local plans that do not follow the original objective of a grant. It is particularly difficult for our Directors to stand by and watch what they believe to be an abuse of one of our Scholars. It is only human for them to want to make a rule which would put an end to that particular problem forever. Yet, to do so would send us down a toad that we do not want to travel. "For nearly twenty years we have opposed any action which would turn us into a book of rules. We have always admitted that some errors have been made with our grants--that at times our money has been wasted. But rather than make a rule we have pre- ferred to bide our time, remembering that we do not have to make a grant to that institution the next time. We do not have to make the same mistake twice. Such patience on the part of a foundation, as we intimated earlier, is not easy to maintain. However, it is worth every bit of the effort in order to keep a program from being stifled by bureaucratic regulations. Freedom in its granting powers is still possible for an independent foundation whether it be large or small.,, -13- goes on to use this analogy to talK And then Mr. Russell about the problem that occurs when Congress focuses on a program for broad objectives and when in the implementation of the program these objectives are ignored. He comes to the conclusion that the implementation of the program must be in terms of the broad objectives rather than in terms of the individual needs and wants at the moment. I think this is one of the areas that the Regional Medical Programs is going toIbe tested on very sternly during the next year. You will have a very major part in determining whether it is possible to implement the objectives of Public Law 89-239 in terms of Congress' goals rather than in terms of segments which may fit particular needs. We will be developing within the Division of Regional Medical Programs, as rapidly as possible, those resources that we anticipate will be needed by you. For instance, Dr. Robert Bucher, Dean of the Temple University School of Medicine, is going to be with us for the next six months working with a small team, which indludes algroup at the University of Michigan under Dr. Galliher, to focus on the broad strategic use of operations research and systems analysis in Regional Medical Programs. We are pleased to have Dr. Charles Hitch, experienced in these areas in the Department of Defense and now Vice-President of the University of California, on our Council. We will be focusing on strengthening within the Division a number of other areas such as ability to relate with groups involved in research in health services in other parts of the Government and elsewhere in the country. -14- EMERGING ISSUES IN @HE REVIEW OF OPERATIONAL GRANTS Delivered By: Karl D. Yordy Assistant Directb@ Division .of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Bethesda, Maryland June 16-17, 1967 -15- This is one of those topics that could be global because at this stage in the beginning phases of Regional Medical Programs, all the issues are still emerging. Therefore, in the remarks I'll make, I think we will have to keep in mind that this is certainly not a comprehensive discussion of the issues and implementation of the operational phases of the Regional Medical Programs. What I will do is pick out a few of the most pertinent issues that seem to affect the process of review and approval of operational grants as we see it up to this point in the develop- ment of Regional Medical Programs. We also have the programatic objec- tive that a discussion of some of these emergency situations may be reflected in operational grant applications that can be considered more effec- tively and efficiently in our review process. Later Mrs. Phillips will describe the review process for operational grants as we have worked it out up to this time. One reason for talking about the emerging issues before we talk about how we intend,to review operational grants is the already apparent fact as one would expect in this program., that this re- view process is going to have to change and alter and be modified as new problems arise. Need for Flexibilitv to Respond to Different Approaches We have noticed in some of the subsequent applications that we have received for operational grants and in discussion with people around the country that there are a number of different approaches being developed around the country to the implementation of the operational -16- have phase of the program. This is, of course, what we expected-, and we attempted to maintain flexibility in the program because we have assumed that the operational phase could emerge in different ways with different operational strategies for entering the operational phase. As a result we face the difficult problem of designing a process which is flexible enough to respond to different kinds of approaches. It would be simple for us to determine at this level exactly what kind of approach ought to be taken and then communicate that message to the regions. Everybody could then come back in with applications that followed our ideas and the review process would fit the applications. But what we want to do is have a process that can effectively review and reach decisions on applications which in fact take different kinds of approaches to the development of Regional Medical Programs. - There are,, however, a few essential criteria that have been noted publicly in Dr. Marston's talk at the January Conference in which he discussed the review of operational grantsi We believe that we followed through with these basic criteria in developing the process for re- view-of operational proposals.* This process was developed on the basis of our initial experiences in reviewing the first applications received. Those first applications were used as a learning process. One clear need did emerge from our initial experience--the need for operational applications to articulate the overall approach being used by the regions in developing the operational proposal 1 addition to the de- scriptiot and justification for particular projects. A clear articulation *This refers to an exhibit in the Report to the President and the Congress and will be distributed when available. -17- of these overall factors would facilitate the judgments on the first categories of consideration that we outlined in January--whether or not there is a Regional Medical Program emerging in this application rather than a list of projects which have no relationship one to another, whether there is an overall conce tual strategy which has guided de- .p cision -making and priority choices,the soundness of the administrative framework and finally whether there is coming into being the regional cooperative arrangements.referred to in the Law which will make possible effective implementation of the Regional Medical Program. In reaching the initial determination on these factors we are willing to accept a considerable variety of approaches to the regions'presentation of these overall qualities of their Regional Medical Programs; but it wou cer- tainly make the review process more efficient in addressing itself to these issues if these kinds of factors are articulated in the application. Even though a site visit will be required in making the final determination, some articulation of the overall philosophy and concept of the 'regional medical program lifts from the reviewers the burden of distilling the overall strategy from the array of specific projects Dresented in the application. It has seemed to us, from our initial experience, there are pro- bably several ways to present the sufficient basis for making these overall judgments. One approach would be the evaluation of the readiness to mount the operational program on the basis of the judgment derived from a critical mass of projects that have been submitted. A sizable complex application enables the reviewers to look at the types of activities which have been proposed and the resulting pattern and to reach some judgment on the decision-making process which went into the selection of those activities. The particular projects would be viewed as concrete evidence of the soundness of Regional Medical Programs framework and strategy being used. A second possible approach is the evaluation of the readiness to mount the operational program on the basis of an effective presentation by the region of the framework for that program --the planning, the conceptual strategy, the decision-making process, effective administrative structure, the cooperative arrange- ments --even when the application itself contains relatively few projects or specific activities. In other words, it is probably not necessary, in order to reach these overall determinations, to have a vast complex proposal submitted. On the other hand, in the absence of a proposal t t shows by its composition the shape of the Regional Medical Program, there would need to be a very clear presentation of the framework-and strategy of the Regional Medical Program from which the future operational specifics will emerge. There are some pitfalls in either one of these alternatives. A region following the first alternative might be tempted to submit a great mass of projects saying in effect to the reviewers, "Now here they are and when you go through them you can figure out what we had in mind." Indeed, they probably could, but this would make the process more difficult and would probably cause some unnecessary spinning of the wheels. -19- The pitfall of the second approach is the presentation of a few small operational proposals along with detailed description of a highly theoretical structure to be used by @he Regional Medical Program in developing the program which did not provide for the reviewer a satis- factory basis for determining whether this was a workable process that would be really effective in making good decisions. Whichever alternative is chosen, the articulation of the logic and the process and the structure of the Regional Medical Program will make more effective the kind of review process to be described by Mrs. Phillips. The reviewers will be able to determine more readily whether they are looking at a Regional Medical Program or a long list of projects. It has been clearly said to us several times, by members of Congress among others, that if the Regional Medical Programs are in fact just a list of projects,the rationale for the Regional Medical Programs approach will be difficult to uphold. The scope of authority in other parts of the Public Health Service is broad enough that probably any specific isolated project could be formulated in such a way to be eligible for support through another mechanism. So the Regional Medical Program has to have something more to it than just a list of projects, however worthwhile those individual projects are. Expansion of the Initial Operational Program A second issue that has been emerging is the type of process needed for reviewing additional operational activities submitted subsequent to -20- the award of the first operational grant. We have said previously that the building of a Regional Medical Program will take place in incremental steps, that a Regional Medical Program will rot emerge full blown at one point in time. Many persons have looked at the first operational awards seeking a full description-of what a Regional Medical Program looks like and we have cautioned that in all cases the first operational awards constitute the first step in the development of a Regional Medical Program. I believe that none of those applicants would have wanted to say that what they proposed was in fact a fully developed Regional Medical Program. Building by incremental steps means that the Division is re- ceptive to supplemental applications that propose the additional steps in the building of a Regional Medical Program after the first award has been made. Even though the first award showed the review groups' satisfaction with the initial proposals, we again have a need for several types of information in the supplemental application. When that supplemental application comes in, in order to make it possible for the review groups and staff to cope effectively with rather complex proposals. it would be extremely useful to the staff and reviewers to have presented in the application evidence of the relationship of those activities to the previously approved program and to the overall concepts that were articulated in the first proposal. However worthy the specific projects and however sound the process by which these additional projects were generated at the regional level, our review process needs -21- to have the basis for establishing a relationship of the additional steps to the steps already taken and to the overall process which was e. This is almost essential to prevent our review described at that tim process at the national level from becoming a project-by-project consideration. It will become a very different type of decision-making process at the national level if it has to become a project-by-project consideration rather than essentially a monitoring of the soundness of a development of a Regional Medical Program in that region. We feel that it is very important to maintain the right kind of relationship between the national review process and the regional review process, but we need the help of the applicants in presenting the applications in a way that enables the national review process to play its appropriate role, preserving for the regions their essential decision-making re- sponsibilities. One of the byproducts of this approach for looking at supplemental requests is that our review groups will use the review of supplemental requests as an opportunity to look at the total progress of the program. They will not look at the supplements in isolation. Do not be surprised when the review process in looking into supplemental requests for operational activities looks at the total progress of the program. An additional implication of this approach is it is going to be difficult tb attract the attention of the review gr6ups to isolated or very small proposals if they are to keep their attention on the overall consideration of the review of the Regional Medical Program and -22- not to replace the detailed decision-making at the regional level. If the supplemental applications are too inconsequential, a strain will be placed on the nature of the review groups' function. Administrative Approval of Small Increases in Grant Amounts Another emerging issue to which we have been giving some thought is the provision of some kind of flexibility for the grantee, in the operational phase, to receive small increases in grant amounts through Administrative approval. Now there have been two types of increases identified. One is the need, at the regional level, to support small new projects which are too small to justify full consideration by our review groups for reasons mentioned previously. The second is a need to provide for some kind of limited expansion of activities previously approved when the need for the limited expansion has been justified to the DRMP staff. One of the questions on which we haven:'t reached a firm conclusion is what the role of the Council in providing this kind of flexibility will be. The Law does, of course, give certain legal powers to the National Advisory Council in terms of making recommendations on grants. We are continuing to develop our approaches to these needs and intend to present our recommendations to the Council in August in order to work out with them the right kind of approach to this flexibility. In order to maintain the legal responsibility of the National Advisory Council, it seems necessary to put some kind of limits on this flexibility, limits perhaps in terms of the percentage of the total activity already approved and/or limits the size of any indi- vidual or specific change in operational grants. These limits will -23- probably need to receive the specific approval of the National Advisory Council so that they are aware of the extent of delegation of administrative responsibility being given to the Division staff. As with most attempts to provide some degree of flexibility, there is the danger of abuse I believe that the successful maintenance and trans- ference of responsibility to the regional level will be made possible by a responsible use of such flexibility by the regions. The surest way to cause withdrawal of such flexibility would'be a feeling by the Council that the flexibility was being used to evade its essential responsibilities for review of program. Feasibility Studies Another issue which has emerged in the review of applications is the matter of feasibility studies. Many of you will recall our early experience with feasibility studies. The term "feasibility studies" is in the law, and there's no doubt that we can provide grants for feasibility studies, in a way that does not make feasibility studies a premature commitment to operational activities. The reason to avoid feasibility studies becoming a backdoor approach to the first steps of operations is the potential for erosion of the review process for operational grants. However, we have identified as the program has developed and emerged the need for some kind of capability for a region to undertake a limited exploration of a.potential operational activity as part of the plan- ning process. This study could be used as an aid to the decision as to whether this activity will be proposed for the operational phase of the program. In orc -24- for us to "resurrect" the concept of feasibility studies, the freed for which has been very clearly stated in some regions, it seems that the proposals which would qualify as feasibility studies need to be clearly related to the planning process and not to be in fact operational activities which the region wants to begin in absence of the kind of overall process which the operational phase will require. This means that the region will need to emphasize a study protocol for the proposed activity which shows how'specific knowledge can be obtained which will then feed into the development of the operational proposal. Prevention of the misuse of feasibility studies as a backdoor approach to operational grants will require judgments to be made by the review process which in some cases will determine that what is pre- sented as feasibility studies as part of the planning process has to be considered as an operational activity. Any kind of definition of this type would require that kind of judgment at the borderline. Other Policy Issues There are a number of policy and administrative issues which are emerging as important during the operational phase which were not as relevant during the planning phase. We are in the process of de- veloping some additional policies for the Division to cover these various areas. This will lead in the relatively near future to a revision of the Guidelines of the Regional Medical Programs--however, as we develop these policies they will be sent out to you to be included at a later time in the total revision of the Guidelines. -25- One of these you have already seen--the policy on renovation and alteration of facilities. This policy raises an issue which is in- herent in designing the policies of this program generally. There is ,4 fairly formidable process of development of Public Health Service grant policy largely having to.do with the traditional grant programs and the NIH framework for the su pport of research and training grant activities and construction programs. This total PHS policy framework has a great deal of relevance to Regional Medical Programs. However, the Regional Medical Program mechanisms are sufficiently different that it was quite obvious from the very beginning that we could not take existing PHS policies, such as those in the Research Grants Manual, and just transfer them to this program without some careful consideration and modification. At the same time, it has been a desirable goal to have the policies emerging from this Division be as nearly consistent with the PHS policies as is appropriate, thereby avoiding unnecessary confusion and differences between the policies. Therefore, even though we have issued a renovation and alteration policy because we need it at this stage, I would like to point out that the Public Health Service is in the process of developing a renovation-alteration policy to apply to research grants. When this policy is put in final form we will probably make some modifications and changes in our policy where it seems appropriate to conform to the approaches taken by the rest of NIH. Among the other policy areas which we are working on and which are at varying stages of development is a policy to deal with the complicated -26- area of patient care costs. This is an controversial issue-as anybody who follows the discussion going on between Medicare administrators and the American Hospital Association can see.: Our policy will go beyond the matter of just how hospitalization costs are calculated to a consideration of the terms and circumstances under which patient care costs can be appropriately paid out of Regional Medical Program grants. Our basic assumption is that we will require special justification in the use of the Regional Medical Program grant to pay patient care costs. It is permitted by the law in somewhat circumscribed circumstances but it has been the feeling of a number of advisors we have talked to that the use of extensive amounts of Regional Medical Program grant funds for the payment of patient care costs could cause a diversion of the program into the financing of medical care. This would probably soak u most of the funds available into that objective and we sta- .p ted in our original Guidelines that this program is not a medical care financing program. Another area in which we are developing policy is training stipends. Here again there are a vast array of training programs support- ed by the Public Health Service. We are in the process of developing a policy which will try to relate the Regional Medical Program training activities to those other activities and to those stipends with pro- vision of enough flexibility to meet the unique type of needs and situations which would emerge in Regional Medical Programs. -27- Another area is the matter of clinical experimentation. There is, as.you know, a policy of the Public Health Service in this area and this policy was developed with particular targets in mind, most of which would not ordinarily arise in the context of Regional Medical Programs. We have been exploring the possibility of some kind of modification of the policy as it applies to Regional Medical Programs. However, I think it would be misleading not to indicate that the PHS policy will obviously apply to Regional Medical Program grants where appropriate. In the operational phase of the programs it will be necessary to conform with that policy in situations where that po 3'.Cy is applicable and appropriate. The extent to which we can achieve some special treatment of Regional Me dical Program under that policy remains at this point still to be worked out. Another issue which has arisen -is the matter of the relationship of the planning grants to the operational grants. We have assumed that at some point in the future the need for a special category grant called "planning grant" is probably going to diminish as the Regional Medical Programs get into the operational phase, we have spoken of the need to have a continuing planning activity as part of the operational activity. However, we would intend to work out this transition in a way that leaves some options open to the grantees. We are clearly authorized to provide separate grants for planning and operational activities and we would continue to do so if that were the grantee's desire. On the other hand, if at some point in the future it becomes -28- desirable on the part of the grantee to merge the two grants because they feel that they can no longer justify the need and desirability of a separate planning grant, this could be worked out. There are actually a number of ways that this can probably be done. I think I have covered some of the issues that have arisen in the review of the applications as we have seen them at this time. There are some broader issues some of which were discussed at the time of the January meeting and which will be discussed in the Report of the Surgeon General to the President and Congress on the extension of the program. Even after the report emerges and is published and even after it takes the form of a legislative proposal to the Congress,,.we will still be operating under the authority of P.L. 89-239. Changes which might be proposed in the President's Report will not become effective until the new law is passed extending the program. The effective date of new legislation is unlikely to be before expiration of the current authority, which is June 30, 1968. One of the areas that was discussed in January and which will be discussed in the Report of the Surgeon General to the President and Congress is the categorical emphasis and focus of the program. This is an issue that becomes more pertinent when we get to the operational phase of the program. We would repeat at this point what we have said previously--activities supported through a Regional Medical Program grant have to be justified in terms of their contributions to the Regional Medical Program with its focus on heart disease, cancer, stroke, and related diseases. We do not have tL-k legal authority to support an activity which cannot be so justifie -29- DATA COLLECTION: MEASURING PROGRESS OF REGIONAL MEDICAL PROGRAMS Delivered By: Maurice E. Odoroff Assistant to the Director for Systems and Statistics Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Bethesda, Maryland June 16-17, 1967 -31- In the course of visits made to grantees by members of @he Divisionts Development and Assistance Branch, a sizeable number of questions are en- countered which concern mechanisms for evaluation of the progress of Regional Medical Programs and the Division's policy on the extent of support of various types of data gathering activities. Questions have centered around initial survey data of facilities, manpower and patient referral patterns, baseline and subsequent health data collections, data management and processing programs including registries and data representing clinical judgments on the care given to patients, expressed in quantitative terms. In the past two months we have called two conferences of data specialists to help the Division assess the kinds of data useful for regional medical program purposes, to review the extent to which existing data can be used, and to identify the type of guidance the Division can give Regions in developing methodologies and creating mechanisms of data collection associated with planning, feasibility studies and operational projects, and the evaluation of the regional medical programs. The regional medical program is clearly not the vehicle for collection of all conceivable health data, nor is it the vehicle for planning comprehensive health programs. It should therefore not be responsible for the complex data gathering activities associated with such broad purposes. The Law itself, and its legislative history, clearly indicate that activities should not be supported which are the authorized responsibility of another agency. The problem is what, within limitations, are the specific data activities for which regional medical programs should be responsible, what data already available or planned to be collected by other agencies and organizations, can -32- be useful to regional medical programs, and what needs to be supplemented by additional efforts in the regions. The more important fact is the nature of the data which are the cific resdonsibilit on and c e collected only it. We must not be guilty of wasting precious manpower and taxpayers' dollars in gathering data for which other sources can be utilized, nor should we leave to others the responsibility for gathering data necessary to evaluate the success or failure of regional medical programs. In this context we wish to share with you some of the cautions and concerns presented to us by the consultants. These concerns may identify for you.data functions for which others are responsible, and for which regional medical programs are data users. They further identify the kinds of data on which the region ought to focus in the light of the ultimate objective of the regional medical programs--bringing to the patient the latest medical advances through cooperative arrangements. There are two kinds of data necessary--normative data for making decisions about what should be done, and operational data for making decisions about what is being done. In the planni the problem is to id ntify the e region of h t cancer stroke and related diseases--identify the deficiencies in resources and manpower and to decide on how to handle the burden of these diseases on the region. In general, the regions need a gross "fix" on the populations having the conditions which require care, on the resources which the regions have, and some identification of the present practices for handling these conditions, in order to plan for increasing the capability of practitioners in the region to the level of present advanced practice. The planning process -33- is thus aimed at determining the critical areas of need. It was our consultants' concensus that for planning there are sources of base data on population, general morbidity patterns, facilities and manpower, which can, with minimum effort, provide r6gions with a useable base. These data have been developed by governmental and voluntary agencies. All inclusive inventories and massive data operations requiring special technical and statis- tical staffs are thus not a region's most urgent need. Activities of this type should be discouraged. The regions should first identify existing data and available resources. These, when available, can with a minimum effort, provide much needed base planning data. It may be necessary to ask the responsible collecting agencies to package the data differently or add items not normally collected. There has been concern for the possible overlapping conflict in data gathering responsibilities under Comprehensive Health Planning and Regional Medical Programs. Present plans call for pertinent health data necessary for health planning to be coordinated by State Health Planning Organizations created under PL 85-749. The consultants cautioned against the indiscriminate proliferation of registries which can result in excessive costs and/or-little use. Experience in other disease-areas demonstrate this danger. Since the aim of a registry in a region is primarily that of a program management tool, and not epidemio- logical research, the consultants agreed that for regional medical programs, the service functions of a' registry outweigh the research functions. If the region mounts a registry, it should be service oriented and provide specific information to improve the quality of diagnosis-through better definition and characterization of the diseases, with feedback to physicians implicit, whi--'.,. -34- will result in proper management of the patient, assurance that he receives rehabilitative and restorative measures, and adequate followup to assure continued optimum care. The purpose of Regional Medical Programs is, ultimately, improvement of the care of patients with heart disease, cancer, stroke and related diseases, by making available latest advances in diagnosis and treatment. To accomplish this, the regional programs are enlisting cooperation of participating organ- izations and agencies in the goal of upgrading the level of care, by extending and increasing the application of the highest standards of medical care and new advances in medical knowledge through better mobilization of existing re- sources and manpower. The consultants infer that there exists a onal responsibility on the part of regional medical programs for interpreting, according to the best peer judgment, and making known the criteria which will define what constitutes the highest standard of diagnosis and treatment for physicians, cooperating h6gpitals and organizations. The steps that are pertinent as a start involve establishing means for obtaining continuous patient information and creating the mechanism for. interchange of professional d ments, practically applicable to patient care. The regional medical centers provide the nucleus of activity, capable of providing education to the staffs of the community hospital regarding the essential criteria which constitute the latest advances in diagnosis and treatment. The consultants point out that this would also provide a mechanism for communicating between the "medical centeril and the insitutions and physicians in the participating organizations. As these arrangements become operative, the regions will develop means of continuous evaluation of the -35- activities established in the cooperating institutions. These can be expressed in quantitative terms since the criteria agreed upon identify the data parameters. They need not be all inclusive, but should express minimums which define what is necessary to diagnose, treat and rehabilitate the patient. The consultants note that there now exists in every region a core of patient data of varying quality and adequacy, which gives some idea of the content of care now being provided and on which regions can build. The regions have the responsibility for establishing a mechanism for the quantitative measurements of the degree of application of accepted criteria by the hospitals and physicians in the region, which are essential elements of operational data. Planning and operational data together contribute as base data for evaluation and for predicting future direction and long-time needs. Although evaluation will take years, it is necessary from the beginning to create the design for measurements of outcome. -36- DATA C@ Delivered By: Margaret H. Sloan, M.D. Chief, Development and Assistance Branch Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Bethesda, Maryland June 16-17, 1967 -37- I am very pleased to have the opportunity to add some comments to Mr. odoroff's very fine presentation. First, however, I wish to express appreciation on behalf of the entire staff of the Development and Assistance Branch for the great consideration and courtesy with which you have welcomed our representatives into your respective regions. Your willing- ness to share with them your problems, your successes, your experiences in the great effort of establishing cooperative arrangements to improve the level of medical care for patients with heart disease, cancer and stroke has been exciting and most rewarding. As you know,, this program has been launched without any real precedents. We are developing policy as we go along and, herefore, it has been extraordinarily helpful to know what t difficulties you have encountered in the interpretation of the Guidelines, in the development of effective programs within the limitations of the present legislation, and in the cooperative arrangements you have endeavored to achieve within your regions. Our staff has been eager to bring these problems home to the Division for consideration and to carry back to you any help we This two-way communication has been of great assistance can offer. -38- ion of Policies and procedures and to the Division in the evolut at a regional medical progr am could in the conceptualization of wh and should be. morning to add a little I thought it might be useful this Division has special activities in which the information about further guidance been involved which may, in the future, give you ff. I would emphasize that along the lines discussed by Mr- odoro d I have been involved the activities in which Mr. odoroff an collection hospital record systems, regarding problems of data 9 nuing activities which are being ex- registriesg etc,3, are conti We have requested leading plordd in many parts of the country. dvise us regarding registries. The experts in the country to ae of the National Hedrt Institute joint Subcomraittee onstrok ases And Blindness and the National Institute of Neurological Dise involved in estab- ive discussions on the problems has held extens . of undertaking an fishing stroke registries and the desirability ii, would require a major commitment activity which, if done we coming forward funds and interest. They will be of manpower9 able by the early with a policy statement which should be avail fall. We have had several meetings$ as Mr- odoroff has said, blem of hospital records systems in regarding the general pro trying to develop information a-id advice. -39- I think you are aware that Section 907 in the legisla- lation calls for the Surgeon General to 11 ... establish, and maintain on a current basis a list or lists of facilities in the United States equipped and staffed to provide the most advanced methods and techniques in the diagnosis and treat- ment of heart disease, cancer, or stroke, together with such related information, including the availability of advanced specialty training in such facilities, as he deems useful..." To obtain the best advice available to approach this requirement of the Law, we have entered into a contract with the American College of Surgeons Cancer Commission to deve lop criteria for standards in the cancer field. They have established a committee which will be review- ing the requirements for such institutions in the field of cancer. This committee is headed by Dr. Warren Cole, whose wisdom and good judgement are familiar to many of you. He has brought together a committee of experts representative of the different disciplines involved in the diagnosis and treatment of cancer, and they are moving -40- forward very energetically. If this effort proves satisfactory we shall probably enter into similar contracts to establish comparable criteria for standards in the fields of heart disease and stroke. There is one area in which our staff representatives have found that it might be advisable to make a general request to all of you now. Often in the review of applications, the question of overlap with activities in the region supported from other sources arises. We should like to urge you to make sure that you indicate the relation of Regional Medical Program funding requests to the other funding sources whi@h are supporting activities in your re'ions at the present time which are relevant 9 to your Regional Medical Programs. Such activities may in- clude community activities being supported through other parts of the Public Health Service, through the State and local health departments and the voluntary health agencies in your particular region. We urge you to make the effort to ecome familiar with these other resources because these are things on which you can build. In this connection, we hope you are investigating the con- tribution which can be made to your regional program by the community health clinics supported by the Office of Eco'nomic -41- Opportunity and by the multiphasic screening activities supported by the Division of Health Resources Research which are just be- ginning. Four contracts have now been awarded for the evaluation of the large scale application of multiphasic screening and you will be fortunate if one is located in your particular area. The Vocational Rehabilitation Administration has extensive activities in all your regions. Some of you have not been too familiar with these because in general they are conducted through the State Office or Department of Education. The fact that ex- tensive assistance in rehabilitation medicine is possible through the State Vocational Rehabilitation Program and the State and regional vocational rehabilitation consultants has not been ad- equately recognized. We have had various complaints brought to us from different groups across the country who have felt that the potential con- tribution of their activities to regional medical programs has not always been adequately recognized. Many nursing groups in the country have felt keenly that they are not involved, an even worse, that they are being ignored. They fear that plans which will involve them are being made without consulting them or providing any opportunity for them to participate in the planning. At the Medical Librarians Association meeting in Miami, which Dr. West and I attended on Tuesday, I was concerned to find -42- brarians had felt that Regional that many Of the medical ii itinterested in the contributions which Medical Programs weren I'm sure this isn't true. be made by the libraries. could ips with the f you have had excellent relationsh know that many 0 National Library of Medicine and have been stimulated by their expanded concept of what a medical. library should offer in terms discussed carefully with of learning resources. But have you the medical librarian in your own institution the contribution ther her library could make to your own RMP@ and what of the o medical libraries in your region? As you may be aware the.National Institute of Neurological indness is supporting stroke.research centers. Diseases and Bl demon- can be very valuable sources of new informations, of These strations of the highest quality of cares and an outstanding resource for continuing education in your region. Similarly, the National Heart Institute is establishing myocardial infarction research centers and, wherever these are located, it would be our hope that you would find ways of u- tilizing them in your continuing education program. Finally, obviously resources are available from your State Health Departments, and from the Regional Health Offices of the Public Health Service. The Regional Health Offices do have counterparts in their regions to the Public Health Service programs in the National Center for Chronic Disease Control here in Washington. -43- The Regional Health Office staff are aware of all the programs of the National Center and can help you relate to these programs. This is particularly important in their programs for the control of heart disease and cancer. They have extensive literature, Publications and guides for many programs in which you would be involved, and we would urge you to consult with them at your earliest opportunity. Because of the role which the Regional Health Offices and State Health Departments will have in the comprehensive health planning activities of P.L. 89-749, such contacts will be essential. May I say again that our Branch is eager to work with you in every constructive way possible and we look forward to a very rewarding continuing relationship. -44- CONTIN ING EDUCATION Delivered By: William D. Mayer, M.D. Associate Director for Continuing Education Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Betbesda, Maryland June 16-17, 1967 -45- e difficulty in From the outset we have had som explaining the role of continuing education in Regional Medical Programs. The perceptions have varied from: "this is primarily a health care program and education ought to constitute only a miniscule part" to "in one sense regional medical programs is nothing more than a continuing education program'. Obviously, the truth lies somewhere in between these two polar perceptions. It is in the definition of where that in between lies that problems have arisen. Obviously, because of the flexibility of the program the definition of the appropriate role of continuing education in an individual regional medical program will vary from region to region. This is as it should be. As we have watched the program emerge over the last 16 months it is apparent that there may be some general guidelines which may be of assistance in the definition of relationships. The intent of Congress in enacting Public Law 89-239 I think was clear to all of us, that is, to improve the quality of health care provided for, and the health status of, citizens of this country with heart disease, cancer, stroke and related diseases. In this very real sense -46- reg7ional medical programs are health care programs. if these ultimate goals are not achieved then we have not met the expectations of the Congress. There are many things which go into the improvement of that health care and health status -- only one of these is continuing education and training. Obviously then continuing education and training are a @rt of regional medical programs. The opportunity of continuing education and training in regional medical programs however, is that they can become an intergral part of health care rather- than a part which is separate and distinct from health care. The planning process in Regional Medical Programs should uncover unmet needs in the area of health care and health maintenance. From those primary health care needs can be -@- d a whole sequence of secondary needs which, when met, derive assist in meeting the originally determined health care objectives. It is here that continuing education fits into the picture of regional medical programs. It is hoped that the deteminiation of educational and training needs will be derived from the larger framework of health care needs, it is hope that the educational objectives will be derived from health care objectives, it is hoped that the educational programs will be related to the health care programs, and finally it is assumed that educational evaluat ion will ultimately be related to health care evaluation. It is this intimacy -47- of relationship of continuing education to health care that is possible under regional medical programs which provides the real opportunity for the construction of meaningful, educational programs. I do not want to seem to belabor this realtionship, but I do feel that it was one that was'contained within the intent of Congress and one which was wisely incorporated into the Law and the Guidelines. I would say then that the first major point I would like to make is that under regional medical programs we are talking about continuing education and-triining for the purposes of improved health of individuals with heart disease, cancer, stroke and related diseases. The second major point I would like to make is that in this context of continuing education and improved health we are talking about development of continuing education and training 'activities not only for physicians, but for all health personnel and even associated professions when their activities bear directly on the improvement of health care. The majority of continuing education activities to date within the health professions have been focused on the physician and many of the organized programs have this as a s6le purpose. Although a very great deal needs to be done in improving continuing education and training for physicians, I would suspect that the ultimate -48- refinement of this aspect at the expense of others who are intimately related to health care might not be the most- efficient route to the improvement of health care. The medical profession may be in a position to provide leader- ship in the establishment of these kinds of educational programs and where this is the case under regional medical programs I would feel that it would be their obligation to provide whatever assistance might be possible. These then are the purposes of continuing education under regional medical programs and the audience to whom the contin- uing education is addressed. What then is the mechanism of preceding? In the guidelines we suggested the following steps in the planning process: (1) identification of existing educational training programs within the region, (2) identif- ication of additional education and training needs in the region, (3) projections of methods of meeting those nee s, (4) prelimin ary thoughts relative to the mechanism of evaluating the effectiveness of future programs in meeting the needs, (5) relationship of continuing education and training programs to the overall objectives of regional medical programs including their anticipated effectiveness in bringing about cooperative arrangements between various health institutions and personnel within the region. If these are appropriate approaches to activities during the -49- planning process what then should the review process consider as it goes over applications for continuing education and training support. Some general questions to which answers might appropriately be sought are in two broad categories (1) those relating to regional medical program philosophy and, (2) those relating to the educational value or import of the specific program. Under regional medical program philosophy the following questions are being asked: A. Does it have a regional genesis predicated on a determination of needs peculiar to or in evidence in the proposing region? B. Does-it foster and extend cooperative arrangements in,which two or more facilities, agencies or organizations interact positively in the develop- ment and utilization? C. Does it bear a positive relationship to other projects in the region? D. How close is the project to the actual bringing about of better patient care? E. Is the project within the scope of categorical restrictions? F. What are the inter-regional implications and have they been considered? -50- In respect to the Educational Value of specific programs, one might ask the following questions: , A. What data have been used to identify the need for the educational activity? B. Are the objectives to be accomplished clearly defined? C. What are the criteria for success? D. How will success be measured? E. Are personnel available either on staff, through consultants, or ot to carry out the program including its evaluation. If the request contains a significant investment in hardware, another series of questions might be asked in addition. A. Will the requested hardware fit the stated objectives? B. Has cost effectiveness been given adequate study? 1. Is there a less expensive sVstem which will accomplish the same ob ective? 2. Is there less expensive "brand" of hardware which will accomplish the stated objectives? C. Is this a new project or the expansion of an existing project? -51- 1. If new, is there evidence of the application of knowledge of similar activity in other related educational ventures? 2. If expansion, is the request warranted on the basis of: a. Past experience? b. Maximal use of existing equipment in the region? D. Have other sources of funds for this project been explored? 1. Such as: Public Health Service, other Federal sources, or State sources? 2. Are any of the above currently supporting this activity via a grant, contract or other funding mechanism? 3. Are there plans to seek other than RMP support now or in the forse6able future? E. Are sufficiently qualified personnel or staff requested to realize a high quality utilization? 1. Professional personnel? 2. Technical personnel? 3. Production personnel? 4. Other support of personnel? F. Is the hardware system compatible with existing systems within the region? In adjacent regions? 1. lihere the system is not compatible, has the implication of this been considered? -52- G. Are adequate "space" provisions identified to house equipment and supporting staff? Obviously, this is not a complete list by any means of questions which might appropriately be asked for the details will and should vary in relationship to the specific proposal in hand. I might comment, however.' that the key issues in most instances boil down to the quality of people who have been attracted to the continuing education activity and the ideas that are generated. Questions have been asked about recruitment of personnel in certain scarce categories and about who might appropriately be supported in continuing education programs under regional medical programs. In the first category, one of the scarce manpower areas is that of individuals with experience in educational evaluation who also have some understanding of health affairs. In an attempt to correct this problem we have undertaken a contract with Dr. George Miller, of the University of Illinois to establish a one year training program in educational evaluation, to provide a series of 6-week seminars and to develop a series of one week workshops in medical education. We are now in the process of reviewing proposals for four other such educational programs around the country. It is felt that the opport- unities presented by these training programs and the -53- development of similar relationships between various health institutions and the colleges of education in various regions may gradually meet this critical manpower need. In terms of,who may be appropriately supported under regional medical programs in the area of continuing education the following general thoughts might be useful. This is a categorical program; granted broadly categorical, but categorical nevertheless. To deny this in continuing education or any other aspect of the program is to deny reality. Under the law there is authorization for grants to plan for and establish Regional Medical Programs to carry out the purposes of the law. The degree to which activities, or people or hardware are related to the Regional Medical Program should be the determinant of the degree to which they are supportable under P.L. 89-239. Thus a continuing education function which is contained within the same- geographic area of an existing regional medical program, but is not a part of a regional medical program is not supportable. @While that which is a.part is supportable. What I am saying is that the fact that a function is Lo t@iiLu@n education does not ipso facto qualify it for support under regional medical programs. It is its relatedness to Regional Medical Programs which is the determinant. Within the context of Regional Medical Programs, however, it must be fully realized that this does not mean 100% -54- hard line accountability to heart disease, cancer, stroKe and related diseases. If, for example, an individual is being supported as a director of continuing education some of his activities may well be spent developing institutional and organizational educational relationships. These are not strictly accountable as heart disease, cancer, stroke, or related diseases, but might well be critical to the establishment of the continuing education program within the RMP. I could go on citing examples of supportable and non-supportables, but I am not sure how profitable that woul'.d-be. The ultimate determination of these issues will be made by the National Advisory Council and the Surgeon General. They will make their decisions on how well they feel a given individual or item is justified in relation to the Regional Medical Program. I realize this brief presentation does not begin to answer all of your questions in the area of continuing education and training. I hope that the discussion period at the end of this morning's program and the smaller group sessions over the next day will provide an opportunity to seek greater clarity. As some of you may realize, this is my next to last day with the Division of Regional Medical Programs. As I have looked back over the last 15 months, I have become -55- even more convince tnat wiunln Lne I[IUCRIULL-L@LLI UL medical programs a great and significant advance in continuing education and training of all health professionals in this Nation can occur. I sincerely hope that the mechanism will be used for it does relate education to health care and in the final analysis it is the improvement of the health status of the people of this Nation to which Regional Medical Programs and the health professionals of this country are addressing themselves. Thank you ... -56- SOME ASPECTS OF COMMUNICATIONS AND PUBLIC INFORMATION FOR REGIONAL MEDICAL PROGRAMS Delivered By: Edward M. Friedlander Assistant to the D3rector for Communications and Public Information Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Bethesda, Maryland June 16-17, 1967 -57- Today we live in a world of proliferating information and everyone, it seems, is "communicating" -- at least that's the word they use. No one seems to talk any more, or meet together, write a etter,, educate, or teach, televise or.broadcast over radio, discuss an idea, talk on the..tel one, convinc omeone, clarifv an issue, gain a ment, influence, or do a thousand other things that relate them to other people, including I suspect) even smilin2 at someone, They no longer say they do any of these things. Rather, they say they "communicate". Within this same context, the one thing that everyone today is busy buying, dissemin tinq, gathering., transmitting, sitorinq, evaluating, and wbat-have-@u -- even communicating is a commodity called "information." If these two observations are true, and all of these activities are really represented by either or both of these two words, then my office is the most important in the Divisionl This illogical logic stems from the fact that our title is: "COMMUNICATIONS and (PUBLIC) INFORMATION". The fact is we are not the most important office in the Division, but I sometimes think we are the most misunderstood. -58- The problem is that the words "communication" and "intorMatioii are overused and their basic meanings abused. We really can not and ao not do all of the things that others infer our title might mean,by their loose s@e of the words. By the:same token, our activities are not limited to or prescribed by the defini- tion these same people would ascribe to us, based on their interpretation of cmmunications and @ul as they relate it to such terms as public rela ions, publ it sin -science writ@, advertis@, publis@n , etc. Contrary to both of these views, the general and specialized activities of our office include everything_ which in our view and those.to whom we are res ible -- will help meet the .....--Communications, information, and related needs of our new program. By agreement, the major goal of our office is to utilize all established communications and informational techniques to help develop Divisional activities which will achieve under- standingi acceptance, support and cooperation of institutions, organizations and individuals at the national level in the initial growth period of the program. Then, to mount an additional effort toward these same ends that will not on y maintain these relationships, but expand them to include all participating and benefiting publics. -59- Not to be overlooked is the important function of providing specialized types of know-how to Program activities to achieve maximum results. For example, continuing education and training can only be as effective as the participants it attracts to learn from its programs. The educators know what to teach, but studies show (1) that their programs leave something to be desired in terms of presentation and use of teaching media; and (2) the old-fashioned methods of promoting such programs on y nets the usual 10-15 percent of the potential audience. Communications and public information public rela ions, advertising and promotion, if you will make the "continuing education package" attractive, and help promote the "sale" and effectiveness of that package to an ever-increasing audience -- provided the disciplines represented by Dr. Mayer's office and mine can be brought together in our Programs at both the national and local level. Also included in this broad spectrum of activities is the important need to work through all lay and professional news, editorial and other media, oh the one hand, and to establish and maintain the best possible Divisional relationships with you, as well as our relationships with our communications and information counterparts in each of your Programs, on the other. -60- By these measures, the major goals of my office in the Division of Regional Medical Programs, and those of its counter- parts in each of your Programs, are essentially the same. Only the scope of our separate activities,,and the publics with whom we must deal, are different. For this reason, each of us must assume responsibility for his own specific areas, geographically and in terms of activity. Together, however, we should plan and integrate our activities so that they will achieve the common desired result of understanding acceptance, support And cooperation at every level throughout the entire country. On this basis, I encourage you to gain as comprehensive an understanding as possible of what we as communications and public information people do; to take immediate steps to develop such an activity; to staff it carefully and adequately; to budget for it as provided for in Guidelines; and to make it as integral a part of your Program as it is becoming a part of the Division's activities. If this can be done in your Regions, each will benefit from greater community support and cooperation, and collectively this same end can be achieved throughout the country by.means of a national communications and public information network for Regional Medical Programs which we plan to organize and implement. -61- To illustrate how we already 'have this sort of "Regiona-L network concept" in operation in terms of calling public attention to the Programs through the news media, I would like to quote from two letters received from the public relations people in two of the Regions with which we worked closely in announcing their operational grants in April: "....The KANSAS CITY STAR, and both AP and UPI had been alerted approximately three days prior to announcement... and then 're-alerted' in early afternoon on April 17 when the announcement seemed to-be hatching", wrote Helen Sims, Director of Informational Services at the University of Kansas Medical Center in her letter to me on April 18. ...... On April 17, 1 left my office at 4:30 p.m. and-started making the rounds to all the media. I had copies of both Kansas and Missouri Region releases". (I should note that Miss Sims, Dr. Joye Patterson, her counterpart at the University of Missouri, and my office had worked together in a tbree-way plan to cover both grants simultaneously in Kansas City.) "All three were waiting for me ... because the announcement bad come over the wires at approximately 4:30 p.m. At UPI I was handed the teletype to read. It was brief, some 100 or so words, stating the announcement was made by Senators Long and Carlson... and the casual reader would have interpreted that -62- the grants were for medical school support. There was no explanation of programs ... and following RMP amounts, the message was cluttered up with two or three additional non-related grants for various schools in the two states. I amplified and clarified the story before it went out over the wires to both states... I did not see the actual words at AP..but they wanted more information. At the STAR, I was handed the story from their Washington bureau to read, and then asked to enlarge, localize, and tell them about the Kansas City metropolitan area. Since they had a copy of the Kansas story (given them during the interim when Kansas was second and Missouri was third) they were most hungry for Missouri Region news From Alan Davis, former Assistant to the Dean for Public Relations at the University of Utah Medical Center and now a member of the full-time staff of the Intermountain Region, came a similar letter. In it, he wrote... "All went well here in Zion and surrounding points. We bad communication from two Senators' offices requ sting further information right after you released it in Washington and indirect communication with several others from the region. We alerted the local AP and UEq offices and they agreed to hold our detailed release until they had received the word from -63- Washington offices. AP then put our story, in length, on the B wire, which covers everything west of Kansas City, then it is rerouted on the various State wires. "...Your help was greatly appreciated. We did not break, but respected the 24-hour restriction religiously, until queries came in resulting from information gained from various Congressional offices. Then we figured it was free, and really piled it on, out here. Our release included Dr. Marston's complete statement ... We will forward the clips as they come in.to As you can see from these two letters, a number of valuable purposes were served. The Congressional delegations had been involved. This made the Department of Health, Education, and Welfare happy -- not to mention the Senators and Congressmen whose names were associated with significant amounts of money going to their States and their Districts in them. The accuracy and completeness of the stories of the grants and what they meant to the Regions were insured. The major news media were involved and conditioned for later news and feature stories and editorial support, if and when it seems indicated. And, last but not least, a cooperative arrangement among the Regions themselves and our office is a working reality. -64- general point of view, at now these Looking from a more kinds of activities fit into a Program, or can be deve7loped for a Region, some excerpts from a recent letter to me from Mrs. rector of Information at the University of Beverly Wood, Di Arkansas Medical Center may be helpful'. oe,,.j wonder if the Program Coordinators, Conference would orum for a discussion on the relationship Of provide a good f ith the ongoing Public the local RMP communications specialist w ams of the University and/or other agencies relations progr involved...", she wrote. ... Our Division of Information - as you know - aided in the preparation of the grant application. We followed through with stories announcing our grant, and generated attention in other media. We have also submitted the names of candidates for a communications person for the Program. I would envision that Project Information Officer as one who would automatically touch base with my Office. We have much to learn from the planning phase, and in turn, we can provide information be will need... I've discovered that several other Regions are experiencing similar difficulties. They are moving ahead, in to be involved in a cooperative way in the information program ... I have a feeling that they (the Coordinators) are so involved with the many other facets o -65- 0 their Programs that they are failing to take advantage of what should be a distinct asset to them.... Now I would like to speak specifically about the issue of communicating information which must form the base of our relationship with you and your relationship with your people. In mid-May you began to receive copies of'a publication which we developed and implemented as a new concept to meet our needs for purveying information to the ever-growing number'of people involved with and interested in Regional Medical Programs. To avoid the term "newsletter", and all of the built-in arbitrary limitations of schedule, space and content that go with that term -- and fo-r-want of a better name -- we called our publication a "communications device". As such, it provides us with the opportunity to develop and distribute as much informa- tibn as possible on any subject, in a form which is flexible in numbers of pages it can accommodate,- requires a minimum of editing; and can be scheduled for mailing as soon as each piece of information is developed, or, if necessary, to meet an emergency. It is, in fact, almost a literal example of the Marshall McLuhan philosophy of "the medium being the message". -66- To date, we have prepared and mailed 14 issues. (The two most recent, which should be arriving in your offices-today are included in your notebooks -- just so you can keep up-to-date) The response to the approach has been most favorable and encouraging. This is not only fla ttering, but meaning u We are obviously filling an important need, and doing so in a way which people seem to like. However, I am afraid they are liking it almost too much. On the one hand, we are being over- whelmed with requests for additional copies of various issues; on the other hand, we are being inundated with demands to add names to our mailing list for this publication. Because of the large and unwieldy lists and quantities of material that could become involved, and some other implications, we have decided not to try to find ways to meet these requests. Rather, we have developed some ideas to help you help us solve the dilemma; keep our growing publics satisfied and informed; and help develop your own communication and information programs at the same time. Naturally, it is our hope that you and your staffs weLcome and read our material. However, we also hope that you are making a judgment of which issues are of particular interest to specific publics in your Region, and either routing your copies -67- or duplicating and distributing them. If you are not-already doing this# we suggest that you consider doing so. You can utilize your own ingenuity, resources, and budget available for this sort of activity to reproduce these materials. How you can most effectively distribute them under your own auspices is up to you. In fact, we suggest that you develop your own way to do this which will not only be most appropriate to your Region, but will be most helpful in achieving the goals of understanding, acceptance, support and cooperation amon.g the individuals and groups for your Program. As for our mailing lists, we have already tripled the number from its original nucleus of the 800 people who were invited to our January Conference. However, the list is now at a critical stage, and we need your help in this too. Therefore, we have included in your notebook a list of the people in your Region who are now on our General A to Z mailing list and who are receiving all of the same materials you do. We would be grateful if you would review this list for accuracy and indicate those whom you would like to continue on it, and suggest additional names of those whom@u think should receive all issues of our publications and other materials of general -68- interest. Because this might best be done in consultation with your staff after your return home, we have enclosed forms for submitting such names and addresses and a return envelope.for our mutual benefit in getting both the corrected lists and the new names back to us as soon as possible. Please bear in mind, however, our su stion that ou develop vo r own distribution for selected materials Therefore, exercise care a names of people to whom the input of informal egional Medical Programs might better be controlled by Vou. Also, please be assured that we will honor any names you may wish to change or add to our lists at any time, now or in the future. This is only the beginning, And we want a firm, meaningful and accurate base list. One other area in which you can be helpful is the development of a Directorv of Regional Medical Pro rams. Although we recognize that it may well be accurate when it goes on the press, and, because of day-to-day changes, be inaccurate when it comes off the press, there is sufficient demand for such a document to merit its development and distribution. -69- The copy we have provided you is only a draft which you may use until we publish the next issue, To help us, please review the extra copy of the page representing your Region inserted in the draft copy along with your listing. Suggest any changes and additions as they may be necessary for accuracy, and return it in the addressed envelope together with your lists. 9 0 0 . a'. 4 a . 0 . 0 The most important act3.vity of any program is the one with which each of us is involved. The fact that it is not, is usually a difficult fact to accept. I have come to accept it in the belief that the whole can be greater than the sum of its parts -- as long as one of its parts is communications and public information, at least as I have tried to describe it, and as we want to practice it in the Division of Regional Medical Programs. -70- GRANTS MANAGEMENT Delivered By: James A. Beattie Chief, Grants Management Branch Division of Regional Medical Programs At: Conference of Coordinators of Reaional Medical Programs Governor's House Bethesda, Marvland June 16-17, 1967 -71- I would like to take this opportunity to express some of the interests and new developments from the grants management aspect. Recently, staff of the Division of Regional Medical Programs met with Mr. Nathaniel Karol, Director of Grants Administration, Department of Health, EducAtion and Welfare. During this meeting, the Department expressed a desire that grants administration between the grantee and the participating institutions be strengthened. Along these lines, it wag brought out that the grantee's responsibility is to advise the participating institutions, that the Guidelines Regional Medical ProgEaas, Public Law 89-239, and the ReRulations-Grants for Regional Medical Pr2aEams apply to all funds disbursed by the participating institutions. Further, your attention is callf!d to the terms and conditions contained in the Application for a Regional Medical Program grant. Wherein it indicates that the applicant organization is responsible for assuring that the participating institutions are advised of and in compliance with the terms and conditions of the -72- award. Along these lines, it would be advisable for the grantee institutions to draw up agreements with participating institutions which delineate and fix certain areas of responsibility, especially as related to budgeting, reporting and accounting matters. In addition, efforts should be made to establish monthly fiscal reporting systems and progress reporting and evaluation systems. By using these techniques, the communication link will be strengthened in the region - and tie the region more closely together. I would like to mention a few Regional Medical Program policies in areas were there would be a chance of an audit exception. First and foremost, institution policy should be followed in all instances, except when DRMP policy is more restrictive. Another item is that consultant fees may not be paid for part or full-time employees of the grantee or participating institutions. When new facets are added to the program, supplemental funds should be requested by preparing a supplemental grant request. Approval of supplemental applications follows the regular review procedure in that the application goes through the special RMP Review Committee and through the National Advisory Council. -73- Another item is budget justification--funds requested in the original, continuation or supplemental budget must be justified. The use of these funds and the basis for the estimate must be stated. In addition, a justification of carry-over funds from the previous grant year and transfer of funds between budget categories within the grant year must be thoroughly justified before approval action can be taken by the Division. I would like to remind you that the grantee and participating institutions should be prepared, through vouchers and invoices, to sub- stantiate all purchases and expenditures from grant funds. Time and effort reporting should be in accordance with the DHEW policy outlined in the Department's publication of March 1967 on this subject. Copies are avail- able from the Division of Regional Medical Programs upon request. In view of the expanding number of grantees and participating institutions in Regional Medical Programs and the concern expressed by many of these institutions that a single indirect cost rate apply to all government-sponsored work at an institution, the Division has requested that the Division of Grants Administration in HEW centrally handle the -74- establishment of indirect cost rates. Until such time as the Department assumes this function, it is incumbent upon us to follow the DRMP Guide- lines regarding the establishment of indirect cost rates and for the grantees and participating institutions to present proposals for an institutional type indirect cost rate. We hope that in the very near future, the transfer of this function will be effected and the institutions will have a single source to deal with for the establishment of all indirect cost rates pertaining to Department of Health, Education and Welfare grants. O-.i May 16th of this year, the Division issued the policy on Alterations and Renovations. The policy states in part that operational grant funds may-,..-,. not support more than 90% of the costs of altering and renovating buildings and that allowable alterations and renovations exclude the construction of new buildings or expansion of and existing building. If you did not receive a copy of this policy, we would appreciate your advising us, so that we may insure your na-,na is on the list tD receive all policy issuances. The Grants Management Branch encourages you to seek our advice and assistance in- areas where the policies are not clearly spelled out or -75- where there is a conflicting interpretation. We will be glad to assist, in the establishment or review of the accounting and reporting-systems utilized for administration and coordinating a regional medical program. Our aim is to keep policies clear and simple, avoid audit exceptions, keep an open line of communication and assure that there is a proper balance and relationship between the budget and program. -76- ION AND PLANNING, EVALUAT THE REPORT TO THE PRESIDENT AND CONGRESS Delivered By: Stephen J. Ackerman Chief, Planning and Evaluation Branch Division of Regional Medical Programs At: Conference of Coordinators of Regional Medical Programs Governor's House Bethesda, Maryland June 16-17, 1967 -77- One of my basic beliefs is that the microcosm at the Federal level in any given program area tends to be a reflection--in reduced size--perhaps like obtained by looking through the wrong end of a telescope--of the full scale, life size macrocosm in the real life out there (as we used to say) in television land where you people are. The degree to which there are distortions in the image of the reflection at our end in my opinion tends to reflect imperfections in our perception, willingness or capability. The degree to which the things that most concern us are the things that are of equal importance in your spectrum of problems is indicative on the other hand of insight, balance and lucidity in our outlook. Consequently, in presenting the areas of interest and concern in planning and evaluation from the standpoint of the microcosm that exists in the fourth floor, B wing in Building 31, I would hope to therehy'by presenting an accurate and clear reflection of the interests and concerns that you see from the cockpit of the RMP in your particular area. To the extent that it does not present an accurate reflection, I would appreciate your suggestions and adjustment; to the extent that there are significant omissions of areas of significance, I would appreciate the benefit of your advice. The dominant function in the planning and evaluation sector--as well as in all other functional areas of DRMP during the first nine months of the program--was.recruiting a staff and helping to get the program off the ground--and I daresay that it was a safe bet that this is your experience, too. I would say that we had appropriate seminal input into the birth of the babyi The major involvements in this regard wore interpretation of the program -78- through speeches and meetings with interested groups, substantive participation in the development of program guidelines and procedures and participation in the review process and its development. In addition, the fundamental job of analyzing the national picture of the evolving development of the RMP movement throughout the country was begun. The second nine months of activity in this area has been devoted to another nascent objective--the propagation of the Surgeon General's Report to the President and Congress on Regional Medical Programs as required under Section 908 of P.L. 89-239. Our in-house activities in connection with the Report tend to reflect the activities that you have encountered in the early planning phases of our RMPs in the field. That is to say, they involved planning, data gathering and coordination activities. We developed the plan of approach for the development of the Report. We analyzed the data submitted with applications and the progress reports as well as developing a special re ort which contained 14 p items of information which we asked all regions to submit. Let me be sure to thank you for your prompt and great help in completing and returning the 14-point form. We also made special trips to certain regions to feel out our information And in some cases obtain a more complete picture of regional activities. As is characteristic of this program, we felt that the real insight can only be obtained from the real world, so that we developed additional mechanisms for capturing this input. The first was the appointment of a special ad hoc committee of expert consultants for the development of the report. The names are all familiar to you and they speak for themselves. Drs. DeBakey, Everist and Howell represented a subcommittee of the -79- National Advisory Council on RMPs in order to assure the close communication and involvement of the Council in its development. The second major channel to the outside world was through the National Conference held in January of this year, which was attended by some 650 persons from all parts of the country and representing a very wide spectrum of the various health interests. As you know, the report to Congress was a major agenda item and issue papers were distributed which formed the basis for the group discussion. The issues posed at that Conference and the distilled essence of the discussions at the Conference offered a pretty accurate forecast of the evolving nature of the Report. The importance of achieving the status of an established program and the assurance of its continuity was given major focus both by the Conference and in the considerations of the ad hoc committee. Closely related to this position was the attitude that it was not necessary to contemplate radical substantive change in the program direction or authorization. In this regard, both the ad hoc committee and the national conference--which incidentally showed a considerable degree of correlation in.their views--felt no need at this time to change the categorical emphasis of the program as originally enacted--their belief being the categorical approach as administered was broad and flexible enough to provide for a meaningful action and initial program development without over-burdening the fledgling programs. This issue remains open, of course, for further examination and consideration in the coming years. However, one issue has tended to generate some difference of opinion and that has been that of construction authority. As you know, Congress deleted the authority for construction, other than for renovation, from the original Administration bill indicating at the time that it did not believe the lack of authorization would hurt the program in the first few years and adding that the issue could be reconsidered at the time of renewal of the Law. -80- The special ad hoc committee and the National Advisory Council on Regional Medical Programs have very strongly expressed the conviction that the program ought to have construction authority if it is to achieve.its natural growth and potential. At the national conference, while there was not a great deal of question on need, there were two schools of thought on timing and mechanisms. The one, a strong assertion of the desirability of having the authority to construct the facilities required to achieve RMP objectives for which other sources of funds were either non-existent or inadequate. A frequently mentioned need in this regard was for educational facilities in community hospitals. On the other hand, there was a line of opinion expressed at the conference that the inclusion of construction authority at this time would con'ute up the recently quieted fears of the "Federal centers" approach i and impair the rising spirit of cooperation with practicing physician groups. In addition, the fear of the problem of duplication and overlap with the. well-established Hill-Burton program mechanism was mentioned in some quarters. In response to these diverse pressures, a modified policy on construction appears to be evolving. A selective construction authorization limited as to both amount, type of project, specifically tailored to the specific program requirements for RMPs and effectively coordinated with the Hill-Burton program could offer an effective compromise approach. Such an approach could provide a means of meeting real program needs and avoid stirring up the undue fears mentioned. While these appear to be the pivotal issues of the Report as it has evolved, the problem of achieving flexibility in funding authorities for important activities related to RMP program development for which authorization -81- are not expressly covered in the original Act. These would include in certain interregional activitives and/or resources designed to serve national or multi-regional RMP needs as well as the full participation of Federal hospitals in the program. The report is now out of the hands of the Division of RMPs and the NIH and is on the hierarchial pathway to the President, which means clearances through the OSG, HEW, BOB and the White House. By Law, the Surgeon General must deliver the report to the Secretary by June 30, 1967. It is probable that consideration will be given to submitting and implementing legislative proposals in sequence with the Report. The completion of our part of the transcendent project of the Sii-kgeon General's Report now brings us to the point where we must, even as you in the field, move on to the phase of basic operations. In other words, we must now get down to the bread and butter functions of planning and evaluation in re RMPs on A national basis. The substantive sine qua non of planning is the delineation of the major objectives that are inherent in the program mission. The procedural sine qua non in the employment of a disciplined framework or environment in which this can occur. We plan a major staff effort in this regard for DRMP and we are greatly interested in both the substance and the methodology of the planning efforts that are underway in your regions. The primary mechanism in the Federal sphere at the present time is known as the PPBS system, and its adaption to the mission of the DRMP is a challenging area for emphasis. We will be greatly interested in learning of your experiences in the application of this or other planning techniques in the regions, and hope to be able to make a series of field visits to the regions in this regard in the coming months. One of the great values of the PPBS system is the linking -82- of the planning for objectives of accomplishment to the budgeting and decision making process. You have all been appraised of the_ Secretary's policy statement on medical care prices. I want to emphasize the important of this statement and solicit your most serious consideration and appropriate action in this issue. We will need your help in developing the required progress reports on this matter. Certainly one of the major factors in achievement of the RMP objective as stated in 900(b) as stated in P.L. 89-239 of enabling the physician and medical institutions of the nation to make available the best in scientific advance for their patients depends upon a more efficient utilization of all resources available. Therefore, we are greatly interested in studying the costs of RMP activities, their projection into the future and their relationship to the objectives of accomplishment. In keeping with both RMP nature and its young but exciting tradition we do not view this an an in-house staff production, but a plan to be working with you getting the benefit of your experience and mutually sharing the insight and information derived. We also plan to involve the consultation of some of the outstanding resources in the country. The assistance and consultation of the Rand Corporation, Brookings Institution, and appropriate university resources will be elicited when and were we can. Dr. Charles Hitch, the famed propagator of the PPBS system in the DOD has recently been appointed to our Council. Another question that we believe to be of considerable importance is the matter of the dependence and interaction of RMP program activities with related programs. In this connection, we have recently sent out to some of you a preliminary guide to some of the pertinent related Federal programs. -83- We have also been asked to furnish staff assistance in the review process for the model cities grants program. In this connection we have some tables which show the current state of development of these programs by region, which we will distribute to you for information and possible future action. The capability of RMPs to counteract the problems of fragmentation and pluralism in the local level through the development of creative and cooperative interrelationships among the various Federal, State and local government programs, as well as the private and voluntary sources, constitutes one of the great potentials of the program. We have great needs for basic information on both the nature and volume of the problem and the methods for approaching its solution. We seem the same staff involvement with the regions and outside resources in probing this problem. Another functional objective that we see in the planning and analysis area concerns a study of the characteristics of the RMP as an instrument of synthesis. We want to study the RNP as an instrument of interaction, planning and decision making;to probe the nature of the power structure and the system of checks and balances that are built into the processes. We see particular focus in this regard on the new corporate forms that have emerged in a number of regions. Some like HOWNY appear to be a fairly broad based local instrument. Another type has involved fairly exclusively academic interests. You may be expecting to hear more from us in this regard and here we see as the expert resources, the political scientists, systems analysts, and we will tap these resources as appropriate. Finally, a word about evaluation--that function which,so frequently gets such magnificent lip-service and such miserable performance. In a hopeful -84- cess, I will give it fairly minimal lip-service attempt to reverse this pro in comparison to the other areas of interest cited and hope to-generate some real action as we go forward in the coming year. We view evaluation to be in 3 broad areas with levels of concentration within each area. The 3 major areas are specifically delineated in the Act as follows: (1) cooperative arrangements; (2) making available to patients the latest advances in the diagnosis and treatment; and (3) to improve generally the health manpower and facilities available to the Nation. Within each area we perceive several qualitative levels of evaluation. The first level is the occurence of pertinent events; for example, it is fairly easy to count the number of cooperative arrangments made. The second level, however, involves the measurement of the qualitative reality of the event, i.e., to what extent does real cooperation and participation exist among all of the major elements in the area and how does this facilitate steps to upgrade treatment for diagnosis and treatment. And lastly, the ultimate test of what measurable difference is made in the health status of the pati,ents andareas served by the RMI?. Each successive stage gets progressively more complex and difficult. They involve increasing amounts of data, the application of sophisticated techniques and longer time periods. But is is my conviction that the important thing in evaluation is to make the effort and not to accept the excuse of deferment or diversion because we do not have adequate and precise tools and criteria at the beginning. We believe the assignment of specific resources and a mission responsibility for evaluation is an essential prerequisite to real progress in the field. Harking back to my opening comment, about lip-service and performance, I would like to close by suggesting in paraphase to the old adage that if we are to accomplish anything in program evaluation, we must put our manpower where our mouth is. -85- U.S. OOMWMM 7 PRJ.NTIN 0 OMCES 1967 30h-835 (3003)