Chapter 4. Discussion

We begin with two important caveats that apply to this and to any evaluation of this type. For example, it may take more time than has elapsed during the evaluation period for certain impacts to occur or to be apparent yet. In addition, it is always possible that further evaluation resources could allow more sensitivity in identifying impacts. We now review the objectives guiding the evaluation, which were to:

• Assess progress of the Portfolio towards meeting Agency and DHHS objectives in the past four years.
• Assess impact of Portfolio research on state and federal health care policy making.
• Assess adequacy of Portfolio progress reporting.
• Assess contribution and role of the Duke Coordinating Center (CC), Steering Committee, program office, and other partners to the CERTs.
• Identify strengths of the program and most successful or promising research, especially with respect to the PART goals.
• Assess role of Portfolio relative to other AHRQ and DHHS priorities.

In the following sections we briefly discuss the evaluation finding in the context of these objectives.

4.1. Progress of the Portfolio towards meeting Agency and DHHS objectives in the past four years.

The Portfolio's goals map well to AHRQ's mission to improve the quality, safety, efficiency, and effectiveness of health care for all Americans and at least some objectives within DHHS' goals 2-5. Besides the alignment of the Portfolio's goals to those of the agency and DHHS, the Portfolio has also contributed to the progress toward these goals. Specifically, the Portfolio has funded research that contributed to the knowledge about the safety and quality of therapeutics as well as expanded the capacity of the research enterprise.

The CERTs have made contributions to understanding the safety of drugs, biological products, and medical devices through the further understanding of the safety issues and risks of therapeutic agents already on the market. The CERTs have contributed to new knowledge about drugs and their risk profile. Besides the identification of unsafe medications in particular populations the CERTs have also committed extensive resources to understanding risk assessment, management, and communication in therapeutics. The Portfolio as a whole has expanded the knowledge of therapeutic efficacy and effectiveness as well as included research that aimed to further understand and improve the efficiency of health care. Additionally, the Portfolio included research that primarily contributed to the furthering of knowledge and future research (a Level 1 Impact). Furthermore, a number of projects also had a Level 2 Impact or informed policies. The level 2 impact studies were primarily on drug safety issues and risk management, change clinical practice guidelines, and quality measures. The Portfolio had only a few examples of a level 3 and 4 impact on clinical practice and health outcomes and these were usually attributed to intervention studies.

Portfolio research and awards have contributed to the knowledgebase of diverse areas in therapeutics, clinical practice, and research methodologies. The research findings of the Portfolio funded grants and research funded through the CERT program have contributed to the field of therapeutics research. The research outcomes relate to the specific areas of CERTs specialization, including drugs, biologics, and devices. The contributions are aimed at various end users including: patients, consumers, health care providers, HMOs, PBMs, government agencies, professional organizations, and other. The research has focused on diverse populations, including: children, women, minorities and ethnic groups, and the elderly. Examples of disease and organ system areas of focus include the cardiovascular and musculoskeletal system. The work of the CERTs and the Portfolio also includes the advancement of methodologies for education and research on therapeutics.

The Portfolio also demonstrated progress towards meeting the Agency and DHHS goals as evident in the outputs from the awards. The outputs included publications, presentations, as well as educational outputs. The CERTs research outputs between 2002 and 2005 included nearly 400 publications, over 200 presentations, conferences, workshops, proceedings, committee roles, and testimony to federal agencies.⁶⁸ Besides these outputs, the CERTs also developed registries and infrequently used data sources for health services and pharmacoepidemiological research.

The CERTs also aim to provide education to advance the optimal use of drugs, medical devices, and biological products. Consequently, the CERTs have provided education on clinical topics and research methods in therapeutics to researchers, practitioners, patients, and policymakers; developed educational resources (e.g. toolkits, continuing education), fostered the development of future researchers and practitioners; and initiated unusual educational initiatives. Educational outcomes of the CERTs have included the development and training of future therapeutics researchers.

The Portfolio has also contributed to the progress of the Agency and DHHS particularly via the CERTs mechanism. The CERTs are centers focused on a research theme in therapeutics funded as cooperative agreements. The CERTs yielded additional outcomes through the development of partnerships with both private and public entities. Additionally, the CERTs program was designed to create a network of collaborators, although the extent to which the investigators collaborated across the CERTs was limited. There were a few examples of cross-CERT collaboration that were viewed positively.

The structure is pretty consistent with original intent of the CERTs program. There are people doing a variety of work around these areas. We capitalize on that body of work and the available collaborations... from a funding standpoint, it creates the possibility of tapping into other funding streams. You can leverage multiple resources (and multiple people's networks). (CC)

The awareness, diffusion, and dissemination of Portfolio research varied. Hundreds of manuscripts were published and hundreds of presentations were given. The Portfolio grantees and CERTs investigators employed atypical dissemination venues as well, particularly with regard to patients and consumers although the success of those was difficult to determine. The extent to which the dissemination of research important to other researchers, practitioners and policymakers was successful is unclear. It appeared from a very small number of contacts with "external" individuals that the CERTs program itself was not always well known.

Different CERTs stakeholders (Steering Committee, AHRQ, investigators, and policymakers) identified the CERT PIs in particular as recognized experts in their respective fields. Additionally, external respondents and policymakers who were familiar with the CERTs thought they were not as successful as they could be given the capable researchers and experts in their fields, and the level of collaboration was seen as minimal. Additionally, an outside researcher believed that the CERTs did not come across as a national resource in terms of taking a stand on therapeutic issues, like Vioxx, on the market. A federal agency representative gave the example that the CERTs would have done well to demonstrate a responsiveness if for example they studied the drug needs and issues of Hurricane Katrina victims (e.g. how long took to get medications or where they unable to get critical medications). However, the representative also acknowledged that academic researchers and institutions are not always "nimble to respond." That said, every respondent who said something critical of the CERTs also said that the CERTs were given a large, perhaps impossible mandate, encompassing a large research area (drug, biological products, and medical devices) that with their level of funding would be difficult to fulfill.

"The limitation of funding, the difficulties of working in the context of a federally funded public-private partnership, and all of those represent constraints, which are a given because of who we are. But given who we are, the productivity in my view has been extraordinary." (SC)

Return to Contents

4.2. Impact of Portfolio research on state and federal health care policy making

The impact of the Portfolio research ranged from Level 1 to Level 4 impact. The majority of Portfolio research had an impact at Level 1 or future research findings whereas only a few studies had a Level 4 impact—impact on clinical outcomes. There were additional studies that had level 2 impacts, leading to changes in clinical practice guidelines, quality measures, and drug regulation and risk management. A brief description of the research for each impact level is provided below:

The Portfolio has conducted extensive research that has had level 1 impact, specifically impact on further research studies. The areas of particular contribution have been in the following areas: advancing research methods on therapeutics, medication adherence, medication safety, medication errors, identifying prescription drug trends, cost and economics of therapeutics, QT prolonging medications, HIT, and antibiotics and antimicrobials.

The Portfolio has had level 2 impacts that inform policies or programs as a result of their research findings and interventions. The specific areas where the portfolio has contributed have been in quality measures and indicators (e.g. HEDIS), drug regulatory activities, clinical practice guidelines, and medication errors.

The Portfolio research has had some examples of impact on clinical practice, primarily with intervention studies. The CERT studies have included interventions to reduce errors, order entry alerts, educational interventions on antibiotic use and prescribing, and continuing medical education.

The level four impacts on patient outcomes were rarely attributable to the Portfolio research. The few examples include the Portfolio grantees intervention studies aimed at improving antibiotic use and reducing antimicrobial resistance.

Additionally, four case studies of CERTs projects were described to illustrate how CERTs research can have or has the potential to have: the black box warning; QT interval and prescribers' knowledge; antibiotic prescribing tensions; and rickets, vitamin D deficiency and guidelines.

Return to Contents

4.3. Adequacy of Portfolio progress reporting

Progress reporting is an expected part of a grant or research funding. The available data did not allow Abt to assess the timeliness of the progress reports. Additionally, the completeness of the progress reporting was assessed from the perspective of the stakeholders. The CERTs progress reports were different in length, content, and level of detail. Besides the variance in the progress reports across CERTs, the progress reports also varied within a CERT across years in terms of factors beyond content.

The CERTs progress report commented on reporting as an administrative burden and mentioned that the timing was sometimes poor when it coincided with other key dates. The progress reports, though sometimes burdensome, were able to accurately capture and represent the advances the CERTs had made, although not all discussants felt this way. The utility of the progress reports for internal purposes varied across CERTs. However, a number of investigators were unclear on how they were used, if at all, by AHRQ. There were some comments on how the process could be improved; however the Coordinating Center is also developing the CIT program. This may be a step in the right direction to charge an entity with coordinating the projects and outputs of seven different centers made of many more investigators and their projects and publications.

Return to Contents

4.4. Contribution and role of the Duke Coordinating Center (CC), Steering Committee, program office, and other partners to the CERTs.

A Social Network Analysis was conducted to examine inter-relationships among the network constituents. In the original CERTs plan, as devised by AHRQ in conjunction with the CERT Steering Committee and its partners, the CERTs Coordinating Center was to have the role of liaison between the CERTs themselves and AHRQ, the Steering Committee, preferred partners, and other government agencies. This analysis found that the Coordinating Center is functioning very much as it was envisioned in the original CERTs plan, acting as the bridge between the CERTs and the other actors within the program. The Coordinating Center is the focal node in this network, dispersing information from AHRQ and the Steering Committee to the CERTs as well as bringing together outside partners with the CERTs based on research needs and interests. This has been both an efficient and effective way to manage the CERTs network to avoid unnecessary resource expenditures or duplication of effort to spread information and create collaborative connections.

Social network density measures indicate that the actors within the network are connected most directly with the Coordinating Center. Other measures support the liaison role the Coordinating Center plays among the CERTs, AHRQ, the CERT Steering Committee, and other partners.

As noted in interviews within this study, the role of the Coordinating Center appears to be changing and evolving into the network structure as depicted in Volume 2 Attachment 9.

Additionally, the relationships of individual members of each CERT can be vital in expanding the CERTs network under the conditions of finite resources. If a principal investigator within a CERT has worked with an individual or organization prior to being involved in the CERT, that relationship can be accessed in the future without the same level of resources as would be needed to initiate and maintain a new connection. Access and trust have already been established with that potential partner that mitigates costs and geographic proximity. Thus, an actor who has a history with a CERT or member of the CERT will be more likely to work with that CERT despite potential geographical limitations. In this situation, the Coordinating Center plays a vital role in maintaining past relationships of the CERT partners to decrease the individual resource costs to each CERT and to provide for future opportunities with the CERTS and those partners. The Coordinating Center currently does a good job maintaining these linkages and bringing together CERTs with those partners who have similar interests or particular needs. If the Coordinating Centers role as the liaison between AHRQ, the CERT Steering Committee, and the CERTS is diminished or diluted through more direct contact with each CERT, the cost to each CERT to maintain relationships and create new connections may increase. Additionally, the burden of the information processing that the Coordinating Center currently undertakes would be shifted to the individual CERTs as well, as there would be duplication in effort in providing information from both AHRQ and the Coordinating Center. It appears that AHRQ is leaning towards having more direct contact with the CERTs. The role of the Coordinating Center becomes vital to maintaining coordination and communication among and between the CERTs so that the network does not become fragmented. In most all of the CERTs network diagrams; the Coordinating Center is seen as vital actor within the networks whose removal would separate the CERT from the macro CERTs network. This confirms the vital role that the Coordinating Center plays in connecting the CERTs to each other and the broader network, providing further evidence that the Coordinating Center is functioning as originally designed within the program.

Return to Contents

4.5. Strengths of the program and most successful or promising research, especially with respect to the PART goals.

The value of engaging in an Appreciative Inquiry exercise is both in the process as well as in the output. By working together to identify the strengths and past successes, as well as future opportunities for the CERTs program, Participants were able to build upon each other's ideas, better appreciate the underlying forces that have contributed to the program's success, and then use this information to envision an exciting and inspired future. The CERTs program possesses numerous assets, which participants believe have been and continue to be crucial to the success of the program, including the level of cooperation among Participants, the cross-disciplinary backgrounds of the key stakeholders, the flexibility of the program, the role of the Steering Committee, and the role of the Coordinating Center. The CERTs program's greatest successes to-date have involved the creation, development and fostering of partnerships to further advance the education and research agenda of the CERTs. In addition, the CERTs program has had significant success in disseminating its findings, and influencing local, regional and national policies and national agendas.

The "Visions of the Future" for the CERTs program entails building upon the work that the various CERTs investigators and that. AHRQ and key stakeholders have already done. It also strengthens the role and prominence of the CERTs as a national resource for practical research on the safety and effectiveness of therapeutics. All of the Participants are deeply committed to this vision and believe it is possible.

The Appreciative Inquiry exercise yielded findings that can serve as a tool for the CERTs program as it continues to strive for its goal of providing education and research on therapeutics, by capitalizing on the strengths, acknowledging the successes and furthering discussions on potential opportunities. It is hoped however that this exercise serves as the first of many such AI exercises that the participants will engage in as they continue to build upon and leverage the program's strengths.

The Portfolio appears to be making progress toward the first two PART goals. More projects are oriented toward decreasing the inappropriate use of antibiotics in children, and some toward reducing congestive heart failure re-admission rates. This may be due to the thematic focus of the CERTs. We are not aware of any projects addressing the PART goals to reduce hospitalizations for upper gastrointestinal bleeding and to improve the appropriate use of therapies to address peptic ulcer disease. The CERTs investigators, and key leaders on the Steering Committee and senior investigators and PIs all stated a commitment to the PART goals.

Return to Contents

4.6. Role of Portfolio relative to other AHRQ and DHHS priorities.

The Portfolio is one of many portfolios within AHRQ. It has the unique focus of research on pharmaceutical outcomes. The FDA has regulatory oversight and management of pharmaceuticals, but AHRQ's Pharmaceutical Outcomes Portfolio provides a unique venue to focus on pharmaceuticals in the market and taken by patients, and the different types of understanding and research which that requires. AHRQ as an agency was identified by various stakeholders as the federal agency that focuses on areas that other agencies do not, specifically translation of research into practice, patient-focused research, practice-based research, applied research, and in the case of pharmaceuticals the understanding of pharmaceuticals post-market. An FDA representative spoke to the role of AHRQ explicitly and the role of the Pharmaceutical Outcomes Portfolio implicitly.

They're doing interesting and valuable work on products in the market, and how they're used, and how to basically do the research that FDA doesn't support and sponsors [pharmaceutical companies] don't do. When you have competing products out there what's the best standard of care and what's evidence based for choosing one over another or one drug class over another. There needs to be more of it done...but what they're doing is making a substantial contribution to both quality care and patient safety.

Within the U.S. Department of Health and Human Services and its priorities, AHRQ and the Portfolio make a special contribution to the research on safety of therapeutic agents on the market, applied in practice, prescribed by providers, and taken by patients.

4.6.1. Case Study Impact

Mechanisms of Impact

To understand the process from research findings to impact on policy or a field, processing tracing informed how the researcher identified mechanisms that led or contributed to the impact the research had. To better understand the mechanisms Checkel (2005) describes: "Mechanisms connect things. They are 'recurrent processes linking specified initial condition and a specific outcome' (Mayntz 2003, 4-5)."⁶⁹ The process tracing is when "one carefully maps the process" for example from findings to impact in the case of these case studies.³⁹ A discussion of potential mechanisms that seemed important to the impact the CERTs research case studies were able to have are discussed below. Some of the mechanisms are clearer than others as is the impact for some of the case studies greater than others.

Most examples have not been as clear-cut as UNC. (CC)

Nature of the Findings

The findings appeared to be an important factor in the impact the results may have, both in the case studies and in the other research projects. Particularly when research findings indicated harm or safety issues and new knowledge on these topics, so for the rickets study it was a "new old disease" a disease of a previous era, that had essentially returned. With respect to the impact it was able to have, it was an actionable change, (in part change in prescribers' behavior, but also just simply providing an inexpensive vitamin). The ACE Inhibitors in pregnancy study from the Vanderbilt CERT was recently published; however, it was identified by respondents as a recent example of impact (potential), in part because it determined that not only should these medications not be used in the third trimester but their study revealed that they should be avoided in the first trimester as well—a population for which safety is a large concern—pregnant women. Actionable findings also seemed to be an important factor. For the rickets study the action was clear-cut and relatively inexpensive. However, the black box warning example refers to 200 drugs for which warnings are not heeded by prescribers. It would be a great challenge to change the behavior for prescribers of 200 different drugs for a diverse group of individuals.

End User

Study impact is determined by the type of end user of the findings. For example, if research findings were regarding the clinical guidelines published by a professional organization, then the end user (the professional organization) is clear and therefore easier to target, whereas if the findings are targeted to general practitioners as a large population, it is more difficult to have an impact. An AHRQ representative makes this point.

You can communicate with certain specialty organizations. They have a vested interest because they have a subset of patients that are fairly identifiable... kids, or, people with heart disease. So that I think, it makes for an easier sell if you know whom you're selling to. It's a lot harder when you're trying to deal with something like all primary care doctors...

So both the rickets study and the QT interval involved the researchers working with the respected, specialized professional societies to impact or understand providers.

Time

Time is apparent for understanding impact. FDA stated that for the black box study it was too early to determine impact. The rickets study began around the time that the CERTs began (1999) and the clinicians' initial identification of cases of rickets was occurring even prior to that. So time has elapsed, and Dr. Schwartz pointed out that it has taken the AAP 5-6 years to change its guidelines on vitamin D supplementation.

Level of Impact

Some research findings are important to a region, however most have transferable knowledge to other regions. The rickets study is an example in NC which impacted policy, the coverage of vitamin D by the WIC program. Although the question may be an issue in other parts of the country, where they are working towards changing guidelines, this may indirectly impact national practice. Local and regional impact is easiest to address.

Dissemination/Partnerships/Networking

True dissemination is identifying who are the true end users and who can do something with the information. This was how the rickets case study clinicians and their partners came to think about the WIC. The WIC program had also given them the additional data they needed for resubmission to the Journal of Pediatrics.

It is difficult to identify mechanisms that result in a study having an impact, particularly beyond an impact on further research to impacting policies, clinical practice, and health outcomes. However, it is a worthy effort to explore the cases in which studies were able to have an impact in case there are commonalties that are mechanisms worth repeating in future research studies to further ensure an impact. Time was identified as a critical factor and when research identifies a safety concern there is a natural time lag before it can impact health outcomes, even in the best of circumstances. Context was another key variable in the ability of a research study to have impact and certainly the research findings contributed to that context. Specifically, the investigators researched topics that were timely or responsive (or at least coincided) to current issues in therapeutics, particularly safety. Connections or networking with colleagues and key decision makers was an important factor in terms of the dissemination efforts of the investigators and allowing them to impact policy and launch dissemination initiatives.

This evaluation has described the pharmaceutical portfolio from a variety of quantitative and qualitative perspectives, and in the final section above has discussed Portfolio performance against a variety of goals and objectives. Some additional summary observations follow:

• The seven CERTs evaluated appear to be making progress toward the original program goals and objectives.
• The CERTs as centers, have devoted considerable resources to what appear to be useful programs contributing to the development and training of future researchers.
• While progress reporting is one of the more mundane processes of the CERTs network, it seemed to generate a disproportionate share of potential opportunities for improvement.
• One of the advantages of the network structure is that it has led to productivity and collaboration.
• The CERTs appear to have great potential to further leverage their expertise in networking and collaborating.
• There was some indication that researchers, practitioners, or policymakers may not be as aware as they should be about the work of the portfolio as Portfolio or CERTs "products" per se.
• Participants in the AI workshop identified the dissemination of information, agenda setting, and building partnerships as three key areas in which to focus future priorities.

Return to Contents
Proceed to Next Section