Chapter 3. Findings (continued)

3.5.3. Opportunities

In order to uncover new opportunities for the CERTs, AI participants were asked to do the following:

1. Identify those areas that they believed were most important to the success of the CERTs program and where they would like to dedicate the majority of their future efforts.
2. Develop a vision for the future of the CERTs program at its best in five years.
3. State their commitments, requests or offers in order to ensure this future vision is realized.

Future Investments

Participants of the AI workshop were asked to identify those elements of the CERTs program that they hoped would continue into the future because of their overall contribution to the success of the program. Participants generated the following unedited list:

• Agenda setting, e.g. setting the national agenda.
• Partnerships.
• The dissemination of information.
• Practical research.
• Strength of cross-disciplinary membership.
• Structure of the Coordinating Center.
• Strong vision.
• Innovation and creativity.
• Resourcefulness.
• Flexibility.

Participants were then asked to identify the three areas in which they would like most to concentrate their efforts. They were:

• The dissemination of information.
• Agenda setting.
• Partnerships.

Finally, participants formed groups around each of the above themes and were asked to identify future possibilities or opportunities in each of these areas. Their ideas ranged from the specific to the more general as evidenced by the lists below. In some cases, for example, their ideas reflected guiding principles for the future more than concrete possibilities.

Group One: Dissemination of Information

• Given its broad public appeal and high ROI, leverage the North Carolina Rickets work by introducing similar initiatives to other states across the U.S.
• Develop more "patient-focused" initiatives.
• Ensure information that is disseminated meets the following three criteria: strategic, integrative and evidence-based.
• Ensure the dissemination process is dynamic and sustainable.
• Conduct an evaluation of the effectiveness of each education initiative.
• Proactively disseminate information.
• Disseminate information to a wide audience including patients, physicians, public health system, media and Internet.

Group Two: Agenda Setting

Participants felt there were many opportunities both in the short term and long term to influence local, regional and national agendas as well as ultimately affect policies, but only mentioned two strategies or approaches to do so during the AI workshop.

• Identify specific gaps in the evidence base and knowledge base to guide future therapeutic research.
• Better leverage the extensive amount of science based research that currently exists in order to inform policy decision-making.

Group Three: Partnerships

While partnerships were seen as one of the CERTs greatest strengths, participants also thought that they could be further strengthened and expanded if the following initiatives were pursued:

• Build or strengthen the "CERTs" brand image in order to strengthen its identity and reputation with partners.
• Invest in building CERTs as a "national resource" or "brain trust".
• Encourage AHRQ to proactively facilitate CERTs as a resource to other government agencies.
• Seek more and greater partnership opportunities between:
     – CERTs—AHRQ,
     – CERTs—CERTs,
     – CERTs—Government partners and
     – CERTs—Private Sector Partners.

Visions of the Future

Appreciating that individuals express themselves differently and that while some individuals are most creative and imaginative when they write their ideas down, while others are most creative and imaginative when they draw, during the AI workshop, participants were given the choice to either:

1. Draw an image of the CERTs program at its best, or
2. Develop a bold provocative statement of the CERTs program at its best.

In addition, participants were encouraged to think about the CERTs program 5 years into the future in order to encourage them to think about new possibilities and opportunities rather than focus on any current limitations or constraints.

The following provocative statements serve as a representative sample of the "Visions of the Future" that participants developed:

Visions of the Future Provocative Statements Provocative Statement #1 "CERTs is the virtual place (with 51 state centers) where government, academia, business providers and patients come together to advance the safe and appropriate use of therapeutics." Provocative Statement #2 "As a result of CERTs education and research, Americans will receive the best possible outcomes of healthcare through the optimization of therapeutic interventions and the minimization of therapeutic risk." Provocative Statement #3 "CERTs will be seen as the premier program to conduct health services research on therapeutics in partnership with both the private sector and various government constituencies (FDA, CMS, Etc.) As a group, CERTs will contribute to setting the research agenda on therapeutics. CERTs will be willing to debate emerging controversial issues in therapeutics (e.g., COX-2 Inhibitors) and disseminate informed summaries." Provocative Statement #4 "Industry supported post marketing surveillance will decrease as a result of increasing capacity and requests of the CERTs to answer critical questions in Phase IV drug development/safety assessment." Provocative Statement #5 "Headline News! Today, CERTs, the nation's oldest and most trusted resource for improved therapeutics, issued its long awaited annual report on the State of the Nation's Therapeutics: 'Healing the Nation.' This year's CALIFF award goes to the state of ..for its fully automated EMR-based real time Therapeutics Assurance and Knowledge Enhancement (TAKE) system to take therapeutics to a whole new level says the Coordinating Center's 20 year director." Provocative Statement #6 "The CERTs are a strategic interagency asset of HHS in the domain of therapeutics. They serve as a brain trust, research enterprise and developers of dissemination strategies. In these roles, they partner with both federal agencies and private organizations." Provocative Statement #7 "AHRQ was asked by the HHS Secretary to take the lead on an HHS-wide working group to plan a National Pharmaceutical Outcomes database that would support multiple studies and analysis to better inform therapeutic decision making. CERTS served as a key resource for the working group, as representatives of CMS, FDA, NIH, CDC and other agencies worked together to define options for linking and utilizing data from multiple administrative claims, clinical and survey course to build this infrastructure." Provocative Statement #8 "AHRQ was asked by the HHS Secretary to take the lead on an HHS-wide working group to plan a National Pharmaceutical Outcomes database that would support multiple studies and analysis to better inform therapeutic decision making. CERTS served as a key resource for the working group, as representatives of CMS, FDA, NIH, CDC and other agencies worked together to define options for linking and utilizing data from multiple administrative claims, clinical and survey course to build this infrastructure." Provocative Statement #9 "FDA and others were concerned about spontaneous adverse event reports related to the use of drugs in novamab class for treatment of fascinoma, amid indications of widening off-label use in patient subgroups where risk might exceed benefit. Several CERTs centers collaborated to conduct drug utilization and outcomes studies, working with their large observational datasets and their healthcare system partners. They also helped develop specs for an evidence-based review of prior studies. Results of the evidence review and new studies showed clearly that risk exceeded benefit among patients with comorbid cryptosis. As a result, the American Academy of Fascinoma Physicians and the Fascinoma Foundation developed new treatment recommendation and national media covered the published paper. The cooperating CERTs centers worked with several large health plans, Medicaid programs and Part D PDPs on prescriber mailings, academic detailing initiatives and PRODUR edits. Use among patients with comorbid cryptosis declined 75% while it increased in subsets where the risk/benefit ratio was favourable." Provocative Statement #10 "CERTs is sought after by policy makers of all types: research policy, health plan administrators policy, pharmaceutical regulatory policy, public payer, therapeutics.

Interpretation

The members of the CERTs and Steering Committee were asked to illustrate the visions of the CERTs program at its best. The participants drew pictures representing those visions, below are the interpretations of those pictures.

Member Illustration 1
"We are a group of individuals, centers and CERT totality made stronger because we partner with ourselves and others. We seek to improve public health by advancing knowledge, affecting policies and directly improving outcomes.

Member Illustration 2
"The CERTs program will be a trusted national resource for all with two way communication between CERT Centers and the following:

• Health Systems.
• Health Providers.
• NIH.
• CDC.
• Policy Makers.
• General Public.
• Research Community."

Member Illustration 3
CERTS as a guiding light in the form of a constellation of stars for MDs, patients and payers who are lost in a sea of therapeutic questions, uncertainty and confusion.

Commitments, Request, and Offers

Finally, in order to realize this vision, participants offered the following commitments and/or requests:

Commitments

• "I commit to facilitate this process through my own work & through participation in the partnership."
• "I commit to doing the work."
• "I commit to getting us there" (i.e., being a trusted national resource in the eyes of the public).
• "I commit to scientific validity, collaboration and being responsive to consumer needs."
• "I commit to being proactive in bringing resources to the program."

Requests

• "I request an integrated and increased emphasis on education and dissemination"

Offer

• "I offer to help translate and disseminate findings."

3.6. Portfolio Funding

During the period 1999-2005 the Pharmaceutical Outcomes Portfolio awarded 22 grants, 8 of which were the awards to the CERTs research centers and Coordinating Center (U18: cooperative agreement). Two grants were also awarded to CERTs research centers for the Risk Series program (Duke) and the Prescribing Safety Program (HMO). There was an additional non-CERT related cooperative agreement grant (U18). There were 12 individual Portfolio grants including: eight R01 (research project) grants; two R13 (conference) grants; and one K02 grant (career development award).⁵⁹

3.6.1. CERTs Funding

We analyzed estimates for the annual and overall percentages of CERTs research centers and the Coordinating Center (CC) funding, by source, for September 1999-October 2005 using data obtained from the Coordinating Center and individual CERTs.⁶⁰ A description of the Coordinating Center and individual CERTs research centers' funding follows.

Financial support of the Coordinating Center outside of CERTs grants and Duke University Medical Center (DUMC) contributions were generally provided for specific program-wide projects organized and led by the Coordinating Center and approved by the CERTs Steering Committee (SC). Examples of such projects include: the Risk Series, the Device Assessment think tank meeting, and the Eisenberg Lectureship.⁶¹ CERTs grants were the largest funding source accounting for 74% of the Coordinating Center's overall funding and as much as 94% of funding in 2000. 2002 and 2003 were by far the years with the most diversity of funding sources, with six and seven different funding sources respectively. In addition to AHRQ CERTs grants, the largest (in dollars) sources of funding were industry contributions, DUMC contributions. In 2004 and 2005 the DUMC accounted for over 35% of the Coordinating Center's funding.⁶²

We also examined financial source data from the individual CERTs (Exhibit 12). AHRQ CERT grants were the largest funding source for five of the seven CERTs examined: HMO Research Network, ARIZONA, UNC, Duke, and Vanderbilt. The Penn and UAB CERTs were primarily funded through National Institute of Health (NIH) grants. Penn funds were the most diversified, reporting eight different funding sources. The remaining funding sources for all seven CERTs are reported in Exhibit 12.

Funding source reporting varied greatly among the seven CERTs examined. Duke's data comprised funding received for budget years 2004-2008 (2002-2007). UNC's data comprised funding received between 9/30/02-9/29/05. Penn's data comprised funding received between 10/02-9/03.

3.6.2. CERTs Investigators' Perspectives on Funding

The stakeholders offered perspectives on the mechanisms and sources of funding for the CERTs specifically regarding the nature of CERTs funding, under-funding, and consequences of insufficient funding. The CERTs are funded as cooperative agreements awards (U18) with AHRQ. As an AHRQ representative said:

The legislation described the intent and was specific about what we needed to do. It wasn't really specific about where we were supposed to obtain the funding... there was a large number of goals - essentially everything having to do with drugs, biologics, and devices marketed. It essentially covers the entire gamut of marketed products. So, that made it challenging to ascertain exactly how you would approach that very broad mandate with extremely limited funding.

Stakeholders indicated that AHRQ released RFAs for cooperative agreements and expected CERTs to leverage these funds and to obtain external funding. For example, one CERT PI described the funding for the CERT from AHRQ as sufficient to support primarily the CERT's core infrastructure and some of its research, but that additional outside funding would be needed.

Each CERT varied with respect to how investigators and which projects were supported by the CERTs funds. In some cases individual investigators were partially funded by CERTs, while in others an investigator's research was completely funded outside the CERT, with the investigator using the resources (e.g. databases) and infrastructure of the CERT on an as-needed base. An UAB CERT investigator provided an example:

We have investigators who are funded directly by the grant, and we have people who we consider affiliate members who attend our meetings and have a looser relationship but may not have direct CERTs level funding.

Many CERTs investigators, regardless of how much funding they received from the CERT, attributed any research that was topically related to their individual CERT as "CERTs research" even if it was largely funded outside the CERT.

Of the CERT investigators who commented on funding, most characterized the amount as insufficient. Additionally, when CERTs investigators were asked about barriers or challenges to their CERTs achievement, under-funding was the most common answer (followed by HIPAA or privacy issues and its effect on research). As mentioned above, AHRQ's expectation is that investigators and CERTs centers should strive to obtain outside funding. Hence a small number of (usually junior) investigators described their CERT as having funded or supported a pilot study which they subsequently leveraged to obtain a larger study funded by another government agency (e.g. NIH, CDC). A CERT PI acknowledged the importance of the core funding they received as a center:

As long as we continue to get support for our infrastructure from AHRQ, we're fine. Should the CERTs program falter or should we not be competitively renewed we'd obviously be in some trouble, but we have broadened our base of support substantially from when we became a CERT so we do have substantial moneys outside of the AHRQ umbrella. We are, however, dependent on AHRQ to continue to help us to do what we've been doing.

A few CERT investigators identified the CERTs as having been able to conduct their research because it was smaller scale, practice-based, or local and would not readily be funded by NIH or CDC. Some investigators indicated that available funding served to 'seed' important research in the form of pilot studies. One CERT investigator put this into context:

If it's a core-funded CERT project, the money primarily funds, a fairly small amount of research— enough for a pilot project. It's not enough funding to give a definitive answer. If the focus is to get rigorous results out there as fast as possible, $10K pilots aren't the best mechanism. But if the goal is to get pilot projects out to then use to apply for other funding, then the mechanism works very well... it is good for developing projects appropriate for an NIH grant.

There was considerable agreement among stakeholders that some opportunities are missed due to the CERTs resource constraints. Many CERTs investigators identified such opportunities as ancillary research activities (e.g. dissemination), the method selected (e.g. evaluation method), or the geographic focus (e.g. regional versus national).

A number of individuals identified education and/or dissemination as the elements that usually suffered because of limited funding in the research process. As a member of the Coordinating Center stated, " We don't really have the necessary resources for dissemination and education." Two CERTs investigators with educational projects indicated that limited funding dictated the type of evaluation they were able to conduct. One investigator described the evaluation of an educational intervention:

We focus mainly on process evaluation because a major thrust of what we're trying to do is to design the messages and get them out there. We don't have a much money to conduct a thorough effectiveness evaluation.

The other investigator described limitations for evaluating the educational module:

Ideally, we would develop, disseminate, and test the module. We only had enough money to develop it and to do small scale testing. AHA posted it to their Web site. But we don't have a mechanism to broadly test the module. We would like to be able to measure whether information leads to changes in practice.

A few CERT investigators expressed their gratitude for the funding while acknowledging the missed opportunities.

I'm not complaining about our funding because we're grateful to AHRQ for what we get, but what we do is basically limited by our funding. We've got plenty of work, it's useful work, and we're very grateful for the funding we get. The more funding, however, the more that can be done.

More money would be better... it would support more research. We have been effective in using our funds. We've leveraged them into other grants and opportunities.

A few CERT PIs indicated that the limited funding had consequences for the principles and mission of the CERTs program as a whole, including the vision of collaboration, interdisciplinary collaboration, and independence:

Lack of money makes it difficult to coordinate all these centers into one superstructure with major collaborations.

A CERT problem is limited funding, not enough to fund even one project because an interdisciplinary team is required, and the level of funding in relation to salary makes it difficult to complete a study without some level of partnership.

As described above, leveraging funding is an expectation and necessity for CERTs to conduct research. One CERT PI identified developing funding as an overall goal. A CERT PI also raised the concern about leveraged funding and its possible effect on the credibility of the CERTs and their ability to be seen as unbiased, independent research centers. That PI explained:

The Level of funding does not allow work to be totally independent, and there are several aspects to that. One can be dependent because a drug company provides support. Due to low funding levels, an alternative is to stay focused with a small number of researchers. That's not what CERTs are supposed to do; there is supposed to be broad collaboration, which takes a lot of money. The only way we can do projects is to partner with someone who has a stake in the project. We also have to write grants and get outside funding; this is working but it is a very slow process — 3 years to get an NIH grant. There is much slippage in the system because we don't have adequate funding to be totally interdisciplinary and independent.

A few CERT investigators raised a point about the change in AHRQ's priorities and its impact on where funds are available and for what area of research. A CERT investigator characterized the changing focus of AHRQ in the recent past:

Funding is always going to be a barrier and it is. There is some "trendiness" to AHRQ's priorities... patient safety was a focus... and then there was a change in focus to Medicare Part D. It is difficult to anticipate this flow in AHRQ's interests.

The large majority of respondents who commented on funding included steering committee members, policymakers, and CERTs investigators; the great majority believed that AHRQ and DHHS were receiving much from the CERTs in return for the limited funding.

AHRQ is very lucky to have the group that they have do this work, especially given the funds going into the program. CERT investigators spend a lot of time giving back to the program more than they take... It's a roaring success given the... the limitations on funding and the difficulties of working in the context of a federally funded public-private partnership... but given who we are, the productivity has been extraordinary.

One CERT PI stated that he was proud that his CERT has been able to have "any influence at all" given the limited funding.

In summary, key CERTs stakeholders, including AHRQ, Steering Committee members, and policymakers acknowledged that the CERTs program is trying to fulfill a mandate for which there are high expectations with inadequate resources. As one AHRQ representative remarked "It really is a billion dollar mandate." Given this mandate and the resource constraints a few investigators provided recommendations on how those limited funds should be prioritized. One suggested putting CERT money toward nationally applicable and generalizable work, and the other suggested funding existing CERTs to spread funds less thinly. One stakeholder summed up the recognition of the importance of the core funding in advancing their research agendas:

It matters that Congress mandated and continues to fund the existence of independent academic centers to move this field forward. That is a statement about the importance of the field and the importance of a center supported by core grants... that no other strategy can do. It is important to create centers if you want to move a field forward.

Return to Contents

3.7. Portfolio Progress Reporting

Comments were made on the progress reporting system by four groups involved in the AHRQ Pharmaceutical Outcomes Portfolio: AHRQ, CERTs investigators and support staff, the Coordinating Center, and Impact Case Study Researchers. The opinions of these groups can be divided into seven categories of comments:

• Administrative burden.
• Timing.
• Accurate capture and representation of CERT advances.
• Utility.
• AHRQ's performance.
• Communication (between AHRQ and the CERTs).
• Recommendations for improvement.

3.7.1. Administrative Burden

Based on the comments made by CERTs investigators, the CERTs perceive the progress reporting process as placing a sizeable administrative burden on the CERTs. Six investigators from four of the seven CERTs commented on the administrative burden of the progress reporting process. All six highlighted that the process is an extensive one. An investigator from one of the CERTs perceived the onerous nature of the progress reports as presenting a large challenge for the CERTs. Two separate investigators from one CERT indicated that the process is overly burdensome, overly bureaucratic, and inefficient.

Although many investigators perceived the burden of the progress reporting process to be substantial, these views were generally accompanied by the perception that some system progress reporting process is a necessity for a program such as the Pharmaceutical Outcomes Portfolio, as well as recommendations for improving the process.

The Coordinating Center staff echoed these sentiments. One staff member highlighted the annual report as being particularly time consuming and the progress reporting process as requiring much work, but as with most of the CERTs investigators, this staff member perceived this burden to be necessary. Another Coordinating Center staff member perceived the progress reporting process to be reasonable and not particularly onerous.

3.7.2. Timing

Investigators from five of the seven CERTs commented on the timing of progress reporting. Three respondents focused on the timing of reports in relation to research being conducted, highlighting that reports can be redundant when a project finished long before the report is due, as well as stating that the timing of reports can result in the omission of research that is not yet 'rolled out.' Three investigators addressed the timing in reports as it related to the administrative burden of the reports. One investigator from a CERT highlighted the difficulty of completing reports within such a short turnaround period. Another CERT investigator who highlighted the difficulty of meeting internal progress report deadlines echoed this concern. An investigator from another CERT indicated that because the administrative burden of the reports is large, it is helpful that reports do not need to be completed more often.

Another CERT investigator believed the timing of the reports is appropriate. This sentiment was echoed by an investigator from another CERT who perceived the timing of progress reports to be typical of most funding agencies.

One Coordinating Center staff member commented on the timing of the reports, stating that the process is timely.

3.7.3. Capturing CERTs Advances

Fourteen investigators from five of the seven CERTs commented on whether or not annual progress reports adequately and appropriately captured advances made by the CERTs. Thirteen of the fourteen commenting investigators believed the reports captured advances made by their CERTs as adequately and appropriately as possible. One CERT investigator emphasized that, because their CERT covers a large amount of both medicine and science, it is difficult to summarize everything into one report.

No investigators suggested that the progress reports failed to adequately capture CERTs achievements, although an investigator from one CERT indicated that it is sometimes difficult to visualize the "big picture" from the progress reports. Two investigators from another CERT highlighted the difficulty of translating scientific information into lay terms. One of these investigators emphasized the great deal of effort it takes the Coordinating Center to ensure that scientific information is appropriately translated.

The perceptions of CERTs investigators on this topic are contrast with those of the Coordinating Center staff. One Coordinating staff member stated that more often than not CERTs advances are not captured by the progress reports. Two additional staff members commented on the issue, stating that the reports capture CERTs advances as accurately as possible, but that the amount of work undertaken by the CERTs is not necessarily reflected.

3.7.4. Utility

Comments on the utility of progress reports generally fell into one of two categories: utilization by the CERTs themselves, and utilization by AHRQ. Seventeen investigators from all seven CERTs commented on the utilization of progress reports, making this the most widely commented on category of the eight categories addressed.

Internal (CERTs) Utility

Thirteen investigators from all seven CERTs addressed the internal utilization of progress reports; all but one of these investigators emphasized the many positive ways the progress reports are used within the CERTs. Many investigators commented on this issue in broad terms, however, a number of investigators specified the following ways in which progress reports were internally used:

• Reflecting on current/past work (Duke, UAB, ARIZONA, UNC).
• Analyzing possibilities for future work and advancement (Duke, UAB, UNC).
• Providing progress reports to institutional IRBs.
• Preparing statements for dissemination/paring information down to the important message (Penn).
• Goal-setting (UAB).
• Organizational tool (Arizona, UNC).
• Collaboration between CERTs (UNC, Vanderbilt).
• Accountability (UNC, Vanderbilt).

Two stakeholders described the progress reports as having limited internal CERTs utility.

AHRQ Utility

Seven investigators from four of the seven CERTs commented on the use of the progress reports by AHRQ. Three investigators at three CERTs highlighted the necessity for AHRQ, as well as for other funding organizations, to have a progress reporting mechanism for holding CERTs accountable for their research goals, but also so that AHRQ may better understand the key issues affecting the CERTs and assist in promoting collaboration among them.

However, despite the recognition of the necessity of a progress reporting process, a number of investigators from two CERTs indicated that it is unclear how AHRQ has made use of these reports. One CERT investigator added that AHRQ does not have any knowledge regarding what their CERT has accomplished while another from the same CERT expressed surprise at the fact that a recently submitted report resulted in helpful feedback from AHRQ. Another CERT investigator stated that although they have not yet received feedback from this year's report, the reports generally do generate feedback from AHRQ.

One CERT investigator stated that the annual progress reports for stakeholders, Congress, and others gives all CERTs equal advertisement and appropriately translates scientific language while still accurately delivering the message.

An AHRQ respondent perceived the progress reports as allowing AHRQ to understand what had transpired within the CERTs over the previous year and to determine if any issues existed that required AHRQ's attention. AHRQ also saw the progress reports as a method to determine if any CERTs findings/products required further dissemination.

One out of four Coordinating Center staff members interviewed stated that the annual report highlights the major findings and initiatives of each of the centers and allows non-scientists to better grasp what the CERTs have accomplished over the past year. However, uncertainty about the use of the progress reports by AHRQ was also expressed.

Coordinating Center Utility

Two out of the four Coordinating Center staff members interviewed highlight that the coordinating staff prepares an annual report of their own volition. One of these staff members additionally highlights the large number of reports the Coordinating Center has compiled for the CERTs Information Technology Transfer (CIT) project. This project is further addressed in the section below.

An additional Coordinating Center staff member stated that the progress reports are "very helpful" for the Coordinating Center.

Return to Contents
Proceed to Next Section