CERTs is a peer-reviewed program that conducts state-of-the-art research to increase awareness of new uses of drugs, biological products, and devices; ways to improve the effective use of drugs, biological products, and devices; risks of new uses; and risks of combinations of drugs and biological products. For more background, select to access the CERTs Fact Sheet.
On January 5, 2005, the Agency for Healthcare Research and Quality (AHRQ) held a conference call to provide potential applicants with background information and to respond to questions about the preparation of applications in response to the CERTs request for applications (RFA). This conference call is summarized below.
Background
Conference Call Agenda
CERTs Technical Assistance Slide Narration
Question-and-Answer Session
The CERTs program began with the release of RFA HS-99-004 in 1999, and the subsequent award of grants pursuant to the Food and Drug Administration (FDA) Modernization Act of 1997. The legislation authorized what is now AHRQ to carry out a demonstration program with the FDA to conduct research and provide objective information on drugs, biologics, and devices. AHRQ continues to manage this program in coordination with the FDA under the Agency's reauthorizing legislation of 1999, P.L. 106-129.
CERTs was created in response to a legislative mandate to conduct state-of-the-art research to increase awareness of:
The CERTs provide objective clinical information to:
The CERTs investigate ways to improve the quality of health care while reducing its cost through an increase in the appropriate use of drugs, biological products, and devices and the prevention of adverse effects of drugs, biological products, and devices. The CERTs also conduct research on the comparative effectiveness, cost-effectiveness, and safety of drugs, biological products, and devices. The purpose of these awards is to:
Lynn Bosco, AHRQ's Director of Pharmaceutical Studies, Center for Outcomes and Evidence, greeted participants and summarized the agenda:
Good afternoon, I am Lynn Bosco. This is a technical assistance conference call about the CERTs RFA. Thank you very much for joining us today. The people around the table are:
Next on our agenda will be the slide presentation (PowerPoint® file, 200 KB). Text description follows:
Slides 1 and 2: The agency's mission is to improve the outcomes and quality of health are services reduce costs improve patient safety,and broaden access to effective services.
My talk is divided into two parts. The first part is about the existing CERTs program. If you were to be funded through this RFA, you would be joining the network of CERTs. In the second part, I will briefly summarize the RFA.
Slide 3: The CERTs program is a research demonstration program funded as a U18 by AHRQ in partnership with FDA. It focuses on research and education on therapeutic products. By therapeutic products, we mean drugs, medical devices, and biologics already marketed in the United States. Essentially, these products are regulated by FDA. (We don't deal with pre-marketing issues.)
The centers also work with and provide research for the activities of other agencies in the Department of Health and Human Services (DHHS).
By way of an overview of the current program, the Food and Drug Modernization Act of 1997 was the authorizing legislation, but CERTs authorization is now contained within the AHRQ Reauthorization of 1999. It is a permanent AHRQ program. The CERTs are to perform state-of-the-art clinical and laboratory research.
There are currently seven funded CERTs research centers, a coordinating center, and a steering committee. The CERTs are networked by shared activities and by steering committee activities.
Slide 4: These are cooperative agreements. By cooperative agreement, we mean that the government is involved in the various projects (as is the steering committee), but the government does not control or direct them, as would be the case with a contract. It allows us to share very limited resources, such as dissemination activities, and to be involved in the steering committee. These are also public-private partnerships.
Again, the issue is to leverage very scarce resources. The current CERTs, like the proposed CERTs, receive $800,000 per center total cost on an annual basis.
Slide 5: This is a diagram of the activities of the CERTs program as it exists now. As you can see, if you were to become a research center, you would join a network of friends. Federal agencies are involved in the various research centers. The coordinating center works with the research centers. AHRQ, FDA, private industry, and foundations all have active roles within this particular program. The RFA lists the various roles that these boxes, so to speak, have within the program.
Slide 6: The vision of the CERTs is to be a trusted national resource for people seeking to improve health through the best use of medical therapies. It's very important that the CERTs be able to act both globally and locally. The mission is to conduct research and provide education that will advance the optimal use of drugs, medical devices, and biological products.
Slide 7: We are going to switch to the issue at hand, which is funding for the new RFA and the new CERTs program. We hope to fund four new CERTs. We have an annual budget of $3.2 million—which means $800,000 total cost per CERTs center per year—for 5 years of funding. This is pretty much non-negotiable. We aren't planning on funding any 4-year CERTs or 3-year CERTs. We're funding for 5 years, and we will not exceed $800,000 total cost per center in this program. The purpose of this new RFA is to expand the existing CERTs program from its existing 7 centers to 11 centers.
Slide 8: We wish to address existing gaps through the specific themes that we've identified. However, no CERTs center exists as an island, and therefore, we expect to integrate new centers with our existing centers, so there may well be some overlap. We will develop additional research in educational strategies and translate knowledge into practice both at the global and local level. We also expect to be able to develop some additional program outcome goals to add to our existing goals.
We address therapeutic issues determined to be complementary to AHRQ, FDA, and DHHS. We are, of course, particularly interested in some of the newer programs, including the Medicare Modernization Act coverage of therapeutics, and we have listed those in the RFA. We also expect to improve geographic diversity and to give priority to areas where we haven't funded a CERTs Research Center.
Slide 9: This describes a center. However, there is some flexibility in terms of what a center looks like, and existing centers certainly don't look the same. However, when we fund a center, we fund the infrastructure rather than large-scale studies. You probably know that, since $800,000 doesn't fund very many large studies.
The centers support the mission of DHHS agencies, including the National Institutes of Health, FDA, AHRQ, and others. The centers have some flexibility to modify projects to support programmatic needs, and AHRQ and FDA will certainly contribute to improving that flexibility by improving and modifying programs. The centers are to have identifiable and quantifiable program goals based on patient outcomes and to develop outreach to improve both local and national health outcomes through partnerships.
Slide 10: This RFA, unlike previous RFAs, has themes that fulfill the requirement of filling the gaps within the CERTs program. All projects must fit within one of these themes. We will not take anything that doesn't fit one of these themes. The themes are therapeutic medical devices, mental health, consumers and patients, and the elderly.
Slide 11: As for key dates, the next key date is our letter of intent receipt date, which is February 11th. Letters of intent are requested but not mandatory. The application receipt date, which is March 11th, is mandatory, not just suggested. The peer review date is sometime after May of this year, and our earliest anticipated start date is July 2005.
This is the end of our formal presentation. During our next segment we will respond to questions that people sent in. We omitted program-specific kinds of questions because we will want to answer those off-line.
First, I will mention a disclaimer, and I may repeat this a number of times and we all may repeat this a number of times. We are really trying to provide accurate information. However, if you receive conflicting information from us, the published request for application (RFA) is the primary source of information. Seek your answers within the RFA. If the answer is in the RFA, we have pulled out that information and included it in our answer today.
If the RFA is silent on the topic, then your next source of information is the PHS 398 Grant Application Form. Some people call it the NIH Application Form, but it is, in fact, the PHS 398, and it has a goodly amount of information about what you should and should not do when you're applying for a grant.
In cases where both the RFA and the PHS 398 are silent or if some issues appear subject to interpretation, program staff here will provide their opinions. Sometimes, grants management staff may also provide information from grant regulations. Again, if we express an opinion that is somehow in conflict with any of the documents that you have as part of the application process, then the documents are right and we're wrong.
I will be answering the program issues first, then Carl Ohata from Review will talk about the questions that came in about the actual review process, and then Michelle Burr from the grants management staff will follow with issues related to grants management. This is also the team that manages your grant should you be funded.
Program Questions
Scientific Review Questions
Grants Management Questions
Questions From Call Participants
Applicants may propose projects to study behavioral interventions and develop new research methods, including the ones cited in the question above. This falls into the requirement that the CERTs develop methods for dissemination and improving outcomes of therapeutics. Behavior change on the part of both the clinicians and patients is an important CERTs program goal. For example, an applicant may propose a study that aims to identify factors associated with higher research participation by minorities, and a potential companion project might be the development of an educational program that will be implemented by the community-based partner.
I received this question from multiple people. It's obviously critical. However, there are no set answers to that question. The RFA does not address that issue purposely.
Currently, funded applications have widely varying numbers of projects and add projects if funds become available. The applicant should evaluate the availability of resources and partners and the ability of the project to leverage other resources. This will help determine an optimal number of projects. So we have left it to you to determine what the appropriate number of projects is for your application.
We recommend in the RFA that the CERTs do pilot studies or studies that are feasible in terms of the limited resources. We are looking for centers that are able to cover a variety of topics within a given theme and able to demonstrate the flexibility to approach any topic within the theme. Additionally, since these projects are cooperative agreements, AHRQ may wish to work with the program to modify individual projects.
The RFA states that a senior-level individual is a person with experience in both research and implementation of change within the health care setting. The investigator is appropriately trained and well suited to carry out this work. The experience level of the principal investigator is appropriate to the requirements of a research center. The investigator and a research team have an advanced level of knowledge and have research education and dissemination experience related to the chosen theme.
The study section that reviews the applications will assess the PI's ability to successfully manage and complete proposed projects. The investigator section of the review will identify the applicant's track record of funding, publications, and research experience. RO1 funding is not required so long as the person has other comparable experience—for example, private-sector research experience instead of government-funded research experience.
At this time, AHRQ has this funding in its budget.
Leveraging and adding onto ongoing studies is a good way to make use of scarce resources. The study section at AHRQ will evaluate the value added by such partnerships. The RFA summarizes issues related to partnerships in the following paragraph.
The applicant should summarize plans in the application for additional funding through private partnerships, institutional support, and additional government funding—for example, training award to be used to assist in completing projects. For those projects where a nongovernmental partner has been identified, the applicant should include in the application a description of the steps that we currently use to assure adherence to the principles set forth for CERTs public-private partnerships. We give you these in the RFA Web site, where you can take a look at these principles, but it's quite critical that no conflict of interest or other issues be found when it comes to these kinds of partnerships; that's quite important.
We can answer questions regarding ideas. However, there are no guarantees that the study section will always see things in a similar fashion. We can assist you in ascertaining whether your project fulfills the requirements of the RFA. We are available to answer project-specific questions. We cannot, however, review grant applications and submissions before they are sent in.
There is some overlap between different centers, as has been noted. In cases where there is overlap, a research center may work cooperatively with other CERTs. Although it is not required, a center may have a methodologic or database focus that complements the theme, and a number of the funded centers do.
Specific projects and timelines should be part of this application. The level of detail should be such that the reviewers can ascertain scientific integrity and AHRQ can ascertain whether the project fulfills the requirements of the legislation, is in keeping with the mission, and has a high probability of showing an improvement in patient outcomes. AHRQ may work with the applicant to modify projects prior to funding.
Any individual with the skills, knowledge, and resources necessary to carry out the proposed research is eligible to apply for funds. Individuals from under-represented racial and ethnic groups as well as individuals with disabilities are always encouraged to apply for AHRQ programs. The individual PI must be able to devote a considerable portion, at least 15 percent, of his or her time to this project.
For the most part, the more criteria listed in the RFA that an applicant is able to fulfill, the more likely it is that he or she will be successful. However, team members with strong research skills also improve the probability of success. Although our current funded CERTs are all located at the academic medical centers, this is not a requirement for this program. Information on the seven successful applicants is available on the Web site.
These projects are demonstration cooperative agreements, a U18 mechanism. Thus, they do not need to look like an RO1 proposal. However, this is also a centers program, and the centers are expected to be able to approach a variety of questions within the chosen theme both independently and in conjunction with AHRQ, FDA, and the CERTs program.
Funding from two Federal agencies should not overlap, but Federal funding may be complementary. Federal partners may contribute personnel or other resources for research.
Yes. Projects should be described in enough detail to allow the study section reviewers to apply the criteria listed within the RFA.
Project officers from other agencies may be observers at the review meeting if their agency is interested in cofunding applications. Sometimes, Federal officials may serve on peer review committees if they have the appropriate expertise and no conflicts of interest.
I believe this pertains to the previous rounds of review. Only information on previously funded programs is publicly available. Information on previously unfunded applicants is not publicly available.
In general, yes, you should follow the rules set forth in the PHS 398 unless the RFA supersedes any of these instructions.
Finally, I'd like to make some general comments about the review committee that will be organized. Anyone listed on any of the applications is ineligible to serve on a peer review committee. Since the steering committee is available to all applicants, an external advisory committee should not be organized and listed in applications. It may be organized after the review if deemed necessary. All consultants should write a letter of support to the PI to confirm their role.
There is no requirement that a certain amount of funds be allocated for each key project. The requested budget for each project should be appropriate to the proposed study and should include only allowable, allocable, and reasonable costs.
Copies of grant applications can only be obtained under the Freedom of Information Act (FOIA). The AHRQ Web site, which is www.ahrq.gov, has a FOIA reading room, where you can find out how and where to submit a FOIA request to AHRQ.
However, if you're interested in seeing abstracts of the existing CERTs, you can go on our Web site to the grants online database, which is also called GOLD. Using the keyword CERTs you can search the database, and you will get to see the abstracts of all of the existing CERTs centers. That same information is available through the NIH CRISP system.
DHHS does not recognize co-PIs on a project. One individual must be the designated PI who directs the project and is responsible and accountable to the recipient organization for the proper conduct of the project. The organization is, of course, in turn legally responsible to the Public Health Service (PHS) for the performance and all financial aspects of the grant-supported activity.
The existing CERTs research centers currently do not receive more than $800,000 total costs per year in funding from AHRQ.
Copies of grant applications can only be obtained via FOIA. However, if you would like to see what they were proposing, you can access their abstracts through GOLD or CRISP.
Training costs, such as stipends and tuition and fees, are not allowable costs on research grants. Payments for educational assistance, such as scholarships, fellowships, and student aid, cannot be paid from PHS research funds. Funds may, however, be used for career development—for example, to support the effort of junior faculty involved in the project and for tuition remission—meaning that it is part of an individual's overall institutional compensation package.
A discrete budget page and budget justification must be submitted for each project. A composite budget page for the overall grant should also be submitted. The applicant may submit a separate research plan for each project or may submit a comprehensive research plan, provided it clearly delineates work to be performed under each of the projects. In any case, the research plan must conform to the page limitation stated in the RFA.
The answer is that if there's more than one institution participating in the CERT, one of the organizations has to be designated as the applicant organization. It will be legally responsible and accountable for the performance and financial aspects of the grant. The other institutions would be considered consortia to the project if they're conducting substantive, programmatic activities. Separate budget pages and budget justifications are required for each consortium.
The applicant organization would have to establish consortium agreements with each consortium institution that protected the interests of itself, the third party, and the Federal Government. The applicant, not the Federal Government, is responsible for handling all administrative and fiscal matters with its third-party institutions. If selected for funding, all funds would be awarded to the applicant organization, who would in turn administer funds to the third parties.
An indirect cost rate, which is also called a facilities and administrative, or F&A, rate, is a rate negotiated between a grantee organization and the Federal Government to cover costs incurred by the grantee organization for common or joint objectives, basically for overhead costs.
Most large research institutions already have a negotiated indirect cost rate; you should check with your institution-sponsored project office. Then, if you do need more information about establishing a rate because your organization does not have one, the DHHS Program Support Center has a Division of Cost Allocations, and that Web site includes information about how to go about negotiating a rate. If you don't have a rate, I would suggest that you just contact me directly (my contact information is in the RFA) so that we can discuss how to go about submitting your application and complying with the $800,000 total cost cap. Also, applicants do have, but often don't desire, the option of waiving indirect costs.
There is no dollar or number limit per se on the use of outside consultants. However, it is expected that applicants will seek appropriate expertise within their own organization before seeking the services of outside consultants. Any consultant agreements must be made in accordance with both the applicant's and the consultant organization's policies on the use of consultants.
A consultant is defined as an individual hired to give professional advice or services for a fee. A consultant generally does not carry out programmatic activities. When an individual is programmatically involved in the project and could affect the direction of the research, the usual procedure is to establish a consortium agreement with that individual's employer. For true consultants, the fee requested should be the individual's normal hourly or daily consulting fee, should conform to fees for similar services in the applicant organization's marketplace, and should be documented by the consultant in his or her consultant letter that's included in the application.
K awards cannot be considered part of a CERT, although expertise gained through projects conducted under K or other career development awards might be pertinent to the CERT application and, if so, could be included. Please note, though, that factors, such as the type of the K award, the awarding agency's policy, and the length of time remaining on the K award all impact whether or not K award recipients can receive salary support from PHS sources.
In general, a CERT application should not include in its budget any salary support for a K award recipient while the K award is active. If salary is requested and it's not appropriate to pay for it from the CERTs award, the award will be reduced accordingly and these unallowable costs will not be available to be used for another purpose. K recipients who are participating on the CERT should be shown on the CERT budget page with effort but with no salary during the period of time that the K award is active. Applicants should not propose budgetary overlap with any other current projects.
Question: We're trying to understand a little bit more about what constitutes training. Our understanding in talking to other CERTs is that they had put fellows on to CERTs. They have fellowship positions and post-doctoral positions put into there. What you said earlier made it sound like that wouldn't be allowed. Can you help me understand that?
Answer (Michelle Burr): A traditional post-doc or fellowship, someone who is not an employee of the organization, should not be being supported under a research grant. If there is an employer-employee relationship, it's something that we would have to look into in greater detail. If you have very specific questions about that, I would probably want to talk to you directly and find out more about your particular situation. Anyone who is in a training status usually is not an employee and usually is receiving a stipend, and that is not allowable under research grants.
Question: I have a question about the FOIA information request. I made a FOIA information request. It's not clear to me how long it's going to take to obtain a response. In the request, I did ask it to be expedited, based on the information on the Web site, but I still haven't heard anything.
Answer (Michelle Burr, Lynn Bosco): You could contact Nancy Comfort, our AHRQ FOIA Officer, directly and ask her about that. That's really out of the control of any us around the table.
Right, we don't do the FOIA requests here. I think the timing depends on how many requests they have.
Question:. My question has to do with the educational project, which I gather is meant to be included in the proposal, and whether that would include a needs assessment in order to determine what educational interventions might be most appropriate.
Answer (Lynn Bosco): I think what we say in the RFA is that each CERT application should have some kind of a needs assessment/strategic plan that looks at the overall needs of the theme and the educational needs. So, yes, you are correct.
Question: It was clearly stated that research proposals outside of the general themes would not be accepted. As I look at these themes, they seem to be fairly broad, especially the one that lists consumers and patients. I'm not sure of the best way to clarify this, either listing examples of themes or proposals that fit these themes or giving specific proposals that don't. I'm trying to get more concrete information or examples of what constitutes proposals that would fit within the themes that are listed in the initial slide presentation.
Answer (Lynn Bosco): Yes, they are quite broad themes. You should take a global look at the requirements within that particular theme; then you need to narrowly focus on some feasible projects within the theme. We may or may not work with you, should you be in the fundable range, to adjust those to develop some appropriate studies. Obviously you are not going to be able to cover everything within a theme, and so you will have to pick and choose, and then justify why you've chosen the particular projects that you want to do and why you've given them priority. We give you some assistance within the RFA on how to develop those priorities. You might want to look very closely at how we suggest you go about developing the projects around the theme.
Question: Would you be able to give specific examples of themes, of proposals that fall outside of these themes?
Answer (Lynn Bosco): I guess one example would be that we have a child health CERT. There are overlaps with some of these themes. We expect that there will be some overlap; clearly, there are child mental health issues. When there is overlap, CERTs work together.
Question: I wanted to clarify something I thought I heard earlier, which had to do with the external advisory committee. One approach would be to have that very detailed and specific group of people—it could be people with certain expertise in the four areas—that could serve as external advisors, as well as potentially bringing into the fold new partners who are not part of the partnership group that AHRQ CERT centers now have. Could you give some advice as to how one might approach that for the initial grant application?
Answer (Carl Ohata): The purpose of not establishing the external advisory committee beforehand is to allow the steering committee to handle all such activities. If you feel that you actually do need a steering committee, we ask that you withhold organizing it until after the review. The point to be made is that we would like to use some of the best people in the country to serve on the peer review committee. If we listed these people ahead of time, they might be excluded from serving on the committee.
Question: May I ask a follow-up question in that regard? Would it be reasonable in the grant application to say that in cooperation with AHRQ and the CERT steering committee, we would establish an external steering committee that might include these areas of expertise and certain professional organizations that are pertinent to one of the four themes?
Answer (Carl Ohata): Yes, by all means. The expertise of the external advisory committee may be listed, but do not list the people you intend to use.
Question: To give a more specific example, I'll pick something that already has a CERT theme. Say that one of the four new themes was pediatrics, just for argument's sake. The American Pediatric Association would obviously be a natural group of people to bring in who would be knowledgeable upfront. You would prefer for us not to list the specific organization, but to list our examples of professional organizations that have a high number of its members that are specific and complementary to one of the four themes?
Answer (Lynn Bosco): I think that you're perhaps mistaking or confusing having a partner versus having an advisory group, because that actually would be perfectly appropriate as a partnership with a written agreement with the CERTs Center, as opposed to coming in under the rubric of an advisory committee. Certainly a signed letter of what you're planning on doing with that group would be perfectly reasonable. If you were doing a child health CERT and you wanted to work with the American Academy of Pediatrics, then a letter talking about that cooperation would be perfectly appropriate.
Question: From my limited point of view, a perfectly reasonable way to organize those outside partners that you could describe in that way would be to make them members of an external advisory. That's where I got confused.
Answer (Lynn Bosco): Yes. It sounds more like what we've been defining as partnerships rather than advisory committees.
Lynn Bosco: Thank you very much, everybody. As I said, we will have a transcript within one or two weeks which we will distribute. If you didn't get the preliminary information that we sent out, like the slides and the questions, then let us know and we'll get those to you. If you have had problems with the FOIA, I would call the FOIA people directly: Nancy Comfort, here at the Agency. Thank you very much for participating. Bye.
Current as of February 2005
Internet Citation:
Centers for Education and Research on Therapeutics (CERTs). Technical Assistance Conference Call Summary. February 2005. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/clinic/certtacall.htm
Return to CERTs Web site
Return to Outcomes & Effectiveness
Clinical Information
Agency for Healthcare Research and Quality