Chapter V. Contribution of AHRQ and Program-Wide Infrastructure

The PFQ program structure had elements that sought to contribute to the success of individual grantees and to help the program achieve its overall goals. In this chapter, we assess the role that grantee oversight played, what PFQ's infrastructure within AHRQ contributed, and how effective AHRQCoPs and other cross-grantee elements were in contributing to both grantee efforts and the success of the program overall. Our analysis is based largely on what we learned in our interviews and reflects the perceptions of AHRQ staff and grantees.

A. Grant Oversight

1. The Project Officer Role

As with other grants, an AHRQ project officer was assigned to each PFQ grant.  Decisions over assignments were made at the beginning of the program by AHRQ's management.  The assignments made an attempt to match grantees with AHRQ staff who had expertise in the grant area, though this was not the case for all grantees.  In many cases, AHRQ staff from particular centers may already have been involved at the application stage and these relationships continued. PFQ was one of the first AHRQ programs, in addition to TRIP I and II, to draw project officers from diverse centers.

Project Officer Perspectives. In our interviews with AHRQ project officers, we found substantial diversity in how they defined their roles and also in the time they put into overseeing each grant.  Traditionally, project officers have been expected to perform in administrative capacities.  One project officer depicted grantees as "customers" and said, "My role is to be a facilitator and answer their questions, and I should be able to ask them questions in return." Another described his role as, "You do as much as you can to help people."  Project officers often had many grants and spent limited time with any one of them.  This was only slightly modified by the fact that PFQ was, as project officers told us, a cooperative agreement and thus included more legally sanctioned interaction than the agency's traditional grantees.  For the most part, such project officers saw themselves as facilitating a process, not necessarily as substantively contributing to the work.  

Some PFQ project officers were exceptions, with strong substantive interest and authority in areas addressed by particular grants (for example, market forces, home health and long term care, and bioterrorism preparedness). These project officers aimed to leverage their knowledge and relationships to help grantees make connections with other efforts and resources that could help the grantees make progress or spread their impact.  Typically, such resources were outside the PFQ program and sometimes they were outside AHRQ itself.  While this subset of project officers did not necessarily spend a lot of time with any particular grant, they concentrated their efforts in ways that they hoped might leverage the substantive contributions of that potential grantee. While oriented this way, they also reverted to a more traditional project officer role when overseeing grants in areas outside their expertise, as might happen in PFQ, particularly as some grants had multiple purposes.  The project officers also triaged their time by providing more support at points where they viewed grantees needed it (like early in the project when it was being refined).

Grantee Perspectives.  Not surprisingly, grantees had different perceptions about how valuable their project officers had been. Those whose project officers were able to help them make substantive connections with others working in similar areas clearly valued the contribution. A grantee said of one such project officer, "___ has added so much to what we've done. Our project officer has made such a difference....  Our project officer is wonderful, gives us fabulous ideas, has a vision for dissemination and hears what people are saying."  Another said of a different but similarly focused project officer, "___ has been terrific—our project officer's been broadly involved.  Early on, we had weekly leadership calls and our project officer actually participated in several of these."  Similarly, others cited help the project officer had provided in making connections elsewhere in AHRQ that ultimately led to related work at DHHS. 

Bioterrorism preparedness grantees were particularly grateful for the support of their project officer, the sole AHRQ staffer for that externally funded bioterrorism preparedness work.  This project officer had what one grantee characterized as "an encouraging attitude that has been very important to the project team. It gave the team the flexibility to let their work evolve from findings in the field... The team was initially concerned about whether AHRQ would see value in this type of work, but the deeper they got into the project, the team realized that AHRQ couldn't help but see the importance... "

Grantees' also were appreciative when project officers brought other assets to their roles. One said they "loved and adored" their project officer who had been "wonderful and encouraging, always giving good advice and as laid back as possible in the parameters as the project officer could be." Another appreciated that their project officer always responded to reports, questions and thoughts, participated in some calls, came to many meetings, and helped when it was time to renegotiate the budget.  Enthusiasm also was valued in a project officer viewed as a "cheerleader" whose role was also to "make sure that we were hitting the mark."

However, almost all grantees' comments were negative when they received little feedback from their project officers.  One expressed this by saying, "I got no substantive feedback at any time in response to any of the reports I submitted... .Maybe there was nothing to say. After you've worked so hard on reports, however, some acknowledgement and feedback would be good.  I never even got an E-mail saying they got the progress reports."  Another grantee was disappointed by never being called by the project officer who was the only expert in their area at AHRQ. "Every time we call, we don't get a response.... It's always back and forth 20 times." One grantee felt differently: "___ and I have a very good relationship.  I don't bother my project officer and my project officer doesn't bother me. I do what I say I'm going to do and my project officer helps out when necessary."

Over time, some project officers were changed due to departures from the agency or problems.  One grantee said the first project officer (no longerwith the agency) was "very poor, wasn't supportive of our efforts, showed no interest in coming to our conferences, didn't provide any useful feedback on progress reports and was summarily unhelpful."  But the replacement was found to be supportive, sending out reminders when things were due and making suggestions for progress reports which the project officer also looked over and commented on.  

The principal investigator for this project suggested that AHRQ "needs to figure out what a project officer should provide in terms of support."  From its perspective, the grantee said, "project officers should function as advocates for their projects.  To do that, they need to understand the projects better, spend some time with the projects' principal investigators to craft appropriate reports... and maybe provide information on other grant possibilities or presentation opportunities. Furthermore, a project officer should function as a point person for a particular grant and help the grant better integrate with AHRQ and other national groups."  They also should not be obstructive, using as an example the actions of the first project officer who, the principal investigator felt, did not understand the project, asked for a lot of extra things that were irrelevant, and was viewed as acting in an adversarial rather than advocacy role.

One PI suggested that AHRQ invest in better training and monitor the role project officers play.  But in doing so, we perceive, AHRQ will have to address the personal preferences of its staff in a climate that appears not to value the project officer role or the time and energy demands needed to spend on any one grant.  Perhaps AHRQ might invest in training specifically to help project officers identify how they can be most strategic and effective in their support. 

2. Grants Management

For the most part, fiscal aspects of grants management within PFQ appear to have operated smoothly, though our ability to assess this is limited by the fact that our evaluation began several years into the program. The main criticism the grants office had was that PFQ, like most other agency programs, worked with a calendar that had renewals at the end of the fiscal year, thus creating imbalances in the workload.  Grants staffers indicated that memories of any earlier problems may have been erased by time or personnel reassignments, though they perceived the program to have been fairly ordinary in its experience. 

Grants Management Structure. AHRQ's grants management office told us that they typically have about 500 active grants, not including ones that need to be closed out and others on no-cost time extensions.  Though their role is administrative rather than programmatic, they see themselves as taking "care of everything from cradle to grave," with broad functions that include helping the agency determine funding mechanism, helping draft RFAs and answer questions from potential applicants, and monitoring awarded grants. PFQ grants were awarded as "cooperative agreements," which the grant office views as appropriate because of the targeted interest. While the grants management function does not change, they said, with cooperative agreements, there is more post award burden as grantees have less flexibility.  A good example is the request to use carry-over funds—which under cooperative agreements but not traditional grants—must be supported by a budget, funding memo, and explanation of why the funds were not used.

Cooperative agreements are more closely monitored than grants.  PFQ had an additional burden because PFQ decided to require grantees to submit progress reports quarterly, something that is rare with grants but more common under cooperative agreements.  PFQ evidently was one of the first AHRQ programs to require quarterly reports, which required the grants management office to establish processes to track receipt.  Problems arose when project officers did not forward the quarterly reports to the AHRQ grants management office or when turnover among project officers occurred.  The office has subsequently automated the system for tracking progress reports so that submissions are automatically tracked for other AHRQ programs.  PFQ reporting is discussed further in the next section on overall program management.

The grants management office at AHRQ uses about 4-5 specialists to help manage programs like PFQ, which has 20-21 grants, assigning a "coordinator" who is responsible for creating consistency across the information specialists provide, for example, standardized grant terms and reporting requirements.  The coordinator has participated in some PFQ meetings. 

Agency Perspectives. Grants management staff perceives that things went fairly well. There were "a few new grantees that needed a little more hand-holding," but the amount was not inordinate. Grants management and program staff worked well together in addressing the most serious grantee issue that arose in PFQ: the need to terminate a project because data to support the research was unavailable. They also processed the grantee applications and paperwork needed annually within PFQ because grantee funds are awarded annually based on amounts set at the outset of the grants.  Grantees seeking to use carry-over funds had to provide additional justification that these funds would be well-used. (Carry-over funds did not diminish the next year's award.) While the office experienced some challenges in getting project officers to be equally diligent in moving funding memos and other issues involved in grant renewal, the problems were not regarded as any different from those typically encountered. Because project officers may not necessarily spend much time in that role, sometimes, the grants management office said, they may not be as aware of the rules as they should be and thus provide grantees poor advice. For example, they might tell a grantee that its grant would follow it to another institution without realizing that this does not happen automatically.  The grants office might not learn of this until the grant renewed the next year.

From its perspective, the grants management office perceived that both the PFQ program director and individual grantees were working hard to make the program a success.  While staff believed there was some disappointment among grantees because of limited program interest by AHRQ leadership and the program's end, the office also viewed this as a generic problem for grants. At some point, office staff said, you had "to cut the apron strings and the people with good, sustainable initiatives will be able to self-sustain."  The office acknowledged that attracting general agency funds for PFQ grantees to build on the work in future efforts might prove difficult given the current agency priorities.

AHRQ's project officers were the primary interface between individual grantees and the grants management office; the program director was mainly involved in setting general policies or problem-solving.  AHRQ's PFQ project officers appear to have worked well and closely with the grants management office.  The project officers differed on their perspectives on the value of grants versus contracts and which one they preferred.  One project officer felt that PFQ was pushing grantees to work almost as contractors because of the commitment to joint meetings, conference calls and tool development.  One preferred contracts to the PFQ mechanism because of the additional control the former allows. Another, in contrast, thought quarterly reports did not add much and mainly used the annual reports.

Grantee Perspectives. Grantee perspectives on the grants management process varied. Most said the process went relatively smoothly or "as expected."  Some grantees were more negative. More than one investigator said that the grants management office might tell them they never received anything several months after it was sent, and they were annoyed at having to resend it. At the beginning, there seems to have been a problem authorizing funding for several grantees, resulting in a delayed start (nine months for at least one grantee).

Organizations new to the federal-funding process seemed to have more difficulty knowing how to proceed than others.  As one said, "This was our first AHRQ grant.  It was a nightmare. It was so hard to get answers to questions... it was confusing to figure out the requirements: When things were due, the format they wanted etc—it felt like a black hole."  Principal investigators from academic institutions whose grants were held by another organization to meet AHRQ requirements tended to perceive that situation as less than ideal.  One noted that because the grantee had never done this kind of thing before, errors in the paperwork were frequently made. Another felt that requiring the non-academic partner to be the lead was a hardship because it required a new infrastructure.  While grantees commended AHRQ on its support, they still felt that the agency had made their team go through "contortions."

While the feedback suggests the grants management went relatively smoothly, we believe the findings also suggest that AHRQ may need to think more carefully about how to orient grantees and project officers to AHRQ cooperative agreements.  Additional attention to both the burden of reporting requirements and how reports are transferred, stored and used also could be valuable.

Return to Contents

B. Overall Program Management

AHRQ uses a variety of models to support its programs, in some cases providing support with an external resource center, in others handling direction internally with limited resources, and sometimes using a mixture of the two to support different functions.  For the most part, PFQ program support followed the second model and was funded from existing operational funds. AHRQ's solicitation required grantees to cover, within their budgets, travel to attend an annual PFQ meeting; when twice-yearly meetings were held, AHRQ assumed grantees would re-budget to cover the costs of the additional meetings.  AHRQ drew upon the agency's pool of meeting support funds to cover the costs of PFQ meetings and upon its existing staff to oversee the program.

While a fair amount of energy went into thinking about the PFQ program goals and design, less attention appears to have been placed on how the PFQ would be supported within the agency.  A former agency official said the agency spent some time discussing program management infrastructure at the inception of the program because it had learned that cooperative agreements require substantial agency staffing.  However, actual decisions on PFQ oversight were made after the grants were awarded, which executives said created some confusion at the beginning, though perhaps not an abnormal amount.  At AHRQ staff's suggestion, and because it makes sense, our evaluation focuses on assessing the infrastructure that AHRQ eventually built to support the PFQ, rather than the process it took to get there.

1. Program Management Structure

PFQ is directed by a member of the AHRQ staff residing in one of its centers—the Center for Primary Care, Prevention and Clinical Partnerships (CP3).  While project officers in other AHRQ centers oversee individual grants, the program director has lead responsibility for program-wide elements.  This includes working with the grants office and project officers on decisions that affect all grantees, like reporting requirements.  It also includes oversight of program-wide elements like the Council of Partners (AHRQCoPs) and other mechanisms of communication, like the Web site.  The current director, who has been there since the first year of the program, was not deeply involved in soliciting grantees or structuring the program, but was asked later to take the program director role.  She also served as project officer for several PFQ grants.  AHRQ management was kept apprised of the program through weekly reports to and quarterly meetings with the CP3 center director.

AHRQ staff, across the board, perceived that PFQ was not very high on the agenda of AHRQ's leadership.  The CP3 center director communicates any important news about the PFQ program in regular meetings with the AHRQ Director.  Once or twice a year, PFQ is on the AHRQ Director's meeting agenda and PFQ activities are discussed. Because the PFQ program is not big, and "there are new kids on the block that take up... focus (i.e., attention by top agency leadership)," the PFQ program is not closely monitored.

The PFQ program director worked almost full-time on the program in its first 12-18 months. The program director developed the program-wide elements, such as AHRQCoPs and Contracts. She convened weekly meetings with PFQ project officers and other staff during the first several months of the program.  Project officer participation in these meetings varied, with some more likely to attend than others. But participation declined over time, particularly when meetings became less predictable due to varied scheduling. To our knowledge, decisions about the overall PFQ infrastructure (for example, role of AHRQCoPs and how often it was convened) were made at the staff level with relatively little input from AHRQ leadership on broad concepts or goals.  

PFQ used two strategies to facilitate regular communication among grantees and AHRQ, in addition to AHRQCoPs meetings, which are discussed later in this chapter.  The two strategies were:

• Grantee Reporting. As discussed previously, each grant is required to report quarterly on its progress, with annual reporting that also serves as the application for the next year's funding and request for use of any carry-over funds. Later on in the program, a PFQ progress report checklist was created (and posted on the PFQ Web site). Grantees were encouraged to fill out and submit in order to make it easier to track the progress and status of projects. 
• PFQ Web site. The Web site was the primary tool PFQ created to facilitate cross-grantee communication and interaction outside of in-person meetings. Grantees were encouraged to use it as a message board and place to store cross-cutting PFQ documents.  The site also included an events calendar for AHRQCoPs and its subgroups. 
• PFQ Staff Perceptions.  PFQ staff within AHRQ found it hard to get necessary resources to adequately support the overall program.  A good example was the Web site, which was delayed by difficulties securing resources and whose functionality was limited as a result. In addition, managing a program like PFQ can be difficult for a staff member located in a complex agency. Without stronger links to the other parts of the organization, it was hard to connect all grantees with related activities elsewhere in the agency.  The structure of AHRQ also means that program directors must rely on the interest and goodwill of project officers in other centers in helping support the program. While the PFQ uses a matrix management structure, individual AHRQ staff are evaluated by the center director without input from others.  Thus, a program director has little formal authority over who oversees individual grants or their performance. Structurally, this means that the program director's effectiveness depends on an ability to work through the informal system of relationships, and on the cooperation, participation and support he or she gets from project officers.

The absence of strong input from agency leadership also appears to have limited how well project officers understood and supported the PFQ program. Some POs had content expertise but weak administrative skills or little interest in participating in PFQ project officer team meetings. Thus, many decisions and tasks were left to the program director.  One project officer believed that PFQ "started out with a bang and ended up with a whimper," with limited attention to partnerships. Several said they perceived the program was not well-thought out and some grants were not appropriate. Another said that project officers did not know what the original goals of the program were and that the concept morphed as it went along.

Grantee Perceptions. Though none of the grantees was enthusiastic about reporting requirements, some seemed to accept them as part of the routine cost of doing work.  Grantees with less experience typically found these requirements more demanding as they had to learn how the system worked.  Some perhaps took them too literally and created more work than was necessary.  Grantees did not use the PFQ Web site and did not like the reporting requirements of the PFQ program.  The majority of grantees we interviewed said they did not use the Web site, mostly because the site was difficult to navigate and PIs did not have the time to learn its functions. Moreover, since grantees perceived that the Web site was only used for communicating and delivering documents, most found it easier to perform necessary activities by E-mail and phone call. 

Most also said they did not use the progress report checklist, which impeded AHRQ staff from regularly updating the database with project information.  The PFQ Web site was needed to access the checklist, and the fact that PIs found the Web site difficult to navigate may have been one reason why the checklist remained unused.  In addition, some PIs had issues with the design of the checklist.  One PI indicated that the terminology for the checklist was ambiguous, and would have benefited from a glossary, and another said the tool's categorical type responses lacked meaning or context.  Lastly, PIs did not appear to understand the purpose for the database, given that they were already submitting quarterly reports to update the agency on their projects' progress. Filling out the checklist for the database seemed like a "waste of time," said one grantee.  We tried to make use of the database in this evaluation and can confirm that there is no updated information after the initial entries.

Return to Contents
Proceed to Next Section