Chapter V. Contribution of AHRQ and Program-Wide Infrastructure
The
PFQ program structure had elements that sought to contribute to the success of
individual grantees and to help the program achieve its overall goals. In this
chapter, we assess the role that grantee oversight played, what PFQ's
infrastructure within AHRQ contributed, and how effective AHRQCoPs and other
cross-grantee elements were in contributing to both grantee efforts and the
success of the program overall. Our analysis is based largely on what we
learned in our interviews and reflects the perceptions of AHRQ staff and
grantees.
A. Grant Oversight
1. The Project
Officer Role
As
with other grants, an AHRQ project officer was assigned to each PFQ grant.
Decisions over assignments were made at the beginning of the program by AHRQ's
management. The assignments made an attempt to match grantees with AHRQ staff
who had expertise in the grant area, though this was not the case for all
grantees. In many cases, AHRQ staff from particular centers may already have
been involved at the application stage and these relationships continued. PFQ
was one of the first AHRQ programs, in addition to TRIP I and II, to draw
project officers from diverse centers.
Project
Officer Perspectives. In our interviews with AHRQ project officers, we
found substantial diversity in how they defined their roles and also in the
time they put into overseeing each grant. Traditionally, project officers have
been expected to perform in administrative capacities. One project officer
depicted grantees as "customers" and said, "My role is to be a facilitator and
answer their questions, and I should be able to ask them questions in return."
Another described his role as, "You do as much as you can to help people."
Project officers often had many grants and spent limited time with any one of
them. This was only slightly modified by the fact that PFQ was, as project
officers told us, a cooperative agreement and thus included more legally
sanctioned interaction than the agency's traditional grantees. For the most
part, such project officers saw themselves as facilitating a process, not
necessarily as substantively contributing to the work.
Some PFQ project
officers were exceptions, with strong substantive interest and authority in
areas addressed by particular grants (for example, market forces, home health
and long term care, and bioterrorism preparedness). These project officers
aimed to leverage their knowledge and relationships to help grantees make
connections with other efforts and resources that could help the grantees make
progress or spread their impact. Typically, such resources were outside the
PFQ program and sometimes they were outside AHRQ itself. While this subset of
project officers did not necessarily spend a lot of time with any particular
grant, they concentrated their efforts in ways that they hoped might leverage
the substantive contributions of that potential grantee. While oriented this
way, they also reverted to a more traditional project officer role when
overseeing grants in areas outside their expertise, as might happen in PFQ,
particularly as some grants had multiple purposes. The project officers also
triaged their time by providing more support at points where they viewed
grantees needed it (like early in the project when it was being refined).
Grantee
Perspectives. Not surprisingly, grantees had different perceptions about
how valuable their project officers had been. Those whose project officers were
able to help them make substantive connections with others working in similar
areas clearly valued the contribution. A grantee said of one such project
officer, "___ has added so much to what we've done. Our project officer has
made such a difference.... Our project officer is wonderful, gives us fabulous
ideas, has a vision for dissemination and hears what people are saying."
Another said of a different but similarly focused project officer, "___ has
been terrific—our project officer's been broadly involved. Early on, we had
weekly leadership calls and our project officer actually participated in
several of these." Similarly, others cited help the project officer had
provided in making connections elsewhere in AHRQ that ultimately led to related
work at DHHS.
Bioterrorism
preparedness grantees were particularly grateful for the support of their
project officer, the sole AHRQ staffer for that externally funded bioterrorism
preparedness work. This project officer had what one grantee characterized as
"an encouraging attitude that has been very important to the project team. It
gave the team the flexibility to let their work evolve from findings in the
field... The team was initially concerned about whether AHRQ would see value in
this type of work, but the deeper they got into the project, the team realized
that AHRQ couldn't help but see the importance... "
Grantees'
also were appreciative when project officers brought other assets to their
roles. One said they "loved and adored" their project officer who had been
"wonderful and encouraging, always giving good advice and as laid back as
possible in the parameters as the project officer could be." Another
appreciated that their project officer always responded to reports, questions
and thoughts, participated in some calls, came to many meetings, and helped
when it was time to renegotiate the budget. Enthusiasm also was valued in a
project officer viewed as a "cheerleader" whose role was also to "make sure
that we were hitting the mark."
However,
almost all grantees' comments were negative when they received little feedback
from their project officers. One expressed this by saying, "I got no
substantive feedback at any time in response to any of the reports I submitted... .Maybe
there was nothing to say. After you've worked so hard on reports, however, some
acknowledgement and feedback would be good. I never even got an E-mail saying
they got the progress reports." Another grantee was disappointed by never
being called by the project officer who was the only expert in their area at
AHRQ. "Every time we call, we don't get a response.... It's always back and forth
20 times." One grantee felt differently: "___ and I have a very good
relationship. I don't bother my project officer and my project officer doesn't
bother me. I do what I say I'm going to do and my project officer helps out
when necessary."
Over
time, some project officers were changed due to departures from the agency or
problems. One grantee said the first project officer (no longerwith the
agency) was "very poor, wasn't supportive of our efforts, showed no interest in
coming to our conferences, didn't provide any useful feedback on progress
reports and was summarily unhelpful." But the replacement was found to be
supportive, sending out reminders when things were due and making suggestions
for progress reports which the project officer also looked over and commented
on.
The
principal investigator for this project suggested that AHRQ "needs to figure
out what a project officer should provide in terms of support." From its
perspective, the grantee said, "project officers should function as advocates
for their projects. To do that, they need to understand the projects better,
spend some time with the projects' principal investigators to craft appropriate
reports... and maybe provide information on other grant possibilities or
presentation opportunities. Furthermore, a project officer should function as a
point person for a particular grant and help the grant better integrate with
AHRQ and other national groups." They also should not be obstructive, using as
an example the actions of the first project officer who, the principal
investigator felt, did not understand the project, asked for a lot of extra
things that were irrelevant, and was viewed as acting in an adversarial rather
than advocacy role.
One
PI suggested that AHRQ invest in better training and monitor the role project
officers play. But in doing so, we perceive, AHRQ will have to address the
personal preferences of its staff in a climate that appears not to value the
project officer role or the time and energy demands needed to spend on any one
grant. Perhaps AHRQ might invest in training specifically to help project
officers identify how they can be most strategic and effective in their
support.
2. Grants
Management
For
the most part, fiscal aspects of grants management within PFQ appear to have operated
smoothly, though our ability to assess this is limited by the fact that our
evaluation began several years into the program. The main criticism the grants
office had was that PFQ, like most other agency programs, worked with a
calendar that had renewals at the end of the fiscal year, thus creating
imbalances in the workload. Grants staffers indicated that memories of any
earlier problems may have been erased by time or personnel reassignments,
though they perceived the program to have been fairly ordinary in its
experience.
Grants
Management Structure. AHRQ's grants management office told us that they
typically have about 500 active grants, not including ones that need to be
closed out and others on no-cost time extensions. Though their role is administrative
rather than programmatic, they see themselves as taking "care of everything
from cradle to grave," with broad functions that include helping the agency
determine funding mechanism, helping draft RFAs and answer questions from
potential applicants, and monitoring awarded grants. PFQ grants were awarded as
"cooperative agreements," which the grant office views as appropriate because
of the targeted interest. While the grants management function does not change,
they said, with cooperative agreements, there is more post award burden as
grantees have less flexibility. A good example is the request to use
carry-over funds—which under cooperative agreements but not traditional
grants—must be supported by a budget, funding memo, and explanation of why the
funds were not used.
Cooperative
agreements are more closely monitored than grants. PFQ had an additional
burden because PFQ decided to require grantees to submit progress reports
quarterly, something that is rare with grants but more common under cooperative
agreements. PFQ evidently was one of the first AHRQ programs to require
quarterly reports, which required the grants management office to establish
processes to track receipt. Problems arose when project officers did not
forward the quarterly reports to the AHRQ grants management office or when
turnover among project officers occurred. The office has subsequently
automated the system for tracking progress reports so that submissions are
automatically tracked for other AHRQ programs. PFQ reporting is discussed
further in the next section on overall program management.
The
grants management office at AHRQ uses about 4-5 specialists to help manage
programs like PFQ, which has 20-21 grants, assigning a "coordinator" who is
responsible for creating consistency across the information specialists
provide, for example, standardized grant terms and reporting requirements. The
coordinator has participated in some PFQ meetings.
Agency
Perspectives. Grants management staff perceives that things went fairly
well. There were "a few new grantees that needed a little more hand-holding,"
but the amount was not inordinate. Grants management and program staff worked
well together in addressing the most serious grantee issue that arose in PFQ:
the need to terminate a project because data to support the research was
unavailable. They also processed the grantee applications and paperwork needed
annually within PFQ because grantee funds are awarded annually based on amounts
set at the outset of the grants. Grantees seeking to use carry-over funds had
to provide additional justification that these funds would be well-used.
(Carry-over funds did not diminish the next year's award.) While the office
experienced some challenges in getting project officers to be equally diligent
in moving funding memos and other issues involved in grant renewal, the
problems were not regarded as any different from those typically encountered.
Because project officers may not necessarily spend much time in that role,
sometimes, the grants management office said, they may not be as aware of the
rules as they should be and thus provide grantees poor advice. For example,
they might tell a grantee that its grant would follow it to another institution
without realizing that this does not happen automatically. The grants office
might not learn of this until the grant renewed the next year.
From
its perspective, the grants management office perceived that both the PFQ
program director and individual grantees were working hard to make the program
a success. While staff believed there was some disappointment among grantees
because of limited program interest by AHRQ leadership and the program's end,
the office also viewed this as a generic problem for grants. At some point,
office staff said, you had "to cut the apron strings and the people with good,
sustainable initiatives will be able to self-sustain." The office acknowledged
that attracting general agency funds for PFQ grantees to build on the work in
future efforts might prove difficult given the current agency priorities.
AHRQ's
project officers were the primary interface between individual grantees and the
grants management office; the program director was mainly involved in setting
general policies or problem-solving. AHRQ's PFQ project officers appear to
have worked well and closely with the grants management office. The project
officers differed on their perspectives on the value of grants versus contracts
and which one they preferred. One project officer felt that PFQ was pushing
grantees to work almost as contractors because of the commitment to joint
meetings, conference calls and tool development. One preferred contracts to
the PFQ mechanism because of the additional control the former allows. Another,
in contrast, thought quarterly reports did not add much and mainly used the
annual reports.
Grantee
Perspectives. Grantee perspectives on the grants management process
varied. Most said the process went relatively smoothly or "as expected." Some
grantees were more negative. More than one investigator said that the grants
management office might tell them they never received anything several months
after it was sent, and they were annoyed at having to resend it. At the
beginning, there seems to have been a problem authorizing funding for several
grantees, resulting in a delayed start (nine months for at least one grantee).
Organizations
new to the federal-funding process seemed to have more difficulty knowing how
to proceed than others. As one said, "This was our first AHRQ grant. It was a
nightmare. It was so hard to get answers to questions... it was confusing to
figure out the requirements: When things were due, the format they wanted
etc—it felt like a black hole." Principal investigators from academic
institutions whose grants were held by another organization to meet AHRQ
requirements tended to perceive that situation as less than ideal. One noted
that because the grantee had never done this kind of thing before, errors in
the paperwork were frequently made. Another felt that requiring the
non-academic partner to be the lead was a hardship because it required a new
infrastructure. While grantees commended AHRQ on its support, they still felt
that the agency had made their team go through "contortions."
While
the feedback suggests the grants management went relatively smoothly, we
believe the findings also suggest that AHRQ may need to think more carefully
about how to orient grantees and project officers to AHRQ cooperative
agreements. Additional attention to both the burden of reporting requirements
and how reports are transferred, stored and used also could be valuable.
Return to Contents
B. Overall Program Management
AHRQ
uses a variety of models to support its programs, in some cases providing
support with an external resource center, in others handling direction
internally with limited resources, and sometimes using a mixture of the two to
support different functions. For the most part, PFQ program support followed
the second model and was funded from existing operational funds. AHRQ's
solicitation required grantees to cover, within their budgets, travel to attend
an annual PFQ meeting; when twice-yearly meetings were held, AHRQ assumed
grantees would re-budget to cover the costs of the additional meetings. AHRQ
drew upon the agency's pool of meeting support funds to cover the costs of PFQ
meetings and upon its existing staff to oversee the program.
While
a fair amount of energy went into thinking about the PFQ program goals and
design, less attention appears to have been placed on how the PFQ would be
supported within the agency. A former agency official said the agency spent
some time discussing program management infrastructure at the inception of the
program because it had learned that cooperative agreements require substantial
agency staffing. However, actual decisions on PFQ oversight were made after
the grants were awarded, which executives said created some confusion at the
beginning, though perhaps not an abnormal amount. At AHRQ staff's suggestion,
and because it makes sense, our evaluation focuses on assessing the
infrastructure that AHRQ eventually built to support the PFQ, rather than the
process it took to get there.
1. Program
Management Structure
PFQ
is directed by a member of the AHRQ staff residing in one of its centers—the
Center for Primary Care, Prevention and Clinical Partnerships (CP3). While
project officers in other AHRQ centers oversee individual grants, the program
director has lead responsibility for program-wide elements. This includes
working with the grants office and project officers on decisions that affect
all grantees, like reporting requirements. It also includes oversight of
program-wide elements like the Council of Partners (AHRQCoPs) and other
mechanisms of communication, like the Web site. The current director, who has
been there since the first year of the program, was not deeply involved in
soliciting grantees or structuring the program, but was asked later to take the
program director role. She also served as project officer for several PFQ
grants. AHRQ management was kept apprised of the program through weekly
reports to and quarterly meetings with the CP3 center director.
AHRQ
staff, across the board, perceived that PFQ was not very high on the agenda of
AHRQ's leadership. The CP3 center director communicates any important news
about the PFQ program in regular meetings with the AHRQ Director. Once or
twice a year, PFQ is on the AHRQ Director's meeting agenda and PFQ activities
are discussed. Because the PFQ program is not big, and "there are new kids on
the block that take up... focus (i.e., attention by top agency leadership)," the
PFQ program is not closely monitored.
The
PFQ program director worked almost full-time on the program in its first 12-18
months. The program director developed the program-wide elements, such as
AHRQCoPs and Contracts. She convened weekly meetings with PFQ project officers
and other staff during the first several months of the program. Project
officer participation in these meetings varied, with some more likely to attend
than others. But participation declined over time, particularly when meetings
became less predictable due to varied scheduling. To our knowledge, decisions
about the overall PFQ infrastructure (for example, role of AHRQCoPs and how often
it was convened) were made at the staff level with relatively little input from
AHRQ leadership on broad concepts or goals.
PFQ
used two strategies to facilitate regular communication among grantees and
AHRQ, in addition to AHRQCoPs meetings, which are discussed later in this
chapter. The two strategies were:
-
Grantee Reporting. As discussed previously, each grant is required to
report quarterly on its progress, with annual reporting that also serves as the
application for the next year's funding and request for use of any carry-over
funds. Later on in the program, a PFQ progress report checklist was created
(and posted on the PFQ Web site). Grantees were encouraged to fill out and
submit in order to make it easier to track the progress and status of projects.
-
PFQ Web site. The Web site was the primary tool PFQ created to
facilitate cross-grantee communication and interaction outside of in-person
meetings. Grantees were encouraged to use it as a message board and place to
store cross-cutting PFQ documents. The site also included an events calendar
for AHRQCoPs and its subgroups.
- PFQ
Staff Perceptions. PFQ staff within AHRQ found it hard to get necessary
resources to adequately support the overall program. A good example was the
Web site, which was delayed by difficulties securing resources and whose
functionality was limited as a result. In addition, managing a program like PFQ
can be difficult for a staff member located in a complex agency. Without
stronger links to the other parts of the organization, it was hard to connect
all grantees with related activities elsewhere in the agency. The structure of
AHRQ also means that program directors must rely on the interest and goodwill
of project officers in other centers in helping support the program. While the
PFQ uses a matrix management structure, individual AHRQ staff are evaluated by
the center director without input from others. Thus, a program director has
little formal authority over who oversees individual grants or their
performance. Structurally, this means that the program director's effectiveness
depends on an ability to work through the informal system of relationships, and
on the cooperation, participation and support he or she gets from project
officers.
The
absence of strong input from agency leadership also appears to have limited how
well project officers understood and supported the PFQ program. Some POs had
content expertise but weak administrative skills or little interest in
participating in PFQ project officer team meetings. Thus, many decisions and
tasks were left to the program director. One project officer believed that PFQ
"started out with a bang and ended up with a whimper," with limited attention
to partnerships. Several said they perceived the program was not well-thought out
and some grants were not appropriate. Another said that project officers did
not know what the original goals of the program were and that the concept
morphed as it went along.
Grantee
Perceptions. Though none of the grantees was enthusiastic about reporting
requirements, some seemed to accept them as part of the routine cost of doing
work. Grantees with less experience typically found these requirements more
demanding as they had to learn how the system worked. Some perhaps took them
too literally and created more work than was necessary. Grantees did not use
the PFQ Web site and did not like the reporting requirements of the PFQ
program. The majority of grantees we interviewed said they did not use the
Web site, mostly because the site was difficult to navigate and PIs did not have
the time to learn its functions. Moreover, since grantees perceived that the
Web site was only used for communicating and delivering documents, most found it
easier to perform necessary activities by E-mail and phone call.
Most
also said they did not use the progress report checklist, which impeded AHRQ
staff from regularly updating the database with project information. The PFQ
Web site was needed to access the checklist, and the fact that PIs found the
Web site difficult to navigate may have been one reason why the checklist
remained unused. In addition, some PIs had issues with the design of the
checklist. One PI indicated that the terminology for the checklist was
ambiguous, and would have benefited from a glossary, and another said the
tool's categorical type responses lacked meaning or context. Lastly, PIs did
not appear to understand the purpose for the database, given that they were
already submitting quarterly reports to update the agency on their projects'
progress. Filling out the checklist for the database seemed like a "waste of
time," said one grantee. We tried to make use of the database in this
evaluation and can confirm that there is no updated information after the
initial entries.
Return to Contents
Proceed to Next Section