Step 4: Communicating and Reporting

Step 4 will focus on communicating with and reporting research to three primary stakeholders: funding agencies, health services research professionals, and the community (public). Each is an important stakeholder; each is an important audience for your research findings.

Communicating with a Funding Agency

Communicating with a funding agency is important and can result in:

• Information about research and funding opportunities.
• Information about relevant conferences, webinars.
• Technical input and advice.
• Practical advice about dealing with administrative requirements and reports.
• Guidance and advice about preparing future solicitations.

Project Progress

Regularly provide updates on the progress of your initiative, program, or project. Plan for and collect information for the annual progress report.

• Begin planning at the outset of your initiative on how you will monitor and collect information for the annual report.
• Before the annual report is due, be sure that you are clear on the expectations and format for the annual report.
• Consider asking a colleague or your funding agency to share exemplary progress reports that may serve as a model for your report.

Budget

It is important to be aware of and monitor the status of the budget throughout the project period. Identify and develop a working relationship with the financial/business office at your institution. Keep abreast of the following elements:

• Total amount spent.
• Amount spent by project or task.
• Labor hours and costs expended for each staff member.
• Other direct costs expended, such as consultant fees and supplies, broken down by type.
• Percentage of total budget remaining.
• Percentage of time remaining on the grant or contract.

Methods for Communicating

When possible, funding administrators suggest face-to-face meetings. Plan to have face-to-face meetings with staff when they are going to be at the same conferences or workshops. Although face-to-face meetings are preferred, they often aren't possible because of distance and funding constraints. Use other means to stay in contact, such as video conferences, teleconferences, and email.

Frequency

Scheduling communication should be done to fit the requests and preferred style of the funding agency: Do they prefer to meet on an unscheduled, as-needed basis or to hold meetings by a defined schedule? Adjust the frequency of communication based on needs and the level of activity on the project.

Other Communication Options

If possible, engage in opportunities to communicate during site visits and PI meetings. Site visits consist of project officers visiting funded institutions or States and usually involve interaction with project staff at multiple levels. Meetings with PIs involve gathering all of the currently funded PIs from a particular program to facilitate networking and information exchange. PI meetings can be in-person or by teleconference.

Return to Contents

Communicating with Health Services Research Professionals

In research, publications and presentations are a major metric for assessing how successful you and your organization are. Publishing in peer-reviewed journals will bring positive attention to your organization and showcase your staff's research abilities. Go to Figure 6 for a list of leading health services research publications. Presentations at conferences will allow you the opportunity to get feedback on your research and network with others in your field. Presentations are often the basis of journal articles. Conferences also give you exposure to funding agencies that sponsor the type of work you do.

Engender success by establishing clear expectations for publishing and conference presentations from staff. Given the advanced planning required for getting publications and conference abstracts accepted, it is advantageous to allocate sufficient time and resources to the activity. To maximize success and the quality of presentations and publications, consider using some of the technical research center support staff. Technical writers, biostatisticians, mentors, and advisory group members can all assist in the development of materials for submission.

Figure 6. Leading health services research publications

Leading journals for health services research
American Journal of Public Health Health Affairs Health Policy Health Services Research International Journal of Health Sciences Inquiry Journal of the American Medical Association Journal of Health Economics Journal of Health Politics, Policy, and Law Medical Care Medical Care Research and Review Milbank Quarterly New England Journal of Medicine Social Science and Medicine

Return to Contents

Sharing Findings with the Community

Translating your research into practice and policies that benefit the community is the ultimate goal of most health services research. This requires communication to stakeholders, such as providers, patients, and policymakers. Various media sources can be effective, such as web sites, blogs, and social media networks; newspaper articles and editorials; mailings; radio; and television. Presentations at town hall-style meetings also may be appropriate. Sharing your research can garner community support. Communication specialists can provide you with necessary guidance.

Maximizing Success

How you choose to communicate with funding agencies, the research community, and the community as a whole will vary. What is most important is that you are communicating with all of these audiences. Be open to varied methods to reach each audience.

Return to Contents

Step 5: Evaluating the Infrastructure Support Initiative

Purpose of Evaluation

Evaluation lets you see where your program stands in relation to the goals you've set. It can help you see if you are where you expected to be and doing the things you expected to be doing at a given point in your program. It is important, therefore, to plan for the evaluation at the outset of your project or initiative. Evaluation results can be used to:

• Support both short-term and long-term planning.
• Improve strategies and activities.
• Justify allocation of human, physical, and financial resources.
• Provide visibility for your program or research center to the community, funding agencies, in proposals and grants, and to your organization's leadership.
• Guide the development of best practices in your field.

Return to Contents

Planning for Evaluation

It is impossible to evaluate every aspect of your program or initiative. Consider what is most important to get out of the evaluation and how you will use the results. You might consider going back to your goals or your funded proposal to help guide your decision.

Develop a logic model^b to help organize your evaluation (Figure 7). A logic model provides a visual showing of the relationships between the resources your program invests, the activities that take place, and the outcomes that result. It can be thought of as a roadmap for your program and for your evaluation.

Figure 7. Basic logic model framework

Return to Contents

Types of Evaluation

A process-based or formative evaluation examines how a program works. This type of formative work enables the detection and correction of problems and inefficiencies. This type of evaluation looks at things like:

• How did administrators, faculty, staff, and students become aware of the resources available through the research center?
• What made it easier to implement capacity-building activities?
• What has made it harder to implement capacity building activities?
• What improvements or changes do administrators, faculty, staff, students, and/or advisory board members recommend?

An outcomes-based or summative evaluation helps you answer questions about the effects your program has had. Outcomes can be thought of as the benefits from the activities your program provides. This information tells how successful your project has been with respect to the attainment of desired outcomes and goals. First, you will want to identify the major outcomes for your program. There are several ways you can do this. For example:

• It may be helpful to think about what activities you are able to implement or are planning to do and then define your outcomes from that perspective, asking yourself “What do I hope to achieve from this activity?”
• Look at your organization's or department's mission, vision statement, or strategic plan to identify your major outcomes: What is it that your organization aims to do?
• Refer to your funded proposal for the program or initiative and examine the outcomes you proposed there.

It is not always feasible to evaluate every major outcome you have identified. You may need to prioritize and choose a smaller number of outcomes to examine. To help prioritize which outcomes to evaluate, select those that are: specific, observable, measurable, realistic given your program activities, and meaningful to your stakeholders.

Return to Contents

Creating an Evaluation Plan

As you did for the Assessment (Step 1), create an evaluation plan. The evaluation plan should include:

What information will be collected?

Develop a list of variables. The evaluation items in the Appendix may provide you with some assistance in developing/selecting variables and questions. Your variables list should include information on potential sources of the data. In deciding how to collect information, balance the quality of information with the cost-effectiveness of data collection. Methods for collecting data might include: surveys, collecting archival data, getting information from Federal/public databases, discussions with staff, and advisory group feedback. Use the variables list for developing data collection tools such as forms, discussion guides, and surveys.

When will data collection occur?

Evaluation can be a resource-intensive activity. Consider how frequently you will conduct a formal evaluation. Perhaps you will decide to collect some data more frequently than others. For example, you may decide to collect publication and presentation data from staff on an annual basis. However, you made decide to collect information on policies every 3 years, since policies are more static. You will probably want to collect information early on in your initiative and then on an ongoing basis to look at changes. Formative data are valuable and can inform what program modifications may be necessary. Determine your schedule for collecting data based on how you plan to use the information.

Who will do the evaluation?

Consider what resources you have to conduct the evaluation and who will conduct it.

• An independent evaluator frees up staff time, eliminates potential conflict of interest, and ensures expert analysis of program data.
• Internal staff members are familiar with the organization and the research center, but they may not be familiar with evaluation techniques or they may lack the perspective that an independent evaluator brings.
• An advisory group may be able to provide useful program review, feedback, and guidance, particularly if you are unable to conduct a formal evaluation.

Maximizing Success

• Begin planning for the evaluation at the outset of your initiative. Allot the time, costs, and labor needed to conduct an evaluation. To focus the evaluation, develop a logic model, revisit your goals, or review your funding proposal.
• Develop an evaluation plan and variables list to guide your data collection efforts.
• Share the results of the evaluation widely to promote your work and show how the feedback you got was meaningful.

Return to Contents

Step 6: Planning for Sustainability

Importance of Sustainability

Maintaining a health services research program can be an arduous task. While there are many challenges to sustaining a program, the greatest challenge is locating and obtaining subsequent funding.^{ii, vi, vii, viii, ix} You can overcome this challenge by creating a sustainability plan at the outset of a new program.

Return to Contents

Creating a Sustainability Plan

Begin by creating your sustainability plan at the start of your program. Implementing a sustainability plan is a five-step process.^x

Five Components of a Sustainability Plan

1. Assessment.
2. Development.
3. Implementation.
4. Evaluation.
5. Reassessment/ modification.

Start by assessing your vision of the program in the next 2, 5, and 10 years. Establish the business case or reason for your program; what makes your program worth the time, expense, and effort? Answer these questions:

• What do you want your program to look like 2, 5, and 10 years in the future?
• What will be the research priorities or research focus for your program
• What type of staff and equipment will you need to conduct your research?
• Who will be your partners? What type of support will they provide?
  How much money will it take to support this vision? What will be the source(s) of funding?
• What impact will your program have in the research community?
• What will your program's community impact be?

Elements of a Sustainability Plan

Based on the answers to the questions above, develop the following elements to be inserted into the sustainability plan. For each of these elements:

1. Set measurable, quantifiable goals.
2. Discuss the means for achieving these goals.
3. Set timelines for achieving your goals.
4. Set new goals once previous ones are achieved.

One or more of the following elements may make up your program's sustainability plan:

• Short and long-term goals:
  • Short-term goals are those set for 2-5 years. These goals may be process-based, particularly for seed organizations, or outcomes-based for fertilizer organizations. Some examples of short-term goals include: meeting timelines, completing project activities, developing project-related publications, generating research/future funding ideas based on current project work, receiving positive feedback from internal and external stakeholders, and increasing student interest in health services research.
  • Long-term goals are primarily outcomes-based goals and can be accomplished in 5-10 years. Some examples of long-term goals include: retaining a team of 10 doctorate-level health services researchers, creating a health services educational track in a university setting, and obtaining a particular amount of research funding each year.
• Research priorities. Define and limit what your program's thematic focus will be. Although this may seem limiting, it can help you to develop experience and expertise in a specific area or niche. Take into consideration funding opportunities and priorities of current and potential funders.
• Staff retention and recruitment. Attrition is a natural part of any organization's growth. Retain staff by supporting their efforts and recognizing their work via verbal appreciation, awards, promotions, and raises. Recruit new staff with applicable skills and interests in health services research.
• Partnerships. Identify strategies for improving your relationships and the effectiveness of the partnerships. Create opportunities to sustain and strengthen the partnerships. Establish new partners by networking at conferences. Recognize the unique knowledge, skills, and capabilities your organization possesses. Summarize these in a one-page document to distribute to potential partners.
• Funding opportunities. Use your research priorities to identify appropriate funding agencies. Monitor potential funding opportunities released from these sources on a weekly or daily basis. Store the information on the funding opportunities in a searchable database that is accessible to key stakeholders in your organization.
  • Consider ways to expand on your current work that can lead to future funding. Explore whether a current funding agency has additional monies or suggestions about other agencies that may fund related projects.
  • It can be particularly challenging to maintain your desired level of infrastructure or research center support without adequate financial resources. Possible sources of funding include your organization's overhead, individual research project funding, and new infrastructure support funding.
• Publications and presentations. Publications in leading, peer-reviewed health services research journals will bring positive attention to your organization and demonstrate your research abilities. Presentations at conferences will also demonstrate your skills and allow you the opportunity to network with others in your field. Conferences also give you exposure to funding agencies that sponsor the type of work you do. Budget and schedule time for writing, travel, and registration fees.
• Community impact. Positive community impact can make it easier for you to engage participants in the future. Community impact may include: enhancing health care through the research projects, informing government policies with research project findings, or distributing health-related information to the public. Share the results of your research widely. Go to Step 4 for more information.

Return to Contents

Implementing the Plan and Monitoring Progress

Build the support systems necessary to implement and achieve your plan. For example, to achieve your publications and presentations goals, budget funds to support attendance at conferences or employ the use of a technical writer to assist with writing and editing publishable articles. Information for building support systems can be found in Steps 2, 3, and 4 of this Guide.

Use the assessment and evaluation plan to ascertain the gap between your current position and the vision for your program. Revisit the plan every 6 months to 2 years to make appropriate modifications to the plan. Monitor your progress by reviewing the results from your assessment and evaluation activities. Be sure to include key measures from the sustainability plan as variables in the evaluation. Refer to Steps 1 and 5 for more information on conducting assessments and evaluations.

Maximizing Success

• Begin planning for the long-term by creating a sustainability plan at the outset of your program.
• Monitor potential funding opportunities closely.
• Encourage and support staff publications development and attendance at conferences. These activities also demonstrate research skills and give your program exposure to funding agencies and potential partners.
• Look for funding for the research center via your organization's overhead, individual research project funding, and new infrastructure support funding.

---

^b Logic models provide a visual showing the relationships between resources, activities, and outcomes of your program. Additional information about logic models can be found at www.cdc.gov/healthyyouth/evaluation/pdf/brief2.pdf. (Plugin Software Help)

---

Return to Contents

References

ⁱ Lohr K, Steinwachs D. Health services research: An evolving definition of the field. Health Services Research, 2002; 37(1):15-17.
ⁱⁱ Cooke J. A framework to evaluate capacity building in health care. BMC Family Practice, 2009;6(44). Retrieved on February 3, 2009 from http://creativecommons.org/licenses/by/2.0.
ⁱⁱⁱ Zea MC, Belgrave FZ. Mentoring and research capacity-building experiences: Acculturating to research from the perspective of the trainee. American Journal of Public Health, 2009; 99(S1):S16-S19.
^iv Yanagihara R, Chang L, Ernst T. Building infrastructure for HIV/AIDS and mental health research at institutions serving minorities. American Journal of Public Health, 2009; 99(S1):S82-S86.
^v Carey TS, Howard DL, Goldman M, et al. Developing effective interuniversity partnerships and community-based research to address health disparities. American Medicine, Management Series: Strategic Alliances in Academic Medicine, 2005; 47-54.
^vi Harrow J. ‘Capacity building' as a public management goal: Myth, magic or the main chance? Centre for Public Services Management, 2001; 3(2):209-230.
^vii Northridge ME, Sidbe S, Gohel TJ. Environment and health: Capacity building for the future. American Journal of Public Health, 2004; 94(11):1849-1850.
^viii Wing KT. Assessing the effectiveness of capacity-building initiatives: Seven issues for the field. Nonprofit and Voluntary Sector Quarterly, 2004; 33:153-160.
^ix Yung B, Leahy P, Deason LM, et al. Capacity-building needs of minority health nonprofits. Evaluation and Program Planning, 2008; 31:382-391.
^x Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: a sustainability planning model. Evaluation and Program Planning, 2004; 27(2):135-149.

Return to Contents
Proceed to Next Section