Research and Evaluation

To organize concepts for both program development and evaluation purposes, theories of change and logic models serve as “roadmaps” that graphically depict the connections between the community context, the actions to be undertaken, and the desired outcomes. Please see the documents below for examples of theories of change and logic models as well as guidance for creating your own.

• Theory of change
• Logic Models
  • Template with guiding questions
  • Example logic models from the Center of Excellence
    • Early childhood education
    • Home visiting
  • Example logic models from program evaluation reports
    • ACBH Logic Model
    • CCEP Michigan Logic Model and Theory of Change
    • MDSE Logic Model
    • Smart Support Logic Model and Theory of Change

The evidence base for Infant and Early Childhood Mental Health Consultation has grown significantly in recent decades. Learning about evaluations of other IECMHC programs helps teams to develop their own questions that build upon the foundational evidence.

• Evidence Synthesis: This brief summarizes the status of the evidence for IECMHC, including both peer-reviewed journal articles and evaluation reports. Future directions for research are provided.
• Annotated Bibliography: Many key findings from innovative studies can be found in peer-reviewed journal articles, but many professionals do not have access to these journals. The Center of Excellence has created a comprehensive Annotated Bibliography that summarizes the important contributions of each known, published study of IECMHC, cutting across different settings and outcomes of interest. In addition, papers that describe IECMHC practices or theories but do not present new analyses are also included. This resource will be updated annually to provide consultants, administrators, students, and all other interested individuals with current research findings on IECMHC. The Annotated Bibliography is intended to build general knowledge about IECMHC, facilitate writing about IECMHC (for grant applications, reports, etc.), and to demonstrate future directions for research on IECMHC. In this resource, you will have access to:
  • Descriptions of each article’s unique contribution to the empirical knowledge of IECMHC
  • APA-format citations for each article
  • Abstracts for each article

Program Evaluation Reports

• To guide your research questions, identify and consult with stakeholders, including families, funders, and providers. Ideally, representatives from all interested parties should be included in all aspects of the evaluation. Families and community members from diverse populations can offer key insights to help formulate relevant questions and identify variables to measure.
• Enhancing equity is a foundational goal of IECMHC; all evaluations should use their data to answer questions related to closing disparities, addressing biases, providing culturally and linguistically appropriate care.
• Primary questions will be driven in part by the funders, the policy climate, the model, and what can be measured accurately using reliable and valid tools. But useful program evaluations should also address questions that are relevant to the early childhood program, administrators, providers, and families. It is important to consider several key factors.
  • What questions MUST you answer (to meet reporting or other requirements)?
  • What questions would you LIKE to answer and WHY (how will you use the information)?
  • Expected effects—what is realistic, based upon your IECMH program implementation? Identify those aspects of early childhood practice that can be expected to change as a result of the “dose” of IECMHC your program is actually delivering. Consider collecting data about possible barriers to successful implementation as well.
  • What information can you readily collect? What reliable tools are available to collect these data?
• For additional information on community-based participatory research specific to evaluations in tribal communities, please access A Roadmap for Collaborative and Effective Evaluation in Tribal Communities.

• When developing the data collection plan, it is necessary to consider the participants that you will likely be able to engage (e.g., parents, home visitors, children, etc.) as well as the languages and cultural background of all participants. Evaluators should learn about the psychometrics of each measure (if available) and translation procedures, and should examine measures item-by-item to ensure that they are appropriate for the participants and measuring what is intended.
• Measure PROCESS and OUTCOME variables
  • The Center of Excellence created a searchable tool for selecting Outcome Measures for IECMHC for evaluators to explore outcome measures used in prior ECE-based IECMHC evaluations.
  • Process variables are critical in moving beyond asking whether consultation had an impact to answering WHY and HOW it has the effect that it had. Process variables can lead to fruitful conversations about strengths and possible areas for improvement for the consultant or the consultation program overall. They can also allow programs to explain null findings and explore predictors of success. Questions that can be answered with process variables include:
    • What is the “dose” of consultation provided to each consultee?
    • Is consultation being delivered with fidelity (i.e., consistent with the program’s written guidelines or program manual)?
    • How strong is the relationship between the consultant and consultee?
    • How often does the consultant receive reflective supervision, and what is the quality of that supervision?
  • All evaluations should collect demographic information from all participants to allow for disaggregated analyses
• Consider Qualitative Data
  • Gathering qualitative data involves the collection of non-numerical information by various methods such as recording, transcribing, and analyzing interviews or holding focus groups with staff members, family members, or consultants. Qualitative data often take the form of personal stories or case studies that convey the details of an individual’s experience with IECMHC. These data can stand alone, or can help contextualize the results from quantitative analyses. Qualitative data are critical for weaving in a range of ways-of-knowing, honoring the wisdom of all stakeholders, and incorporating clinical insights and cultural values.

• Data collection and management
  • It is essential to have a well-organized plan for your data that is exempt or approved by the relevant Institutional Review Board (IRB) prior to beginning data collection. The written plan should address:
    • What information will be collected
    • Who will ensure that consent is obtained from all participants
    • The schedule for data collection (e.g., baseline, and then after every three months of consultation)
    • Who will be responsible for collecting data
    • Where the centralized data will be stored; and
    • Who will take responsibility for ensuring data are recorded, compiled, and reported, including monitoring for missing data and ensuring that they are managed in a HIPAA-compliant manner.
  • Given the likelihood of barriers to data collection, including staff turnover, create a system for double-checking that data collection is happening on schedule for each participant. If you are gathering data pre- and post-intervention, it is essential to have procedures to ensure that post-intervention data are collected – analyses hinge upon having a sufficient quantity of data at the post-intervention time point.
• Data analysis
  • Depending on the skills and expertise of the evaluation team, consider a formal relationship with a university-based researcher or similarly trained professional. The analyst should have the ability to select a particular statistical test or qualitative analysis strategy that fits the type, quantity, and quality of data that was collected and answers the research questions.
  • Create a data analysis plan that addresses:
    • The type of data analysis to be performed
    • Who will complete the data analysis
    • How often the data will be analyzed
    • The format and process for sharing preliminary and final results
    • Expectations for disseminating the findings
• Communicating Results
  • The contents of any evaluation report depends on its purpose and the intended audiences.
  • Evaluation data can be crafted into a message by blending science, marketing, communications, and graphical skills. The information can convey concrete take-away messages, comprehensible facts, and ideas for promoting consultation as a valuable service to young children, families, staff, and programs.
  • Make it accessible and meaningful! Can the public, policy-makers, and other stakeholders, including families and providers, easily understand what the data mean and what the implications are for the child, family, staff, programs, and ultimately the community well-being?
  • All system stakeholders should have an opportunity to review the data to provide interpretations. The community members should be informed of not only the results, but also the potential implications of the findings.
  • Publication in a final report or professional journal should not be the primary means of dissemination of community-based research. Ideally, the results should be published in user-friendly formats accessible to all diverse populations.
• Ongoing data collection is valuable not only for reporting to an external audience, but also for internal program improvement and development efforts. This is called Continuous Quality Improvement (CQI). Preparing interim, and periodic reports provides ongoing opportunity for reflection, reviewing program performance, making mid-course corrections, and hopefully, celebrating successes.
• Examples of programs that effectively communicated their evaluation results:
  • Using Data to Show Effectiveness and Promote Sustainability
    • Key staff from Maryland explain how they used data to illustrate their IECMHC efforts and outcomes, leading to the formation of an effective message to share with decision makers.
  • Using Evaluation Data to Make the Case for Securing or Expanding IECMHC Funding
    • The Maryland State department expanded IECMHC funding based on positive outcomes and proven effectiveness.