|
|
|
July 2008
Assistant Secretary for Planning and
Evaluation (ASPE)
U.S. Department of Health and Human Services
(HHS)
| As a pioneer in broad-based marriage initiatives, the Oklahoma Marriage Initiative (OMI) has charted new territory. Recognizing that there was little prior information to guide implementation designs and strategies for pursuing its goals, OMI planners enlisted the help of a range of research experts from around the country. These experts did not necessarily have ready answers to the challenges faced by the OMI, but were willing to help analyze emerging issues and provide input based on the best available information. This advisory panel has remained engaged since the OMI’s beginning, with its function adapting to evolving needs. With the panel’s guidance, Oklahoma was the first state to conduct a statewide survey of its citizens’ attitudes and behavior with respect to divorce and marriage. Survey findings were used to inform OMI program decisions and to educate Oklahomans about marriage and divorce in their state; they also may be used as a baseline against which to compare later outcomes. As the initiative has unfolded, the OMI has used research to assess and inform progress, continue expansion, and explore outcomes. |
Printer Friendly Version in PDF format (10 pages)
This Research Brief is available on the Internet at:
http://aspe.hhs.gov/hsp/06/OMI/Guide/rb.shtml
As the Oklahoma Marriage Initiative was being developed, its planners made a commitment to rely on research to guide its development. Research has been integral to its evolution, starting with the findings on family structure that first stimulated the idea for the initiative, to the development of subsequent strategies and approaches for implementation. Since 2001, the OMI has been guided by a panel of state and national experts on marriage, divorce, and low-income families. This interdisciplinary Research Advisory Group (RAG) includes academic scholars, university-based practitioners and researchers, and policy experts and evaluators who meet annually and sometimes contribute to other OMI research activities throughout the year. Their ongoing activities, including, for example, conducting small-scale studies of OMI programs and assisting in the development of dissemination materials that translate research findings for a broad audience, have two main benefits: they provide data on which the OMI can base continued development and improvement of program operations, and they lend credibility to and build awareness of the OMI within and outside of Oklahoma.
As the initiative has developed and matured, the research advisory group has considered strategies for assessing the effects of the OMI’s overall approach on Oklahoma families. Given the interest in obtaining evidence about statewide change, OMI planners have considered an assessment of broad shifts in attitudes toward marriage and reductions in nonmarital childbearing and divorce rates. More recently, OMI staff have begun to sponsor studies of short-term outcomes and participate in evaluations of the long-term impact of OMI services on particular targeted populations. These latter, more rigorous, evaluations are assessing the impact of services on the quality and stability of couples’ marriages and relationships and the well-being of their children. OMI officials also view research as a resource for providing information about families, marriage and divorce to the broader population, as exemplified in the series of “tip sheets” they have issued.
This brief describes how the OMI has used research in planning, ongoing operations, expansions, and evaluation of outcomes. It also discusses the creation and use of a research advisory group and the development of strategies for incorporating research in the context of a dynamic, evolving, broad-based initiative.
[ Go to Contents ]
The idea of a panel of experts guiding or providing advice for large-scale policy change efforts is not a new one. The OMI expert panel, however, took a somewhat different approach. Because the OMI is an ongoing initiative with ambitious goals, OMI planners felt the greatest need was for guidance that would help it grow, particularly during the first several years. Therefore, the RAG focused primarily on issues related to development, with evaluation of outcomes as a secondary and more long-term goal. Another distinction that has set the RAG apart from other similar panels is the ongoing and active involvement of a highly placed state-level decision maker. In light of the initiative’s goal of broad social change, the OMI’s management and RAG members have viewed this high-level involvement as especially useful.
Particularly during the early years of the initiative, the RAG focused on contributing to development rather than evaluation. The OMI was a unique concept, so the RAG focused on how prior related research could inform development of the initiative. OMI leaders believe that this emphasis on development, rather than evaluation of outcomes, contributed to the OMI’s forward momentum during its first five years. OMI leadership has used the research advisory group to help develop implementation strategies and to help it think through and address early challenges as they emerged. The RAG also has been engaged in assessing the potential for expansion into new areas, such as programs targeted to TANF or Medicaid recipients.
The active involvement of a key policymaker has promoted research-informed decisions about new directions for the initiative. Since its inception and continuing today, the annual RAG meetings have been attended by the state’s Secretary and Director of the Department of Human Services (DHS)-the agency that sponsors the OMI. Secretary Howard Hendrick brings many years of experience and special knowledge of Oklahoma’s families as well as the political and policy context. As the initiative’s key decision maker, he has actively proposed and responded to new ideas, engaging in lively discussions with RAG members. This exchange of ideas and information among decision makers, implementers, and researchers has had the effect of bringing research to bear on new directions of interest to the state, and has promoted the initiative’s development.
The RAG has served as a sounding board for new ideas, supplied new information, and provided credibility to the initiative’s efforts. During its annual meetings, RAG members have provided input on research that may be relevant to OMI goals and implementation design. The members also have responded to presentations by Public Strategies, Inc. (PSI), the firm that manages the OMI’s operations, on the current state of progress and new directions that may be proposed by OMI leadership. RAG members have proposed new ideas and areas in which the initiative might expand or improve. Beyond its substantive contributions, OMI leaders believe the RAG’s ongoing involvement has given credibility to the initiative’s efforts. In their view, the active participation of nationally known experts and scholars has demonstrated the initiative’s commitment to using the latest research to continuously refine the initiative.
| RAG Member | Affiliation |
|---|---|
| Paul Amato | Professor of Sociology, Demography and Family Studies, College of Liberal Arts, Pennsylvania State University |
| Robin Dion | Senior Research Psychologist, Mathematica Policy Research, Inc. |
| Kathryn Edin | Professor of Public Policy and Management, John F. Kennedy School of Government, Harvard University |
| Dave Fournier | Professor of Marriage and Family Therapy and Family Sciences, Oklahoma State University |
| Norval Glenn | Ashbel Smith Professor of Sociology, Stiles Professor of American Studies, The University of Texas at Austin |
| Ron Haskins | Senior Fellow of Economic Studies, Co-Director of the Welfare Reform and Beyond Initiative, Brookings Institution |
| Howard Hendrick | Cabinet Secretary for Human Services and Director of Oklahoma Department of Human Services |
| Christine Johnson (Co-chair) | Associate Dean for Research and Graduate Studies, College of Human Environmental Sciences, Associate Professor, Department of Human Development and Family Science, Oklahoma State University |
| Pamela Jordan | Associate Professor of Family and Child Nursing, University of Washington |
| Howard Markman | Professor of Sociology, Co-Director of Center for Marital and Family Studies, University of Denver |
| Mary Myrick | President, Public Strategies, Inc., Project Manager, Oklahoma Marriage Initiative |
| Steve Nock | Professor of Sociology, Director of Marriage Matters, Co-Founder of Center for Children, Families, and the Law, University of Virginia. (Dr. Nock passed away in early 2008.) |
| Theodora Ooms (Senior Consultant) | Independent Consultant, formerly at Center for Law and Social Policy |
| Scott Stanley (Co-chair) | Co-Director of Center for Marital and Family Studies, University of Denver |
[ Go to Contents ]
One of the first activities of the RAG was to develop a survey of Oklahoma’s citizens, focusing on attitudes and behavior regarding marriage and divorce. In 2001, the OMI contracted with the Oklahoma State University Bureau of Social Research (OSUBSR) for the statewide survey, which consisted of a random sample of 2,020 adults in the general population and 303 Medicaid clients. The survey was designed in collaboration with members of the RAG, who also contributed to the data analysis and report (Johnson, et al. 2002). Although the motivation for the survey was to assess Oklahomans’ openness to a statewide marriage initiative and attitudes about marriage and divorce, it ultimately served other purposes as well. In particular, it helped inform OMI planners about key population groups that might be targeted. For example, one finding from the survey was that the average age of first marriage was lower in Oklahoma than the rest of the country, a fact that suggested the implementation of services for young population groups, such as high school students. An oversampling of low-income individuals revealed that although many were single, they (like other Oklahomans) would be interested in marriage education services. This finding contributed to a focus on services for the low-income population.
Using the OMI baseline survey data, several RAG members collaborated on an analysis of the association between participation in premarital preparation and marital quality and stability (Stanley et al. 2006). Because this analysis was based on data collected prior to widespread implementation of OMI workshops and is nonexperimental in nature, it does not assess the impact of the OMI, but confirms the potential usefulness of premarital preparation for Oklahomans. Since Oklahoma conducted its statewide survey, other states, including Florida, Louisiana, and Utah have followed suit.
[ Go to Contents ]
To track progress toward statewide saturation, the OMI team developed a management information system to collect and maintain data on workshops and workshop participants. The OMI uses this information to monitor progress throughout its service delivery system, and to update the RAG on the foregoing year’s activities. Although not designed for research purposes, the data also shed light on such questions as which among the education, faith, social services, corrections, and health sectors are most active, what proportion of the state’s population has participated in a workshop, the geographic distribution of workshop activity, and how many workshops volunteers tend to lead.
[ Go to Contents ]
As the initiative’s operations have unfolded, other data collection activities have been carried out to inform the development and expansion of OMI services. Intended primarily for internal planning and decision making, projects were undertaken to learn what motivates individuals to participate in services and to better understand the needs and circumstances of special population sub-groups, such as Medicaid recipients, TANF recipients, and prison inmates.
[ Go to Contents ]
Because of the diverse range of populations served and the very specific venues in which programs operate, the OMI can be considered a laboratory for exploring program strategies that, although drawing on research for their design, have not previously been tested. First steps toward understanding their potential effects on families included OMI-sponsored studies of implementation and outcomes in two programs - in Oklahoma’s prisons and in retreats for couples with adoptive and foster children.
[ Go to Contents ]
The OMI is part of a demonstration program in two national multi-site evaluations funded by the Administration for Children and Families at the U.S. Department of Health and Human Services. As such, it is helping to generate estimates of the impact of marriage education on a targeted group of unmarried and married couples involved in controlled experimental studies.
[ Go to Contents ]
There are numerous obstacles to rigorous evaluation of statewide impacts. Chief among these challenges is the lack of a counterfactual, that is, another state that is the same in every way except for presence of the OMI, whose social outcomes could be compared to those observed in Oklahoma. One alternative approach is to compare Oklahoma’s current divorce and nonmarital childbearing rates to measures taken prior to, or at the beginning of, the initiative. While this “pre-post” method would provide valuable information, it would not determine the extent to which observed changes occurred as a result of the OMI as opposed to other factors, such as changes in the economy. Certain evaluation techniques, however, may be developed and applied to reduce these concerns. The design of such an evaluation would require special attention and tailoring to the OMI’s unique circumstances.
A second important issue to consider in any assessment of state-level outcomes is that broad social change in the culture of marriage is likely to take some time to emerge, regardless of the quality and depth of OMI program implementation. The pervasiveness of divorce and nonmarital childbearing suggests that changes in deep-seated values and attitudes are not likely to occur quickly or easily. It is possible that assessments of state-level outcomes of divorce and nonmarital childbearing may not capture change until the OMI’s reach extends well beyond current implementation.
Assessing cultural change in attitudes and norms about marriage will similarly require careful thought about research design. It is possible to design and conduct a survey to assess change in knowledge and attitudes, such as whether more people think healthy marriage is something that can be “learned,” and OMI planners have begun to take steps in that direction. For the same reasons described above, such an approach would not permit observed changes to be confidently attributed to the influence of the OMI, but it may be possible to design a quasi-experimental approach to reduce threats to the validity of findings.
[ Go to Contents ]
OMI leadership considers research to be useful not just for informing development, assessing progress, and understanding impacts, but also for educating the public about relationships and marriage. The RAG has engaged in developing practical information about marriage and divorce derived, in part, from its statewide baseline survey, for distribution to Oklahoma families. This effort resulted in a series of information sheets that distills research into consumer-friendly questions and answers, addressing common questions about couple relationships.
Each single-page “tip sheet” focuses on a topic of interest to couples who are considering marriage or are already married, often pointing out or debunking common misconceptions about marriage and divorce. Examples of topics include: how cohabitation correlates to later success in marriage; marriage and age; how divorce affects children; what a healthy marriage is; the value of premarital education; how to decide whether you and your partner are ready for marriage; identifying an unhealthy relationship; and how having a child changes marriage. These tip sheets are released at regular intervals and posted on the OMI website (www.okmarriage.org). OMI managers expect that the information in these documents could be used by individuals, and in a variety of settings and with different audiences, such as in churches, newspapers or newsletters, and public addresses. Workshop leaders also may use these products as supplemental handouts for participants.
[ Go to Contents ]
The OMI’s research advisory group has remained involved in the initiative since its inception, and is viewed by OMI leaders as an essential component of the initiative. Its usefulness is likely linked to two factors: flexibility and synergy.
Flexibility: The RAG has shown flexibility at multiple levels, including its willingness to respond to the initiative’s developmental stage and its ability to work across disciplines toward a common goal. RAG members have strived to support the OMI by providing or collecting information for the initial design and the initiative’s development, expansion, and evaluation. The ability to work across disciplines toward a common goal has been a hallmark of the RAG, and has resulted in multiple collaborations that often further the research interests of individual RAG members as well as those of the initiative.
Synergy: The composition of the RAG-including researchers, practitioners, and policymakers-has created a synergy that supports the ongoing evolution of the initiative. The annual RAG meetings present an opportunity for the state to emphasize its policy and program interests, for program implementers to bring forward the issues they are confronting, and for experts to contribute their knowledge from research and past experience. The common interests and cohesiveness of the group contribute to the dynamics and staying power of the initiative.
Among the challenges facing the OMI are how to rigorously evaluate the impact of the initiative on state-level outcomes, and how to collect information from workshop participants so that a better understanding of who is participating can be obtained. Future research efforts will likely be designed to address both of these issues.
[ Go to Contents ]
Einhorn, L., T. Williams, S. Stanley, N. Wunderline, and H. Markman. “PREP Inside and Out: Marriage education as an intervention for prisoner reintegration.” Denver, CO: University of Denver, in press.
Fournier, David, and Kelly Roberts. “Examination of Help Seeking For Couple Relationships in Oklahoma.” Oklahoma City, OK: Oklahoma State University Human Development and Family Science, 2003.
Hershey, Alan A., Barbara Devaney, M. Robin Dion, and Sheena McConnell. “Building Strong Families: Guidelines for Developing Programs.” Washington, DC: Mathematica Policy Research, Inc., 2004.
Johnson, Christine, Scott Stanley, Norval Glenn, Paul Amato, Steve Nock, Howard Markman, and M. Robin Dion. “Marriage in Oklahoma: 2001 Baseline Statewide Survey on Marriage and Divorce.” Oklahoma: Oklahoma State University Bureau for Social Research, 2002.
MDRC. “Guidelines for Supporting Healthy Marriage Demonstration Programs.” New York, NY: MDRC, 2005. [PDF - 14 pages]
Stanley, Scott, Paul R. Amato, Howard Markman, and Christine Johnson. 2006. “Premarital Education, Marital Quality, and Marital Stability.” Journal of Family Psychology 20: 117-126.
| Information reported in the OMI research brief series is based on analysis of data gathered during a multiyear study of the initiative’s design, development, and implementation. Study methodology included semi-structured interviews with individuals and groups, direct observation of program operations, focus groups with staff and participants, and secondary analysis of data from existing reports and surveys. The research team met directly with 163 individuals involved with the OMI in various ways, focusing on implementation in the education, social services, corrections, health, and community volunteer sectors. Mathematica’s research team is led by M. Robin Dion, and includes Alan M. Hershey, Debra A. Strong, Heather Zaveri, Sarah Avellar, Nikki Aikens, and Timothy Silman. |
[ Go to Contents ]
| This ASPE Research Brief discusses the Oklahoma Marriage Initiative’s
use of research to guide the development and improvement of program services,
assess progress, and disseminate information to the community about marriage.
The brief draws on findings from an in-depth process evaluation conducted
by researchers at Mathematica Policy Research, Inc. under contract to ASPE.
This brief was prepared by Heather Zaveri and Nikki Aikens.
Office of the Assistant Secretary for Planning
and Evaluation
Melissa Pardue |
To obtain a printed copy of this report, send the title and your mailing information to:
Human Services Policy, Room 404EFax: (202) 690-6562
Home Pages:
Human Services Policy
(HSP)
Assistant Secretary for Planning and Evaluation
(ASPE)
U.S. Department of Health and Human Services
(HHS)
Last updated: 09/30/08