U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Administration on Children, Youth and Families
Family and Youth Services Bureau

July 2008

Letting Data Tell The Story of Runaway Youth

Stan Chappell never thought he would be so pleased to hear that one simple word—‘effective.’ As the director of research at the Family and Youth Services Bureau (FYSB), Chappell, along with Bureau staff and grantees, have been working diligently over the past several years to prove that the Runaway and Homeless Youth (RHY) Program, and the hundreds of runaway and homeless youth agencies it funds, deserved the Office of Management and Budget’s (OMB) highest rating.

As recently as a few years ago, the RHY program had no ability to track outcomes, earning it a “results not demonstrated” in the Program Assessment Rating Tool system, known as PART, that the OMB uses to measure the effectiveness of Government programs (see How is FYSB Rated?).

“Results not demonstrated means you don’t even know,” Chappell says. “The Office of Management and Budget in the Executive Office of the President that manages the money and executes fiscal and funding policy is not just unimpressed, but greatly disappointed.”

The reasons for the rating change? “FYSB has more than a dozen professional staff in the regions who are able to provide technical assistance directly to grantees on ways to improve program effectiveness,” says Jacqueline Baker, team leader for FYSB’s RHY Programs.

And secondly, FYSB grantees have made great strides in recording data into the Runaway and Homeless Youth Management Information System, or RHYMIS, the system FYSB uses to track information about young people served by its Basic Center, Transitional Living, and Street Outreach Programs.

That diligence has paid off—helping show that FYSB-funded agencies are making a difference in the lives of the youth they serve.

Fifteen Years of RHYMIS

Congress created RHYMIS in 1992 when it authorized funding, through the Runaway and Homeless Youth Act, to implement a national reporting system for programs receiving Federal runaway and homeless youth funds. Programs funded under the RHY Act must agree to keep adequate statistical records for profiling youth and families served under their grant. Grants funds can even be used to acquire the computer equipment needed to keep such records.

“A lot of grantees come up to me and say, ‘RHYMIS used to be a pain, now it’s easier.’ They tell me they like using it for their own internal tracking and to record other categories of youth, not just the ones they report to FYSB.” —Stan Chappell, FYSB Director of Research

But until the end of the 1990s, only about 43 percent of grantees reported on the numbers of youth they served. Part of the problem, grantees said, was that filling out all the paperwork took too long and sapped their ability to provide youth with the highest quality services.

In response to those complaints, Chappell says, the Bureau streamlined the RHYMIS data reporting system in fiscal year (FY) 2000. The new software eased reporting for grantees and brought the response rate to a remarkable 99 percent. Through the release of two additional versions of the software, FYSB has maintained that high level of reporting. Effective technical support by Arlene Calabro, a RHYMIS contractor, also makes a major contribution to the response rate.

“A lot of grantees come up to me and say, ‘RHYMIS used to be a pain, now it’s easier,’” Chappell says. “They tell me they like using it for their own internal tracking and to record other categories of youth, not just the ones they report to FYSB.”

FYSB refines RHYMIS from time to time, sometimes simplifying some aspects of the system and clarifying others. For example, in FY 2008, FYSB added new categories to the basic center report, allowing grantees to more easily report on preventive services, such as in-home family counseling, meant to prevent youth from running away and on other services provided at home, in the community, or other settings outside the Basic Center. The Bureau continues to educate new and continuing grantees about how to input data so that program achievements are accurately recorded, Chappell says.

FYSB has also made an effort over the past 5 years to improve access to national RHYMIS data. Each year, RHYMIS data is uploaded to the Internet (https://extranet.acf.hhs.gov/rhymis), where grantees and others can access and analyze information about youth served by FYSB programs. Many grantees cite the data in their efforts to attract non-Federal funding or to evaluate the successes of their programs over time.

As FYSB continues to improve the system, Chappell says keeping RHYMIS from becoming a burden for grantees is a primary goal. “We don’t want you to spend all your time filling out forms,” he tells RHY practitioners. “We don’t want your staff to stay up all night inputting data when they should be home with their families or with program youth shooting hoops. We want you to be on the front lines.”

How Is FYSB Rated?

Every year, the Office of Management and Budget (OMB) assesses all Federal programs using the Program Assessment Rating Tool (PART). The OMB awards an “effective” rating when a program sets and improves efficiency. Lower ratings include moderately effective, adequate, ineffective, and results not demonstrated. In 2006, only 17 percent of the 977 Federal programs rated under PART received an “effective” rating.

Since 2002, FYSB’s Transitional Living Program (TLP) has been rated on the following “strategic triangle” of measures:

Measure 1. Increase the proportion of youth living in safe and appropriate settings after exiting ACF-funded TLP services.

Why it matters: Providing older homeless youth (ages 16-21) with safe places and developmental experiences that lead to safe and appropriate exits is a fundamental goal of the TLP. “Safe and appropriate” exits include entry into independent living and housing situations, residential apprenticeships, higher education, military service, and other destinations with positive opportunities. Mental health or correctional systems can also be appropriate or necessary placements. The street, the unknown, and homeless shelters are not considered safe or appropriate (i.e, successful) exits.

Measure 2. Increase the proportion of TLP youth who are engaged in community service and service-learning activities while in the program.

Why it matters: Research on Positive Youth Development shows that youth thrive when they are given opportunities to belong, develop positive social norms, express civic engagement, and learn skills. Such youth appear more likely to successfully navigate toward independence and other suitable living situations.

Measure 3. Increase the number of youth who complete the TLP by graduating or who leave ahead of schedule based on a positive opportunity.

Why it matters: Young people who stay longer or complete TLPs are more likely to be successful in school and on the job. By completing the program, youth also have a greater chance of a safe and appropriate exit, compared to young people who drop out with no plans or are expelled. Starting in 2008, FYSB’s Basic Center Pro-gram will be tracking the following measure (see article on page 6):

Measure 4. Increase the proportion of youth who are prevented from running away through BCP in-home/offsite services as a percentage of all youth receiving such services.

Why it matters: Young people who are prevented from running away avoid the dangers of the street. Interventions can also help families in crisis deal constructively with conflicts and other causes of risky or destructive decisions and behavior.

It May Not Be Glamorous, But Data Entry Matters

Not too long ago, Doug Tanner at the New England Network for Youth decided to take a closer look at where young people went after leaving the transitional living programs in his area. He was startled by what he found – data showed that many of the youth wound up back on the streets, in shelters, or in other unknown, but presumably unsafe, places.

At first, Tanner thought he was looking at serious programmatic failures. But upon further inspection, he noticed that all the programs had two things in common: incorrect data entry and a lack of standardized data-entry procedures.

“You think you have a safe exit problem,” he says. “You don’t. You have a data entry problem.”

Indeed, interviews with 26 Family and Youth Services Bureau (FYSB) grantees last year revealed that agencies with data management policies were more likely than agencies without such policies to have high safe exit rates—one of the key measures FYSB uses to gauge the effectiveness of its programs.

The interviews highlighted how important it is for grantees to correctly enter information into the Runaway and Homeless Youth Management Information System, or RHYMIS, the system FYSB uses to track information about young people served by its Basic Center, Transitional Living, and Street Outreach Programs.

“Some of the best performing agencies didn’t necessarily have data in RHYMIS that reflected their success,” says Gary Hammons, executive director of the Northwest Network for Youth in Seattle, Washington. “Good record keeping and skill in recording information in RHYMIS is essential for agencies wishing to provide an accurate picture of their programs.” (It should be noted that FYSB never uses RHYMIS data to lower the score of a grantee reapplying for funding.)

In addition to taking a snapshot of the performance of each individual program, RHYMIS data shows how runaway and homeless youth served by FYSB programs are faring on regional and national levels. That data is reported to the Office of Management and Budget through PART (see box on page 1) and to Congress through the biannual Report to Congress on the Runaway and Homeless Youth Programs (the report for the 2004 and 2005 fiscal years is available at http://www.acf.hhs.gov/programs/fysb/content/aboutfysb/publications.htm).

“National performance levels affect funding and advocacy,” Tanner says. “We find that when staff understand that, they do a better job of entering the data.”

Based on the findings, FYSB and its technical assistance team have been working with grantees to improve their data management—and their ability to capture how well they are performing certain services for youth.

One of the problems with safe exit reporting, FYSB found, was that agencies often selected “don’t know”—an unsafe exit—if a youth left without immediately providing new contact information, even if the young person subsequently got back in touch or returned to the program.

“One of the things we didn’t realize was that we would have the opportunity to go back and make changes,” said Dale Holder, director of the Janus Center for Youth in Crisis in Bridgeport, Connecticut.

Now, the Janus Center staff must update a young person’s records a certain number of days after they are first referred to the program, as well as a certain number of days after they leave. The agency also has data entry rules regarding youth who leave the program and later come back.

With these rules and policies in place, the agency has a more complete picture of its achievements, Holder says.

The reporting problems go beyond safe exits, as well. FYSB found that community service learning, one of the other measures that it tracks and reports to the OMB, was also being wildly underreported in RHYMIS because agencies were unclear about what counted as community service.

“Once we explained to them that community service learning didn’t have to be an elaborate, sophisticated undertaking,” Stan Chappell, FYSB’s director of research says, “their response was ‘we’ve been doing that all along, but we haven’t been claiming it!’”

Another problem Tanner found was that some grantee agencies weren’t aware that updated versions of RHYMIS had additional reporting requirements.

“When you’ve been inputting things in a certain way, you don’t notice changes,” Holder says. Until Tanner pointed it out to him, Holder didn’t know that RHYMIS now collects information about aftercare planning and services provided to young people both in preparation for leaving a program and in the months after they leave.

Those exit planning, aftercare, and “connection” services can affect whether young people make a successful transition after they have departed the TLP. They are often essential for youth to complete their transitional living programs, the third of FYSB’s strategic triangle of measures.

So while Holder’s program did provide aftercare to runaway and homeless youth, information about that service wasn’t making it into RHYMIS. With a few simple changes, they were able to get credit where credit was due.

For a copy of “RHYMIS: Goals, Tips, & Traps,” a fact sheet that explains data entry procedures that can help reflect agencies’ progress toward FYSB’s national performance goals, contact Doug Tanner, dtanner@nenetwork.org or (978) 544-2067.

For guidance on creating or updating data entry policies or on training staff to use RHYMIS, contact Arlene Calabro, RHYMIS Support, (888) 749-6474, option 1 (Monday - Friday, 8am - 5pm EST), or e-mail rhymis_help@csc.com.

RHYMIS Data At A Glance

What’s Next for RHYMIS

Most people recognize basic center programs (BCPs) for the crucial role they play in sheltering young people who’ve run away from home. But that’s only part of the work they do every day to keep the potential runaways in their communities safe.

“For BCPs, prevention is actually more important than the intervention,” says Jacqueline Baker, team leader for FYSB’s Runaway and Homeless Youth Programs. “Even one night on the street exposes homeless young people to danger and a host of risky behaviors.”

That’s why some of the most successful BCPs work hard to get to young people before they’ve run away. Staff meet with family members for mediation or conflict resolution services. They provide individual or family counseling. And they work with young people to find relatives or a safe place they can stay with if they need to cool off for a night.

In order to document those successes, FYSB has recently started asking BCP grantees to record a small amount of additional information in RHYMIS.

“We want BCPs to be able to tell the full story of this kid in crisis,” says Stan Chappell, director for research at FYSB. “If programs place a potential runaway back with their family, they should be recognized for that. If the young person enters the shelter, that prevention effort would also become part of their story.”

And it’s not just basic centers that provide services to youth who aren’t officially enrolled. Transitional living programs often help young people on their waiting lists. Or kids who are still deciding if they want to join the program.

Other young people may be taking a break from the program, whether by their own choice, or the agency’s.

“There are kids on the verge of burning out or rebelling or have broken one too many rules,” Chappell says. “So the agency may say ‘You aren’t ready for this. We are going to take you someplace where you can be safe to think about this. In the meantime, you can come back for dinner every night, or once a week. You can get counseling or participate in some of our other services. When you’re ready, come back to the program.”

Before, Chappell says, those cases were closed in RHYMIS and considered failures. Often, though, the kids do come back, and they are usually highly motivated and ready to finish the program.

“Programs were asking us how they could record this kind of situation,” Chappell says. “So we urged them not to close the individual RHYMIS record too soon and added a new area in RHYMIS where programs can document the services they provide to kids in this type of therapeutic respite.”

While not all basic center and transitional living programs provide these types of intervention and respite services to young people, Chappell says FYSB encourages them as a way to make a greater impact on runaway and homeless youth in the community.

An impact, he says, that will eventually find its way into hands of policymakers.

“These are our goals. We invite you to make them part of your goals. Use these goals to tell your story,” Chappell says.

FYSB Update is developed for the Family and Youth Services Bureau by Johnson, Bassin & Shaw (JBS), Inc., under Contract # GS10F0205K from the Administration on Children, Youth and Families; Administration for Children and Families; U.S. Department of Health and Human Services, to manage the National Clearinghouse on Families & Youth.

National Clearinghouse on Families & Youth
P.O. Box 13505
Silver Spring, MD 20911-3505
Tel./TTY: (301) 608-8098
Fax: (301) 608-8721
E-mail: ncfy@acf.hhs.gov
ncfy.acf.hhs.gov