Meeting Summary
Child and Adolescent Effectiveness Research in Clinical Practice and Community Settings: Needs, Challenges, and Opportunities
January 24, 2008 – January 25, 2008
Washington, DC
Sponsored by:
National Institute of Mental Health (NIMH)
Division of Services and Intervention Research
Aims
This workshop provided a forum for reviewing accomplishments and challenges in conducting effectiveness research in practice and other community settings and served as an opportunity for informally discussing promising approaches to further research. The focus of the meeting was on testing the effectiveness of treatment interventions for children and adolescents when delivered in clinical practice and other community settings, such as schools.
The main goals of this workshop were to: 1) examine the current need for conducting research in clinical practice and community settings in order to evaluate the effectiveness of treatments for children and adolescents; 2) review the progress made thus far in this area; 3) discuss challenges to this type of research; 4) describe promising approaches; and 5) identify opportunities for further research.
Background
Collaborations and partnerships between researchers and community health care providers, including clinical practice, school, and public health delivery systems, are important for studying the effectiveness of mental health interventions. Traditional clinical trials conducted in academic settings, although highly successful in answering critical efficacy questions, use selected samples of patients and have limited generalizability. Moreover, their sample size is too small for allowing adequate testing of possible predictors and moderators of treatment effects, take a long time to complete, and are expensive. Hence, it is appropriate to consider practical clinical trials to test treatment effectiveness. Over recent years, NIMH has launched initiatives to stimulate effectiveness research, such as funding dedicated community research networks as part of the NIMH Advanced Centers for Innovation in Services and Intervention Research. This area of investigation, however, remains in an early phase of development.
Effectiveness Research
Effectiveness studies are methodologically distinct from efficacy and explanatory research, although there are hybrid studies that include elements of both. The effectiveness, cost-effectiveness, and adverse effects of treatment interventions are best evaluated with practical trials, which imply randomizing a large number of patients under typical practice conditions (i.e., including usual clinically referred patients and usual practitioners to deliver the interventions). While no large, multisite, practical trials have been conducted with children as yet, progress has been made in forming networks of practitioners providing psychiatric care to children and adolescents; forging local collaborations between mental health researchers and pediatric primary care practices; conducting small trials based on health care organization clinical structures; and developing Internet-based applications for treatment delivery and testing. In general, there is a need for more practical, evidence-supported models to guide development and maintenance of child/adolescent treatment effectiveness efforts.
The design, methods, and operational structure of a few adult and youth practical trials were presented at the workshop as possible examples of feasible approaches to conducting practical trials in children and adolescents. Statistical methods applied to effectiveness research were also reviewed. What constitutes a meaningful effect in the context of an effectiveness study may need to reconsidered and redefined. In fact, treatment effect sizes might not be as large as with efficacy studies, but smaller effect sizes might still be meaningful in an effectiveness context.
While randomized clinical trials are the best method to derive causal inferences, there is room for other approaches as well. Some questions could be answered through data mining of naturalistically treated children, especially if the studies are sufficiently large and representative and if clinical data are linked to claims and pharmacy records. These data mining studies could generate hypotheses about predictors of outcome and identify associations between treatment and safety outcomes. Other questions, when randomization is not ethical or feasible, will require prospective longitudinal cohort designs.
Challenges
A number of challenges were identified, including:
- engaging practitioners and administrative staff, who may have little research background, low interest in research activities, and time constraints and productivity requirements that limit time available for research
- providing sufficient incentives for participating in research to patients and families whose financial and case management needs complicate the use of structured research protocols
- addressing concerns about the feasibility of randomization and data collection with both families and practitioners in practice settings and concerns about the validity of diagnosis and assessment
- addressing the loss of control over participant contact, case assignment, and other procedures that are more easily managed in traditional trials
- identifying appropriate comparison and control conditions
- maintaining high quality of data, experimental validity, and assay sensitivity to discriminate between treatment groups
- addressing health insurance and liability issues that may compete with research plans
- ensuring compliance with ethical, Institutional Review Board, and other regulatory requirements in both clinical practice and school settings
- containing costs, for instance by billing health insurance for usual care services (as separate from research-specific costs)
- balancing strategies for effective implementation against research rigor (e.g., selecting “best therapists” vs. randomly assigning therapists)
Conclusions
Although no specific model for conducting child and adolescent treatment research in practice settings has been fully validated for feasibility, a number of promising approaches have been developed. These include partnerships between mental health researchers and psychiatric practice networks, primary care pediatric practices, health management organizations, and State mental health offices. The increasing interest in evidence-based practices among both public agencies and private health care organizations can provide momentum and incentives to build a research agenda to generate practice-based evidence about the effects of treatments. Successful implementation will depend on building and maintaining collaborations among researchers, health organizations, providers, patients, and families. Ideally, there should be a seamless integration of clinical care and research in practice settings. Discussion highlighted the importance of including resources and infrastructure to support stakeholder involvement (e.g., as members of steering committees, consultants, collaborators, treatment trainees/supervisors) and the potential utility of qualitative approaches to facilitate efforts to refine and test evidence-based approaches in community/practice settings. Effectiveness research in practice and other community settings is an area of considerable potential and in need of further development.
