Chapter 3. Findings (continued)

3.3.5 Impact Case Studies (continued)

Impact Case Study III: Rickets, Vitamin D Supplementation and AAP Guidelines

Rickets is everybody's favorite [project/study] because clearly it was one of those "a-ha moments" for policymakers. The research was immediately taken up, programs were adopted to change things and it has changed things. So that was fabulous. (SC)

Project Title: Effect of AAP Guidelines on Vitamin D Supplementation recommendations in Practice⁴⁷
PI: Marsha Davenport and Ali Calikoglu
CERT: UNC
Partners: NC WIC Program, America Association of Family Physicians (AAFP), American Academy of Pediatrics (AAP)

This case study is about a line of research that identified and understood vitamin D deficiency and rickets in breast fed infants, explored the impact of AAP guidelines on vitamin D supplementation, and contributed to the change in the guidelines.

Formation of the Research Question

Dr. Calikoglu and Davenport were co-investigators on these CERTs studies. They described how the study began, specifically that they started diagnosing rickets in African-American children in their practices. Additionally, Dr. Calikoglu recalled that there was one father, a professional athlete who was "up in arms" about his child having rickets, perhaps providing an extra impetus to explore what was happening. Dr. Schwartz at the Wake Forest University Baptist Medical Center was a key partner on the research with CERTs investigators, and he recalled seeing more and more rickets and vitamin D deficiency cases, such that he and another colleague wrote the cases up in the NC Pediatric Society bulletin. Together they compiled information on 30 cases of vitamin D deficiency rickets, which they submitted to the Journal of Pediatrics. Dr. Schwartz describes:

Initially when submitted to Pediatrics, in the review process they sent it back and asked for more data and "this is where the review process worked well in that they told them they needed more information which inspired them to get more data." So they went to get two diff data 1) Dr. Schwartz got the data from the WIC program about the breastfeeding rate (increased significantly) from 1989 to 1999, Apparently the rate of breastfeeding had quadrupled among African American children and tripled in all, something like from 5%-30% during the period 2) Dr. Davenport looked at prescribing practices.

The article was published in 2000.⁴⁸ After the publication of the report and manuscript, the researchers looked into resources they could mobilize in NC and Dr. Schwartz at Wake Forest had close connections to NC WIC program. Dr. Schwartz described the good relationship with the WIC program and NC health department. They come to the Pediatrics society meetings and Dr. Schwartz has known the WIC representative for a long time at WIC they have a long term, trusting relationship. The WIC representative gave Dr. Schwartz the WIC breastfeeding data.

Research Findings

The 30 case dataset they collected revealed the following, as described in the published article:

Thirty patients with nutritional rickets were first seen between 1990 and June of 1999. Over half of the cases occurred in 1998 and the first half of 1999. All patients were African American children who were breast fed without receiving supplemental vitamin D. The average duration of breast-feeding was 12.5 months. The age at diagnosis was 5 to 25 months, with a median age of 15.5 months. Growth failure was common: length was > 5th percentile in 65% of cases, and weight was > 5th percentile in 43%.¹⁰

After they had their cases and the finding, they approached WIC and presented to them these data. The WIC program agreed to provide nursing mothers with free vitamin D (in NC no vitamin is covered by Medicaid). The program provided $1.50/baby/month vitamin D supplement (in a vitamin D formula with A, C, D). Around the same time, the CDC convened a vitamin D expert panel meeting:

On October 11 and 12, 2001, scientists, health practitioners, and policy makers from the Centers for Disease Control and Prevention (CDC), the American Academy of Pediatrics (AAP), academic and professional institutions, and government agencies met in Atlanta, Georgia to discuss vitamin D supplementation of infants. CDC convened the meeting to examine scientific issues and policy implications regarding vitamin D supplementation and to identify current research needs. Experts presented information on the incidence of rickets in the United States; the role of sunlight in preventing vitamin D deficiency and in the occurrence of skin cancer; and the risks and benefits of vitamin D supplementation, including its impact on breastfeeding; alternatives to supplementing infants with vitamin D; and development of a communication strategy to promote a new policy on vitamin D.⁴⁹

Dr. Calikoglu served on the expert panel to which he presented on more cases of vitamin D deficiency rickets. It was also at this panel meeting where he started learning that the AAP was looking at revising its guidelines for breastfed infants and vitamin D supplementation. So Dr. Calikoglu and Davenport developed and fielded a survey of pediatricians in NC that found that they were not supplementing vitamin D in breastfed infants in part because the AAP guidelines did not clearly address the issue, and that more recently trained pediatricians were less likely to supplement because they were following newer guidelines, while older physicians were practicing more appropriately because of older guidelines. Dr. Calikoglu or Davenport did not participate on the guideline revision committee. Given the likely, upcoming change in guidelines they sought to determine whether physician prescribing behavior would impact the prevalence of vitamin D deficiency rickets and to assess the impact of the changed guidelines. So in 2002 they sent a survey to members of AAP and member of AAFP before guidelines were changed. The UNC CERT described the results of the initial survey in a progress report:

"The results of initial survey indicate that inadequate Vitamin D supplementation may be an important contributing factor to the development of nutritional rickets in many infants and toddlers in the United States. It also suggests that the vagueness of established guidelines was a likely factor in inadequate vitamin D supplementation practices." (UNC PR04-05)

Then in April 2003 the AAP published their new guidelines that recommended 200 IU/day for all breast-fed infants. So the investigators, one year post AAP guidelines randomized a follow-up survey to AAP and AAFP groups and sent an intervention package with a) letter strongly recommending vitamin D supp for infants b) new guidelines from AAP c) a magnet d) a calendar 2004 with the statement "breastfed infants deserve vitamin D supplements – it's not just about bones anymore" at the top. The recipients of this package were the intervention group. After a year the investigators sent a second survey to the same population to identify how family practitioners compared to pediatricians in supplementing vitamin D. They found that family practitioners only supplemented vitamin D 32% of the time in 2002 and 58% in 2004.

Dr. Schwartz said "... WIC nationally is revising their package of food. IOM report 2005 talk about the need for Vitamin D in their " breastfeeding promotion and that daily vitamin D and supplement need a recommendation on the national level that the WIC program include them.

Dissemination

Publication 1: Kreiter SR, Schwartz RP, Kirkman HN, Jr., Charlton PA, Calikoglu AS, Davenport ML. Nutritional rickets in African American breast-fed infants. Journal of Pediatrics. 2000;137:153-157.
Journal IF: 3.837

Publication 2:
Davenport ML, Uckun A, Calikoglu AS. Pediatrician patterns of prescribing vitamin supplementation for infants: do they contribute to rickets? Pediatrics. 2003 Apr;111(4 Pt 1):908-10.
Journal IF: 4.272

Media: There was significant pickup of this work by the media and reporters calling from Canada and throughout the US. Dr. Schwartz presumed that this is because people can directly relate to this, rickets, and broken bones, and real disease and can understand "something from 1800s and diet."

CERTs and AHRQ publications:

Other publications: Stated from Australia to Japan news agencies cited the study.

Impact of the Research Findings

This case study and the initial study in particular were consistently and most frequently identified by investigators from across the CERTs as an example of CERTs work having an impact.

Level 1 Impact: Research
The studies revealed the estimated incidence of rickets, which was thought to not to exist to any significant degrees. However, with the trends in promoting breastfeeding at the end of the 20th century and without the appropriate supplementation in part because of the AAP guidelines, breastfed infants were not receiving sufficient vitamin D, and as a result, cases of vitamin D deficiency and rickets were increasing. The NC pediatricians and investigators compiled the cases to illustrate this point, along with the data showing the concomitant trend in breastfeeding from the WIC program data.

Additionally, the findings on the guidelines were important for understanding the potential impact of guidelines on practice. Also, they demonstrate the challenge that even when guidelines are changed, change in practice can be slow.

Level 2 Impact: Policy
The research had an impact in terms of contributing to a CDC-convened group of experts. The research has also impacted the NC WIC program policy that resulted in the provision of vitamin D supplementation for breastfed infants. This research also impacted the AAP guidelines on vitamin D supplementation.

Level 3 Impact: Clinical Practice
The investigators demonstrated the educational intervention and the provision of the new AAP guidelines on vitamin D supplementation in breastfed infants did have an impact on the behaviors of physicians.

Mechanisms of Impact

The key factors that led to this project and portfolio of research having the impact included: connections, collaboration, and time. Drs. Davenport, Calikoglu and Schwartz indicated that a key reason for the impact of the findings with respect to changing WIC coverage policy was the connections to the NC WIC program. Dr. Schwartz described a good relationship not only with the WIC program but the NC department of health. They had already asked for the data to show the trend in breastfeeding at the time of their case studies. So when they contacted the WIC program to present on their findings they were more than willing. The CERTs investigators stated, "... key thing was joining forces with Wake Forest group to increase the numbers of collaborators with Dr. Schwartz with more than one institution in the state working on something to have a greater weight with policymakers."

Dr. Schwartz believed that the key mechanisms were communication and mutual effort (collaboration). It was 5-6 years before policy changed (AAP) he'd been working on this for a long time. He had been to the to the AAP forum/congress previously - " ... then we got the data to have them listen."

An AHRQ representative identified time as a key factor:

For one thing [this project] had legs and it's old and so, of course, lots of things can happen within, you know, seven years or so that's been kind of an active issue. The other thing that makes that a little bit easier for it to have legs is that I think the American Academy of Pediatrics is a real groundbreaking group... and very sensitive, I think, to these sorts of things, and the folks at the CERTs have kind of had an in there with Marcia Davenport and other people who've kind of worked with the American Academy of Pediatrics. ...it's a good example, but it certainly works because the specialty organization is a good one. It's also a good example I think of what we were hopeful of the local collaborations that the CERTs would have. You know, when we were going to fund four new CERTs one of the things we said is that we really think that the CERT ought to work regionally, and work with its local authorities and that's a good way to get started. And that worked out really well because they worked with the local Medicaid authority. Vanderbilt does that too, work closely with the state, the ideal situation.

Dr. Davenport and Calikoglu stated that could always use more money to further investigate the magnitude of the problem and its solutions, because they could have devoted more time, although they stated that it would not have happened without the CERTs. Dr. Schwartz felt that this project was the "most important thing he's worked on." He calls it a "new old disease."

Impact Case Study IV: Tensions in Antibiotic Prescribing

"If the only thing you focus on is antibiotic use without looking at person to person spread, then that's not sufficient" (Penn)

Project Title: Tensions between patient and public health values in generalists use of antibiotics
PI: Josh Metlay
CERT: Penn
Partners: Robert Wood Johnson Foundation
Description: Cross-sectional anonymous mail survey. Participants: National random sample of 400 generalist physicians (general internal medicine and family practice) and 429 infectious diseases specialists.

Formation of the Research Question

Dr. Metlay provides the context and background important for understanding the significance of the research:

Overuse of antibiotics is a key force driving the emergence of resistant bacteria. Increasing awareness of the problem of drug resistance may be part of the solution to the problem, however the question is: How important is that for physicians when making prescribing decisions?

In some cases providing education to physicians about the risks and benefits of drugs may result in better individual decisions but in some settings providing information is less likely to change behavior.

A major risk associated with antibiotic misuse is the public health and community cost and not individual patient risk. Dr. Metlay poignantly describes:

"The fact that the benefits happen for patients and the risks happen for society could create an unequal weighting in peoples' minds as to how important" antibiotic prescribing and use are to them or their patients."

He offers another example to illustrate:

"A related example is vaccination policies, where the risks are often for the individual person who gets the immunization but the benefits are on a more societal level. It's been observed that without guiding policies, individuals will opt out from getting vaccinated because they will recognize that they can get the public health benefit without taking the individual risk. And that's why we have policies that essentially mandate vaccinations."

Similarly, individual patients and physicians may opt to keep taking and prescribing antibiotics as long as they know that most people are avoiding them because they will get the individual benefit of the antibiotics and not contribute that much to the harm. This, Dr. Metlay continues, "begs the question as to whether we can really improve the quality of antibiotic prescribing, in hospitals and outpatient settings, simply by educating patients and providers. Or do you really need to impose some kind of control on the whole system because of this imbalance in risks and benefits."

Therefore hospitals may have antimicrobial management programs that may require an individual physician obtain approval before prescribing a number of broad-spectrum antibiotics. Unfortunately, that is less common in the outpatient system because the same kind of information systems and decision support systems are not available.

The study was a survey of physicians (i.e. generalists, internists, family practitioners, and infectious disease specialists) using a nationally representative sample drawn from an American Medical Association database. The objective was to obtain "a snapshot of how physicians viewed current goals around appropriate antibiotic use in ambulatory care settings" (JM). The survey included knowledge and attitude questions and asked respondents to answer how important certain things were in their decision-making. To reveal how the prescribers think rather than having them directly provide their thoughts, so the study could identify what is driving their decision-making the survey included vignettes in which they had to make treatment decisions. The design of the vignettes was such that the study investigators were manipulating some of the key issues to see how those particular issues were influence their decisions.

Dissemination

Publication 1: Metlay JP, Shea JA, Asch DA. Antibiotic prescribing decisions of generalists and infectious disease specialists: thresholds for adopting new drug therapies. Medical Decision Making. 2002 Nov-Dec; 22(6):498-505.
Journal IF: 1.822

Publication 2: Metlay JP, Shea JA, Crossette LB, Asch DA. Tensions in antibiotic prescribing: pitting social concerns against the interests of individual patients. Journal of General Internal Medicine. 2002;17:87-94.
Times Cited:" At one point they published the articles that were the most cited in the journal each year.... for the Journal of General Internal Medicine that was one of the top ten or twenty"
Journal IF: 3.013

Abstracts:One abstract presented.

Other publications:
Leonard Davis Institute of Health Economics Issue Brief, Vol 7, No 7, May 2002. The LDI Issue Brief⁵⁰ is disseminated to "those issue briefs are pretty widely disseminated, particularly to non academic audiences, so to hospital executives, pharmacists, executives" (JM)

Dr. Metlay described presenting his research in CERTs meetings as well as talking to the FDA about the expected and unexpected results of adding warnings to package inserts for antibiotic drugs. He also mentioned speaking to different physician groups and going beyond the traditional dissemination of findings in peer-reviewed journals.

Impact of Research Findings

Level 1 Impact: Research
Findings of the first published article:

Respondents significantly reduced their threshold for switching to a newer antibiotic as disease severity increased. Generalists were more responsive to disease severity than Infectious Diseases specialists. Thus, the adoption of recommendations to limit overuse of newer antibiotics may be variable across clinical settings and providers, reducing the impact of these recommendations on emerging resistance.

Findings of the second published article:

Both generalists and infectious diseases specialists were more likely to prefer newer, broader drugs for the treatment of pneumonia compared to older agents still recommended by national guidelines. Physicians rated the issue of contributing to antibiotic resistance lowest among 7 determinants of their choices. Conclusions: Despite national guidelines and increasing public awareness, the public health concern of contributing to the problem of antibiotic resistance does not exert a strong impact on physician prescribing decisions for pneumonia. Future efforts to optimize antibiotic prescribing decisions will need to consider options for increasing the impact of public health issues on the patient-oriented decisions of individual physicians.

Level 2 Impact: Policy
Dr. Metlay identified two primary areas in which changes are occurring that he could not directly attribute to his research, but believes that it was part of the critical mass of research that may be influencing the changes. The changes that he is observing in the health sector are described below:

In the recent past the CDC had funded intervention studies to reduce antibiotic overuse and improve the quality of antibiotic use that primarily focused on education, however the studies had a small effect on reducing the misuse of antibiotics. More recently, investment is being made in studies and designs that are not just educationally driven. For example, Dr. Metlay stated that, "more studies are trying to provide real time feedback to doctors about what they're doing and develop other kinds of levers, such as computerized decision support tools, that might help improve the quality of prescribing."

In the last few years the rate of antibiotic use for non-bacterial infections in children has become a quality measure for health plans and subsequently, a measure for employers choosing health plans. Whether the fact that there is now a quality measure impacts the rates of antibiotic use is unclear, but it indicates a movement toward using quality measurement and incentives to drive the change and not just education. Dr. Metlay also stated that CMS has now tied performance of antibiotic treatment for pneumonia to payment.

Dr. Metlay stated that there are starting to be stronger interventions and antibiotic management because of a general awareness that certain kinds of structure are needed to really improve quality. He concludes that not unlike other domains besides antibiotics, "maybe in some way this kind of information helps hasten that thinking, not to close the book on education, but to point out the serious limitations to education and knowledge awareness as a quality driver."

The findings from these studies have had an impact on the research community and were recognized as a frequently viewed article on the journal's Web site. Additionally, they raise a critical issue about prescribing behavior and how choices are made in prescribing antibiotics. The topic of antibiotic prescribing and resistance is a critical area for AHRQ.

Return to Contents

3.4. CERTs Program Outcomes

3.4.1. Educational Outcomes

CERTs Value—Education: Education of current and future health care providers, policy makers, and patients is critical to improving health.⁵¹

The CERTs are committed to education to improve the health of patients and the population. The CERTs have provided both formal and informal post-graduate opportunities. Formal training has included research centers supporting traineeships and fellowships (HMO, Penn, Alabama, Arizona). The trainees included: faculty, researchers (i.e. epidemiologists, health services researchers, clinical researchers; and social scientists), graduate students in various disciplines, medical students, pre-medical students, pharmacy students, social workers, and providers. The Alabama and Penn investigators provided many examples of their traineeships and training, and in interviews characterized it as a critical part of the CERTs and as a special opportunity given that the CERTs are centers and have money for pilot studies. "I think a critical aspect of our CERT is our ability to serve as a training vehicle for young investigators," said a UAB Investigator. The Penn CERT developed a pharmacoepidemiology fellowship-training program which had six fellows by 2002-2003 in response to the need for well-trained clinical scientists (Penn PR02-03). Less formally, some CERTs described providing access to data or access to study collaborators to further train newer investigators (e.g. HMO-affiliated students have sought access to CERTs HMO data for their dissertations).

Some CERTs stated that providing junior faculty research opportunities increased their visibility. Some CERTs centers described having internal scientific meetings, journal clubs, seminars on research methods and therapeutics, visiting professorships, and feedback opportunities. The Penn CERT has a strong commitment to education and training about anti-infective therapeutics and epidemiological research. One Penn investigator stated, "Were it not for the CERT many of these people would not be focused on anti-infectives. It has allowed us to provide support to trainees and others within the center."

Some CERTs investigators attributed their involvement with their CERT as being helping their career, for example by providing "seed money", mentorship with experienced investigators, and access to feedback (i.e. CERTs scientific calls). As one CERT investigator stated, "My involvement in the CERT certainly has helped my career" (UAB) and another attributed to the CERT his shift from basic to outcomes research. A few of the more junior CERTs investigators identified the principal investigators of their center and other senior investigators as mentors. A Duke CERT investigator stated his work on practice guidelines and involvement with the CERTs enlarged his perspective (in grad school his focus has been on medical decision making and mostly at the point of care, in physician-patient interactions) and now because of the CERT work is more focused on policy.

Additionally, the CERTs program as a whole fosters the development of junior and seasoned investigators in their monthly scientific calls in which researchers present their research or project ideas to obtain constructive criticism and feedback. Many CERTs investigators, who raised the topic of the scientific calls, characterized the calls as very helpful, collegial, and a safe environment in which to present projects. One investigator stated, "Scientific calls have been a fabulous innovation" (UNC). A CERT PI explained:

A key has been the bonding among the members of the network. We share things on those scientific calls that we never would have shared because we were rivals. Lots of organizations with therapeutics as focus come to those annual meetings.

The CERTs, partly because of the relationships they have built and their focus on education, appear to have created a less competitive environment, allowing for collaboration, networking and collegiality. A number of CERTs investigators and stakeholders identified the minimization of competition and collaboration model as unique and extremely positive, as one describes: "Genuine camaraderie... Not too much competitiveness... Very frank and honest collaboration on scientific calls... has strengthened familiarity and collegiality. We need more sharing and collegiality in science."

3.4.2. Centerness

"Centerness" is a concept that is helpful for understanding program outcomes beyond research and education. Centerness as a concept has been used in federal agencies for funding and evaluating centers. The National Institute of Drug Abuse includes among the characteristics of centerness thematic focus, synergy, and involvement of different disciplines.⁵² An AHRQ representative describes the original RFA for the CERTs, "We wanted them from the beginning to have a centerness about them" and to be centers that already existed. The center structure was intentional, but respondents also responded to a question about the advantages and disadvantages of the center structure. A Steering Committee member said:

Once you're designated as a center then you can let that be part of your marketing to gain a portfolio of research funders, research projects, and to build from strength to strength. Without pre-existing funding as a center, you don't have the critical mass that allows you to move this field forward, so the funding of centers, as opposed to individuals, was a critical conceptual breakthrough. And then there is the willingness to work across centers, to help all of us to move from strength to strength ...

The CERTs investigators offered different interpretations of what makes their CERT a center, including:

• A number of projects that all have the same common theme...that brings people together. People have a tendency to drift on their own... it helps to have this framework. It encourages people to do things they would not normally do and keeps people focused on one area of research. (Duke)
• We know each other's strengths and weaknesses. To know each other and internally critique, there's nothing like proximity...there is a huge advantage being a physical center. (Penn)
• Fosters the sense of collaboration and reduces the sense of competition. (UAB)
• There is a structure... and goals that are common to the center...a center takes the next step to identify interventions...it's not just about research but about intervening" when you're a center... (Duke)

The UAB CERT's situation is different because the University of Alabama has a center structure that is encouraged and has certain requirements. In response to our question about "center," we learned that the UNC CERT is a center in many ways, but not by the University's standards. "It's a center by designation. It's not a center in UNC's organizational status that reports directly to the provost."

The NIDA criterion that a center have a thematic focus is fulfilled for the CERTS by their focus on therapeutics research. A second criterion is multidisciplinarity. The CERTs values include multidisciplinary alliances: "the best research harnesses the collective expertise of medical practitioners, clinical pharmacologists, health services researchers, clinical epidemiologists, pharmacists, clinical researchers, and others involved in health care."⁵³ With this health-focused definition of multidisciplinarity the CERTs centers (with some variability) have involved individuals from these diverse backgrounds and expertise. An AHRQ representative explains the original intention of multidisciplinarity:

The idea was that you would have a multidisciplinary group that could look at areas in a variety of ways. It could be survey research or epidemiologic studies, or pilot clinical trials if there was enough money... the need was to have not just one discipline but many different disciplines ... to focus on a particular theme... increases the probability of success. Having a few senior investigators, a PI, and some trainees continues the development of the field; this was really important to us ...

Most investigators identified the center structure as advantageous, because a center provides opportunities for interactions with different investigators and collaboration. Additionally, a center provides researchers with an infrastructure and support that is helpful, for example, in applying for grants for which the center provides technical assistance. A center structure was identified as key for building data systems as well. Generally resources shared within a center across investigators was raised as helpful by a number of investigators in describing how the CERT center facilitated their work. A few investigators believe their research would not have been possible were it not for their CERT center and its infrastructure and support.

Being a designated center... creates so many opportunities by the people we have involved with the CERTs to learn about other research opportunities... just to have a great team assembled by the PI consisting of collaborators inside and outside the university is a strength of our center. (UAB)

The CERTs bring together researchers and experts from: pediatrics, rheumatology, clinical pharmacology, cardiology, gastroenterology, internal medicine, biostatistics, psychology, communications, and pharmacy, among others. A CERT PI believed that the center structure facilitated involving multiple disciplines in the center. A few respondents stated that a center improves the credibility of the center with its supporting infrastructure. One respondent identified the center as useful for recruiting purposes. The educational components of the CERTs program have helped to "create a critical mass of seasoned investigators to train others" (CERT Investigator).

Given the wide-scoped of the CERTs program mandate, the use of a center structure and the nature of a center to bring in individuals to focus on the same topic, it was stated by some stakeholders that being a center creates synergy and a "whole that is greater than the sum of its parts" because as one respondent explained, "It is important to create centers if you want to move a field forward." An AHRQ respondent further explained that the centers would ultimately help "to have a field that would perpetuate itself."

Return to Contents
Proceed to Next Section