As you know, in August 2011 we issued a final rule in the Federal Register revising the regulations on financial conflicts of interest (FCOI) of extramural investigators, and these regulations were implemented in August 2012. One of the major changes in the new regulations was the requirement that institutions make publicly accessible certain information concerning identified financial conflicts of interest held by senior/key personnel. This requirement was included to help the public monitor the integrity of the research funded by the Department of Health and Human Services (HHS) and to underscore our commitment to fostering transparency and accountability.

This public accessibility requirement in the FCOI regulations is in line with a number of recent public disclosure initiatives. One such example is the “Sunshine Rule”, which was included in the Affordable Care Act passed in March 2010. Today, a final rule implementing the Sunshine Rule provisions was posted in the Federal Register by the Centers for Medicare & Medicaid Services (CMS).

The rule, called the “National Physician Payment Transparency Program: Open Payments”, finalizes the provisions that require manufacturers of drugs, devices, biologicals, and medical supplies covered by Medicare, Medicaid, or the Children’s Health Insurance Program (CHIP) to report payments or other transfers of value they make to physicians and teaching hospitals to CMS. CMS will post that data to a public website. The final rule also requires manufacturers and group purchasing organizations (GPOs) to disclose to CMS physician ownership or investment interests. Applicable manufacturers and applicable group purchasing organizations must begin to collect the required data on August 1, 2013 and report to CMS by March 31, 2014. The final rule contains a lot more information on reporting these payments or other transfers of value and their subsequent publication on the website.

There are some major differences between these two rules. For example, the FCOI regulations cover financial interests of investigators receiving or applying for funding from the Public Health Service including the NIH, while the new CMS rule covers payments from the entities listed above to physicians and teaching hospitals. As I’ve shown before, around 30% of NIH-funded principal investigators have an MD or MD/PhD degree. Therefore, some individuals may be covered by both rules.

It is extremely important that we all strive to maintain the public trust in biomedical research and its applications. The National Physician Payment Transparency Program is part of this evolving landscape.

You may recall that back in February 2012 I blogged about OMB’s request for comment on proposed reforms to federal grant policies contained in Office of Management and Budget (OMB) circulars such as A-21, A-133, and A-122. OMB proposed these reforms to streamline Federal policies relating to grants. The essential idea behind this reform is to reduce the “red tape” and unnecessary or overly burdensome requirements so that grantees can better focus their efforts achieving their research objectives.

The prior Advanced Notice of Proposed Guidance (ANPG) received more than 350 public comments, all of which OMB has considered carefully in drafting the proposed rule issued on February 1, 2013 in the Federal Register (note that there are two primary documents: the preamble and the full text of the proposed rule). I am pleased to note that the public’s feedback has influenced the draft guidance greatly and OMB has opened the latest version of the proposed rule for public comment once again.

Providing comments and feedback on the proposed rule is an important opportunity for the grants community because their comments can be used to further refine the reforms before the final guidance is issued. I described the scope of the proposed reform in my previous blog post, and I suggest that you read the Federal Register notice in full, but I want to highlight some key areas on which OMB seeks specific feedback, as these topics are highly relevant to the NIH research administration community:

• Proposed audit language for time and effort reporting requirements. OMB is asking whether the language proposed adequately provides enough flexibility for institutions to meet these standards in the way most appropriate to their particular organizations.
• Revisions to reimbursement for utility costs to institutions of higher education.
• One time extension of indirect (“F&A”) costs for all types of institutions, i.e. the option of extending negotiated rates for up to 4 years subject to approval of the indirect cost cognizant agency.

There are many more issues raised in the proposed rule; again, I encourage you to read the notice in its entirety. Please note the 26% limitation on reimbursement of administrative costs remains for institutions of higher education and would not apply to other types of institutions such as non-profit organizations.

Also, please note that OMB offers a number of resources and supporting documents comparing current requirements and proposed changes to help you navigate the proposed changes. Some of you may also be interested in viewing the upcoming webcast on Feb. 8 at 11 am EST, hosted by OMB and the Council on Financial Assistance Reform.

Both Federal agencies and the public are invited to submit comments at www.regulations.gov; the last day comments will be accepted is May 2, 2013. My office has been providing comments throughout the development of this guidance, and will do so again now. I encourage you to share your thoughts with OMB too.

As I’ve been discussing the biomedical research workforce with people in the extramural community over the past few months, I’ve heard people say things like “PhDs don’t do much research with human subjects”. I decided to look into data that could support or refute statements such as this, and started with checking data on the use of animal and human subjects in research.

As you can see in the graphs below, we looked at awards by degree over the past 10 years and grouped them into four categories based on the research described in the grant applications: 1) human subjects research only (defined as having been reviewed by an Institutional Review Board), 2) animal subjects research only (defined as having been reviewed and approved by an Institutional Animal Care and Use Committee), 3) no human or animal subjects, or 4) both human and animal subjects.

To put these data in context, you’ll remember that about 70% of NIH-funded PIs have PhDs, with most of the remaining PIs having either an MD or an MD/PhD. Since the number of PIs with other degrees (DDS or DVM only, for instance) is very small, we’ve provided those data separately – see the footnote below.

First, we took a look at human and animal subject use by each degree category: PhDs, MD/PhDs, and MD.  As shown in figure 1, among PIs with a PhD degree, about 45% of PhDs engage in research involving only animal subjects, about 22% engage in human subjects research, a little less than 30% do research with no human or animal subjects, and only 5% conduct research with both. These proportions haven’t changed much over time.  

The breakdown for PIs with MD/PhD degrees is slightly different (figure 2).  A higher proportion – over 50% – engage in research with animal subjects only, nearly 22% conduct research involving human subjects only, while around 10% do research with no human or animal subjects and another 10% with both.

Looking at PIs with MD degrees, we see MDs are likely to engage in research solely involving human subjects (approximately 45%), and also likely to engage in research involving only animal subjects (nearly 35%). However, they are the  least likely to conduct research that involves no human or animal subjects – only about 8% of MDs conduct research that does not involve human or animal subjects (figure 3).

In this last graph we’ve pulled out the human subjects only percentages for each of these 3 degree categories and compared them all on one graph. As shown in figure 4, investigators with MD degrees do tend to work on research involving only human subjects. This is not too surprising, though it is interesting that such a significant and similar percentage of PIs with a PhD and MD/PhD do research exclusively with human subjects.

Taken together, these data support the idea that many PIs are conducting research that fits with their academic training, but a substantial number of PhD-trained scientists are doing research with human subjects and many clinician-scientists (either MDs or MD/PhDs) conduct research with neither human nor animal subjects.  Also, changes in the extramural community over the past decade do not seem to have affected these proportions significantly.  These data may prove useful to the new advisory committee to the director’s working group on clinician-scientists.

—-

Note: The data table corresponding to these figures is posted on RePORT and can be downloaded in an Excel file. The table includes data for other advanced degrees (e.g. DVM, DDS, and DMD), which were not plotted on the charts because of their small numbers.

This week I want to address the field of health research in the lesbian, gay, bisexual, transgender and intersex communities (LGBTI communities). Earlier this month, NIH Director Francis Collins announced additional support of LGBTI health research by renewing and permanently establishing a trans-NIH LGBTI research coordinating committee under the leadership of two NIH institutes: the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) and the National Institute on Minority Health and Health Disparities (NIMHD). This committee was initially established to evaluate the NIH research portfolio and how it addressed health issues unique to LGBT individuals, as a follow-up to the NIH-commissioned Institute of Medicine (IOM) report published in 2011. See Francis Collins’s statement for more information about the IOM and NIH reports and plans to bolster LGBTI health research.

In February 2012, we released three specific funding opportunity announcements (FOAs) for R01, R03, and R21 awards in this research area. One misconception that I want to clarify is whether researchers can submit applications related to LGBTI health to other funding announcements. The answer is of course. While we issued these FOAs to draw specific attention to research needs in LGBTI health, NIH welcomes applications related to LGBTI health for any applicable FOA, and we encourage investigators to discuss their ideas with NIH staff in advance.

As scientists, we’re well acquainted with the importance of protecting our research from factors that could undermine objectivity. And as the nation’s largest single funder of biomedical research, it is vitally important that the public trusts the research NIH supports.

 To this end, NIH recently published a centralized document describing our policies pertinent to scientific research integrity. While NIH has extensive policies and procedures to help ensure the highest degree of scientific integrity, this new document –  NIH Policies and Procedures for Promoting Scientific Integrity –  for the first time consolidates this information from a diverse array of documents into a unified report that’s easily accessible to anyone interested in learning more about NIH research, even those new to working with or learning about NIH.

 An NIH working group, which included members of my staff in the Office of Extramural Research, developed this resource, and I’m pleased to see the outcome of their hard work. This document not only fulfills a White House directive for a public document on scientific integrity, but it is also supports transparency and public accountability that is at the core of NIH’s mission.

Time flies when you’re having fun! I started Rock Talk on January 19, 2011 (see the first two posts here and here) and when Nature asked me to write about my adventures in blogging to commemorate Rock Talk’s two year anniversary I gladly agreed… check out my article here.

As a followup to my recent blog post on fiscal year 2012 success rates, I’d like to post an update of an earlier blog post where I explained how paylines, percentiles and success rates relate to one another. It’s a long one, but should be helpful in understanding what we mean when we look at success rates.

 “Paylines, Percentiles and Success Rates” with updates added:

I have read or heard much about the dilemma of NIH applicants as they struggle to understand their chances of receiving NIH funding. As budgets flatten and tighten, this discussion has heated up. To declare that NIH success rates have hovered around 20% for the past five years does little to calm the storm of concern when we hear about shrinking percentiles and paylines. So how is it possible to have a success rate of 20% but a payline at the 7^th percentile? Let’s take a few moments to sort out what these things mean and think about how these numbers are derived and how they can differ.

IMPACT SCORE

It all starts with the impact. This score is assigned by reviewers to indicate the scientific and technical merit of an application. Impact scores range between 1 and 9. A score of “1” indicates an exceptionally strong application and “9” indicates an application with substantial weakness. (I always wondered why at NIH low = good and high = bad but that predates me!) In assigning an impact score, reviewers consider each of five scored criteria: significance, investigator, innovation, approach, and environment, along with other factors like protection of human subjects and vertebrate animal care and welfare. Read more about scoring.

PERCENTILE RANK

The percentile rank is based on a ranking of the impact scores assigned by a peer review committee. The percentile rank is normally calculated by ordering the impact score of a particular application against the impact scores of all applications reviewed in the current and the preceding two review rounds. An application that was ranked in the 5^th percentile is considered more meritorious than 95% of the applications reviewed by that committee. This kind of ranking permits comparison across committees that may have different scoring behaviors. It is important to note than not all research project grant applications (RPGs) are percentiled. For example, applications submitted in response to a request for applications (RFA) are usually not percentiled. In the absence of a percentile rank, the impact score is used as a direct indicator of the review committee’s assessment. Read more about percentiles.

PAYLINE

Many NIH institutes calculate a percentile rank up to which nearly all R01 applications can be funded. For grant applications that do not receive percentile ranks, the payline may be expressed as an impact score. Institutes that choose to publish paylines in advance (see an example) calculate the payline based on expectations about the availability of funds, application loads, and the average cost of RPGs during the current fiscal year. Other institutes prefer to describe the process for selecting applications for funding (see an example) and then report on the number of applications funded within different percentile ranges at the end of the fiscal year (see an example) Because the NIH is currently operating on a continuing resolution and funding levels for the remainder of this fiscal year are uncertain, most of the NIH institutes have offered less detail this year than in the past.

But remember, even when an IC establishes a payline, applications outside of the payline can be paid under justified circumstances if these applications are a high priority for the particular institute or center. When these select-pay/out-of-order/priority pay/high priority relevance selections are made, it may result that other applications within in the payline are not paid because funds are no longer available to support them.

SUCCESS RATES

The success rate calculation is always carried out after the close of the fiscal year, and it is based on the number of applications funded divided by the number of applications reviewed and expressed as a percent. To better reflect the funding of unique research applications, the number of applications is adjusted by removing revisions and correcting for projects where the resubmission (A1) is submitted in the same year as the original application (A0). Read more about success rates.

THE ANSWER

Now we are equipped to answer our earlier question. How is it possible to have a success rate of 20% but a payline at the 7^th percentile? There are several real-life reasons why paylines (the ones that use percentiles) can be either higher or lower than success rates.

• Applications that are not percentiled are still factored into the success rate calculation. Thus, funding a number of awards that are not assigned percentiles will increase the success rate without changing the payline.
• The success rate for a particular fiscal year is a reflection of the funded applications and can include applications reviewed in the previous fiscal year; whereas, the payline encompasses only applications reviewed in that fiscal year. So awarding applications that were reviewed in the previous year will also increase the success rate.
• The average quality of the applications assigned to an institute will also affect its payline. If an institute happens to receive a set of applications with very good (low) percentile scores, its success rate will be higher than its payline, all else being equal. For example, in fiscal year 2011, the NIGMS R01 success rate was about 24% but the midpoint of the funding curve occurred close to the 19^th percentile.

Check out more reports on RPG success rates broken down by year and IC at report.nih.gov – if you’re interested in other success rates, you can find them on our RePORT website as well.

Whew, you made it through. The difference between paylines, percentiles and success rates remains a confusing topic because of the compounding factors that rule out a simple linear relationship. You need to consider all the factors when assessing the potential for an individual application to be funded. Your best advisor on this issue, because of the differences in the ICs and programs, is your NIH program official. Give him or her call.

The numbers for fiscal year (FY) 2012 are in. Here are some facts about applications and awards in FY2012, compared to FY2011:

This 2012 data, and data from past years, can be found in the NIH Data Book. This is the first place to look for summary statistics on NIH awards — data and charts are exportable for easy incorporation into reports, presentations, or your own blog posts.

Looking back on these data, the first thought that comes to my mind is, “We made it.” Despite a flat budget and complex fiscal times, we maintained last year’s success rate and slightly increased the amount of award dollars that went to research project grants. We continue to strive to maintain a diverse portfolio of biomedical research, and keep this important work moving along quickly.

Even though the current fiscal year remains uncertain, we know that above all, it is critical to support your continued work on innovative science. Stay tuned for more data.

At the Advisory Committee to the Director’s meeting today and yesterday, NIH director Francis Collins, NIH deputy director Lawrence Tabak, and I presented some exciting new initiatives in support of the future of biomedical research.

As I’ve blogged about before, the Advisory Committee to the Director (ACD) formed three working groups to address three important topics in science — harnessing the power of biomedical data and informatics, achieving diversity in the biomedical research workforce, and developing a competitive and sustainable biomedical research workforce. In June, these working groups presented their recommendations, and at the ACD meeting yesterday and this morning, we proposed how NIH would implement them.

Two new initiatives would support the management, analysis and integration of large-scale data and informatics. The Big Data to Knowledge (BD2K) initiative seeks to facilitate broad use of biomedical big data through new data sharing policies, catalogs of datasets, and enhanced training for early career scientists entering the new world of big data.  The NIH InfrastructurePlus initiative will advance high-performance computing, agile hosting and data storage, and modernization of the NIH network, among other approaches.

In response to the diversity working group recommendations, NIH proposed launching several initiatives to meet the great and important challenge of increasing diversity in the biomedical workforce. Two of the programs focus on enhancing mentoring. For example, a new NIH program called BUilding Infrastructure Leading to Diversity (BUILD) provides rigorous mentored research experiences for undergraduate students, resources to help faculty train highly effective mentors, and more. NIH also proposed establishing the National Research Mentoring Network to connect students, postdoctoral fellows, and faculty with experienced mentors — in-person and virtually — as well as provide relevant workshops and training opportunities in grantsmanship. The BUILD program and NRMN will form a consortium to link trainees and investigators from groups that have been underrepresented in science to majority investigators.

In addition to these programs, NIH would test multiple interventions to assess and mitigate the effects of implicit bias in peer review, including diversity awareness training for both scientific review officers and members of review panels.

To spearhead these efforts, NIH would recruit a chief diversity officer to not only coordinate diversity initiatives, but to oversee a rigorous prospective evaluation of existing extramural and intramural diversity programs, and join NIH’s intramural program as a practicing scientist.

Finally, as co-chair of the working group on the biomedical research workforce, I’m excited to share how NIH plans to support this critical component of the biomedical research enterprise, and improve the training experience of graduate students and postdocs alike. We intend to launch a program to support innovative approaches that expand and complement existing research training to include science-related career outcomes, and also encourage the adoption of individual development plans for all trainees. NIH plans to increase the funding of awards that encourage independence like the K99/R00 and early independence awards, and increase the initial postdoctoral researcher stipend. NIH also intends to embark on novel ways of improving the trainee experience, such as looking more closely at, and soliciting community feedback on, postdocs’ access to workplace benefits.

Additionally, NIH plans to expand our ongoing assessments of the biomedical research workforce, including conducting a follow-up study on clinician scientists, and developing a simple and comprehensive tracking system for trainees. Collecting this data will not only help us with evaluating NIH’s current workforce initiatives and planning our future efforts, but also provides valuable data for those considering careers in biomedical research.

These are all ambitious initiatives which will take some time to refine and implement. As they unfold, I will continue to discuss their details here and in other forums. It’s very satisfying to see how the past two years of work is coming to fruition. I think we’re headed in the right direction, and I am looking forward to your feedback.