United States National Library of Medicine National Institutes of Health

HTA 101: Monitoring Impact of HTA

The impacts of HTAs, from market research reports to RCT reports to expert panel statements, are variable and inconsistently understood. Whereas some HTA reports are translated directly into policies with clear and quantifiable impacts, the findings of some "definitive" RCTs and authoritative, well-documented assessment reports go unheeded or are not readily adopted into general practice (Banta 1993; Ferguson, Dubinsky 1993; Henshall 2002; Institute of Medicine 1985).

As is the case for the technologies that are the subjects of HTA, the reports of HTAs can have intended, direct impacts as well as unintended, indirect ones. Some of the ways in which a HTA report can make an impact (Banta 1993) are:

Attributing Impact to HTA Reports

The impact of a HTA depends upon the target groups' legal, contractual, or administrative obligation to comply with it (Anderson 1993; Ferguson, Dubinsky 1993; Gold 1993). FDA market approvals of new drugs and devices are translated directly into binding policy. Most of the HTAs conducted by AHRQ are requested by CMS for use in the Medicare program, although CMS is not obligated to comply with AHRQ findings. The impacts of NIH consensus development conference statements, which are not statements of government policy, are inconsistent and difficult to measure. The ability of NIH statements to change behavior seems to depend upon a variety of factors intrinsic to particular topics, the consensus development process and a multitude of contextual factors (Ferguson 1993; Ferguson 2001).

The task of measuring the impact of HTA can range from elementary to infeasible. Even if an intended change does occur, it may be difficult or impossible to attribute this change to the HTA. A national-level assessment that recommends increased use of a particular intervention for a given clinical problem may be followed by a documented change in behavior consistent with that recommendation. However, the recommendation may be made at a time when the desired behavior change is already underway, when third-party payment policy is shifting in favor of the technology, during a strong marketing effort by industry, or close to the time of announcement of the results of a convincing clinical trial. Given widespread and nearly instant communications in health care, it may be difficult to control for factors other than a particular HTA report that might influence behavior change.

As is the case for attributing changes in patient outcomes to a technological intervention, the ability to demonstrate that the results of an HTA have an impact depends upon the conditions under which the assessment results were made known and the methodological approach used to determine the impact. Evaluations of the impact of an assessment often are unavoidably observational in nature; however, under some circumstances, quasi-experimental or experimental evaluations are used (Goldberg 1994). To the extent that impact evaluations are prospective, involve pre- and post-dissemination data collection, and involve directed dissemination to clearly identified groups with well-matched controls (or at least retrospective adjustment for reported exposure to dissemination), they are more likely to detect a causal connection between an HTA report and behavior change. Even so, generalizing from one experience to others may be impractical, as it is difficult to describe and replicate the conditions of a particular HTA report dissemination.

Factors Mediating Impact

The factors that can affect the impact of HTA reports are many. Beyond the particular dissemination techniques used, characteristics of the target groups, the environment and the HTAs themselves can mediate the impact (Goldberg 1994; Mittman and Siu 1992; Mittman and Tonesk 1992). Examples are shown in Box 41. Knowledge about these factors can be used prospectively. As noted above, assessment programs should consider how to properly target and modify their dissemination strategies to achieve the desired impact given particular characteristics of organizations, clinicians, environments, etc. Systematic attempts to document the dissemination processes and impacts of HTA programs are infrequent (Banta 1993; Goodman 1988; Institute of Medicine 1985), though a few, notably the NIH Consensus Development Program (Ferguson 1993), have been studied in detail. Like other interventions in health care, HTA programs may be expected to demonstrate their own cost-effectiveness, i.e., that the health and/or economic benefits resulting from an HTA program outweigh the cost of the program itself.

Box 41
Examples of Factors That Can Affect Impact of HTA Reports

Target provider organization characteristics

  • Hospitals: general versus specialized, size, teaching status, patient mix, for-profit vs. non-profit, distribution of payment sources (e.g., fee-for-service vs. capitation), ownership status, financial status, accreditation
  • Physicians' offices: group practice vs. solo practice, hospital affiliation, teaching affiliation, board certification, distribution of payment sources

Target clinician characteristics

  • Type of clinician: physician, nurse, dentist, etc.
  • Specialty
  • Training
  • Professional activities/affiliations
  • Institutional affiliations (e.g., community hospital, university hospital)
  • Familiarity with and access to recent literature

Environmental characteristics

  • Urban, suburban, rural
  • Competitive environment
  • Economic status
  • Third-party payment status (e.g., percentage of patients in HMOs, private insurance, etc.)
  • State and local laws, regulations
  • Malpractice activity

Characteristics of HTA findings/recommendations

  • Type: research findings, practice guidelines, standards (e.g., equipment acquisition, use, maintenance), appropriateness criteria
  • Format: printed, word-of-mouth, electronic, etc.
  • Frequency of message
  • Required level of compliance (ranging from mandatory to optional)
  • Locus of decision: general practitioner/primary care physician only, physician specialist only, multiple clinicians, physician with patient input, patient only
  • Perceived inappropriate rigidity (allowance for discretion for differing circumstances)
  • Cost of relevant procedure/management of condition
  • Payment issue(s) addressed: coverage status, payment level
  • Reputation of sponsoring organization, analysts, expert panel
  • Overall strength of evidentiary base (e.g., existence of "definitive" clinical trial)
  • Credibility/rigor of assessment process
  • Existence or potential for malpractice action
  • Timeliness of dissemination, especially compared to degree of uncertainty, most recent research findings, or current levels/change rates of utilization of procedure
  • Existence and nature of other HTA findings on same topic.

Sources: Goldberg 1994; Mittman 1992; others.

Previous Section Next Section Table of Contents NICHSR Home Page

Last reviewed: 08 September 2008
Last updated: 08 September 2008
First published: 17 August 2004
Metadata| Permanence level: Permanent: Stable Content