Quality and Quality Improvement

NRSA Trainees Research Conference Slide Presentation (Text Version)

By Carole Lannon, M.D., M.P.H., and Peter Margolis, M.D., Ph.D.

---

On June 25, 2005, Carole Lannon and Peter Margolis made a slide presentation on quality improvement (QI) at the 11th Annual National Research Service Award (NRSA) Trainees Research Conference. This is the text version of the slide presentation. Select to access the PowerPoint® slides (4 MB).

---

Slide 1

NRSA AHRQ Workshop: Quality and Quality Improvement

Carole Lannon, M.D., M.P.H.
Peter Margolis, M.D., Ph.D.

Slide 2

Session Objectives

1. Overview: what is driving the quality momentum?
2. What do we know about quality improvement:
   1. Examples: VON [Vermont Oxford Network], ESRD [End-Stage Renal Disease ].

The challenges of evaluation:

1. An in-depth example of successful QI:
   1. Methods exist to improve quality.
2. Key components of successful QI methods.

Slide 3

Session Objectives

1. Overview: what is driving the quality momentum?
2. What is quality and quality improvement?
3. What changes clinical care?
4. A conceptual framework for improvement.
5. An in-depth example of successful QI.
6. Challenges with evaluation.
7. Next.

Slide 4

Illustration of the cover of the Institute of Medicine report, To Err is Human: Building a Safer Health System.

Slide 5

The Chain of Effect in Improving Health Care Quality

I. Environmental context.
II. Macro-organization (e.g. health care organization).
III. Microsystem of care delivery (e.g., practice).
IV. Child, family, and community.

Arrows point up and down between each item in the list.

Slide 6

What is driving the quality momentum?

Slide 7

"Drivers" in the current environment

• Clinical teams' motivation to provide best care for patients.
• Consumers (patients and families) expectations to receive best care.
• Leadership:
  • Raising awareness and building will.
  • Supporting change and spreading key improvements.
• Maintenance of certification and ACGME competencies.
• Public and private emphasis on use of quality measures.
• Pay-for-performance programs.

Slide 8

Board certification of physicians

• Certification:
  • Until ~1990: Knowledge-based one-time test.
• Re-certification:
  • 1990s: episodic cognitive examination every 6-10 yrs.
• Maintenance of certification:
  • Continuous rather than episodic.
  • Focus is on six primary competencies.

Slide 9

Six Core Physician Competencies

• Patient care.
• Medical knowledge.
• Interpersonal and communication skills.
• Professionalism.
• Practice-based learning and improvement.
• Systems-based practice.

Adopted by the ACGME and all twenty four ABMS specialty boards.

Slide 10

Two arrows point downward from two boxes, one labeled "PMCP-G" and the other "PMCP-G," to a stack of boxes below that read:

Part One: Professional Standing.
Part Two: Lifelong Learning.
Part Three: Cognitive Expertise.
Part Four: Performance in Practice

• Peer & Patient Surveys.
• Practice Assessment and Quality Improvement Component.

Slide 11

National emphasis on measures and P4P

• National Quality Forum.
  Set consensus-based national standards for measurement and public reporting of healthcare performance data.
• National Committee on Quality Assurance.
• Joint Commission for the Accreditation of Healthcare Organizations.
• AMA Consortium on Performance Measures.
• Ambulatory Care Quality Alliance.
• Leapfrog Group.

Slide 12

National emphasis on P4P and HIT

P4P

• Link quality, measures, and reimbursement.
• Over 100 programs underway; strong push from CMS for legislative mandate:
  • Bridges to Excellence.
  • CMS/Premier.
  • Integrated Health Association.

Office of the National Coordinator for Health Information Technology

Slide 13

2004 National Reports on Quality and Disparities

• Second annual reports focus on quality of and disparities in health care in America.
• AHRQ.

Illustrations of the covers of the National Healthcare Quality and Disparities Reports.

Slide 14

Key Question

• What do we mean by Quality?
• What is the difference between Quality and Quality Improvement?

Slide 15

Inspection Focus

(Is competency an all or none phenomenon?)

Chart shows bell curve displaying Number of Providers against Increasing Quality. A section at the low end of the curve is labeled "Inspection and Regulation for Public Safety," and highlighted with red stripes labeled "Sanctions."

Slide 16

Improvement Focus

Chart shows bell curve, as in Slide 15, but now the low end of the curve is labeled "Standard" and an arrow pointing to the high end labeled "System Improvement" appears above the curve. A second curve in dotted lines rises to the right of the first curve to indicate increased quality.

Slide 17

Improvement Focus

Chart shows bell curve, as in Slide 16, but the second curve with dotted lines has been replaced with a "spiked" curve within the first curve to indicate a sharp rise and fall in quality/competency.

Slide 18

Improvement Focus

Chart shows bell curve, as in Slide 17, but now the "spike" has moved to the left of the first curve.

Slide 19

Percent of Patients Meeting the NKF-DOQI Target Urea Reduction Ratio of 65%

Figure 4.25, prevalent hemodialysis patients, 1999, by HSA.

A map of the United States from the USRDS Atlas, with areas marked in shades of grey to show percentage of hemodialysis patient.

Slide 20

Quality of Care (ESRD)

Line graph showing percentage of patients per Urea Reduction Ratio (URR) for the October-December quarter of 1993 through 1997:

Percentage of Patients	Oct-Dec 1993	Oct-Dec 1994	Oct-Dec 1995	Oct-Dec 1996	Oct-Dec 1997
<50%	7.7	5.4	4.2	2.9	2.5
50-54%	8.9	7.3	4.9	3.5	2.9
55-59%	17.6	13.7	10.6	7.7	6.7
60-64%	23.4	24.9	20.9	18	15.7
65-69%	22.6	27.2	29.9	31.5	28.4
70-74%	13.3	15.5	21.3	25.8	28.9
75-79%	4.9	5.1	7.3	9.3	13.1
80+%	1.7	0.9	0.9	1.1	1.8

Average URR: 1993, 62.7%; 1994, 63.8%; 1995, 65.5%; 1996, 66.8%.

Note: Sixteen Network areas participated in the first ESRD Core Indicators assessment (Oct-Dec. 1993); all Network areas participated in subsequent years.

Slide 21

Adequacy of Hemodialysis

Line graph showing percentage of adequate hemodialysis dosage by race, comparing black and white patients for years 1993-2000:

Patient Race	1993	1994	1995	1996	1997	1998	1999	2000
Whites	46	53	62	70	73	76	85	87
Blacks	36	43	54	63	69	70	83	84

Source: Sehgal A, JAMA 2003;289(8):996-1000.

Slide 22

"Adults received (only) 54.9% of recommended care... strategies to reduce these deficits are warranted."

—McGlynn, NEJM 2003

Slide 23

AHRQ taxonomy 'QI strategies'

• Provider reminder systems.
• Audit and feedback.
• Provider education.
• Patient education.
• Promotion of self-management.
• Patient reminder systems.
• Organizational change.
• Financial incentives, regulation, and policy.

Source: AHRQ Technical Review 9: Closing the Gap.

Slide 24

What Changes Clinical Care?

Bar graph shows which methods are most effective:

Method	Percent of Studies
	Change	No Change
Educational Outreach	100	0
Opinion Leader	100	0
Reminders	84	16
Patient Mediated	78	22
Audit & Feedback	42	58
Educational Materials	36	64
CME	16	84

Source: After Davis, JAMA 1995.

Slide 25

Effectiveness of interventions in improving physician behavior or health outcomes

• Little effect—Didactic lecture-based, mailed unsolicited materials.
• Moderately effective—Audit and feedback, especially if delivered by peers or opinion leaders.
• Relatively strong—Reminder systems, academic detailing, and multiple interventions.

Slide 26

What increases effectiveness of CME [continuing medical education]

• Active (interactive) learning opportunities.
• Longitudinal or sequenced learning.
• Enabling methods to facilitate implementation in the practice setting (e.g., tools, strategies).

Source: Davis, JAMA 1999.

Slide 27

Most effective CME

• Assessment of learning needs.
• Interaction among clinician-learners with opportunity to practice the skills learned.
• Sequenced and multifaceted educational opportunities.

Source: Mazmanian and Davis, JAMA 2002:288;1057-1060.

Slide 28

Paradigm shift!

• Continuing education.
• Measurement.

Slide 29

Partnership for Quality

Illustration of three children holding hands. Below are the logos for The Center for Children's Healthcare Improvement, AHRQ, and the American Academy of Pediatrics.

Slide 30

Project Description

• Setting: 10 American Academy of Pediatrics State chapters.
• Technical assistance and support at local level:
  • Chapter leadership:
    • Monthly calls.
    • Templates.
  • Practices:
    • 6-hour chapter workshop.
    • Monthly conference calls (clinical, measurement, and improvement).
    • Listserv™.
• Use of online CE program:
  • Measurement and improvement tool.
  • Use satisfies Part IV ABP maintenance of certification.

Slide 31

SuperCME ADHD Workshop C7 Goals/Target Outcomes

Bar graph shows number of responses to Workshop C7 Goals/Target Outcomes:

Management Plan includes family goals: 18.18%
Chart documents family goals: 19.79%
Discussion deferred until future visit: 9.09%
Not documented: 53.94%

Slide 32

"Asking busy clinicians to collect data is like adding a brick to a full backpack."

Slide 33

Performance measurement

Doable:

• Track changes in normal care setting.

Valuable:

• Provide baseline for comparison and document improved outcomes.
• Identify opportunities for improvement.
• Identify processes on which to focus effort.

Practical:

• Economic to obtain.
• Can be imbedded in work.
• Contributes to sustaining gains.

Slide 34

Achieving quality through systems change

Slide 35

Deming's System of Profound Knowledge

Illustration of a three-sided pyramid. The three base points are labeled "Theory of knowledge," "Psychology," and "Understanding variation." The top point is labeled "Appreciation of a system."

Slide 36

Deming's System of Profound* Knowledge

• Appreciation of a system.
• Understanding variation.
• Human psychology of change.
• Theory of knowledge.

*Profound—having intellectual depth and insight (Webster).

Slide 37

The relationship between improvement science (Deming) and HSR

Improvement Science	HSR
Appreciation of a system. Knowledge of variation. Theory of knowledge. Psychology of change.	Conceptual model. Epidemiology, biostatistics. Scientific methods, testing changes. Behavior change theory.

Slide 38

An Example

Slide 39

CF physicians are justifiably proud

Bar graph showing median age at death by year:

1940: 0.5
1950: 1
1960: 10
1970: 16
1980: 18
1990: 29
2000: 32

Slide 40

What is a System?

"A network of interdependent components that work together to accomplish a shared aim." (WE Demming)

The aim of the CF system:

To assure the development of the means to control and cure cystic fibrosis (CF Foundation).

Slide 41

Illustration of a three-sided pyramid. The three base points are labeled "Theory of knowledge," "Psychology," and "Understanding variation." The top point is labeled "Appreciation of a system."

Slide 42

System of Care for Children with CF

Cystic Fibrosis Foundation

• Sets research agenda (e.g., genomics):
  • Collaboration with industry, NIH.
  • Provides significant portion of funding.
• Assures availability of high quality care:
  • Accreditation of care centers (provider team, support services, laboratory procedures, diagnostic testing).
  • Provides funding for these mandates.

Slide 43

System of Care for Children with CF

• Disseminates new knowledge:
  • NA Cystic Fibrosis meeting (>3,000 people from all disciplines).
  • Consensus statement and guideline development.
• Tracks patient outcomes:
  • National data registry.
• Advocacy.

Slide 44

The CF Registry

• Contains demographic and clinical data on all patients attending accredited care centers.
• Content and use have evolved over 20 years:
  • Initially, describe the CF population.
  • Hypothesis generation:
    • Epidemiologic risk factors.
    • Disease progression and pathogenesis.
  • Since 1998:
    • Define variability in outcomes among centers.
    • Clinical information system to support quality improvement.

Slide 45

Variation in outcomes among CF Care Centers

Graph shows percentage of children below 5th percentile for weight, by center.

Slide 46

Survival of Children with Cystic Fibrosis University of Minnesota Data

Line graph comparing total U.S. survival rate with Minnesota's. The U.S. median survival in 1998 was 32 years, median survival at Minnesota center was 46 years.

Slide 47

Our Initial Responses as Clinicians

It's not true: The data are wrong. The interpretation is wrong. Our patients are sicker than theirs.

It can't be true because: We are world famous experts and/or researchers at cutting edge of CF care. We work really hard and our intentions are pure.

Slide 48

It's the system!

Every system is perfectly designed to achieve exactly the results it gets.

Illustration of a man seated in a yoga position on a mountaintop.

Slide 49

The CF System at the University of Minnesota

• Multi-disciplinary educational program for newly diagnosed patients and families.
• Written guidelines and standards for outpatient care.
• Detailed planning before clinic visits:
  • Pre-visit patient questionnaires to identify needs.
• Aggressive approach to nutrition.
• Database of all patients linked to lab.
• Regular review of overall population performance.

Slide 50

Maybe we can use some Minnesota ideas?

Slide 51

Examples of QI Studies

• How effective are specific changes (e.g., reminder/recall) in care processes?
• How effective are QI methods in promoting changes in care delivery?
• Which process changes are appropriate in different clinical settings? (e.g., large vs. small practices)

Slide 52

Matching the Study Design to the Question

A framework for QI studies/projects.

Slide 53

Comparing Research and Improvement Studies

	Type of Study
	Research	Improvement
Aim	New knowledge (What?)	Applying knowledge: Improve care in specific setting (How?)
Objective	Estimation of effect	Prediction of future
Outcomes	1-2 dimensions	Multiple dimensions
Sample size	Hypothesis testing	Prior knowledge
Approach to "bias" and variability	Background	Foreground
Confounding	Randomization Adjustment	Structured testing in subgroups
Selection	Restriction	NA
Chance	Random sampling	Control methods (SPC)
Time/secular trend	Comparison groups	Statistical process control Interrupted time series

Slide 54

Model for Improvement

What are we trying to accomplish?

How will we know that a change is an improvement?

What change can we make that will result in improvement?

Illustration of a circle divided into 4 quarters labeled Act, Plan, Do, Study; arrows around the rim of the circle indicate that this is an ongoing process.

Slide 55

How to use the model

• Provides framework for conducting tests.
• Goal: improve care.
• How: Conduct tests to learn which changes to current system will result in improvement.
• Initial focus on aim and measures.

Slide 56

Building Belief in a Change

Illustration of four circles labeled Act, Plan, Do, Study, as in Slide 54, rolling up an incline. The lower end of the incline is labeled "Hunches, Theories, Ideas," and the upper end is labeled "Changes That Result in Improvement." Across the top is an arrow labeled "Data" pointing toward the upper end of the incline. The first circle at the lower end is labeled "Very Small Scale Test," the second is "Follow-up Tests," the third is "Wide-Scale Tests of Change" and the fourth, nearest the top, is "Implementation of Change."

Slide 57

Cystic Fibrosis Aim

• Nutritional goal: reduce the proportion of children in nutritional failure.
  By increasing use of nutritional status classification, self management goals, and appropriate medical interventions.
• ETS exposure goal: eliminate ETS exposure.
  By documenting parental smoking, prescribing a smoke free environment, and counseling parents on smoking cessation.

Slide 58

How will we know if a change is an improvement?

"All improvement requires change, but not all change is an improvement."

Slide 59

Principles of QI measurement

• Enable participants to observe gaps between desired and observed outcomes.
• Facilitate priority setting.
• Track performance over time.
• Facilitate communication between teams working toward common goals.

Slide 60

Screenshot of Progess Report data from the Public Health Preparedness Web site at www.publichealthpreparedness.org

Slide 61

Screenshot of charts showing At Risk data from the Public Health Preparedness Web site at www.publichealthpreparedness.org

Slide 62

Building a Measurement System

Attributes:

• Small number (<8) of key measures that refer to the Collaborative goals.
• Balance—together describe a great system of care.
• Ideally—clearly defined for data collection and reporting (but not necessary).

Slide 63

CF Goals

• Improve nutritional status.
• Reduce exposure to environmental tobacco smoke.

Slide 64

CF Collaborative Measures

Measure	Population Statistic	Typical Levels	Appropriate Goal
Nutritional Outcome Measures
Nutritional Status—Nutritional Failure	Percent of children with "nutritional failure" according to 2001 consensus conference	20% (>1 year of age)	10% (>1 year of age)
Nutritional Status—At Risk	Percent of children "at risk" for "nutritional failure" according to 2001 consensus conference	10%	5%
Nutritional Status—Weight Percentile	Median weight percentile of CF center population	27%	35%

Slide 65

Cycle for Learning and Improvement

Cartoon of two cavemen sitting on rocks. One is holding two fish, and other has a slab of rock and stylus to write with. The caption reads, "Negative results on the fish... Let's try rubbing two sticks together."

Slide 66

Building Belief in a Change

Illustration of four circles labeled Act, Plan, Do, Study, as in Slide 54, rolling up an incline. The lower end of the incline is labeled "Hunches, Theories, Ideas," and the upper end is labeled "Changes That Result in Improvement." Across the top is an arrow labeled "Data" pointing toward the upper end of the incline. The first circle at the lower end is labeled "Very Small Scale Test," the second is "Follow-up Tests," the third is "Wide-Scale Tests of Change" and the fourth, nearest the top, is "Implementation of Change."

Slide 67

Evaluating Effectiveness

Graphical Methods

Slide 68

Improvement in Cycle Time

Bar chart showing Cycle Time in minutes: the Average Before Change is 70 minutes; the Average After Change is 35.

Slide 69

Photograph of a Florida 2000 Presidential election ballot. A punchhole between Pat Buchanan's and Al Gore's names is circled in red with an arrow labeled "Point of Controversy."

Slide 70

Photograph of a chart following the 2000 Presidential election and showing the number of votes cast in each Florida county for Pat Buchanan. Palm Beach County is notable higher than the other counties, with 3,407 votes for Buchanan.

Slide 71

Improvement in Cycle Time

Run Chart: a graphical record of a characteristic measured over time.

Line graph shows cycle time over the period of a year. The "Change Made" is marked as occurring in June, and cycle times decrease dramatically after this point.

Slide 72

Improvement in Cycle Time

Line graph shows cycle time steadily decreasing over the period of a year. The "Change Made" is again marked as occurring in June.

Slide 73

Improvement in Cycle Time

Bar chart showing Cycle Time in minutes: the Average Before Change is 70 minutes; the Average After Change is 35.

Slide 74

QI Tools—Run Chart: Annotations link changes to outcomes

Chart labeled "'Flu shot uptake in Ped CF patients" shows percentage known to be immunized over a period of weeks: The percentage rises steadily from just above 0% in Week 1 (annotation reads "'flu shots arrive") to ~85% in Week 15. The Goal is marked as 95% with a red line across the top of the chart.

Annotation on Week 2 reads, "MDs and nurses reminded to give 'flu shots, clinic nurses screen patients, and attach 'flu shot packet to clinic chart."

Annotation on Week 8 reads, "Letter sent to CF families re 'flu epidemic asked to call office with 'flu shot info."

Annotation on Week 15 reads, "List generated of pts with unknown 'flu shot status, office staff call 60 families, get info on 43."

Slide 75

Trackable nutrition algorithm

A box at the top of the chart is labeled, "Classify every CF patient at every visit!" Arrows from this box point to two smaller boxes labeled "'At risk' or 'failure'" and "Acceptable." Arrows point to and from the "Acceptable" box to and from a box just below it labeled "See every 3 months"; arrows point to and from the "See every 3 months" box to and from a box just below it labeled "Re-eval every visit." An arrow points from the "Re-eval every visit" to "'At risk' or 'failure'."

Below the "'At risk' or 'failure'" box is a smaller box labeled "See every 6 wks." Two arrows point down from the "'At risk' or 'failure'" box to two boxes labeled "Eval calorie intake" and "Eval calorie absorption."

Two arrows point down from the "Eval calorie intake" box to two boxes labeled "Goal met" and "Goal not met." An arrow points down from the "Goal not met" box to a box labeled "Progressive intervention." There are two arrows leading from the "Progressive intervention" box; one points to a second box labeled "Goal met" and the other to a second "Goal not met" box. Another arrow points back up from the second "Goal not met" box to "Progressive intervention."

Two arrows point down from the "Eval calorie absorption" box to two boxes labeled "Goal met" and "Goal not met." An arrow points down from the "Goal not met" box to a box labeled "Progressive intervention/workup." There are two arrows leading from the "Progressive intervention/workup" box; one points to another box labeled "Goal met" and the other to another "Goal not met" box. Another arrow points back up from the second "Goal not met" box to "Progressive intervention/workup."

Slide 76

Improved Tracking: making changes to documentation (Docsite)

Photographs of two versions of a patient tracking sheet, labeled "Before" and "After." On the "Before" sheet, next to the question, "Flu Shot Given?" are the options Yes/No; on the "After" sheet, the options next to the question, "Flu Shot Given?" have been expanded to Yes/Deferred/Ineligible.

Slide 77

Impact of improvement efforts on nutritional status at UNC

Line graph labeled "Nutritional Classification" showing percentage at "Acceptable," "At risk," and "Failure" status for months from February 2003 to August 2004:

Time Period	Acceptable	At Risk	Failure
Feb '03	45.5	27.3	27.3
Mar '03	46.8	17.0	36.2
Apr '03	53.2	19.8	27.0
May '03	54.5	22.1	23.4
June '03	57.8	18.7	23.5
July '03	58.2	17.9	23.9
Aug '03	60.5	16.4	23.1
Sept '03	59.3	17.6	23.0
Oct '03	60.1	17.8	22.1
Nov '03	61.8	14.2	24.1
Dec '03	60.4	15.7	24.0
Jan '04	56.4	19.3	24.3
Feb '04	54.3	21.1	24.7
Mar '04	57.0	19.7	23.2
Apr '04	59.4	16.2	24.5
May '04	62.2	14.2	23.6
June '04	60.7	16.7	22.6
July '04	61.7	15.8	22.5
Aug '04	60.7	17.2	22.2

Slide 78

Comparing Research and Improvement Studies

	Type of Study
	Research	Improvement
Aim	New knowledge (What?)	Applying knowledge: Improve care in specific setting (How?)
Objective	Estimation of effect	Prediction of future
Outcomes	1-2 dimensions	Multiple dimensions
Sample size	Hypothesis testing	Prior knowledge
Approach to "bias" and variability	Background	Foreground
Confounding	Randomization Adjustment	Structured testing in subgroups
Selection	Restriction	NA
Chance	Random sampling	Control methods (SPC)
Time/secular trend	Comparison groups	Statistical process control Interrupted time series

Slide 79

Many studies of QI interventions do not show improvement... why?

Slide 80

Illustration of box labeled "Changes" plus box labeled "QI Method" equals (arrow points to) box labeled "Outcomes."

Slide 81

Variability in changes and methods

Changes:

• Evidence-based.
• Format.

Method:

• Format.
• Dose.
• Frequency.
• Combination.
• Timing.

Slide 82

Evaluation of QI Interventions

• Uncontrolled before/after.
• Time series (A baseline, B intervention):
  • Controlled time series (AA, AB).
  • Multiple time periods.
• Cluster randomized controlled trial.
• Factorial design.

Slide 83

Two photographs, one of a still lake, the other of river water rushing over rocks.

Slide 84

Many opportunities for learning and research

Slide 85

Next steps

Slide 86

If AHRQ offered a series of distance-learning activities on implementing and evaluating quality improvement strategies, would you be interested in participating?

Bar chart shows results of this question:

Directors: Yes, 88%; No, 13%
Participants: Yes, 82%; No, 18%

Slide 87

What would be helpful?

Conference calls:

• Powerpoint®.
• Pre-work: readings.
• Frequency.
• Archive powerpoint/audiotape.
• Series on specific topics.

Topics?

• QI strategies.
• Design of QI interventions.
• Evaluation.

Current as of July 2005

---

Internet Citation:

Quality and Quality Improvement. Text Version of a Slide Presentation. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/fund/training/lannontxt.htm

---