|
BACKGROUND/RATIONALE:
Though education is one of VA's four missions, few studies have assessed the cost of VA's education program, or its impact on patient outcomes, professional education, or contribution to clinical productivity. VA spends $1billion annually to create 8,900 positions through which 30% of all residents in ACGME approved programs do clinical rotations. Although resident supervision is a key component to any residency training program, only a few studies have measured supervision or related it to patient health or educational outcomes. While ordinal measures for procedural care (surgery) are available, there are no scientifically validated measures to assess resident supervision in outpatient care.
OBJECTIVE(S):
Objective: This study will refine and test the Resident Supervision Index (RSI), a survey instrument designed to measure quantitatively the intensity of supervision residents receive in outpatient clinics.
METHODS:
Methods: The RSI was evaluated for content validity through an Expert Panel of 10 nationally selected, Graduate Medical Education experts from VA. The RSI was tested in general internal medicine and two subspecialty surgical clinics the Loma Linda VAMC. Patient care encounters were included if residents delivered care and the patient had a diagnosis of diabetes or depression. Based on these criteria, a VistA report was used to develop a cohort patient list from which 10 to 15 patients were randomly selected weekly during an 8-week period. The RSI was administered to the consenting resident and attending physicians after the supervisory encounter. Total reported minutes of attending supervision was re-tested for reliability within 24 hours. Concurrent validity compared attending physician versus resident responses. Agreement was based on kappa, intraclass correlation, correlation, and bias. Variation in supervision intensity was assessed by computing the percent of total time that the resident spent on behalf of the given patient when the attending physician was physically present in same room.
FINDINGS/RESULTS:
The Expert Panel judged RSI ver. 3.11 as having content validity to measure supervision intensity through minutes in supervision, resident case understanding and changes or confirmation of clinical findings and care. Feasibility: for residents: 75 (93.8%) of 80 residents consented and completed RSIs; for attending physicians: 37 (97%) of 38 physicians consented and completed RSI. Reliability: for residents: 125 of 148 RSI with retests agreed with kappa=.74, ICC=.95, r=.90, bias=.03 minutes of mean 7.7 minutes/encounter; for attending physicians: 132 of 143 captured retests, with kappa=1.00, ICC=.92, r=.85, and bias=.09 minutes of mean 8.3 minutes/encounter. Among 140 encounters where both resident and attending physician reported an RSI on a given supervision encounter, kappa=.72, ICC=.78, r=.64, bias=.22 minutes of mean 7.6 minutes/encounter. Intensity variability: Supervision intensity (percent of resident patient time that was directly supervised by the attending physician) averaged 28% over 110 study patients, with standard deviation of 14%, and a range per case from 2% to 75%. However, 50% of cases had intensity between 20% and 33%.
IMPACT:
The Resident Supervision Index has content validity, is feasible in clinical settings, is a reliable measure of supervision, and did distinguish residents facing different levels of supervision. The RSI provides an opportunity for researchers assessing graduate medical education to assess supervision intensity and its impact on patient (quality, health), trainee (satisfaction, learning), and system-level (costs, patient retention, attending physician burden) outcomes. RSI applications may also contribute to VA policy makers, planners, and administrators by offering insight into graduate medical education (GME) in VA and the role GME plays in caring for the present and future health needs of our veterans.
PUBLICATIONS:
None at this time.
DRA:
Health Services and Systems, Mental Illness
DRE:
Communication and Decision Making, Diagnosis and Prognosis, Quality of Care
Keywords:
Mental Health Care, Education Research, Trainees
MeSH Terms:
none
|
