|
Methodological Challenges
Responding to Reviewer Critiques: A Case Example
One way to learn about strategies for addressing methodological
challenges to conducting management research is to look at a specific case example.
This example provides some insight into specific challenges inherent in study design
and it offers a glimpse into the thinking of scientific review committee members. The
initial proposal was rejected pending modification. The revised proposal, changed
along the lines described here, was accepted.
Martin Charns and Gary Young of the MDRC submitted this study
as an Investigator-Initiated Research project in 1995. The study was designed to
investigate the implementation of quality improvement (QI) practices in Veterans
hospitals. Most VA medical centers had implemented QI or were in the process in
1995. However, there appeared to be much variation regarding the organization and
management of these intiatives. The study was designed to capitalize on this
variation to examine the relationship between the degree to which VA medical centers
have implemented QI and various influence factors. The study used an observation
research design with statistical controls to account for various confounding factors.
The complete abstract is available here.
Below are five critical comments from reviewers about the original proposal (in italics).
For each comment, the original design element being criticized is summarized in parentheses,
followed by the response by Charns and Young.
1. The sampling frame for the site visits does not seem
appropriate for developing a more general understanding of the relationships among the
variables.
(The original study design was to examine the five highest and five
lowest scoring VA hospitals on quality improvement implementation.)
The study hospitals will be stratified into high, medium and low
Quality Improvement (QI) implementation sets and six facilities will be randomly selected
from each strata. Additional criteria may be used if the data indicate that certain
characteristics or implementation design features have a substantial impact on QI
implementation. We will visit at least three high- scoring hospitals that have
experience conducting QI projects.
2. The proposal lacks a discussion of the potential bias associated with the
timing of the adoption of QI techniques by hospitals-e.g., innovative high quality
hospitals might be the first to adopt QI techniques. As a consequence the analysis
with performance measures only after QI implementation might erroneously attribute
continued high quality to QI implementation.
(The original design included a cross-sectional approach assessing
the relation between quality and culture.)
A third performance measure, for which data is available from before
implementation of QI, will be added. We will also add longitudinal data on all measures
and will examine the relationship between degree of implementation and hospital
performance using percentage change in performance on each measure. Thirdly, we
will use several proxy measures to examine bias associated with historical quality.
3. The proposal does not discuss the role unions or
mid-level managers will play in the implementation of QI initiatives.
(Originally there was only one item included to assess this
concept.)
The Medical Center Questionnaire has been revised to include
several questions to gather information from the medical center director about the
degree or resistance or cooperation the hospital has had from its union(s) and QI
implementation. In addition, we have revised our site visit format to include
interviews with officials from the unions at each hospital selected for a site visit.
Mid-level managers will be included in the pool of employees from
which we will select to complete the scales. We have also revised our site visit
format to include employee group interviews that will include mid-level managers.
4. The proposal seems limited by assessing many variables that may be immutable.
(Several of the variables, such as facility size and culture were considered difficult
to manipulate and evaluate scores.)
We recognize that a few of the study variables are immutable,
including urban/rural status, prior innovation experience and hospital mission. If
medical center directors are aware that their facility faces a major barrier to
quality improvement (e.g., the facility has little or no prior innovation experience
and our study demonstrates that this variable plays an important role in facilitating
QI), they may be able to anticipate specific implementation problems and adopt certain
strategies to improve the chances of a successful QI initiative.
5. The strategies for data collection may intimidate
employees from responding as candidly as they might.
(The description in the original proposal did not include enough
details about data collection procedures.)
We have given careful thought to whether the employee questionnaires
should be distributed to employees in a group setting or through the mail. We have
chosen a group setting approach because, based on the experience of several members
of the research team, we believe that it will lead to a substantially better response
rate. The potential drawback of a group setting approach is that it may intimidate
some employees because of concerns about confidentiality. In anticipation of this
concern, we will distribute a letter to each selected employee that will (1) ask
for their participation, (2) outline the primary study objectives, and (3) describe
the procedures for distributing and collecting the questionnaires. We will also assure
employees that all responses will be strictly confidential. We will make clear to
employees that service chiefs and department managers will not be involved in
distributing or collecting the questionnaire.
top | next
|
