260 South Broad Street
Philadelphia, Pennsylvania 19102
Contact: Betty Morrell, MSE
Phone: (215) 985-2554
Key words: management information systems
(MIS); client records; case management
services
The purpose of this study was to determine
whether or not online availability of
data would improve the delivery of health
care services to clients.
METHODS
In October 1996, the Philadelphia Health
Management Corporation (PHMC) initiated
a prospective analysis of the benefit
of automation to social workers and case
managers working on HCH projects at PHMC.
For this study, major enhancements to
the social work features of the HCH Management
Information System (HCH-MIS) used by HCH
projects were completed in January 1997,
with additional modifications made through
August 1997.
The hypothesis of the PHMC study was that
the provision of interactive service plan
and progress note features in the HCHMIS
and the on-line availability of related
data would accomplish the following: 1)
help staff review and monitor clients_
level of functioning and 2) improve staff
documentation, thereby indirectly enhancing
the delivery of services to their clients.
The study design included both an experimental
(pilot) and a control group. The groups
were stratified so that the three HCH
programs and the four supervisors associated
with those programs were represented proportionately
in both groups. The number of staff that
could be assigned to the pilot group was
limited by the number of personal computers
available for staff use. Two of the five
pilot staff were assigned laptops which
could be used in shelters or networked
in the office when needed. However, the
laptop users were not required to do so
in the presence of a client, and usually
used the laptop as a desktop in the office.
Other pilot staff used desktop computers
permanently located in a home office,
rather than a shelter.
Initially, five HCH social workers or
case managers were randomly assigned to
the experimental (pilot) group. The remainder
of the HCH social workers and case managers
(11) were placed in the control group.
Various staff changes caused the composition
of both the pilot and control groups to
change somewhat throughout the data collection
period. Most of the staff changes affected
the control group, which usually included
thirteen staff collecting data during
a sampling period.
Each member of the control group submitted
paper contact forms, weekly service plans,
and related progress notes. The data on
these forms (excluding
progress notes) were entered into the
HCH-MIS by data entry staff. Beginning
in mid-February, staff in the pilot group
were required to enter the same information
(including progress notes)
directly into the HCH-MIS, producing computer-generated
paper forms as needed. Each member of
the pilot group had the on-line ability
to generate summary or status reports
about their specific caseload; the control
group could only request summary reports.
Supervisors could review and/or summarize
any data in the HCH data base, if the
information was collected by their staff
or related to their caseloads.
Automated intervention features were officially
ready for use on February 10, 1997, with
pilot group staff trained during the following
two weeks. Initial enhancements to the
HCH-MIS affected automated caseload reports,
progress notes, and service plan features
and formats. The automation of protocols
governing case reviews and access to automated
client and staff-specific data mirrored
the protocols governing access to paper-based
information. Additional features were
implemented at the request and specification
of pilot staff and supervisors throughout
the study period.
Four types of evaluation tools were used
in the study: 1) Activity Log, 2) HCH-MIS
reports, 3) Supervisor Interview, and
4) Staff Satisfaction Survey. The Activity
Log recorded how control and pilot staff
spent their time on their HCH project
work. Information was recorded by staff
for three sample weeks during the study
period.
The HCH-MIS Reports were produced using
current HCH data, which included all client
and staff-specific information entered
into the HCH-MIS since July 1996 either
directly or indirectly by HCH project
staff. The reports sampled the data base
three times in 1997 for the study. The
first 2-month sample (January and February)
established baseline data; two additional
2-month samples (April-May and August-September)
allowed patterns of change to be documented.
Certain reports were produced using data
for the entire data review period (January-September
1997) or other weekly sample periods.
The Supervisor Interview was used to guide
in-person discussion with a supervisor
from each program regarding their view
of the impact of automation on staff activities.
This structured interview, fielded in
January 1998, replaced the Supervisory
Case Files Record Review Form, an internal
audit tool previously used by the HCH
programs. A records review of a 20 percent
sample of case files randomly selected
from clients served by HCH staff since
July 1996 was completed in February 1997;
however a follow-up review of sample clients
served after February 1997 has not been
completed.
The Staff Satisfaction Survey measured
the self-reported impact of automation
on the pilot group in comparison to the
control group. It also provided information
about possible bias in the selection of
the pilot group. The survey was distributed
in August with responses received back
later in August and September.
Four categories of measurements, and seventeen
measures, were defined to evaluate the
impact of the automated intervention on
staff. The selection, summary and reporting
features of Microsoft (MS) Access and
Excel were the primary analysis tools
used in this study.
The analyses done generally compared pilot
and control group measures across selected
total or sample time periods for each
group as well as across the groups. Results
were derived from HCH-MIS reports, Activity
Logs, Satisfaction Surveys, or Supervisor
Interviews.
The use of automated features by the pilot
group had a positive impact on most of
the measures reported. In terms of availability
of data, the average time between the
service plan date and the entry date of
information for the pilot group was half
(17.6 days) the average time reported
for the control group (37.7 das). The
control groups higher average was most
likely due primarily to the time it took
data entry staff to key control group
service plans. This average also reflects
the added time it took control group staff
to completely rewrite a client service
plan, once a week. The average time between
the contact date and the entry date of
information for the pilot group was only
slightly less than the average time for
the control group. Review of supporting
data showed that pilot contacts were entered
both by data entry staff and by pilot
staff. Pilot staff probably directly entered
the one contact form required to establish
case management for a client so that they
could begin developing service plans,
and passed the other contact forms on
to data entry staff.
All supervisors interviewed agreed that
the quality of the service plans and progress
notes entered by the pilot group was much
improved. Plans and notes were clearer,
more complete, more consistent and more
accurate than those completed by the control
group. The significant difference in the
percentage of goals met by case managed
clients reflects the improved quality
of service plans, rather than differences
in staff performance. Only 14 percent
of specified goals were met by the clients
case managed by the control group, while
23 percent of specified goals were met
by clients case managed by the pilot group.
The ready availability of prior service
plans and the ease of reporting the current
status of goals helped improve the documentation
of client outcomes.
It was difficult to determine if the automated
changes were effective in changing the
pattern of work time. The overall distribution
of client work time and documentation
work time differed slightly, with the
pilot group spending more time with clients
(54.5 percent versus 44.0 percent) and
less time on documentation (27.6 percent
versus 34.1 percent) than the control
group. Not surprisingly, there was a change
in documentation work time between the
first (baseline) and second sample periods,
when the pilot staff switched from paper
to computer usage. Unexpectedly, the work
time patterns of the groups were not similar
in the baseline sample and were inconsistent
in the next two sample periods. Several
factors, such as special project activities
during the sample period, an inconsistent
definition of the _other_ category, and/or
inaccurate recollection of time spent
might have influenced this measurement.
Another indication of the effectiveness
of the automated changes was the change
in importance, by rank order, of project
data collection tools used by staff, as
indicated in the staff satisfaction survey.
While both groups ranked progress notes
as the most important data tool of the
five required forms, the groups ranked
the importance of the other forms in reverse
order. Pilot staff attached equal importance
to the social work contact form and the
service plan form, while control staff
considered the background questionnaire
(client assessment) and the service plan
summary form to be more important than
the contact form and the service plan
form.
From both an automated perspective and
a supervisory perspective, pilot and control
group staff continued to be equally compliant
in meeting contact standards and service
plan standards. Most differences favored
the pilot group, which could be attributed
to ease of documentation and correction
rather than better compliance. There were
two areas of apparent significant difference.
One measure specified that a greater percentage
of pilot group discharged clients (80
percent) compared to control group discharged
clients (59 percent) were discharged because
project goals had been met (i.e., services
were no longer needed). This difference
is mostly likely due to program differences.
A second measure regarding completeness
of files seemed more meaningful. All supervisors
reported that client paper files were
much more complete for pilot group clients
than for control group clients. The HCH-MIS
facilitated this improvement through onrequest
printing of contact forms and service
plans with progress notes attached, including
an indication that the information had
been approved by a supervisor.
Pilot staff training and frequent communication
were an important part of this study.
Although pilot staff had slightly more
self-reported computer knowledge and comfort,
most of that knowledge was related to
their use of word processors. One-on-one
(2-hour long) training sessions with pilot
staff and supervisors helped to ease concerns
about using a computer and the HCH-MIS
software. The project programmer also
responded quickly when staff had computer-related
problems or ideas. Monthly meetings with
the pilot group provided a forum for staff
to raise questions about MIS operations;
to request and specify changes or enhancements
to the software; and to agree on the priority
level of these changes.
This study clearly demonstrated that automation
can productively serve social workers
and their clients. The
automation of the service plans and progress
notes was the biggest time-saver for staff
and supervisors; it eliminated duplicate
entries and simplified corrections. In
addition, the ability to review automated
progress notes and weekly service plan
updates without referring to paper files
had an immediate positive impact on staff
supervision, from the perspective of both
pilot staff and their supervisors.
This study also resulted in the establishment
of measurable goals and objectives that
social workers and case managers could
set when working with a client. The automated
features in the HCH-MIS permitted staff
to tailor a service plan to a particular
client and to easily monitor and ocument
outcomes.
