Slide 1
Evaluating California’s 
Web-based Training for 
Medical Assistants
Leslie Heyden, MA
California Department of Public Health
Immunization Branch
Information and Education Section
February 19, 2009

(CDPH logo-mask.)

Slide 2

Overview
•	The purpose of the CDPH IZ Branch evaluation is to assess the use and effectiveness of the first two immunization skills elearning lessons:

•	Preparing Vaccines
•	Administering Vaccines

(A Image showing a nurse preparing vaccines for a mother and child)

•	The elearning is custom developed, interactive web-based training.
•	The primary target audience is medical assistants in California VFC provider practices.
•	The lessons are available via http://www.eziz.org. 

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 3
eziz.org

•	The new California VFC program website
•	A one-stop shop for California VFC providers and their staffs
•	A repository of a wide variety of immunization skills training and VFC program resources, including:
•	VFC forms (e.g., vaccine order and recertification forms, vaccine temperature and usage logs, patient eligibility screening forms) 
•	Job aids, guidelines, posters, and up-to-the-minute news about vaccine availability 

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 4
eziz.org—home page
(Image of the http://www.eziz.org  home page. Displaying the naviagtion bar for the homepage and where to register for EZ-IZ Training.)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 5

Webinar learning objectives
By the end of this webinar you will be able to:

•	List the objectives of the evaluation plan
•	Explain why CDPH IZ branch selected the EZ-IZ elearning lessons to evaluate
•	Describe the methodology CDPH developed to evaluate the training
•	Indicators
•	Targets
•	Methods, including data sources
•	Describe lessons learned
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 6

Why evaluate EZ-IZ’ elearning?
•	Is IZ Branch’s first elearning project
•	Has potential for high impact re: ability to reach unlimited number of learners with limited resources
•	Primary tool needed to conduct evaluation (LMS) was already being created
•	IZ Branch wanted to compare the 
“reach” of the elearning to existing 
classroom training

(graphic of computer image displaying “www.eziz.org)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 7

Goal of activity to be evaluated
•	To educate MAs in California VFC provider practices about best practices for vaccine preparation and administration

(graphic image showing a how to give a IM injection to a Baby)
(Image displaying students listening to training in california)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 8

Evaluation objectives
1.	By September 5, 2008 the first two elearning lessons of the EZ-IZ curriculum (Preparing Vaccines and Administering Vaccines) will launch, and the learning management system will begin tracking usage and learner performance.

2.	By March 31, 2009, at least one person from 5% (approximately 180) of California’s active VFC provider sites (approximately 3,600) will have completed one of the elearning lessons (either Preparing Vaccines or Administering Vaccines) and have passed its posttest.
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 9

Evaluation objectives (cont’d)
3.	By September 5, 2009, at least one person from 20% (approximately 720) of active California VFC provider sites will have completed the Preparing Vaccines and Administering Vaccines elearning lessons and passed the posttest for each.
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 10

Evaluation objectives (cont’d)
4.	By November 30, 2008 all user feedback about the EZ-IZ website will have been gathered and analyzed, and recommendations made.
•	Recommendations concerning website usability will be forwarded to and resolved by the web developer and/or the web editor. 
•	Recommendations concerning the EZ-IZ Curriculum and the lessons in it will be forwarded to and resolved by the instructional designer. 
•	Recommendations about programming and usability issues within the elearning lessons will be forwarded to and resolved by the elearning vendor.
•	Recommendations concerning the learning management system (LMS) will be forwarded to and resolved by the LMS vendor.
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 11

Stakeholders
•	Management
•	John Talarico,  Interim CDPH IZ Branch Chief 
•	Tiffany Sutter,  Section Chief: Information & Education 
•	Natalie Nakahara, Project Manager and Creative Director
•	California VFC Program
•	Claudia Aguiluz, VFC Program Coordinator
•	VFC field staff and field services reps
•	VFC provider staff (particularly MAs who work in private provider practices) 
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 12

Stakeholders (cont’d)
•	Immunization (federal, state, and local)
•	CDC (NCIRD)
•	Steve Nickell,  Research Coordinator, IZ Branch
•	Sandra Jo Hammer, Nurse Consultant, IZ Branch
•	CDPH IZ Branch Evaluation team (Leslie Heyden, Natalie Nakahara, and Steve Nickell) 
•	IZ Coordinators in LHDs
•	Others
•	California Medical Assistants Association (CMAA) and California medical assistant training programs 
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 13

Table 3: Program Component/Activity Description 

Inputs: Funding, Staff, Time 

Activities:
Initial: Review available materials & choose content
Design training
 Employ vendor to program elearning and create learning management system 
 Make training available via website
 Promote training 

Subsequent:
Review of content by content experts
Beta test on target audience
Collect & assemble feedback
Decide on changes
Revise training 
Create a “one-stop shop” learning and resource portal for CA VFC providers, immunizers and other staff with clinical or administrative immunization-related responsibilities  

  Outputs:
(Live) online elearning lessons, with learning management system tracking

Outcomes:
Short term:
VFC providers will adapt to elearning as a training delivery  method
 Increased  skill and knowledge (pre- posttest comparison) about vaccination among medical assistants & other office staff
Steady increase in the number of MAs who take the elearning lessons

Long term:
Fewer real-time “person hours” required to train MAs
Fewer vaccine administration errors
Higher coverage rates
Fewer missed opportunities
Fewer VPDs
Increased provider/ user satisfaction

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 14

Primary data source—EZ-IZ LMS
5.	An LMS (Learning Management System) is software that automates the administration of training. 
6.	The LMS registers users, tracks courses in a curriculum, and records data from learners
7.	The LMS provides reports to management.
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 15

Evaluation question #1 
8.	Q: Did the Preparing Vaccines and Administering Vaccines elearning lessons launch by September 5, 2008 (target)? 
9.	Indicator and method
•	The Preparing Vaccines and Administering Vaccines elearning lessons launch: request IZ branch staff to access lessons and report any problems accessing the lessons 
10.	Data source: feedback (verbal and email) from IZ Branch staff (and confirmation from LMS vendor)
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 16

Evaluation question #2
11.	Q: Did the learning management system (LMS) begin tracking usage and learner performance data by September 5, 2008 (target)? 
12.	Indicators and methods
•	LMS is operational: verify with vendor that LMS is operational
•	LMS is gathering usage data: obtain registration and login data from LMS report 
•	LMS is gathering learner performance data: obtain pretest, lesson start, lesson completion, and posttest data from LMS report 
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 17

Data source—
registration
data

(graphic image displaying Registration Form, To list  Full name, email, Job title, and check the shipments or information you are responisble for handling)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 18

Learning History Page

(image displaying EZ-IZ learning history page.

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 19

Data source—LMS
(image of LMS sample data, displaying : pretest, lesson, post-test)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 20

Evaluation question #3
13.	Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?
14.	Target and indicator
•	5% of provider sites have at least one person who has completed one lesson and passed its posttest by March 31, 2009
15.	Method
•	Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
16.	Data source: LMS

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 21

Data source—pre- and posttests
(graphic image displaying  pre and post-test) 
(Question 1 to 10:
Which of the following is not a route of vaccine administration? 
IM
IT
SC
Nasal

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 22

Evaluation question #4
17.	Q: How many provider sites have at least one person who has completed one lesson and passed the posttest?
18.	Target and indicator
•	20% of provider sites have at least one person who has completed one lesson and passed its posttest by September 5, 2009 (one year after launch)
19.	Method
•	Generate LMS report of provider sites that shows lessons completed and posttest scores by provider site and user
20.	Data source: LMS

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 23

Evaluation question #5
•	Q: What feedback has been received from EZ-IZ users that could indicate potential obstacles to accessing and completing the elearning lessons? 
•	Targets and indicators
•	100% of  emails and 100% of all other feedback that could indicate technical problems with the website
•	100% of  emails and 100% of all other feedback that could indicate technical or any other problems with the lessons  
•	Method
•	Emails about technical problems and overall feedback about the EZIZ website will be automatically sent to a specific (ezig.org) MS Outlook folder; phone calls will be logged and categorized by November 30, 2008
•	Data source: emails submitted by users via links on eziz.org
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 24

Data source—feedback from users
(Graphic image displaying the following)
Tell us what you think.
Send us an email at eziz@cdph.ca.gov 

If you are having trouble using the EZ-IZ lessons please call 510-620-3738
(Monday-Thursday, 10am- 5pm)

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 25

Development of evaluation plan—Challenges
21.	Very few challenges. IZ Branch Research Coordinator collaborated on and provided very valuable input into plan.
22.	Due to limited resources, not able to build as robust an LMS as originally planned.
23.	Proposed revisions to methodology during plan development required consulting with LMS vendor who then had to modify LMS database or do additional programming.
(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 26

Lessons learned
•	Did not add question about MA’s experience; could not separate primary target audience from secondary target audience.
•	Learned that more data was needed than provided by LMS. Added question to QAR for field reps to administer.
•	The project team learned a lot during this entire pilot effort (including the evaluation plan) and is applying lessons learned during the current development of the next two lessons.

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 27

Added question to QAR
1. Have you or anyone else in your practice completed either of the interactive training lessons (Preparing Vaccines or Administering Vaccines) at EZIZ.ORG, the new VFC website?

YES  NO

2. If NO, what is the main reason?	
Haven’t had time  
Didn’t know about them  
 Couldn’t figure out how to do it  
Not allowed to use computer 
Office has no internet access              
No interest, don’t need to      
There were technical problems with the site   
Other? specify:______________						

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 28

Questions?

(California Department of Public Health, Immunization Branch, CDPH logo-mask.)

Slide 29
Leslie Heyden
Health Educator & Instructional Designer

Leslie.Heyden@cdph.ca.gov
(510) 620-3754

http://www.eziz.org 

This document can be found on the CDC website at:
http://www.cdc.gov/vaccines/programs/progeval/downloads/cali-eval-02-09.rtf