FTS-CDC-NCIRD

Moderator: Dale Babcock
February 19, 2009
1:00 pm CT

Coordinator:	Welcome and thank you for standing by. At this time, participants are in a listen-only mode. During the question and answer session, please press star 1 on your touchtone phone.

	Today’s conference is being recorded. If anyone has objections, you may disconnect at this time. Now I’d like to go ahead and turn today’s call over to (Sashita Loreck). Ma’am, you may begin.

(Sashita Loreck):	Okay, thanks so much. Thanks everyone for joining us for the Webinar and giving your colleagues an opportunity to share their evaluation plans and their experiences related to developing the plan with you this afternoon - it’s afternoon on the East Coast, I guess.

	We picked the plans you’ll be hearing about today in the next month during the next Webinar because they included a good range of process and outcome questions. The activities being evaluated were in varying stages of implementation. And we felt they were diverse enough to be of interest to many grantees out there.

	So we hope that you find these presentation useful and helpful and interesting to hear about with some of your counterparts we’re working on and we hope the presentations will generate some discussion and thinking about future evaluations at your program.

	So just some logistics; we’re scheduled for about an hour and a half today but we may only need about an hour depending on how long our discussion and Q&A goes. And the way we’ll do it is first, we’ll have Leslie Hayden from California present and then we’ll follow that up with a little discussion and Q&A. And then we’ll repeat the same steps with Shana Johnny from Washington.

	And Leslie, if you could just - when you start, just introduce yourself and your role in the evaluation, that would be great. And FYI, hopefully you guys have seen this in the email but the next Webinar is planned for March 12 and we’ll be sending out some details about how to log in shortly.

	Okay, so I think we’re ready for Leslie to take it away.

Leslie Hayden:	Thank you, (Sashita). And thank you very much for giving California the opportunity to present our plan and to tell you about what we’ve been doing. I am an instructional designer in the information and education section and I am the evaluation lead for this project. So we may as well just move forward into the presentation, so next slide.

	Let me give you a little bit of overview about the project. The purpose of this evaluation is to give the immunization branch here in California an opportunity to assess the effectiveness of our very first two eLearning lessons in immunization skills. And those lessons are preparing vaccines and administering vaccines.

	And that little graphic you see there is a screen shot from the introduction of one of the eLearning lessons. In eLearning, we develop this custom-developed, this interactive Web-based training. The primary target audience for the training is medical assistants and California VFC provider practices.

	And the lessons are available at www.eziz.org and you’ll hear me mention that a number of times through the presentation. And I encourage you all to go there and check the lessons out yourself. Next slide.

	So the lessons are housed on www.eziz.org which is the new California VFC program Web site. And we have developed it so that it would be a one-stop shop for California VFC providers and their staff. Obviously, anything we develop for California is available to anyone and we encourage anyone to avail themselves of any of the resources on this site.

	The site contains a huge repository of a variety of immunization skills training, videos, VFC program sources, there’s all kinds of VFC forms, some are California-specific, some can used by anyone. This year, the site was used for VFC certification and most VFC providers did recertify onsite. There’s also a lot of job aids, guidelines, posters and then up-to-the-minute news about vaccine availability on the site. Next slide.

	So that’s the www.eziz.org home page. As you see, on the right column there’s a lot of the resources that are available to California and out-of-state providers, more specifically California. If you look at the red arrows pointing, you will see that that is pointing to the www.eziz.org training which is where the two lessons that we’re evaluating are housed and that’s how they’re delivered.

	So if you go there and you want to check out the training, you can just log in there where it says User Name and Password and it’ll walk you through the process if you haven’t been there before. Next slide.

	Okay, we’ll get back to the Web site but I just wanted to give you an overview of what we’ll be doing during this Webinar. By the end of this Webinar, you’ll be able to list the objectives of the California plan, explain why the branch decided to use those two EZIZ lessons to evaluate, describe the methodology that we use which I assume you all use to develop your evaluations. And that would include the indicators, the targets and the methods, including the data sources. And then you would be able to describe some of the lessons we learned. Next slide.

	Okay, why did the branch choose to evaluate the EZIZ eLearning? Well this is the branch’s very first foray into eLearning. So it was brand new for everyone here but eLearning is the way to go and we began this before the economy tanked, so to speak. So eLearning is a way to - as I say in the second bullet point, it has the potential for reaching a huge number of people with very limited resources.

	So it is the way of the future so we wanted to test it out as a delivery - as a training delivery method also. The primary tool we needed to conduct the evaluation - which is that tool as a learning management system is already being created for us so we would already have had the data. And I’ll talk a minute about the learning management system.

	And at the branch, we had existing classroom training on a lot of the immunization school for medical assistance. The classroom training is labor intensive, as you know, and it requires an instructor to deliver the training each time. And we wanted to compare the reach of this training to the numbers of people that are being trained by conventional classroom training in the same content areas. Next slide.

	So the goal of our evaluation is to evaluate the two lessons that are geared to MAs in California to educate them in VFC provider practices about best practices for vaccine preparation and administration. You’ll see a screen shot on the left is a screen from one - is from the administering vaccines lesson and that‘s a practice exercise on identifying the right site for the injection.

	And the screen - the photograph on the right is from one of our pilot groups when we were piloting the lessons before we finalized them. So we were piloting them live in VFC provider offices with real medical assistants. Next slide.

	Okay, now we’ll get to the evaluation objectives. We have four. So we wanted to make sure that we launched by this past September, by September 5 and that our learning management system would be up and running and would begin gathering usage and learner performance data.

	The second object is by this March, our goal is to have one person from 5% of California’s active VFC provider locations who have completed the lesson and have passed a post test. Now that would just be one of the lessons. And that’s by this coming - by next month. Next slide.

	The third objective is we want to up the ante here so by this coming September, which will be a year after launch, we would like to have one person from 20% of active California VFC provider sites having completed both lessons, preparing vaccines and administering vaccines and have passed both post tests. Next slide.

	By November of this past year, we had intended to gather all feedback and route the feedback to the people who are handling the content areas for that feedback. We are now on an ongoing basis continuing to gather feedback and channel that feedback to the appropriate person. In some cases, it means that feedback has to be communicated from us to the eLearning vendor who helped us develop the training and they need to do some reprogramming. Or it has to do with the vendor who built the learning management system for us and needs to change something about the database work.

	So any changes, any feedback can branch out to any number of people for a fix. Next slide.

	Okay, so there’s a lot of stakeholders in this project. And I divided them into what I thought were logical categories. Certainly the management here at the IZ branch wants to know how effective this program is since it’s, at this point, a pilot program because we’ve never attempted anything like this before.

	So our branch chief and our section chief, (Natalie Nocahara) is our project manager. So she has a huge stake in this. Target audience is the VFC providers so the VFC program coordinator here has certainly a huge vested interest in this training, as does the VFC field staff and field reps. They are the ones out in the trenches and visiting the VFC providers on an ongoing basis. And they are the ones who see the problems and see the needs that our training has been developed to address.

	And the third bullet point there is the actual VFC provider staff, most of whom are MAs and work in private provider practices. Next slide.

	There are also a group of stakeholders such as all of you and you at CDC, we here at the state in California. In California, all the immunization coordinators and all the local health departments because they also see a lot of the problems and work with a lot of the community health centers.

	And there are a few other stakeholders as well. For example, the California Medical Assistance Association and training programs in medical assistants tend to become certified medical assistants. We work with them during the development of the training and even before that during the needs assessment to get a sense of their curricula to see if what we’re teaching for medical assistants, if it’s complementary to it, if it’s - we want to make sure it wasn’t redundant. And in many cases, we are supplementing skills training for medical assistants that doesn’t exist in existing medical assistant curricula.

	Next slide. Okay, so here is the logic model not in as beautiful form as some logic models are. But it has the basic content. And in the Activities column, you’ll see what went before we reach the stage where we were developing the evaluation plan for CDC. We had to design the training, we had to plug through some big vendor selection processes to get our LMX vendor as well as our eLearning vendor. We did a lot of pilot testing. I developed pre-post test and had to validate those.

	The goal is to create, as you saw and as I said, a one-stop shop for training and resources for California VFC providers. And we want to make sure that the skill and knowledge gained can be documented and we’re doing that with pre-imposed tests. And ultimately we’d like to see a steady increase in the number of MAs who goes through this training.

	Long term, we hope to see a reduction in person hours required to train MAs because we’ll be able to take a lot of the training online and that wouldn’t require a person to administer that. One of the results of doing - of preparing and administering vaccine training is we would hope there would be fewer vaccine administration errors. And there would be fewer missed opportunities to administer immunization through the VFC program.

	Ultimately, fewer vaccine preventable diseases and that the providers and the users of the training would have increased satisfaction with the training, and ultimately with the way the training has increased their job skills. Next slide.

	Okay, I’ve mentioned the learning management system. A learning management system is software that automates the administration of training, it registers the users, it tracks the courses they take, it records profiles about them. And then all that information is captured in the database and I can dice and slice it pretty much any way I want to. It’s kind of - it is the heart, it is the nucleus of the evaluation for this training because it captures all the data. Next slide.

	So we’ll be getting back to the learning management system because that is our primary data source. But now I’ll walk you through the evaluation questions and they relate to the objectives obviously. So I’ll go through them one by one.

	So we want to know, did the two lessons launched by the target date, which was September 5? Now how do we know that? Well, we had a number of (unintelligible) make sure that they could access the lessons. One would think that oh, we’d have to do without the LMS vendor if they launched. But we wanted to make sure, just because of the way that technical glitches do happen and sometimes you don’t know the extent of them. But not only the vendor said they launched but we knew the people could actually access and see the lessons. Next slide.

	Our second evaluation question was, is the learning management system tracking the learner usage and tracking their performance and is that in place by September 5 also? So we had three indicators on that. We needed to know that the LMS is operational and that it’s gathering data and we were able to confirm with the vendor that it was, that it is gathering usage data that we know every time someone registers, what information they registered.

	And then the last one, that it’s measuring - that it’s gathering data about learner performance, meaning that it’s gathering pre and post-test data as well as data that shows that the lesson was completed. Next slide

	So the next slide shows you that a screen shot of part of the registration process, this is where the data comes from, that you register to take one of the lessons, the user fills out a registration form and this is part of it. And all that data goes into the learning management system.

	So just kind of FYI, at the bottom of your screen there, we wanted to verify the job responsibilities that the users were responsible for because this is helping us make sure that we are on target of development for subsequent lessons, that we are hitting the right target audience with the right training. Next screen.

	What you’re looking at now is the Learning History page on EZIZ. So once you register, you will be automatically routed to this screen. This screen is driven by the LMS. So for example, for this user, it shows that they want to - they’re in the process of taking the vaccine administration lesson that they have completed the pre-test which is the pre-lesson check, that they have completed - I’m sorry - that’s not preparing vaccines lesson. That they have completed the lesson, that they have passed the post-test and if they choose, they are able to print the Certificate of Completion.

	They are in process on the administering vaccines list and they completed the pre-test, they completed the post-test, they completed the lesson but they have not either completed the post-test or they did not pass the post-test. So all this data goes into the learning management system and this is what the learner sees.

	And every time that learner logs in, they’ll be taken back to this screen so if they quit during the middle of a lesson, they can go back to the exact lesson that they were at if they, you know, if they were doing it during lunch and had to go back to work. Next screen.

	So this is the end of it that we see. This is a cutaway of the learning management system data and you can see the pre-test data, lesson data, lesson completion data and post-test data. And we can slice and dice this data pretty much anyway we want to. Next slide.

	So then to Evaluation Question 3, how many provider sites have completed at least one lesson and passed that post-test. Well, you’ve seen the LMS and you know how that data is gathered. So to meet our target for next month, we want 5% of provider sites to have at least one person has completed one lesson and passed it’s post-test. And the way we know that is by the screen that you just saw in the learning management system. Next slide.

	One of the measurements that we use, as I said, was pre and post-test data. And that’s just a screen shot from one of the post-tests and that’s on vaccine administration routes. And that’s just a question from that test. And that’s what it looks to the user and then the data is captured and imported into the learning management system. Next screen.

	Our Evaluation Question 4, now as I said, we want to up the ante and we would like by the end of September - by September 5, which would be one year after launch, did 20% of VFC provider sites have at least one person who has completed a lesson and passed the post-test. And that data will come from the learning management system. Next slide. Next slide. Thanks.

	I had mentioned that we wanted to get all the data we could from users. We had no idea going out what kind of problems people may encounter, there are different technical issues, there are different issues that come up related to people’s personal computers.

	So on the Web site, we have two different ways for people to communicate with us about technical help or just to give us feedback or whatever. And those are the two ways that we gather that kind of evaluation data from the Web site. Next slide.

	So now that you have the sense of what our plan was, I’ll talk about challenges very briefly. Actually, we had very few challenges developing this plan. I was able to work with the IZ branch research coordinator who is a full-time researcher here and he provided some very, very good input and suggestions into the development of the plan.

	On the down side, as in public health, everybody experiences we had limited resources and we were not able to build as robust a learning management system as originally planned. So we had to scale down what we eventually got. And in an ideal world, I would be able to get every little bit of data that I ever dreamed of but I’m getting enough. But that was kind of a compromise.

	And when we wanted any - while I was working on the plan, any changes I wanted to the methodology might need some data that the LMS was not set up to gather or set up to gather it in a way that I needed to have it to report in a way that I wanted to do in the plan. So there was a constant back and forth with the LMS vendor to revise and modify the learning management system so that I could get the data I needed for the evaluation. Next slide.

	Lessons learned - and we are still learning them. One thing we did not add to the log on screen was a question about how long the MA has been an MA or has been employed by the practice. So what I’m not able to get is a - we are able to see who is an MA, we are not able to see how long they have been on the job or how much experience they have so we can’t correlate pre-imposed test scores to tenure on the job. It would have been nice but it looks like we may not be able to get that.

	We also wanted some data outside the LMS. We wanted to make sure that we were getting everything we could. So we added a question to the QAR that the field reps administer when they do their site visits and their annual quality assurance visits to make sure that MAs on those sites were aware of the training and are they taking it. So I’ll show you what we added to the QAR in a second.

	But I just want to say that we learned so much. And there’s so much that we have learned that we are applying to the development of the next two lessons which are storing vaccines and monitoring vaccine temperatures. Those are in development now and I think it’s really great that we did the preparing vaccines and administering vaccines first because as you know, the who vaccine storage and handling issue is so huge and fraught with opinions and different approaches that I’m very glad that it’s coming second because we learned a whole lot. Next slide.

	That’s the question that we had added to the QAR to find out when the - so the field reps can ask MAs or whoever they do their QA visits with, if anyone in practice has taken any of the lessons. And if not, is it because they don’t know about them or what was the reason.

	So obviously at that point, having asked the question, the MA has already heard that there is training that is an opportunity for the rep to tell the - that practice a little bit about the training. Next slide.

	So that’s all I have to say. Does anyone have any questions about this?

Coordinator:	Thank you. We will now begin the question and answer session. If you would like to ask a question over the phone, please press star 1. Please unmute your phone and record your name clearly when prompted. Your name is required to introduce your question.

	Again, if you would like to ask a question over the phone, please press star 1. Please unmute your phone and record your name. To withdraw your request, please press star 2. One moment please for the first question.

	Your first question come from (Kathryn Harrelson). Your line is open.

(Kathryn Harrelson):	Hi, Leslie, this is (Kathryn Harrelson) from Tennessee. And I just wanted to let you know that the program, I thought it was fantastic. I learned about it through (Edgar Innocot) in conversations with him. And you guys did an awesome job with it.

Leslie Hayden:	Oh, thank you so much. Well, be sure to tell everybody in your branch.

(Kathryn Harrelson):	And we’ve even recommended some of our providers who are in tertiary education that they have their staff use it. So thank you very much for all your hard work.

Leslie Hayden:	Oh, thank you very much. We’re getting great feedback about it from all over the country. There’s a medical center in New Hampshire that’s using it in their curriculum. There’s a nursing school here in the bay area that’s incorporated it into their curriculum. So keep those cards and letters coming, folks.

Coordinator:	The next question comes from (Audrey Muniz). Your line is open.

(Audrey Muniz):	Hi, this is (Audrey Muniz). I do (unintelligible), I do education for MAs, LENs, RNs and usually I have a very limited time to do this. So this is a wonderful program, kudos to you for that. But I’m wondering, as far as this - the MA training goes, how much time involved for students did it usually take to complete those courses. Do you have any data on that?

Leslie Hayden:	Yes. The lessons start to finish, the preparing lesson, pre-test and post-test included can be completed in 15 to 20 minutes. The administering vaccines lesson, during pilot, I’ve seen them do it in as little as 11 minutes. So it’s pretty efficient. We wanted to - our target was 15 minutes and we know a lot of these folks have to do it either after work or during their lunch hour or during a break or from home or whatever.

	So our goal is not to exceed 15 minutes. Where there may seem like extra time is the combined time to register plus that first lesson, preparing vaccines because that’s a longer lesson. But once they’re in there, it shouldn’t take them more than 15 minutes or so to complete either lesson.

(Audrey Muniz):	So for the MA student, like with your, like, you know, your MA colleges and things like that, did they do that in the classroom? Is that when they completed that training?

Leslie Hayden:	We did it in a - we used them as one of our pilot groups. So we took the training to one of their conferences and afterwards we piloted the training with them. But they now have full access to use it in their curriculum and I’m going to soon follow up with them to find out exactly how they’re doing that. But they were all very interested in using it.

(Audrey Muniz):	Wonderful. Thank you.

Leslie Hayden:	Thank you.

Coordinator:	Right now, I show no further questions. As a reminder, if you would like to ask a question over the phone, please press star 1. Please unmute your phone and record your name.

(Pascel Wordley):	Hi, Leslie, this is (Pascel Wordley) at CDC.

Leslie Hayden:	Hi.

(Pascel Wordley):	Thanks for a great presentation. It’s nice to see, you know, both the program and the evaluation piece. And I was thinking it’ll be nice, you know, down the road since this is a new activity and this point you’re looking at some of the immediate outcomes. And it’d be nice to figure out ways to look at some of the longer term outcomes too to document the validity of this effort and all that.

Leslie Hayden:	Right ,for sure. And we see a very big opportunity for that kind of documenting of our efforts in the storage and handling areas.

(Pascel Wordley):	Yeah, so we’ll look forward to, you know, brainstorming about that with you if you want.

Leslie Hayden:	Sure, absolutely. Thank you.

Coordinator:	You do have another question over the phone. Just a moment.

(Martha Ridgker):	Leslie, this is (Martha Ridgker) with Kansas Immunization Program.

Leslie Hayden:	Hi, (Martha).

(Martha Ridgker):	How are you?

Leslie Hayden:	Good, how are you?

(Martha Ridgker):	Great. It sounds like a wonderful program. For new graduates, say they’re LPNs or RNs, not a lot of the nursing schools teach immunizations or good administration techniques. So are - have you set your program up so that if they take the classes they can get continuing education for them?

Leslie Hayden:	No, it is not set up that way. We have done continuing ed in the past and it’s a very labor intensive process.

(Martha Ridgker):	I see.

Leslie Hayden:	So we have not looked at that for this program. Whoever their regulatory organization is handle that?

(Martha Ridgker):	Yes, they could.

Leslie Hayden:	And we would be happy to work with them to help them make that happen. But in terms of us doing it, in terms of lessons learned on other programs, it takes a lot of resources to that.

(Martha Ridgker):	Okay, thank you.

Leslie Hayden:	So we’d be happy to work with anyone who would like to do that certainly.

(Martha Ridgker):	All right, thanks.

Coordinator:	Your next question comes from (Jean Roberts). Your line is open.

(Jean Roberts):	Hi, this is (Jean Roberts) (unintelligible) again, this is a great presentation. I got to it a little bit late. I just want to know how exactly can one get in contact with you instead of (unintelligible) could you have us work with some of our local health departments.

Leslie Hayden:	Okay, great, (Jean). They just put the slide up with my contact information on it. So feel free to email me or give me a call and, you know, go back to the www.eziz.org and see what else there you might find interesting. But please send me an email or anyone or give me a call and I’d be happy to talk with you about it.

(Jean Roberts):	Very good. Thank you.

Leslie Hayden:	Sure, my pleasure.

Coordinator:	And right now I show no further questions.

(Sashita Loreck):	Okay, Leslie, this is (Sashita). I also want to say thanks so much for the great presentation.

Leslie Hayden:	Thank you very much for giving me the opportunity, (Sashita) and I hope it was helpful.

(Sashita Loreck):	I think it was very helpful. But yeah, thanks again for putting it together on such short notice for us.

Leslie Hayden:	My pleasure.

(Sashita Loreck):	All right. So we’re going to move on now to Shana Johnny. And as I mentioned, she’s from Washington. And Shana, if you could just say a couple of things about how you were involved with the evaluation, that would be great. And yeah, take it away.

Shana Johnny:	Okay, thank you, (Sashita) and (Pascel). And it’s - my name is Shana Johnny, I am with the Washington State Department of Health. I work in the immunization program top profile. And I am one of the public health nurse consultants here. And I appreciate you having me on the call. It is an honor to be here to talk with you today about perinatal (unintelligible) program evaluation. And may I have the next slide?

	I thought that I would start with giving you an overview of our program, Perinatal (unintelligible) Prevention because this is a backdrop then to what we look at as far as measuring and evaluating and what you’ll be hearing about today.

	So in 1990, plans were provided to us through CDC and Washington State provides - started receiving these funds and providing case management in conjunction with our local house jurisdictions for infants born to (HBSAV) positive moms. And we started this in March of 1991.

	So the case management process is very well established here. We do contract with 35 local house jurisdictions to provide services for perinatal (Hep B) prevention. And these components include, under the vaccines for children’s programs, state supplied Hepatitis B vaccine, laboratory testing. Also information, education, consultation and technical assistance. And we again enlist and our local house jurisdictions in providing this information and education out to providers.

	Next slide please. Well why did we select this evaluation activity? Because it’s one of many indicators that we continually evaluate in this program. But it was a CDC program requirement and certainly to evaluate the completion of post-vaccination serology testing for infants.

	Although we have made some improvements on and off over the years, it still remains a gap. And this is one area that we have pinpointed that really needs our attention. So we would like to identify the barriers that may exist to post-vaccination serology testing. We want to increase our understanding of barriers our providers are facing in practice and we want to develop strategies to increase post-vaccination testing.

	And this just gives you evidence here, data-wise, that it pinpoints a percent of infants broken down by year who have been tested. We have a denominator, a different denominator for each year. So in 2002, out of 279 infants born to (HPV positive) pregnant women, we have 158 of those tested. So that’s about a 57% that were tested.

	In 2003, we have 344 infants reported with 203 tested - that’s 59%. And in 2004, 321 infants, 194 of those tested which gives us 60%. In 2005, we had 317 reported and 192 tested for 61%. And in 2006, 377 reported in our state with 217 tested with a 58%. So you can see that we do have an area that needs improvement. Next slide.

	Well, our primary aim here is certainly to increase awareness, knowledge and practice with perinatal Hepatitis B post vaccination serology testing. And the evaluation goal will be directing all my activities listed later in the PowerPoint with the broader goal certainly to increase our rate of post-vaccination serology testing.

	The next slide shows heavy involvement with our stakeholders and partners. We have both internal stakeholders as well as external stakeholders. And a key factor certainly in excess of any plan like this is our ability to identify, understand and engage key stakeholders and certainly to get their buy in to help reach our goals and objectives.

	And so you will see a few of our stakeholders. We have a non-profit agency that we contract with called Within Reach. It’s our API taskforce and they are involved with this work as well as our local health jurisdictions and we have epidemiologist, both communicable disease as well as internal and other healthcare professionals and internal staff. Next slide.

	With these objectives, I have both process and outcome objectives. And our work certainly is around these objectives, developing our strategies for improving our rates of post-vaccination serology. So our first objective is we want to determine the number of providers identified by local health jurisdictions as caring for infants of (HBSAV) positive moms and the number of providers who do have standing orders in place for testing.

	We want to survey the providers, identified as caring for infants to determine the percent of those providers who are aware of the value of early detection and the criticality of post-serology testing for infants born to (HBSAV) positive moms.

	We also want to look at - we want them to be educated on the value of early detection as well. And continuing, we want to increase the number of providers identified as caring for infants of positive moms who are aware of the value of early detection and the criticality of post-serology testing. We want to increase this by 5% and we want to also increase education on the value of early detection by 5%.

	We also want the percent of providers who have written standard orders for post-testing, we want to increase those by 10%. And the rate for a three-year period from December 2006 to December of 2009 for post-testing for infants born to positive moms will increase from 58 to 70%. And that comes out to about 42 infants more per year. Next slide.

	Please note that if you put your slide show here on View 100%, you might be able to see this a little clearer. And so this is not self-evident but this is our logic model. And it’s very busy but it does explain a process that we use in, you know, conducting our activities, including resources - that’s under the Input column.

	The activities that we both currently do in perinatal Hep B prevention and the ones that we will be doing with these activities, that’s under Subsequent. And because of the activities, we’re able to have the outputs. And so we would like to see certainly health care providers have standing orders that local health jurisdictions utilize our module which is our surveillance system and so forth.

	We obviously want to impact both knowledge and behaviors of providers since this is directed toward providers. And certainly long term we want to be able to, you know, get those infants post-tested because without that, we will never know if they’re positive or negative. And that feeds into a whole other algorithm. And we want to certainly track the efficacy of the vaccination and decrease vaccine preventable disease. Next slide.

	Now our evaluation activities include identifying local health jurisdictions and geographical areas where our testing is low. We have already done this and they are part of our stakeholder group. And we will also identify providers where post-vaccination serology is low.

	We will also plan outreach to providers to assess awareness of why infants born to positive moms are not getting post-tested. We will be doing this with a pre-test as a survey and we will have basic questions on it, perhaps six or seven, simple survey. It will be directed toward pediatricians. And this will be really determined by my stakeholder group.

	So we’re not really into this activity right now. But I think we have a lot of discussion around it for both the pre-test and eventually then the post-test that we’ll measure whether our intervention is working. And we will certainly use findings to develop strategies to increase our testing rates. Next page.

	Our short-term outcome. We certainly want our providers to be more aware of the importance of our testing. We want our providers educated on the value of early detection. We want to increase the number of providers who are conducting the testing and we want to increase the number of LHJs providing that consultation piece with providers.

	And this is work that they’re already doing but we want to formalize it or attach it onto other visits that they might be doing - or working with providers, whether it’s (unintelligible) or other site visits. We think that this can be piggybacked onto those visits. Next page.

	I was asked to include data collection and analysis in this format because it’s in the CDC format. And for the Evaluation Question 1, we have framed to what degree did the number of providers who are aware of the valued early detection and post-testing for infants born to mothers increase from March 2009 to December of 2009?

	Our target once again here is 5% increase from those times. The LHJs will be collecting the data from providers. They will in turn communicate that back to us at the Department of Health. The method of analysis will basically be comparing that number of providers who answer the survey positively in January and March. And we will compare that to the number who answer positively in December 2009 and then calculate that percentage change. Next slide.

	The next question would be, to what degree did the number of providers who had been educated on the value of early detection and the criticality of post-vaccine testing for infants born to positive moms increase between March 2009 and December 2009?

	Again, our target is the same; it’s 5%. Our methods are the same for collecting that data. We will be comparing the number of providers who answer the survey and then following up again. And that information will be communicated to us and we will compare the number of providers who answer that positively and then again calculate the percentage change. Next slide.

	And we have the next framed as to what degree is the number providers have written standing orders for post-vaccine serology testing for infants born to positive moms? So that question will be included certainly in the pre-survey and the post-survey. And we really want to increase the providers who actually have standing orders by 10%. And we will be looking again, calculating the percentage of change between the two surveys. Next slide.

	This is framed to what degree did the rate of testing for infants born to moms increase from December of 2006 to December of 2009? Now those dates may look a little funny to those who are not in perinatal Hep B prevention. But the date of which we gather, analyze and report data to CDC is not the same as the actual data of birth.

	So in 2006, we followed those infants and report the post-vaccination serology actually in 2008. So in - for 2007 births, we’ll be reporting those in 2009 because it takes up to two years to - possibly for an infant to receive prophylaxis, the series, the post-vaccination. And possibly, depending on the test result, then we might have to repeat the series and repeat the post-vaccination testing. Next slide.

	Our long term outcomes. We certainly want to see an increase percentage of infants born to (HBSAV) positive moms who receive post-vaccination testing. We want to see an increased ability to track the efficacy of Hepatitis B vaccine through post-testing - or post-vaccine serology testing. And we want to see a decrease in Hepatitis B disease in infants born to (HBSAV) positive moms. These are all long term outcomes. Next slide.

	The plan itself, as far as challenges, the plan itself - the challenges weren’t as great. I really see the challenges more with the intervention piece. And you know, working with the stakeholders because out of the stakeholders and partnership group, my methodology could change slightly. And you know, we certainly want to maintain the rapport with local health jurisdictions and all the other stakeholders. We want to come to a consensus on the plan.

	And certainly this does tie up some resources and fitting it into the overall work load, not just at the Department of Health but at the local level, you know, there is a bit of work to this. And with planning the meetings, attending the meetings that hopefully it will fit into other work that the local house is doing as well. Next slide.

	Well this is very simply put. I think lessons learned in any kind of program evaluation is certainly partnership work. And we - I think in - because perinatal Hep B prevention is a huge area where evaluation is constantly ongoing, targeting a piece or effort indicator like this is important. And I think we’ll learn a lot from this and it will change how we might proceed with program activities.

	So we certainly can’t do all the indicators like this in the perinatal Hep B prevention program but it’ll give us a good start. And you know, to continue looking at some areas and gaps.

	The last page certainly is acknowledgement. And there’s a lot of people here on this page I’d like to acknowledge and certainly they are listed. It’s a big group doing this and that is it to my presentation.

(Sashita Loreck):	Okay, thanks so much, Shana. That was a great presentation. Again, we appreciate you putting it together and I think it’ll be very helpful for others out there hopefully.

	So if we could, I guess, queue up for Q&A.

Coordinator:	As a reminder, if you would like to ask a question over the phone, please press star 1. Please unmute your phone and record your name. Again, to ask a question over the phone, press star 1. To withdraw your request, press star 2. One moment please for the first question.

(Pascel Wordley):	Well maybe while we’re waiting - this is (Pascel Wordley) again at CDC and I wanted to thank you also, Shana for a really nice presentation. And just comment that it’s a great example of how you’re collecting some information before you actually go out and do something different that will allow you to evaluate this intervention that you’re going to undertake and really be able to tell, you know, whether you accomplished something with it. So I think that’s a great example.

Shana Johnny:	Thank you.

Coordinator:	We do have a question over the phone. (Martha Freshner), your line is open.

(Martha Freshner):	Great presentation, thank you so much for doing it. Are these slides going to be available after the discussion?

(Sashita Loreck):	Yeah, we’ll send them through the NCIRD email list. So hopefully you’re on that. If not, somebody at your program can forward them onto you. And we’ll also place them on the ITE Web site.

(Martha Freshner):	Thank you.

(Sashita Loreck):	Okay.

Coordinator:	As a reminder, if you’d like to ask a question over the phone, please press star 1. Please unmute your phone and record your name.

(Sashita Loreck):	I guess while we wait, Shana, I was just going to ask, you know, as we expected there was some burden or a lot of burden placed on you all to meet with all the different stakeholders. So I was just trying to get an idea for maybe how many times the stakeholder group met and perhaps during those meetings, did they result in any other productive conversations around other perinatal or immunization related issues?

Shana Johnny:	Right. I think the challenge is getting the stakeholder group because it’s an extensive stakeholder group together. And so we do have a stakeholder meeting planned in March and we’ll plan about four to six actually iLinks or conference calls on this work. But they are aware of it and I have asked them to, you know, be participants but also, you know, provide data as well.

	So we have not reached that point where, you know, we’ve had a lot of activity with the stakeholders as far as face-to-face.

(Sashita Loreck):	Well, I guess in a big state like yours, that might be a challenge anyway.

Shana Johnny:	And that’s why we’re using the iLink. And I think that’ll work out well. And because perhaps the financial atmosphere, that might work great too for our local level here.

Coordinator:	I do have another question over the phone. (Carrie Sprake), your line is open.

(Carrie Sprake):	Hi, this is (Carrie Sprake) in Idaho. I loved your presentation. Did you guys - are you guys using the perinatal module with your IIS?

Shana Johnny:	Yes. We have a new perinatal Hep B - Hepatitis B module. A module of child profile which is our overall state immunization registry. So within that module, you know, it’s an extensive case management program and - which allows us to receive data from our local health jurisdictions because they have the keys to do that. And that’s how we secure our data.

(Carrie Sprake):	The reason I was asking is we have the same vendor and we are - we’re keying it up to use the module as well. I work - I do the data quality for the registry (unintelligible) already using it. I know who to call.

Shana Johnny:	Right. And so, you know, you can certainly contact (Crystal Alvaret) and - or myself, Shana Johnny. And you know, my phone number is 360-236-3698.

(Carrie Sprake):	Could you repeat that one more time, please?

Shana Johnny:	Yes, 360-236-3698.

(Carrie Sprake):	Thank you.

Coordinator:	The next question comes from (Jean Roberts). Your line is open.

(Jean Roberts):	Hi, this is (Jean Roberts) (unintelligible) from Arizona again, a fantastic presentation. I wish I had it here in Arizona so I’m planning on doing something like this. Before you started the whole process, did you train your group, your partners, your coalition and try to explain exactly the process and how long did it take?

Shana Johnny:	You know, I haven’t trained them per se. I have had some communication with them via email outlining, you know, the program itself, the evaluation piece, what we’re expecting and we’re looking at assessing, the amount of time potentially it will take.

	And what all is involved, so that, you know, they’re aware of the time commitment. And you know, these folks are pretty keen on the indicators already. And they pretty much know locally what their gap areas are. So I think it’s a matter of really communicating with them all at the same time so that we understand all the barriers that these providers are facing.

	I hope that answered your question.

(Jean Roberts):	Part of it. Second of all, were there any grants or funds to track or maintain that interest for such activity?

Shana Johnny:	There are no grants or funds other than we hold a consolidated contract with our local health jurisdiction in that they will perform these activities. And they’re already doing these pieces anyway. But we’re really honing in on one specific indicator and asking them to step up and, you know, communicate with us a little bit more about the exact indicator of the post-vaccine serology testing.

	So we’re just going into a little bit more detail. And you know, so that I have a better understanding at the state of what’s really going on with providers out there right now because we have a lot of issues in our state with parental hesitancy and exemptions. And it may not be the provider. We may be total thrown off on, you know, our survey here. But that’s why the stakeholder meetings are important so that we can all get an understanding of what’s going on in the different geographical regions of our state.

(Jean Roberts):	Thank you.

Coordinator:	And right now I show no further questions.

(Sashita Loreck):	Okay, I also wanted to open it up if anybody had any general questions about the evaluation plans or the implementation while a few of the grantees are on the phone, if you had any general questions you’d like to ask, feel free to do that also.

Coordinator:	Again, as a reminder, any other questions, please press star 1. Please unmute your phone and record your name.

(Sashita Loreck):	And while we wait to see if there’s anymore questions, as a reminder, the next Webinar will be on March 12. That’s also a Thursday and I believe that’s also 2:00 pm. We’ll send out all the details later this week so you’ll have all the log in information. And we urge everyone on the phone to try to pass it on to other staff at your program so that if others are interested, they can join in also.

Coordinator:	And right now I show no further questions.

(Sashita Loreck):	Okay, so thanks again to Leslie and Shana, really for taking time to put together these presentations and sharing all this valuable information with your colleagues on the phone. Hopefully folks on the phone found it useful. And as usual, if you have any questions or comments related to any of the evaluation activities, feel free to email us at www.ipe@cdc.gov. That stands for Immunization Program Evaluation. Again, www.ite@cdc.gov. And I guess we’ll talk you again in March. Thank you.

Coordinator:	Thank you so much for participating in today’s conference call. You may disconnect at this time.

END