|
 |
|||||||||||
 |
|
|||||||||||
|
|
||||||||||||
Training for all levels of project field staff occurred both prior to the start of data collection and throughout the year. Having experienced staff allowed training programs to go beyond the basic steps and focus on enhancing and improving necessary project skills.
Two management training programs occurred to share information and better equip all Regional Directors (RDs), Regional Supervisors (RSs), Field Supervisors (FSs), and survey specialists for their roles on the 2002 NSDUH.
The first 2002 NSDUH management session was held November 18, 2001, in Cincinnati, OH to prepare for the upcoming year. Topics covered during this session included:
In lieu of a May Management Meeting, a series of four management teleconferences was held in the spring to discuss the following topics:
The first two conferences occurred during March, the third in April and the final conference in May.
Training sessions were held about once a quarter throughout the year to train newly hired new-to-NSDUH FIs. These sessions helped maintain a sufficient staff size to complete S/I within the quarterly timeframes. For each session, there were multiple training rooms staffed by a team of three or sometimes four trainers. Occurring in January, February, and then prior to the beginning of each new quarter, a total of 197 new FIs were trained during these replacement sessions. Table 5.1 summarizes the interviewer training sessions held for the 2002 NSDUH.
The new-to-project training program consisted of seven full days of training covering the general techniques of interviewing, screening using the Newton handheld computer, conducting NSDUH interviews on the laptop computer, and general NSDUH protocols and technical support. Spanish-speaking FIs attended an additional one day session to review the Spanish translations of the questionnaire and the Newton screening program.
All trainees were required to pass a certification process as part of the successful completion of training. Conducted individually, each trainee had to demonstrate knowledge of the basic NSDUH protocols by completing a straight-forward screening and interview through the beginning of the ACASI questions. Any trainees who did not pass on the first try received immediate feedback and additional individual training to clarify any points of confusion. During the subsequent re-certification attempt, the trainee only had to re-do the portion(s) done incorrectly the first time. Any trainee failing the re-certification process was either placed on probation, (and barred from working until the proper completion of further re-training/re-certification), or was terminated from the project. Of the 197 new-to-project interviewers trained during 2002, 5 were placed on probation for problems with the certification process and no trainees were terminated for certification issues.
To provide consistency between training classrooms, a near-verbatim guide with 22 sections provided detailed instructions and text to ensure all necessary instructional points were covered. In addition to the guide, trainers also used a videotape that contained multiple segments for use throughout training; a workbook containing exercises on the Newton and laptop computer and printed examples; training segment materials used in exercises that replicated actual segment materials; the FI manuals for reference; and the two computers (the Newton and the Gateway laptop) with accessory equipment.
At each training site, staff included a Site Leader, Logistical Assistant, a Lead Technician, a Certification Coordinator, and one or more training teams. Each of these roles was well-defined to ensure that training proceeded smoothly.
The Site Leader at each training site coordinated all FI registration activities, hotel relations, and logistics; and monitored trainees and trainers. The Site Leader's specific tasks included:
The Site Leader role was filled by a qualified NSDUH supervisor who had extensive experience with project protocols and management goals.
The Logistical Assistant worked closely with the Site Leader throughout training to be sure all trainees were registered properly, all training rooms had all necessary supplies, and hotel services functioned smoothly. Other duties included grading home study tests and distributing training and incentive checks at the successful conclusion of training.
The Lead Technician served as the point of contact for all technical issues including the proper functioning of all equipment and programs. Other duties included supervising training equipment set up and the initialization and distribution of interviewer computer equipment.
The Certification Coordinator managed the certification process, including establishing appointment schedules, monitoring and distributing certification supplies and materials, and reporting the results to the Site Leader.
Each classroom was taught by a training team consisting of a lead trainer, one or sometimes two assistant trainers, and a technical support representative. The lead trainer and assistant trainer(s) divided the responsibility for presenting sections of the training, with the technical support representative often helping with the more technical sections. The lead trainer had the additional responsibility for the logistics and schedule of the training room. In general, one trainer would train from the front of the room while the other trainer(s) would monitor FI progress, assist FIs with questions, and sometimes operate the computer equipment.
The technical support representative's primary role was to prepare and set-up the computers for each FI; to ensure the proper functioning of the Newton, Gateway and Toshiba projection equipment used for the training presentation; to provide in-class technical help; and in some cases, to present the technical sections of the training program (depending on the classroom's training needs and the technical support representative's training experience).
Training teams were selected based on availability and experience. The lead trainer was usually an RS with considerable training experience. Assistant trainers were usually RSs, FSs, Instrumentation Team members, or survey specialists.
After completing the registration process the evening before, training classes began first thing in the morning with an introduction to the history and scope of the NSDUH presented in a video by Project Director Tom Virag. Next, classrooms went through a three-hour introductory computer session. This included instruction in the use of the Gateway computer hardware and a thorough introduction to the basics of the Newton hardware and software, although the actual screening program was not covered. In the afternoon, trainees were introduced to the importance of professional ethics, respondent rights, and the interviewer's role and tasks on the NSDUH. Trainees with little computer experience could stay after class for some hands-on practice in order to build their confidence.
Day 2 included a general introduction to survey sampling and counting and listing, followed by an in-depth discussion of how to locate segments and selected DUs. Trainees also learned how to contact selected DUs for screening and the importance of knowing the study. They were given the opportunity to review supplementary materials and practice effective introductions and responses to respondent questions. Trainers then introduced the screening process using a video of a 'real' screening and explanations of the purpose of each question. Following a trainer demonstration, each trainee had the opportunity to try the Newton handheld computer during a group walk-through screening exercise.
On Day 3, trainees focused on gaining experience and confidence by conducting numerous practice screenings on the Newton. Trainees completed several enumeration and rostering exercises round-robin style as well as individual and paired mock exercises covering the whole screening process. Trainees also learned about the specifics of screening group quarters units. All trainees were invited to attend an evening study hall session for additional practice.
Training on Day 4 began with explanations of adding missed DUs. The next topic of the day was an introduction to the NSDUH interview and the basics of good field interviewing techniques. Next, trainees learned the details of the NSDUH instrument with a complete round-robin read-through of the entire questionnaire, including question-by-question specifications.
On Day 5 trainers presented a brief discussion of the functions of the CAI Manager program on the laptop. An individual practice interview exercise allowed trainees to review both the format and questions in the CAI program at their own pace. The next topic was devoted to information about overcoming reluctant respondents and dealing with difficult situations. This section included informative video segments and group exercises. This was followed by a description of the details required in collecting industry and occupation information. Trainees who were performing well could attempt the certification process the evening of Day 5. Since the training program was not complete, anyone not passing this first attempt was given another opportunity at the conclusion of training.
The next day began with RTI's IRB interviewer training module which covered ethics and regulations involving human subject research, the role of the IRB, and the role of the interviewer in protecting respondent rights. Next, a session on transmitting data had a trainer or technical support representative demonstrate how to transmit from both the Newton and the Gateway. The class then began a series of two paired mock exercises encompassing the entire screening and interviewing process so that trainees could practice the transition from the screening on the Newton to the CAI interview on the laptop. Following each mock interview, a group review session was conducted by the trainer. At some point during the practice mock interviews, trainees attempted a successful transmission on both computers at a station in the training room. The day concluded with a discussion of the project's administrative procedures, project supplies, data quality control, and proper documenting and reporting. Certifications were scheduled for the evening of Day 6.
Day 7 began with the completion of any remaining discussion on administrative topics. The next section on troubleshooting and technical support informed staff about the most common technical problems they might encounter, steps to take to correct them, and when and how to contact Technical Support for additional help. A brief recap of the entire process of screening and interviewing helped trainees review again how all the tasks fit together. The afternoon was designed as a study hall in which interviewers had a number of tasks to complete while trainers were available to assist as needed. These tasks included another individual interview exercise and various exercises completing electronic production, time and expense reports. Any remaining trainee certifications also took place during this time.
A trainer fluent in Spanish conducted a one-day session for RTI-Certified bilingual FIs on the Spanish-language NSDUH materials. These FIs were trained to use the Spanish versions of the screening introduction and rostering questions on the Newton, the CAI instrument, and other 2002 supplemental materials. Only those FIs who were RTI-Certified bilingual interviewers and who had been hired as bilingual interviewers attended this session.
After completing New-to-Project training, all graduates were mentored by their FS, another FS or by an experienced FI. Previously referred to as on-the-job-training and conducted as needed, the mentoring program became more formalized in June of 2002. Mentoring of all trainees was required. Standardized instructions were provided to the mentor to be sure all important protocols learned during training were reinforced.
To prepare the field interviewers chosen to continue from the 2001 NSDUH into 2002, special Veteran FI training sessions were held in January 2002. Having regional sessions throughout the nation served several purposes:
Veteran training sessions were held at 8 sites including: Anchorage, AK; Atlanta, GA; Baltimore, MD; Cincinnati, OH; Honolulu, HI; Houston, TX; Los Angeles, CA; and Newton, MA. Two separate sessions were held at 6 of these sites, with the A groups meeting on January 5-6 and the B sessions meeting January 8-9, 2002. The Hawaii and Alaska sites each had single sessions to train the staff of those two states. In addition to these early January sessions, two special weekend sessions were held later in January to train traveling field interviewers and any veteran interviewers unable to attend the early sessions. Also, throughout 2002, additional veterans who missed the January sessions were trained with permission on an individual basis. Table 5.1 summarizes the January Veteran interviewer training sessions.
The Veteran training program consisted of an initial home study (see Section 4.5.1) followed by two training days covering topics such as changes for the 2002 study, data quality, being comfortable at the door to help overcome objections, respondent rights and RTI's IRB training module, and details on the new incentive process.
In order to successfully complete training and receive a 2002 assignment, all veteran FIs were required to pass a certification process the same as the one for new-to-project trainees. Veteran certifications occurred the day prior to the start of classroom training. One-on-one, each veteran had to demonstrate knowledge of the basic NSDUH protocols by completing a straight-forward screening and interview (through the first ACASI question). Certifiers gave feedback immediately following the certification attempt but did not provide an outcome of either pass or fail. Official notifications of results were distributed to all staff the morning of Day 1. Those failing the first certification were given appointments for the evening of Day 1 for a re-certification. Any veterans failing the re-certification were either placed on probation (without an assignment) or were terminated, depending on the severity of the error and other circumstances. Veterans on probation received thorough re-training from their FS. When ready, the veteran attempted another certification with RTI staff. If successful, probation status was removed and an assignment given. If not successful, the FI was terminated from the project. Results of the veteran certification process—which are the final results of training—are found in Table 5.2.
Standardized certification materials included instructions for the certifier, a verbatim script for the certifier to use during the process, and a form on which to document ratings on all required aspects of the interviewer's performance.
To provide consistency between veteran training classrooms, a near-verbatim training guide with 16 sections provided detailed instructions and text to ensure all necessary instructional points were covered. In addition to the guide, trainers also used a videotape; a workbook containing exercises on the Newton and laptop computer and printed examples; the FI manuals for reference; and the two computers (the Newton and the Gateway laptop) loaded with the new 2002 programs.
At each training site, there was a Site Leader, Logistical Assistant, Lead Technician, and a Certification Coordinator with responsibilities as described in Section 5.2.2 for new-to-project training sessions.
Each classroom was taught by a training team consisting of a pair of FSs. One FS's staff attended during Session A then the other FS's staff came for Session B. The FS pair worked together to divide the responsibility for presenting the various training sections. The presenting trainer usually trained from the front of the room while the other trainer monitored FI progress, assisted FIs with questions, and sometimes operated the computer equipment.
Training experience varied considerably among the FS staff. For classrooms with weaker training teams, Site Leaders assigned available RSs, survey specialists, or Instrumentation Team members to support the FS training team or, in some cases, to lead the training.
To prepare all lead and assistant trainers for their training role and to instruct all project staff in the changes for the 2002 survey, a Training-the-Trainers Session was held in Cincinnati, OH on November 15-17, 2001. Classrooms were led by "master trainers" with assistance from other experienced project staff. The groups reviewed the Veteran training guide and materials as well as logistics for the January sessions.
The master trainers were RDs and other members of the management staff or Instrumentation Team. These master trainers attended a two-day Master Trainers session at RTI on October 29-30, 2001 to learn about the Veteran training program and the expectations for the Training-the-Trainers session. In addition, a training expert consultant led the group in a number of exercises to increase awareness of presentation skills.
During the three day session in November, master trainers first led the training teams through a discussion of training presentation skills. Trainers for January then presented their assigned sections of the guide to the classroom. Presenting to this group allowed for multiple classrooms to review the content and test the accuracy of the guide and the training program, submitting comments to the Instrumentation Team for consideration when making revisions. Most importantly, having the January trainers actually train gave them the opportunity to focus on their presentation style and mastery of the material.
Also during this time period, all project management staff completed the same certification process required of the interviewers.
Day 1 began with a brief presentation of experiences from the 2001 survey (response rates and overview of field observation results) followed by some actual study results from the 2000 survey. After an overview of the changes for 2002, trainers presented the RTI IRB interviewer training module on respondent rights and ethics of conducting research involving human subjects. Next on the agenda were the details of the 2002 changes for the Newton and for the CAI instrument. The majority of the afternoon was devoted to numerous ways to increase an FI's comfort level at the door. At the end of the day, FI computer equipment was returned and a practice screening and interview exercise assigned for homework.
During the evening of Day 1, any necessary re-certifications were completed.
Day 2 began with a review of the homework assignment. The next topic was data quality, which included discussions of NSDUH protocols and procedures including explanations for the reasons behind the various protocols and procedures. As part of the discussion of field observations, interviewers were asked to observe an FI at work as shown in a video segment. Next, classes reviewed the changes related to the use of the cash incentive, including changes in overcoming objections and handling the incentive administratively. Trainers then presented a short module on what makes a good Record of Calls.
FSs then selected and led one of three workshops to spend more time on a topic where the region could use some improvement. Workshop choices included Closing the Deal, Restricted Access, and Using the Segment Kit. The remainder of the training consisted of an FS Team Meeting in which each FS could discuss region specific topics and have time for team building exercises.
Two additional veteran training sessions were held January 12-13, 2002 in Cincinnati, OH and January 19-20 in the RTP, NC area to accommodate those veteran interviewers unable to attend the early January sessions and to train traveling FIs. Various project staff served as the trainers for these sessions, so that FSs could focus on managing data collection.
As the year progressed, several veterans from 2001 who wished to continue working were trained individually via home study and telephone conference with an FS. These veterans missed the January sessions due to illness or pre-approved scheduling conflicts. With special permission, one-on-one training brought these five interviewers up-to-speed on the 2002 NSDUH. Following successful completion of the home study, an FS (who had been chosen based on training ability) worked with the veteran for one to two days covering the content of the 2002 Veteran Training session. While group exercises were excluded, all individual exercises, homework, and discussions occurred. When finished, the veteran completed the certification process via telephone, typically with a Regional Supervisor.
Regional team meetings with particular FS teams occurred throughout the year. As needed, team meetings were held to introduce interviewers to a new supervisor (either FS or RS). In other situations with teams performing below expectations, the focus of these meetings was to provide further training for FIs on refusal avoidance, refusal conversion, and efficiently working case assignments. Additional discussion topics included data quality and specific team performance issues. These "mini-camp" meetings were held in locations central to an FS's team. An RTI project manager was required to be present at these meetings; therefore, an RD and/or RS attended. In addition to team meetings in regions with a new supervisor, one team meeting was held in Arkansas in order to improve regional performance.
Periodic evaluations of interviewer knowledge were conducted via an arrangement similar to the electronic home study for veterans. All FIs picked up the eVal program via transmission and had about one week to complete the 10 item questionnaire. These 10 items were assigned randomly from a bank of close to 100 questions all designed to test interviewer knowledge of basic NSDUH protocols. When finished with the open book evaluation, the computer program scored the answers so that the FIs could receive immediate feedback about their results. To pass, FIs had to score at least 80%. FIs not achieving that score received another set of 10 questions to complete. Any FI not scoring at least 80% on the second set of questions was placed on probation pending the completion of further re-training with the FS.
For the first eVal issued in May of 2002, over 98% of the current interviewers passed on the first try. Of the 12 FIs requiring a second attempt, only one failed. The results of the second eVal issued in August, 2002 were similar: over 99% passed on the first try, and only 1 of the 6 needing a second attempt was placed on probation. Results from the 2002 eVal program are provided in Table 5.3.
After completing training, FIs continued to need opportunities to improve or refine their screening and interviewing skills. During weekly conference calls and at other times as needed, an FI and FS discussed questions or problems. However in some cases, an FS made an in-person visit to work with the FI and increase the FI's skills and experience through on-the-job training. These in-person visits were not always a reaction to a major problem; they were sometimes a proactive measure taken to ensure success in the field and to reduce FI attrition. All such visits were subject to prior RS approval.
While there were opportunities for FSs to enhance their skills of managing NSDUH production through work with the RS and regional management sessions, in some cases, management staff identified a need for more intense one-on-one training. In-person visits by an RS were used primarily for the purpose of observing and coaching FSs in effective methods of managing organizational and administrative tasks. However, these RS visits were almost always for the purpose of troubleshooting and addressing a major problem centered around concern about an FS's performance, or to help a new FS transition into the position. These visits were subject to prior RD approval.
Leading the training sessions held throughout the year required involvement of project staff with other NSDUH responsibilities. These dedicated staff trained each day and then completed their other project duties in the evenings. Training planners tried to rotate staff across the various training assignments throughout the year to avoid overloading any one individual. This seemed to work reasonably well.
Implementing an electronic home study for Veteran FIs involved many staff from question developers to programmers to web designers working to display the results for managers to use. Great teamwork accomplished the task in a timely fashion.
The requirement that all FIs successfully complete the home study process in order to be able to attend training and continue on the project caused concerns for many FIs. Although the materials explained that the questions tested basic knowledge of protocol and were not designed to be tricky, FSs had to spend time providing reassurance.
Similar to the veteran home study, the development of the certification process was very demanding. Significant effort was involved in order to ensure fairness, ease of use, consistency between sites, and accurate reporting of results. Once again, dedicated staff were able to complete the work on schedule.
The implementation of the certification process, particularly at the veteran trainings, caused additional problems. Staff had to train during the day and then serve as certifiers in the evenings. Due to the number of veterans requiring re-certification, the time demand issue was significant. The Certification Coordinator had the difficult job of scheduling the re-certifications and tracking all the results.
Also problematic was the additional stress placed on veteran FIs who depend on their jobs and feared the consequences of failing the certifications.
Table 5.1
2002 NSDUH Interviewer Training Programs
| Month | FI Training Sessions Date & Location |
FIs Trained |
Cum. No. of FIs |
Attrited FIs |
Cum. No. of Attrited FIs |
|---|---|---|---|---|---|
| Veteran Training Sessions | |||||
| Jan | Date: Session A: 1/5-6 Session B: 1/8-9 Location: 8 sites (see text) |
708 | 708 | 14 | 14 |
| Weekend/Make-up Veteran Trainings Dates: 1/12-13 and 1/19-20 Location: Cincinnati (OH) and RTP (NC) |
30 | 738 | |||
| Replacement Training Sessions | |||||
| Jan | Date: 1/26-2/2 Location: Cincinnati |
39 | 777 | 0 | 14 |
| Feb | Date: 2/16-23 Location: Cincinnati |
11 | 788 | 16 | 30 |
| Mar | Date: 3/18-27 Location: Cincinnati |
29 | 818 | 20 | 50 |
| Veterans Trained One-on-One | 1 | ||||
| Apr | Veterans Trained One-on-One | 1 | 819 | 18 | 68 |
| May | Veterans Trained One-on-One | 1 | 820 | 30 | 98 |
| June | Date: 6/22-29 Location: Cincinnati |
35 | 856 | 15 | 113 |
| Veterans Trained One-on-One | 1 | ||||
| July | No training session | 0 | 856 | 16 | 129 |
| Aug | No training session | 0 | 856 | 26 | 155 |
| Sept | Date: 9/20-9/27 Location: Cincinnati |
83 | 939 | 25 | 180 |
| Oct | Veterans Trained One-on-One | 1 | 940 | 24 | 204 |
| Nov | No training session | 0 | 940 | 23 | 227 |
| Dec | No training session | 0 | 940 | 34 | 261 |
Table 5.2
Veteran Training/Certification Results
| Veteran Training Session | Successfully Completed Training/Certification |
Completed Training on Probation due to problems with Certification |
Failed Certification (terminated from project) |
Total |
|---|---|---|---|---|
| Session A | 357 | 22 | 3 | 382 |
| Session B | 308 | 21 | 8 | 337 |
| January Make-Up – Cincinnati | 20 | 2 | 1 | 23 |
| January Make-Up – RTP | 8 | 0 | 0 | 8 |
| Vet One-on-One Training | 5 | 0 | 0 | 5 |
| Total Veterans | 698 | 45 | 12 | 755 |
| Total Veterans Graduating from Training |
743 | |||
Table 5.3
Results from Home Study and Periodic eVals
| Test Name | Passed on First Try | Failed on First Try | Total Passing | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Passed on 2nd Try |
Failed 2nd try* | ||||||||
| Count | % | Count | % | Count | % | Count | % | ||
| Home Study, Dec. 2001 | 754 | 98.6 | 11 | 1.4 | 10 | 90.9 | 1 | 9.1 | 764 |
| eVal, May 2002 | 718 | 98.4 | 12 | 1.6 | 11 | 91.7 | 1 | 8.3 | 729 |
| eVal, August 2002 | 695 | 99.1 | 6 | 0.9 | 5 | 83.3 | 1 | 16.7 | 701 |
|
|
|
 |
Click to Return to OAS Home Page |
|
This page was last updated on May 16, 2008.
|
|
