Assessing a Food Safety Behavior Questionnaire for Criterion Validity

• Patricia Kendall, Mary Schroeder, Kelly Sinclair & Anne Elsbernd
  Colorado State University
• Lydia Medeiros and Gang Chen
  The Ohio State University
• Virginia Hillers and Verna Bergman
  Washington State University

Introduction

• Food safety education programs generally rely on self-reported behavioral questions administered pre and post education to measure program impact.
• Observational studies indicate that errors in food handling are more common than reported on questionnaires (Jay et al. 1999; Anderson et al., 2000)

Purpose

• Develop a validated bank of food safety behavior questions that could be used with confidence when evaluating consumer education programs.
• Determine if self-reported behaviors can be a valid way to assess behavioral outcomes of food safety education programs among low-income groups.

Development of Behavioral Questions

• Sub-group (n=8) from Expert Panel developed behavioral questions for each of 29 behaviors identified by Expert Panel as being important in reducing risk of foodborne illness in the home.
• First draft reviewed for content and face validity by tri-state team, faculty in three states, and 2 groups from target audience.
• Questionnaire revised and shortened based on feedback received.

Question Bank

Question Bank to be tested for reliability and validity contained 52 behavior questions:

• Practice personal hygiene (5 questions)
• Cook foods adequately (12)
• Avoid cross contamination (7)
• Keep foods at safe temperatures (12)
• Avoid foods from unsafe sources (16)

Questionnaire

• Two part questionnaire addressed food safety issues for the general public and those specific to pregnant women
• Contained a variety of question types:
  • 5 point Likert scale (20 questions)
  • Dichotomous Y/N (41)
  • Multiple choice (1)

Reliability Testing

Target audience members included WIC paritcipants (7), Greeley EFNEP participants (9) and University Village residents in a parent/toddler group (4)

• Test/retest:
  • Target audience members (n=20) took questionnaire at 2 time points; responses correlated and compared via paired t-tests
  • Questions considered reliable if:
    • P-value > 0.05 & r ³ 0.70 or Agreements/Agreements +Disagreements ³ 70%
• Internal consistency:
  • Assessed using Cronbach alpha; run on all items within a particular construct
  • Questions with a ³ 0.60 considered internally consistent (Osterhof, 2001; Taylor et al., 2001; Murphy et al., 2001)

Reliability Results

The questions with no variability were those for which respondents answered the same way on test and retest; it seemed silly that these questions could not be reliable despite high correlation coefficients (1.00), A/A+D values (100%) or both.

Alphas for the CC and AF constructs were low because of the nature of the questions- AF included a variety of food preference questions for which people who knew the items were unsafe had varying levels of concern – yes I will eat raw cookie dough but no, I will drink raw milk or untreated water from a stream

CC- all 50 subjects reported “most of the time” or “always” handwashing after handling raw meat but fewer subjects reported washing their fruits (38) or vegetables (47) “most of the time” or “always” before eating them

• Test/Retest:
  • 47 of 52 questions met reliability criteria
• Internal consistency:

Validity

• Degree to which an instrument measures what it is intended to measure
• Assessed several types of validity:
  • Content Validity: Reflects domain of content to be measured
  • Face Validity: Measures what intended to measure
  • Criterion Validity: Correlates with other more accurate instrument

Validity Testing

• Criterion Validity – focus of this study.
• Established by comparing questionnaire response to observed behavior and interview responses during a kitchen activity session held ~ one week later.

Validation Study Subjects

• 70 FSNEP and EFNEP participants in CO, WA, and OH
  • 50 post education only
  • 20 pre and post education
• Primary food preparers
• Had completed an education program that included a 30- to 60-minute food safety component

Study Design Post-Education

• Week 4 - Food Safety Class
• Week 5 - Take Questionnaire - Recruit -
• Week 6 to 7 - Observation + Interview Session

Study Design Pre-Post Education

• Week 2 - Recruit and take questionnaire
• Week 3 - Observation & Interview Session
• Week 4 - Food Safety Class
• Week 5 - Take Questionnaire - Recruit -
• Week 6 to 7 - Observation + Interview Session

Kitchen Activity Session

• Cooking Observation in Community Kitchen:
  • Cook a chicken breast to desired doneness
  • Slice an apple to garnish the chicken
  • Cook a hamburger to desired doneness
  • Slice a tomato to go with the hamburger
• In-depth interview
  • Asked the same questions on the questionnaire in a conversational, open-ended manner

Kitchen Activity Protocols

• Extensive training of research assistants on conducting cooking observations and interview sessions. Mock interviews conducted & videotaped.
• Two research assistants conducted each session (safety reasons).
• Each subject provided with same food items, utensils, equipment, instructions.
• Cooking sessions videotaped and interviews audio-taped.
• Actions and responses coded by research assistant who conducted session, then re-coded by one researcher in Colorado and responses compared to ensure comparability of data. Differences in coding reviewed by 3^rd party and resolved.

Limitations

Behaviors performed incorrectly:

Hand washing after contact with raw poultry or hamburger (76% and 43%), countertop washing after food preparation (76%) incorrectly, cutting board washing after raw poultry (21%) or after raw meat (17%) and before ready to eat fruit or vegetables

People do not clean their countertops before preparing food……..even those with pets wandering around the house all day

Only 18% of subjects used a thermometer with chicken and 20% used a thermometer with hamburger while the majority of subjects( greater than 90%) cooked their chicken or hamburgers to 160 degrees F.
In Anderson et al, only 5% of subjects used a thermometer and fewer cooked these items to 160 degrees F (165F for the chicken) = 52-58% of subjects

• Non-randomized design
• Observations not performed in homes
  • No interruptions
  • Subjects could focus on food preparation/cooking
• Intervention for pre/post design wasn’t controlled

Validation Design

• Percent Agreement
  • Behavioral Questionnaire
  • Face-to-face Interview
  • Observation

*Validation criteria:

• Observable behaviors: 70 % agreement among all 3 instruments (counting correct & incorrect behaviors)
• Non-observable behaviors: 70 % agreement between questionnaire & interview

Validity Results

• Observable Questions: 54.5% (6 of 11) met validity criteria
• Non-observable Questions: 66% (27 of 41) met validity criteria

This table shows the results for each control factors, detailing the number of questions that met our validity criteria

Instrument Sensitivity

• Potential ceiling effect:
  • Good questions are those that capture range of responses
  • Looked for questions that 20-80% gave less desirable response at pre education
• Change in mean scores from pre to post education (n=20)

(Parmenter and Wardle, JNE 32:269; 2000)

Instrument Sensitivity Results

• Among validated questions, several in 4 of 5 control factors showed good response variety pre-workshop, with room for change.
• Pre and post scores on Cross-contamination questions generally high, but improvements needed in skill level.
• Improvements in behavior pre to post seen for washing hands prior to cooking and not leaving meat on counter.

Conclusions

• 33 of 52 behavioral questions met reliability and validity criteria (³ 70% agreement), including several questions from each pathogen control factor.
• Agreement between observed and self-reported behaviors was better when incorrectly performed behaviors were included.
• Further study is needed using these questions in educational settings with controlled interventions.