My Thursday at the American Evaluation Association meeting had a late start due to my participation in a morning webinar.  Still, interesting sessions awaited after lunch and here are some notes:

Evaluation Methods and Experiences on Five Indian Reservations With the Federally Recognized Tribal Extension Program in Arizona and New Mexico (Roundtable Presentation from Linda Masters, Melvina Adolf, Gerald Moore, Matthew Livingston, and Jeannie Benally, all from theUniversity of Arizona )

Extension faculty who work with the Federally Recognized Tribal Extension Program in Arizona and New Mexico presented a summary of evaluation methods that they have used with the Navajo Nation, the San Carlos Apache Tribe, Colorado River Indian Tribes; the Hualapai Tribe, and the Hopi Tribe.  The programs involve agricultural practices.  Emphasis was on how long it can take to establish trust (time spans of years) and the importance of finding an ally in the community to help open doors.  Faculty members are challenged to reconcile university requirements with tribal realities.  In the ensuing discussion, a possible approach of including recognition of the need to establish trust as part of project logic models emerged.  The roundtable also included overviews of approaches to workshops, and these included  setting small, measurable goals and identifying early adopters among class participants (these are people who could help with futher learning opportunities).  Workshop evaluations are usually done at the end of a session, before people leave, and feature questions with Likert scale responses.  individual interviews and success stories have also proven useful.  Pre- and post-tests were not recommended because they could be intimidating.

Indicators of Success in a Native Hawaiian Educational System: Implications for Evaluation Policy and Practice in Indigenous Programs (Roundtable Presentation from Ormond Hammond and Sonja Evensen, Pacific Resources for Education and Learning)

A project to identify valid, reliable, and meaningful indicators of success for the Native Hawaiian Education Council (NHEC) in response to governmental indicators that do not reflect Native goals and values resulted in development of proposed target impacts (Resiliance & Wellness; Hawaiian ‘Ike; Academic Achievement & Proficiency; Employment, Self-Sufficiency & Stewardship) and proposed target levels: Kanaka (individual/group); ‘Ohana (family); Kaiaulu (community); and ‘Onaehana (system).  For example, activities that address issues such as homelessness, life skills, or nutrition could be considered as impacting Resiliance & Wellness at the Kanaka (individual/group) level.  This is an effort to express indicators in culturally meaningful ways.

Improving the Collection, Analysis, and Reporting of Survey Data (Gary Miron and Anne Cullen, Western Michigan University)

The presenters demonstrated tools that they have developed for their clients to help improve the collection, analysis, and reporting of survey data.  These included examples of an Excel approach to analyzing open-ended questions (salient themes are column headers, comments are listed in the first column, and check marks placed in appropriate cells) and a preformatted Excel reporting file that produces graphs and tables of survey results.  Among their lessons learned: sometimes self-ratings of knowledge and skill decline from pre-test to post-test because class participants learn how much they don’t know.