Section VI: Annotated References (Part I)

Accomplishment Records References:

Hough, L. M. (1984). Development and evaluation of the "accomplishment record" method of selecting and promoting professionals. Journal of Applied Psychology, 69(1), 135-146.

Developed the accomplishment record method, which applies the principle of behavioral consistency to selection and promotion and results in self-reported descriptions of accomplishments in highly relevant, behavioral job dimensions. These protocols were reliably rated (.82) using specially prepared rating scales and guidelines. Scores on the accomplishment record inventory were unrelated to traditional psychological measures, such as aptitude tests, grades, and honors, but were correlated with job performance. Correlations with self-perceptions of success, hard work, and self-assurance, and with length of time spent practicing a profession are also provided.

Hough, L. M., Keyes, M. A., & Dunnette, M. D. (1983). An evaluation of three "alternative" selection procedures. Personnel Psychology, 36(2), 261-276.

A content-oriented strategy was used to develop three alternative selection inventories designed to reflect the content domain of positions held by attorneys employed with a large Federal agency. These inventories and three traditional inventories were completed by 329 attorneys of the agency as part of a concurrent validation study. Criterion-related validities of an accomplishment record inventory with a background inventory and an interest and opinion inventory were both statistically and practically significant. The special features and advantages of the accomplishment record inventory as an alternative selection procedure are discussed.

Sackett, P. R., Schmitt, N., Ellingson, J. E., & Kabin, M. B. (2001). High-stakes testing in employment, credentialing, and higher education: Prospects in a post-affirmative-action world. American Psychologist, 56(4), 302-318.

The authors describe the nature of the issues faced by practitioners when trying to optimize both the performance and ethnic diversity of chosen individuals. A review of research on different strategies to address these issues (e.g., adverse impact) is provided. The authors recommend using selection materials assessing the full range of relevant attributes using a format minimizing verbal content as much as is consistent with the outcome one is trying to achieve. They also recommend the use of test preparation, face-valid assessments, and the consideration of relevant job or life experiences. The authors conclude that regardless of strategy adopted, it is difficult to maximize both the performance and ethnic diversity of selected individuals.

Schmidt, F. L., Caplan, J. R., Bemis, S. E., Decuir, R., Dunn, L., & Antone, L. (1979). The behavioral consistency method of unassembled examining. Washington, DC: U.S. Office of Personnel Management, Personnel Resources and Development Center.

This report describes the history, rationale, and development of the behavioral consistency procedure, a methodology very similar to the accomplishment record in format and is based on an applicant's past achievements rather than on credentials. It also describes the results of a study conducted to compare the behavioral consistency method to two traditional rating tools. The results indicated the behavioral consistency method (1) shows greater interrater reliability than the other two methods, (2) is feasible in terms of cost and time requirements, and (3) measures factors different than those measured by the traditional rating procedures.

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262-274.

This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of cognitive ability tests with the 18 other selection procedures. The practical utility of the implications of this article's summary findings are substantial. In addition, the implications of these research findings for the development of theories of job performance are discussed.

Assessment Center References:

Arthur, W. Jr., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125-154.

Used meta-analytic procedures to investigate the criterion-related validity of assessment center dimension ratings. By focusing on dimension-level information, the authors were able to assess the extent to which specific constructs account for the criterion-related validity of assessment centers. From a total of 34 articles reporting dimension-level validities, the authors collapsed 168 assessment center dimension labels into an overriding set of 6 dimensions: (1) consideration/awareness of others, (2) communication, (3) drive, (4) influencing others, (5) organizing and planning, and (6) problem solving. Results show a range of estimated criterion-related validities from .25 to .39.

Caldwell, C., Thornton, G. C., & Gruys, M. (2003). Ten classic assessment center errors: Challenges to selection validity. Public Personnel Management, 32(1), 73-88.

This paper summarizes ten classic errors associated with selection and promotion related assessment center administration. Critical errors covered are: (1) poor planning, (2) inadequate job analysis, (3) weakly defined dimensions, (4) poor exercises, (5) no pre-test evaluations, (6) unqualified assessors, (7) inadequate assessor training, (8) inadequate candidate preparation, (9) sloppy behavior documentation and scoring, and (10) misuse of results. Reducing and/or eliminating the errors in this list will allow more efficient and effective assessment center administration.

Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72(3), 493-511.

A meta-analysis of 50 assessment center studies revealed higher validities were found in studies in which potential ratings were the criterion, and lower validities were found in promotion studies. Validities were higher when the percentage of female assessees was high, when several evaluation devices were used, when assessors were psychologists rather than managers, when peer evaluation was used, and when the study was methodologically sound. Age of assessees, whether feedback was given, days of assessor training, days of observation, percentages of minority assessees, and criterion contamination did not moderate assessment center validities. The findings suggest assessment centers show both validity generalization and situational specificity.

Testing and Assessment: An Employer's Guide to Good Practices. (2000). Washington, DC: U.S. Department of Labor, Employment and Training Administration. Note: Article can be accessed at http://www.onetcenter.org/guides.html.

This guide is written as a source of guidance to managers and Human Resource (HR) professionals. It provides basic yet essential concepts on employment testing and assessment to help managers and HR professionals evaluate and select the most appropriate assessment tool for a specific situation, administer and score assessment tools, interpret assessment results, and understand professional and legal standards that must be followed when conducting personnel assessments. An overview of the development and administration of assessment centers is provided.

Woehr, D., & Winfred, A. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29(2), 231-258.

The present study provides a systematic review of the assessment center literature with respect to specific design and methodological characteristics that potentially moderate the validity of assessment center ratings. In addition, the results of a meta-analysis of the relationship between these characteristics and construct-related validity outcomes are presented. These results provide little if any support for the view assessment center ratings lack construct-related validity, while at the same time demonstrating criterion-related validity. The implications of these findings for assessment center construct-related validity are discussed.

Zedeck, S. (1986). A process analysis of the assessment center method. In B. M. Staw & L. L. Cummings (Eds.), Research in organizational behavior, 8, 259-296.

Discusses the dynamics operating in observing the behavior of managerial candidates in simulated exercises and in processing information for the evaluation of candidates. These dynamics are viewed from 3 perspectives: (1) information processing, (2) categorization and social cognition, and (3) group dynamics. Concepts such as categories and management behavior schema are used to explain how assessors recall information and make predictions and judgments.

Biographical Data (Biodata) Tests References:

Elkins, T., & Phillips, J. (2000). Job context, selection decision outcome, and the perceived fairness of selection tests: Biodata as an illustrative case. Journal of Applied Psychology, 85(3), 479-484.

The present study aimed to verify and extend Gilliland's (1993) proposed model of perceived selection fairness by investigating the role of job context in the formation of fairness perceptions of biodata. A sample of 255 students completed an operational biodata instrument, believing it would be used to hire persons for either international, local, or unspecified entry-level managerial positions. Participants were then presented with outcome information (selected or rejected for further consideration). Consistent support was found for the research hypotheses derived from the Gilliland model. Participants' perceptions of the fairness and job relatedness of biodata were affected by the selection context and decision outcome.

Hough, L. M., & Oswald, F. L. (2000). Personnel selection: Looking toward the future — Remembering the past. Annual Review of Psychology, 51, 631-664.

Reviews personnel selection research from 1995-1999. Areas covered are job analysis; performance criteria; cognitive ability and personality predictors; interview, assessment center, and biodata assessment methods; measurement issues; meta-analysis and validity generalization; evaluation of selection systems in terms of differential prediction, adverse impact, utility, and applicant reactions; emerging topics on team selection and cross-cultural issues; and finally professional, legal, and ethical standards. Three major themes are revealed: (1) better taxonomies produce better selection decisions; (2) the nature and analyses of work behavior are changing, influencing personnel selection practices; (3) the field of personality research is healthy, as new measurement methods, personality constructs, and compound constructs of well-known traits are being researched and applied to personnel selection.

Mount, M. K., Witt, L. A., & Barrick, M. R. (2000). Incremental validity of empirically keyed biodata scales over GMA and the five factor personality constructs. Personnel Psychology, 53(2), 299-323.

Sought to determine whether empirically keyed, cross-validated biodata scales account for incremental variance over that accounted for by the five factor model (FFM) of personality and cognitive ability predictors. A concurrent validation study was employed using 376 employees in a clerical job (222 in the developmental sample and 146 in the cross-validation sample). Building on the F. A. Mael and A. C. Hirsch (1993) and M. A. McManus and M. L. Kelly (1999) studies, the authors examined the joint use of cognitive ability, biodata, and personality as predictors of four different criteria: quantity and quality of work, problem solving, interpersonal facilitation, and retention probability. Results for the cross-validation sample provided support for the hypothesis that biodata predictors accounted for substantial incremental variance beyond that accounted for by the FFM predictors and cognitive ability for three of the four criteria. Support was also found for the hypothesized zero-order correlations between cognitive ability, FFM, and biodata predictors and the four criteria.

Rothstein, H. R., Schmidt, F. L., Erwin, F. W., Owens, W. A., & Sparks, C. P. (1990). Biographical data in employment selection: Can validities be made generalizable? Journal of Applied Psychology, 75(2), 175-184.

The hypothesis was examined that organizational specificity of biodata validity results from the methods typically used to select and key items. In this study, items were initially screened for job relevance, keying was based on large samples from multiple organizations, and items were retained only if they showed validity across organizations. Cross-validation was performed on approximately 11,000 first-line supervisors in 79 organizations. The resulting validities were meta-analyzed across organizations, age levels, sex, and levels of education, supervisory experience, and company tenure. In all cases, validities were generalizable. Validities were also stable across time and did not appear to stem from measurement of knowledge, skills, or abilities acquired through job experience. Finally, these results provide additional evidence against the hypothesis of situational specificity of validities, the first large-sample evidence in a noncognitive domain.

Schmitt, N., Cortina, J. M., Ingerick, M. J., & Wiechmann, D. (2003). Personnel selection and employee performance. Handbook of Psychology: Industrial and Organizational Psychology, 12, 77-105. New York, NY: John Wiley & Sons, Inc.

The authors of this chapter suggest personnel selection research has clearly expanded from its early interest in documenting predictor-criterion relationships. They discuss progress made in considering a broader range of predictors, testing more sophisticated performance models, and in paying more attention to the social significance of personnel selection and the reactions of examinees. Biodata measures are discussed as part of this new trend in personnel selection and employee performance.

Cognitive Ability Tests References:

Hunter, J. E. (1986). Cognitive ability, cognitive aptitude, job knowledge, and job performance. Journal of Vocational Behavior, 29(3), 340-362.

A research review indicates general cognitive ability (GCA) predicts supervisor ratings and training success as well as objective, rigorously content-valid work sample performance. Analyses carried out in several previous studies by the present author showed much of this predictive power stemmed from the fact GCA predicted job knowledge and job knowledge predicted job performance. However, GCA predicted performance to a greater extent, verifying job analyses showing most major cognitive skills are used in everyday work. Evidence showing GCA and not specific cognitive aptitudes predict performance is discussed. Findings support classic learning theory over behaviorist theories of learning and performance.

Murphy, K. R., Cronin, B. E., & Tam, A. P. (2003). Controversy and consensus regarding the use of cognitive ability testing in organizations. Journal of Applied Psychology, 88(4), 660-671.

Over 700 members of the Society for Industrial and Organizational Psychology indicated agreement or disagreement with 49 propositions regarding cognitive ability tests in organizations. There was consensus that cognitive ability tests are valid and fair, they provide good but incomplete measures, different abilities are necessary for different jobs, and diversity is valuable. Items dealing with the unique status of cognitive ability were most likely to generate polarized opinions. The data represented two factors: (1) societal concerns over the consequences of ability testing and (2) emphasis on the unique status of cognitive ability.

Outtz, J. L. (2002). The role of cognitive ability tests in employment selection. Human Performance, 15(1-2), 161-172.

Cognitive ability tests correlate with measures of job performance across many jobs. However, cognitive ability tests produce racial differences three to five times larger than other predictors--such as biodata, personality inventories, and the structured interview--that are valid predictors of job performance. Given (a) cognitive ability tests can be combined with other predictors such that adverse impact is reduced while overall validity is increased, and (b) alternative predictors with less adverse impact can produce validity coefficients comparable to those obtained with cognitive ability tests alone, sole reliance on cognitive ability tests when alternatives are available is unwarranted.

Ree, M. J., Earles, J. A., & Teachout, M. S. (1994). Predicting job performance: Not much more than g. Journal of Applied Psychology, 79(4), 518-524.

The roles of general cognitive ability and specific abilities or knowledge were investigated as predictors of work sample job performance criteria in seven jobs for U.S. Air Force enlistees. The interaction of general cognitive ability and specific experience were defined by scores on the first and subsequent principal components of the enlistment selection and classification test (the Armed Services Vocational Aptitude Battery). Analyses revealed cognitive ability was the best predictor of all criteria and specific abilities or knowledge added a statistically significant but practically small amount to predictive efficiency. These results are consistent with those of previous studies, most notably Army Project A. The study also extends the findings to other jobs and uses traditionally more acceptable estimates of cognitive ability.

Schmidt, F. L., & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality & Social Psychology, 86(1), 162-173.

The psychological construct of general mental ability (GMA), otherwise known as cognitive ability, introduced by C. Spearman (1904) nearly 100 years ago, has enjoyed a resurgence of interest and attention in recent decades. This article presents research evidence GMA predicts both occupational level attained and performance within one's chosen occupation and does so better than any other ability, trait, or disposition, and better than job experience. The sizes of these relationships with GMA are also larger than most found in psychological research. Evidence is presented that weighted combinations of specific aptitudes tailored to individual jobs do not predict job performance better than GMA alone, disconfirming specific aptitude theory. A theory of job performance is described explaining the central role of GMA in the world of work.

Emotional Intelligence Tests References:

Brackett, M. A., Rivers, S. E., Shiffman, S., Lerner, N., & Salovey, P. (2006). Relating emotional abilities to social functioning: A comparison of self-report and performance measures of emotional intelligence. Journal of Personality and Social Psychology, 91, 780-795.

Two distinct approaches to measuring emotional intelligence (EI) have emerged: ability-based (performance) tests and self-report questionnaires. The purpose of this study was to explore the relationship between these two different measures of EI and whether one approach is better than the other at predicting an important outcome measure (namely, social competence). The findings indicated ability-based measures and self-rated EI were not strongly related, suggesting a self-assessment of one's own EI may not constitute an accurate measure of EI. These findings were compared to other research showing individuals are also bad judges of their own mental abilities. In a follow-up study, the researchers examined the extent to which self-rated and ability-based measures of EI were able to predict a key outcome variable, social competence. The results showed women score higher than men on ability-based measures of EI and self-rated measures came out about the same for the two groups. The self-report measures of EI did not predict social competence for either men or women. The ability-based measure of EI predicted social competence scores for men but not women. Possible explanations for the gender differences were considered.

Frost, D. E. (2004). The psychological assessment of emotional intelligence. In J. C. Thomas & M. Hersen (Eds.), Comprehensive handbook of psychological assessment, Volume 4: Industrial and organizational psychology (pp. 203-215). Hoboken, NJ: John Wiley & Sons.

The author first describes the growing public and scientific interest in the concept of emotional intelligence (EI) and the historical and social context behind EI's recent emergence. Debates concerning the proper definition of EI are considered and whether the concept is little more than a renaming of personality variables as aspects of traditional intelligence (i.e., IQ). The author concludes most researchers and practitioners see the psychological construct of EI as being made up of distinct mental abilities or capacities. The rest of the chapter describes attempts to measure EI and its many workplace applications (e.g., selection, placement, and training). The author also considers what types of occupations would be most suitable for EI assessment.

Mayer, J. D., Salovey, P., & Caruso, D. R. (2004). Emotional intelligence: Theory, findings, and implications. Psychological Inquiry, 15, 197-215.

This article provides a detailed and comprehensive review of findings regarding emotional intelligence (EI) measured as an ability. The first major conclusion is EI is distinct from other commonly measured variables related to personality and intelligence. Evidence for this is based on studies finding very low correlations between EI and other types of intelligence. Also, EI has rather low correlations with measures of social and emotional traits. A growing body of evidence indicates EI, measured as an ability, predicts a variety of important outcomes. For example, studies have shown people higher in EI are more likely to avoid drug problems, interpersonal arguments, and violence. Such individuals are also more satisfied with their social networks and appear to receive more social support. These predictive relationships are at levels typically found for similarly complex concepts (e.g., personality variables). The authors argue that because high EI involves the more successful resolution of personal conflict and lower levels of aggression, it is a highly desirable, and often personally valuable, attribute to possess.

Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition, and Personality, 9, 185-211.

Introduces the original framework for the emotional intelligence concept, a set of skills hypothesized to contribute to the accurate appraisal and expression of emotion, the effective regulation of emotion, and the use of feelings to motivate, plan, and achieve. Adaptive versus maladaptive qualities of emotion are discussed. The literature on intelligence, and especially social intelligence, is reviewed to examine the place of emotion in traditional intelligence conceptions. The article also provides a framework for integrating the diverse research on emotion-related skills.

Integrity/Honesty Tests References:

Cullen, M. J., & Sackett, P. R. (2004). Integrity testing in the workplace. In J. C. Thomas & M. Hersen (Eds.), Comprehensive handbook of psychological assessment, Volume 4: Industrial and organizational psychology (pp. 149-165). Hoboken, NJ: John Wiley & Sons.

This chapter provides a comprehensive summary of integrity testing research and practice. Topic areas include a review of validity evidence for personality-oriented and overt integrity tests, the relation between integrity tests and other assessments (e.g., cognitive ability tests, standard personality factors), the effect of faking on integrity test scores, applicant privacy right issues, and subgroup score differences by race and gender. The chapter's central conclusions are integrity tests: (1) show consistent evidence of validity for predicting counterproductive behaviors, as well as overall job performance, (2) do not show subgroup differences by gender, race, or ethnicity, (3) have some overlap with standard personality factors (i.e., the Big Five) but also appear to measure other unique factors, and (4) are subject to faking by applicants but response distortion (i.e., providing socially desirable answers) does not seem to undermine the usefulness of the tests.

Ones, D. S., Viswesvaran, C., & Schmidt, F. L. (1993). Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679-703.

The authors conducted a comprehensive meta-analysis that quantitatively summarized the results of multiple validity studies. Generally, the researchers found integrity tests were useful predictors of overall job performance and counterproductive behaviors on the job, such as theft, disciplinary problems, and absenteeism. Both personality-based and overt (i.e., clear purpose) integrity tests proved to be valid predictors of job performance. Results from predictive validity studies conducted on applicants using external performance measures (i.e., excluding self-reports of counterproductive behavior) indicate integrity tests predict an overall measure of counterproductivity (a combination of violence on the job, absenteeism, tardiness, and other disruptive behaviors) better than they predict employee theft alone.

Sackett, P. R., & Wanek, J. E. (1996). New developments in the use of measures of honesty, integrity, conscientiousness, dependability, trustworthiness and reliability for personnel selection. Personnel Psychology, 49(4), 787-829.

This article provides a fairly extensive summary of issues related to the use of integrity tests for personnel selection. The authors conclude personality-based and overt (clear purpose) integrity tests are valid predictors of counterproductive behaviors (admitted theft, dismissals based on theft, various illegal activities, tardiness, absenteeism, and on-the-job violence). Both types of tests also predict measures of overall job proficiency for a wide variety of jobs. Limiting the predictive analysis to only applicants (as opposed to current employees) and actual detection of theft lowers the validity substantially. Integrity test scores are related to some of the Big Five measures of personality, particularly conscientiousness, agreeableness, and emotional stability. Integrity tests do not demonstrate adverse impact against women or minorities and are not generally perceived as negative by applicants. Integrity tests are relatively unrelated to cognitive ability (i.e., intelligence), implying they can be used in conjunction with cognitive ability measures to enhance selection quality while also reducing the adverse impact associated with cognitive ability tests.