Introduction - This symposium will discuss assessments and measurements of behaviors and other noncognitive factors as it relates to college and career success.  Research has consistently shown the efficacy of measuring noncognitive factors overall, and the incremental value added when combined with cognitive assessments in the workplace.  Similar research has begun to emerge in educational research as well.  Yet, many organizations and policymakers nationally and globally focus predominantly on academic skills for accountability, international comparisons, and casual studies of college and career success.  This symposium will review efforts to define, develop, and implement rigorous assessments of behavior and noncognitive factors in a variety of contexts, as well as the current state of such measures.

Contributions - Three papers which review the state of such noncognitive assessments across college and career readiness comprise the symposia.  The first paper reports on over a decade of research by ACT in developing noncognitive measures which are used to supplement the admissions tests in identifying students on track to be successful in college.  The second paper provides a real world example of how to identify and define such constructs for a specific career domain represented by culinary arts.  The third paper focuses on measurement invariance across countries as it relates to the assessment of noncognitive skills.  This study focuses on the assessment of the Big 5 personailty traits.

Conclusions - This symposium highlights the importance of noncognitive measures in the prediction of college and career readiness and success while also addressing important measurement-related issues associated with the assessments of these types of factors.

*****

Paper 1 - The need for a Holistic Model of Educational Success

Wayne Camara, ACT, USA

Krista Mattern, ACT, USA

Introduction - Ensuring that all high school students graduate ready for postsecondary success, has become a priority in the United States as research illustrates that 30-40% of entering college students require remediation and U.S. enrollment has stagnated while other developed countries have increased (OECD, 2014).  Standards-based reform has emphasized college readiness in K-12 assessments, common core state standards, and international benchmarks.  These efforts address the core academic skills required to transition from high school to college and work, but they fail to fully account for other types of skills and knowledge students may need to fully prepare for education and work.  Behaviors, interests, and generalized cognitive competencies include important tasks such as choosing a college and major persisting through degree completion, and choosing a career that fits well with abilities, interests, and goals.  Many students flounder at these important milestones, in spite of their academic preparation, often resulting in poor decisions and the derailment of their goals and aspirations.

Objectives - The goal of this research was to identify the most important factors related to success in school and work in order to develop a holistic model of college and career readiness that encompasses additional domains beyond core academic skills.

Methods - This paper is based on a review of empirical and theoretical reseaarch related to college and career readiness.  The results of the research led to the development of a holistic model of college and career readiness and the extent that validity eveidence for such noncognitive constructs and measures can be transported across counties and cultures.

Results - Academic and cognitive skills have traditionally been viewed as the sole indicator of college readiness, but research supports the importance of noncognitive factors such as personality, career interests, and self-beliefs.  Our framework of CCR incluedes skills in four domains: 1. Core academic skills. 2. Cross-cutting capabilities such as critical thinking and collaborative problem solving, and technology skills. 3. Behavioral skills such as dependabiility, working effectively with others, adapting, and managing stress. and 4. Education and career skills including self-knowledge of abilities, interests, values, and knowledge about majors and occupations.

Conclusions - The ultimate goal was to build and validate a holistic model of CCR based on research and to validate components of the model in terms of specific criteria or outcomes related to postsecondary success that is more personalized and developmentally appropriate.

Paper 2 - Taking the Heat in the Kitchen: Assessing Culinary Students' Noncognitive Skills

Gabriel Olaru, Center for Innovative Assessments, USA

David Livert, Professional Examination SErvice, USA

Kathleen Merget, The Culinary Institute of America, USA

Jeremy Burrus, Professional Examination Services, USA

Richard D. Roberts, Professional Examination Services, USA

Introduction - Preparing food in kitchens, bakeries, or restaurants is an emotionally laden, complex event, likely to require the management of emotions, high levels of work ethic and interpersonal skills, and a propensity to cope with stress.  This is certainly vindicated in the US Department of Labor's Occupational Information Network, where dependability, stress tolerance, and cooperation are valued most highly by both employers and employees in this job sector.

Objectives - Despite the preceding, this population has been little studied.  The purpose of this investigation was to bridge this gap and first examine the psychometric properties of a variety of noncognitive measures, then their relationahip to meaningful outcomes, using students from the Culinary Institute of America (CIA).

Methods - In two studies, a total of 301 first and second semester CIA students (152 males; Mean age=21.3 years, SD=5 years) completed a battery of noncognitive assessments measureing the aforementioned constructs, as well as allowing access to externship and course data.  The battery included measure of emothional intelligence (i.e., a contextualized version of the Situational Test of Emotional Management [STEM]), personality (i.e., the Big Five Inventory [BFI]), and coping with stress (i.e., the Emotional Reactions to CIA Inventory).

Results - Compared to a sample of over 132,000 people drawn from across the globe, CIA students reported higher levels of Conscientiousness, Agreeableness, and Emotional Stability as assessed by the BFI.  The effect sizes were medium.  By contrast, the average probability of CIA students correctly solving the STEM was 50%, indicating that the BFI results may be somewhat confounded with social desirability.  Although a measure created for this study, the contextualized STEM was related to personality, coping with stress and gender in theoretically meaningful ways.

Conclusions - This disjunction between what typical (i.e., BFI) and maximum (i.e., STEM) performance measures yield, bears further scrutiny.  We continue to track these students along their CIA trajectory, into the workforce to ascertain which of these measures better predict success in the food preparation industry.

Paper 3 - Challenging the Measurement Invariance Approach

Daniel Danner, GESIS - Leibniz Institute for the Social Sciences, Germany

Beatrice RAmmstedt, Department Survey Design and Methodology at GESIS and University of Mannheim, Germany

Introduction - Measurement invariance of scales across countries can be tested via multi-group structural equation models.  This approach focuses on the internal item structure and tests whether factor loadings are identical across groups.  However, different factor loadings may not necessarily reflect the lack of comparability of a scale but also idiosyncratic structures of the constructs itself.

Objectives - We propose that a lack of measurement invariance is only problematic if it biases the association between scale scores and external criteria.  Hence, we suggest combining testing measurement invariance with testing the association with external criteria.

Methods - We use a short version of the Big Five Inventory that was used in the International Social Survey Programme (25,912 respondents from 19 countries) to illustrate this approach.  Measurement invariance was tested with multi-group structural equation models.  The associations with external criteria were investigated with multi-level regression models.

Results - Results suggests that (1) the factor loadings of the Big Five items differ between countries, (2) the associations between the Big Five, education and income differ between countries, but (3) the differences in factor loadings do not bias the association with real life criteria.

Conclusions - The present results challenge the relevance of measurement invatiance tests and suggest not only focusing the internal item structure but also considering the association with external criteria. Theoretical assumptions and practical limitations are discussed.

Discussant - Nancy Tippins, Senior Vice President, CEB-SHL Talent Measurement Solutions, USA, ntippins@executiveboard.com