Open Conference Systems, ITC 2016 Conference

Font Size: 
POSTER: Concurrent Construction Designs may not Overestimate the Predictive Validity of Situational Judgment Tests (SJTs)
Mats Englund

Building: Pinnacle
Room: 2F-Harbourside Ballroom
Date: 2016-07-03 11:00 AM – 12:30 PM
Last modified: 2016-05-21

Abstract


Screening candidates with situational judgment tests (SJTs) is becoming increasingly popular among companies worldwide. Attractive aspects of using SJTs include effective corporate branding and realistic job previews for candidates while maintaining high predictive validity. A common method to establish which items to use involves a validation study and picking the most predictive items from an item pool. Using this method, capitalization on chance may lead to overestimation of the predictive validity, potentially making concurrent designs inaccurate for estimating true predictive validity. However, in practical cases, data is often collected using classic concurrent validation study designs, where participants already may have learned on the job what behaviors are desirable. This leads to the risk of restriction of range, attributable to inside knowledge rather than true predictive ability/potential, which leads to underestimation of the predictive validity, opposing the overestimation mentioned above. Here, results are presented from the development and validation of an SJT for service positions in a large Swedish organization. Comparisons of results from the construction of the SJT (n=800, concurrent design) with results from high-stakes testing (n=550, predictive design), and a supplementary student-sample study (n=128), suggest that both capitalization-on-chance overestimation and incumbent-knowledge-related restriction-of-range underestimation of the predictive validity were present. Mental ability scores predicted SJT scores better for students, suggesting SJT restriction of range for employees was attributable to learned desired behavior, and estimated predictive validity proved close to the true predictive validity. These results suggest one reason using a concurrent designs may not always overestimate SJT predictive validity.


An account with this site is required in order to view papers. Click here to create an account.