Introduction

New technologies for administering psychometric instruments open up new opportunities for obtaining paradata. Paradata of a psychometric instrument are data describing the process of completing an instrument such as response times or clicks (Stieger & Reips, 2010). Thus, new technologies allow for tracking the candidate’s behaviour during the entire completion process.

Objectives

The purpose of the session is to give an overview of different opportunities of using paradata to assess data quality particularly in unsupervised online settings and to increase the reliability and validity of psychometric online tests and questionnaires. There will be an overview of various research findings, followed by a discussion of implications for research and practice of employment testing.

Design/Methodology

In Study 1 N = 100 students gave a true and a forged self-description on a questionnaire that assesses job-related competencies. In Study 2 we logged different positions of a slider on a visual analogue scale (first click and subsequent changes of slider positions) that was used in a questionnaire assessing integrity.

Results

Study 1: Scale and item variance increased significantly as an effect of intentional faking while total scale point distribution decreased and the pattern of response times changed. Study 2: Using the position of the slider after the first click instead of its final position for calculating reliability yielded higher internal consistency of the instrument.

Conclusions

It is possible to assess and improve data quality and thus validity using paradata. However, using them raises a number of questions such as: What do the instructions have to explain to candidates on what kind of data is collected about the completion of an instrument? Which effect will the candidate’s awareness of their behaviour being tracked have on their actual test performance?