Introduction

Test security continues to be a growing problem in the assessment industry - across all sectors - and data forensics to find security issues remains a nascent topic.  This presentation will seek both to educate newcomers on the topic as well as provide comparative research on existing approaches.

Objectives

This presentation will provide an overview the security threats that we face, such as collusion, teacher/proctor help, item harvesting, and preknowledge.  It will then match forensic approaches that fit well to investigating these.  For example, preknowledge can be indicated by unusual time/response patterns (examinees spend an unusually short time while responding correctly) as well as collusion indices (examinees will provide a similar response string to other examinees that used the same brain dump site).  We will also discuss classification and nomenclature of forensic methods, contrasting how they can be used at the intra-individual, inter-individual, and aggregate levels.

Design

We will demonstrate use of data forensics on both a real data set and manufactured data set.  The real data set is higher-fidelity but of course does not have known issues.  The manufactured data set is constructed to have known issues and can therefore be used to evaluate whether the forensics can find problems that we know are there.  It will have examples of student collusion, teacher/proctor help, item harvesting, and preknowledge.

Results

Results will include comparison of eight different collusion indices (e.g., g2 and omega), response time indices (e.g., Response Time Effort), joint time/response indices (e.g., bivariate distribution), and aggregate indices (e.g., standardized pass rates and test times).

Conclusions

Unsurprisingly, results indicate that some collusion indices are better than others.  Results are mixed with intra-individual and aggregate analyses, but they are still far better than not doing data forensics at all.  It is therefore a vital need of the assessment field to have more organizations utilizing this approach.