Open Conference Systems, ITC 2016 Conference

Font Size: 
POSTER: Regression Methods to Study Differential Item Functioning In the Case of No Internal Matching Variable and Continuous Item Ratings
Michelle Chen, Wendy Lam, Bruno D. Zumbo

Building: Pinnacle
Room: 2F-Harbourside Ballroom
Date: 2016-07-03 11:00 AM – 12:30 PM
Last modified: 2016-06-08

Abstract


Many techniques and procedures have been developed to investigate differential item functioning (DIF: Zumbo, 2008); however, performance assessments, such as writing tests, often pose two unique challenges in DIF investigations: lack of an internal matching variable and a continuous rating score. It is difficult to find an appropriate internal matching variable for writing tests because of the small number of tasks within writing tests; therefore, typical DIF methods are not feasible. Building on work by Zumbo (2008), a method to test DIF for a continuously scored writing test with only two writing prompts on each test form is proposed and demonstrated with real test data. Data from a high stakes English proficiency test is used for this study. The overall test has four components: listening, reading, writing and speaking. Our focus is on gender DIF for the writing prompts. All test takers respond to two different writing prompts. Prompt-level scores for each test taker are on a continuous metric ranging from 0 to 12.3, with over 260 possible scores. A multiple matching strategy was applied. Given that the correlations among the four different components of the tests are fairly high (above 0.77), listening and reading scores were used as multiple matching variables to investigate writing prompt DIF. In light of the continuous prompt scores, linear regression models are used to test DIF. Each prompt was answered by at least 120 test takers from each gender group. A total of 56 writing raters were involved in rating these writing samples, with each sample rated by two to three raters. Three writing prompts out of 81 were flagged as DIF items with moderate magnitude. Details of the analysis procedures, decision rules to flag DIF items, and results from this DIF investigation will be presented in the final paper.


An account with this site is required in order to view papers. Click here to create an account.