Open Conference Systems, ITC 2016 Conference

Font Size: 
PAPER: Enhancing our Interpretation of Sources of DIF in Post-Secondary International Surveys
Stephanie McKeown, Maria Elena Oliveri, Debra (Dallie) Sandilands

Building: Pinnacle
Room: 3F-Port of San Francisco
Date: 2016-07-03 11:00 AM – 12:30 PM
Last modified: 2016-05-21

Abstract


Along with the growing number of international students enrolling in Canadian institutions is an increased demand for information about the international student experiences in post-secondary.  Such data is typically collected using institutional surveys where results are compared across domestic and international students to better understand the student experiences.  Valid interpretations of group comparisons are dependent on ensuring that the surveyed students understand the questions in comparable ways and that sources of construct-irrelevant variance (e.g., due to differential interpretation of terms) are minimized.  Thus, quantitative approaches are used to determine if differential item functioning (DIF) exists on item-level endorsement across groups who otherwise have similar attitudes on the overall trait.  While quantitative approaches flag DIF items that function differentially, qualitative analyses are conducted to elucidate its construct-relevant or irrelevant sources.  Traditionally, content experts review survey items prior to its administration and revisit DIF items post-survey administration.  Expert reviews can be enhanced through focus group discussions with survey respondents to identify qualitatively why DIF might occur.  In this presentation, we used both qualitative and quantitative approaches to analyzing survey data with domestic and international students at a large Canadian university.  We found that respondents suggested differential interpretations of the term “faculty†used in the survey.  Some international students interpreted “faculty†with a broad interpretation to include any individual within their Faculty, such as staff and students.  On the other hand, domestic students interpreted “faculty†narrower, which solely included their professors as faculty.  After identifying which terms were differentially endorsed, we conducted post hoc analyses across all survey items using analysis of variance and logistic regression.  We found differential endorsement for the flagged questions.  Results imply that qualitative analyses can greatly enhance our understanding of DIF sources and yield new insights into possible sources of differences that may go undetected by expert reviewers.


An account with this site is required in order to view papers. Click here to create an account.