Statistical detection and evaluation of response styles has been the main focus of research aimed to improving validity of cross-cultural comparisons. There is a increasing consensus about the need of integrating quantitative and qualitative methods in order to understand how response processes, context and cultural factors interact, and provoke cross-cultural differences in response styles. The symposium gather innovative quantitative and qualitative researches that can illustrate benefits of a mixed approach to analyzing and understanding effects of response styles on measurement equivalence.

The collection of four papers will address the following general research questions:

1. How can response processes, context and cultural factors interact to cause response styles bias in cross-cultural research?
2. To what extent statistical methods can identify and adjust date for response styles within and across cultures?

The two first contributions to the symposium present a comprehensive mixed approach to detecting and understanding response styles bias in international survey research projects. The first one integrates quantitative indexes of extreme response styles derived from Quality of Life measures, with cognitive interviewing evidence of differences in response processes between Spanish and Dutch respondents. The second paper investigates response styles with factual items from the PIAAC project in 21 countries and explores the associations of response styles with individual, situational, and cultural factors. The last two papers address innovative developments of quantitative methods: the mixed Item Response Theory (IRT) Partial Credit Model (PCM), and Latent Class Factor Analysis, to identify response styles effects and how they can come from cultural factors.

The presenters will encourage discussions on how to improve equivalence and validity in cross-cultural research.

*****

Paper 1: Linking Extreme Response Style to Response Processes: A Cross-Cultural Mixed Methods Approach
Isabel Benítez, Jia He, Fons A. R. van Vijver, and Jose-Luis Padilla (presenter)

Introduction

The importance of evaluating bias for assuring valid cross-cultural comparisons has been clearly settled in the last years. One important source of method bias is response styles, which are defined as the systematic tendency to use response options on some basis other than the target constructs. The main focus of response style research has been statistical detection, whereas the cognitive, context, and cultural factors underlying the presence of response styles have not been systematically studied.

Objectives

The aim of this study is to contribute to a better understanding of extreme response style in cross-cultural research by integrating quantitative and qualitative evidence in a mixed methods design.

Design/Methodology

In the quantitative phase, indexes of extreme response style, derived from Quality of Life measures from different international studies, were compared between Spain and the Netherlands. In the qualitative phase, Cognitive Interviewing was conducted involving 25 participants in each country.

Results

Quantitative results indicated that extreme responding was more common among Spanish than among Dutch in endorsement of items, but that the opposite was found for frequency scales including never as an anchor. The integration of quantitative results and qualitative findings suggests that country differences in extreme response style may stem from various sources, such as the more independent evaluation of each item by Dutch or the stronger connotations of never for Spanish, and stronger emotions triggered by specific topics such as work satisfaction that was more strongly associated with insecurity for Spanish.

Conclusions

It is concluded that the integration of quantitative and qualitative evidence can help to understand cross-cultural similarities and differences in extreme response style.

 

Paper 2: Response Styles: Individual, Situational, and Cultural Correlates
Jia He, Fons J. R. van Vijver (presenter)

Objective

This study investigates response styles with factual items in 21 countries and explores the associations of response styles with individual, situational, and cultural factors.

Methods

Responses on various factual questions, cognitive tests, and interviewers’ observational data during the survey responding from a total of 152,514 respondents in 21 countries in the Programme for the International Assessment of Adult Competencies (PIAAC) were analyzed.

Results

We first extracted the general response style with a positive loading of extremity and negative loadings of midpoint responding and acquiescence at both individual and country level. In a multilevel analysis, the individual level response style was predicted by literacy competency, education being male, age cohort, third person presence, background noise at individual level, and country affluence at cultural level.

Conclusions

We confirmed the general response style from factual items, and here it is also interpreted as culturally moderately communication filters. Implications on how to perceiving and dealing with response styles in cross-cultural surveys are discussed.

 

Paper 3: The Accuracy of the Partial Credit Model for Identifying and Adjusting Data for Response Styles
Bruce Austin (presenter), and Brian French

Introduction

Cultural response styles (RS) are a source of error that result from how individuals interpret survey response scales. These responses are independent of the construct measured. The comparison of groups with different RS can lead to erroneous results and faulty conclusions. Adjustments to survey data for RS can remove this error. The mixed Item Response Theory (IRT) Partial Credit Model (PCM) is promising for the identification and adjustment of data for RS. Unfortunately, empirical analysis of the accuracy of this procedure is lacking.

Objectives

This study examines the behavior of the mixed IRT PCM for identifying and adjusting data for RS within and across cultures. The accuracy of the model is examined under different conditions that reflect real data.

Design/Methodology

In this simulation study, sample size is focal and manipulatedwith 3 levels (i.e., 600, 2000, 10,000)while maintaining a set number of RS and a consistent presentation across scales. The number of latent RSis four and include whole scale responding, acquiescence, extreme responding, and mid-scale responding. Likewise, the consistency with which respondents exhibit a RS across eight content domain scales is set at six. Regression models are estimated using adjusted and unadjusted data. Outcome variables focus on (a)the proportion of respondents correctly classified into latent response style class, and (b) the accuracyof a regression model (e.g., effect sizes; standard errors) estimated using RS adjusted and unadjusted data.

Results

The results will be available in March, 2016. Based on our past work, we expect more accuracy with adjusted regression models for RS and accurate identification of RS across content domains.

Conclusions

RS adjustment methods that are easy to implement and interpret can improve the accuracy of cross-cultural comparisons. This study will demonstrate how the IRT PCM can assist in fair comparisons.

 

Paper 4: The effects of extreme responding and individualism on measurement equivalence
Meike Morren, Jose-Luis Padilla, Isabel Benítez, Irene Gómez, and Ana María Hernández (presenter)

Introduction

Surveys are useful in studying attitudes and opinions across various cultural settings. One important requirement for interpreting these differences substantially is that the attitudes are equivalently measured across cultures. Response styles can threat the comparability of attitudes measures across cultural settings.

Objective

The aim of this paper is to provide evidence of the relations between culture factors, extrem response style and measurement equivalence. In the quantitative phase of the research project, we explore the measurement equivalence of two scales across three countries differing on Hofstede's value of individualism (Hofstede, 2001): Mexico (30), Spain (51) and the Netherlands (80). We expect that individualist countries (those scoring high on individualism, such as the Netherlands) are more likely to use the endpoints than countries scoring low on individualism (e.g. Mexico).

Design/Methodology

Two weakly related multi-item survey questions included in the World Value Survey intended to measure emancipation and religious attitudes are analyzed. Extreme response style is included in the model by a separate latent factor relating to both scales. We use the Latent Class Factor Analysis (LCFA) method to a) estimate the response style, b) assess the impact of the response style on the substantive measurement of emancipation and religious attitudes, and c) simultaneously test for measurement equivalence.

Results

The results show that extreme response style improves model fit, religion is metric inequivalent while emancipation is metric equivalent, and controlling for ERS does not lead to higher levels of equivalence. Additionally, we observe in each country that emancipation and religion are multidimensional as some items load very weakly on the factor.

Conclusions

Controlling for extreme response style should lead to higher levels of equivalence between the three nations. In the qualitative phase, Cognitive Interviewing methodology will be applied to obtain evidence of how individualism could explain country difference in extreme response style.