Comparison of Preferences and Data Quality between Discrete Choice Experiments Conducted in Online and Face-to-Face Respondents

Ruixuan Jiang; Eleanor Pullenayegum; James W Shaw; Axel Mühlbacher; Todd A Lee; Surrey Walton; Thomas Kohlmann; Richard Norman; A Simon Pickard

doi:10.1177/0272989X231171912

Comparison of Preferences and Data Quality between Discrete Choice Experiments Conducted in Online and Face-to-Face Respondents

Med Decis Making. 2023 Aug;43(6):667-679. doi: 10.1177/0272989X231171912. Epub 2023 May 18.

Authors

Ruixuan Jiang¹, Eleanor Pullenayegum², James W Shaw³, Axel Mühlbacher⁴, Todd A Lee⁵, Surrey Walton⁵, Thomas Kohlmann⁶, Richard Norman⁷, A Simon Pickard⁵

Affiliations

¹ Center for Observational and Real-World Evidence, Merck & Co., Inc, Rahway, NJ, USA.
² Child Health Evaluative Sciences, Hospital for Sick Children, Toronto, Canada.
³ Patient-reported Outcomes Assessment, Bristol-Myers Squibb, Princeton, NJ, USA.
⁴ Duke Department of Population Health Sciences and Duke Global Health Institute, Duke University, Durham, NC, USA, Germany.
⁵ Department of Pharmacy Systems, Outcomes, and Policy, University of Illinois at Chicago College of Pharmacy, Chicago, IL, USA.
⁶ Institute for Community Medicine, Medical University Greifswald, Greifswald, Germany.
⁷ Curtin University School of Public Health, Perth, Australia.

Abstract

Introduction: Discrete choice experiments (DCE) are increasingly being conducted using online panels. However, the comparability of such DCE-based preferences to traditional modes of data collection (e.g., in-person) is not well established. In this study, supervised, face-to-face DCE was compared with its unsupervised, online facsimile on face validity, respondent behavior, and modeled preferences.

Methods: Data from face-to-face and online EQ-5D-5L health state valuation studies were compared, in which each used the same experimental design and quota sampling procedure. Respondents completed 7 binary DCE tasks comparing 2 EQ-5D-5L health states presented side by side (health states A and B). Data face validity was assessed by comparing preference patterns as a function of the severity difference between 2 health states within a task. The prevalence of potentially suspicious choice patterns (i.e., all As, all Bs, and alternating As/Bs) was compared between studies. Preference data were modeled using multinomial logit regression and compared based on dimensional contribution to overall scale and importance ranking of dimension-levels.

Results: One thousand five Online respondents and 1,099 face-to-face screened (F2F_S) respondents were included in the main comparison of DCE tasks. Online respondents reported more problems on all EQ-5D dimensions except for Mobility. The face validity of the data was similar between comparators. Online respondents had a greater prevalence of potentially suspicious DCE choice patterns ([Online]: 5.3% [F2F_S] 2.9%, P = 0.005). When modeled, the relative contribution of each EQ-5D dimension differed between modes of administration. Online respondents weighed Mobility more importantly and Anxiety/Depression less importantly.

Discussion: Although assessments of face validity were similar between Online and F2F_S, modeled preferences differed. Future analyses are needed to clarify whether differences are attributable to preference or data quality variation between modes of data collection.

Keywords: EQ-5D; discrete choice experiment; face-to-face; online.

MeSH terms

Choice Behavior
Data Accuracy
Health Status*
Humans
Quality of Life*
Surveys and Questionnaires