To investigate the effect of using negatively oriented items, we wrote semantic reversals of the items in the Rosenberg Self Esteem Scale, UCLA Loneliness Scale, and the General Belongingness Scale and used them to create four experimental conditions. Participants (N = 2,019) were recruited through Amazon's Mechanical Turk. Data were assessed for dimensionality, item functioning, instrument properties, and associations with other variables. Regarding dimensionality, although a two-factor model (positively vs. negatively oriented factors) exhibits better fit than a unidimensional model across all conditions, bifactor indices were used to argue that a unidimensional interpretation of the data can be employed. With respect to item functioning, factor loadings were found to be nearly invariant across conditions, but thresholds were not. Concerning instrument properties, inclusion of negatively oriented items results in lower mean scores and higher score variances. Instruments with both positively and negatively oriented items demonstrated lower reliability estimates than those with only one orientation. For associations with other variables, path coefficients in a model where loneliness mediates the effects of belongingness on life satisfaction and self-esteem were found to vary across conditions. Findings suggest that negatively oriented items have minor impact on instrument quality, but influence measurement model and path coefficients.
Keywords: bifactor; confirmatory factor analysis, CFA; dimensionality; item orientation; item wording; measurement invariance; psychological measures.