Development and validation of indices for detecting careless responding in multidimensional forced-choice questionnaires

  • As an alternative to rating scales, in the multidimensional forced-choice (MFC) format, participants are asked to rank items according to how well the items describe them. However, researchers have not yet determined how to detect careless responding in MFC questionnaires. Careless responding is a response behavior where participants provide an answer without considering part (or any) of the content of the items (e.g., Nichols, 1989). This dissertation provides a thorough investigation of careless responding in the MFC format. We aimed to develop indices that could be used to detect careless responding in MFC questionnaires and that would be suitable for differentiating between thoughtful and careless responses and investigating their validity. To achieve these ends, we conducted three studies. In Study 1, we developed indices for detecting careless responding in MFC questionnaires and examined their performance and intercorrelations. We conducted an online study with N = 1,169 participants who completed a long survey that included a series of personality questionnaires in the MFC format. We applied the newly developed indices to this data set and computed several analyses to explore their performance and to gain initial insights into careless responding in MFC questionnaires. The correlations among the MFC indices were somewhat lower than those reported in previous studies that used indices to detect careless responding in rating scales. A latent profile analysis indicated that the majority of participants in the online study, approximately 76% to 84%, did not engage in careless responding. However, those who did respond carelessly displayed different levels of careless responding. In addition, we conducted a simulation study to investigate whether the indices worked as conceptually intended. In this study, we simulated three careless responding patterns and different proportions of careless respondents in the sample. With one exception, the indices performed as conceptually intended. In Study 2, we evaluated the newly developed indices under conditions that are assumed to be related to careless response behaviors, and we explored strategies for responding to MFC questionnaires when feeling tired, listless, or unsure about how to rank items. For this purpose, we conducted a laboratory study including N = 430 participants. We manipulated careless responding and compared its manifestation on 11 careless responding measures. Participants were assigned to one of three conditions: a control condition or two careless responding conditions in which we either instructed the participants to respond carelessly or distracted them to induce careless responding. The survey consisted of five inventories for assessing personality traits in the MFC format and concluded with an oral interview in which we asked participants questions about the response process. We found significant differences between the control condition and the instructed careless responding condition on all of the indices except for survey completion time. However, the control condition and the distracted responding condition were not significantly different from each other on most indices. In the interview, commonly reported strategies when feeling unsure, tired, or listless were reading the items superficially, thinking less about the item content, choosing a random order, or copying the presented order. In Study 3, we thoroughly investigated the construct validity of the indices for the MFC format in an online study that included three subsamples and a total of N = 2,163 participants. We examined the convergent, discriminant, and criterion-related validity of the indices for the MFC format. A total of n = 1,513 participants completed a self-report survey with several personality questionnaires in the rating scale (RS) and MFC formats and several criteria, such as the motivation to participate, an intelligence measure, and educational level. Moreover, we assessed n = 650 friends or partners of the self-report participants who completed an other-report questionnaire in the RS format about their friend’s or partner’s personality. Except for one index, convergent validity was established for all indices in the MFC format. Moreover, three indices showed higher correlations with their counterparts in the RS format than with other indices in the MFC or RS formats, reflecting discriminant validity. Results for criterion-related validity were mixed. Whereas no support was found for hypotheses on personality traits and educational level, self-reported test motivation and the intelligence measure were related to the indices in the MFC format as hypothesized. The latent profile analysis in Study 3 yielded slightly different results compared with Study 1, identifying one careless class that comprised 13% of the sample. A comparison of the proportions of careless respondents identified by the indices in the RS and MFC formats showed smaller proportions of careless respondents in the MFC format on three indices, whereas no differences were found on three other indices. Taken together, the three studies suggested that careless responding plays an important role in MFC questionnaires and thus should be addressed not only in RS questionnaires but also in the MFC format. The validated indices are publicly available and can easily be applied to questionnaires in the MFC format using triplets. Moreover, the results informed the definition of careless responding in MFC questionnaires.

Download full text files

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Rebekka KupfferORCiD
URN:urn:nbn:de:hbz:386-kluedo-86484
DOI:https://doi.org/10.26204/KLUEDO/8648
Advisor:Eunike WetzelORCiD, Thorsten Meiser
Document Type:Doctoral Thesis
Cumulative document:No
Language of publication:English
Date of Publication (online):2025/01/23
Date of first Publication:2025/01/27
Publishing Institution:Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau
Granting Institution:Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau
Acceptance Date of the Thesis:2025/01/13
Date of the Publication (Server):2025/01/27
Page Number:XI, 157 Seiten
Faculties / Organisational entities:Landau - Fachbereich Psychologie
DDC-Cassification:1 Philosophie und Psychologie / 150 Psychologie
Licence (German):Creative Commons 4.0 - Namensnennung (CC BY 4.0)