Assessing the visual Q method online research tool: A usability, reliability, and methods agreement analysis

Abstract

The traditional paper-based Q method was introduced to the social sciences in the 1930s. However, despite its unique capability for measuring peoples’ subjective opinions, the method has not been broadly employed by researchers. Moreover, despite recent efforts to administer the Q method surveys via the Internet, they have been mostly limited to the usage of textual prompts. Besides, there is no concrete evidence on the usability, reliability, and agreement analysis between the online Q method research tools and the traditional paper-based Q method. Therefore, the visual Q method online research tool was developed to resolve these deficiencies and add new dimensions (audio and video) to the Q method research. The development of the visual Q method platform was a three-stage process that commenced with a usability test on 31 visitors to a local museum. Second, a test–retest reliability analysis with a convenient sample of 37 students resulted in a high reliability coefficient superior to that of the traditional paper-based Q method. Third, the analysis of agreement between the visual Q method and the traditional paper-based Q method on a sample of 10 students revealed substantial similarities between their generated (V)Q sorts. Overall, the study findings provide substantial evidence on the usability and rigor of the visual Q method online research tool.