Reviewers of papers in business research often reject papers that use a single method for data collection, arguing that the results cannot be trusted on account of Common Method Variance (CMV). This idea is inspired by Campbell and Fiske (1959). Recently, management journals ORM among others devoted particular attention to this problem. In that issue, it transpires that it is not so clear what CMV is and how great its effect is. In this context we will report on a within-subject, split-ballot multi-trait multi-method experimental design for estimating CMV for a specific method (length of response scale) and the effect of a common data collection method (self-reporting of behavior) related to Emotional and Social Competencies.
Results show that there is indeed CMV, but the effect of this common variance is much smaller than the effect of the random errors. The design of this study allows some nuances to the Lance et al (2010) conclusions of that these two effects cancel each other out. This study agrees with previous research suggesting that random errors should receive at least as much or even more reviewer attention than CMV.