On being misquoted

Although I am not a statistician, through some quirk of Google’s search algorithm, it appears that I have become promoted to the status of a go-to internet expert on Likert scales. This is sometimes awkward, especially when a less-than-perfect blog post is cited in a peer-reviewed publication, but I can live with that. On the other hand, I tend to be somewhat more frustrated I come across work that misunderstands and misrepresents my writing.So, the purpose of this post is to set the record straight after one of these instances.

So what’s the problem?

I recently came across the PhD thesis (or dissertation, to go by US usage) of Dr Carolyn J. Hamblin, who cites some of my remarks on Likert scales and levels of measurement.

In the methodology chapter, Hamblin states that “[s]ome scholars, such as Kostoulas (2015), asserted that any numerical calculation applied to the data [produced by Likert scales] are [sic] invalid in all cases.” (p. 57). After a “comparison of medians and interquartile ranges (Kostoulas, 2015) with means and standard deviations” (p. 58), Hamblin concludes that it’s quite safe to ignore my recommendations, since her calculations (mean and standard deviation) produced similar results with mine (median and interquartile range) most of the time.

From minor mistakes…

Before engaging with Hamblin’s argument in a more substantive manner, I want to correct a minor point. The in-text citations to ‘Kostoulas (2015)’ are, as far as I can tell, in reference to two distinct blog posts I wrote in 2013 and 2014. Of these, I was able to find only one in the bibliography, with an incorrect year of publication and a wrong URL.

…to disingeniousness

Moving on to a less trivial issue: I never stated that one cannot subject Likert scale data to any kind of numerical calculation. I have emphatically claimed that “ordinal data cannot yield mean values”, which is, I should think, an uncontroversial thing to say. I have stated that, in my opinion, Likert-type items produce ordinal data, but I have also written that Likert scales (which are composites of several items) allow for more flexibility. Elsewhere, I have explained that:

Under certain circumstances, a Likert scale (i.e., a collection of Likert items) can produce data that are suitable for calculating means, or running statistical tests that rely on the mean. These can be called ‘ordinal approximations of continuous data’.

In all, I think that the selective presentation of my writings in Hamblin’s thesis does little justice to either my views or her research.

This is not the only instance where Hamblin is being disingenuous. Further in the same paragraph, she writes that: “Grace-Markin (2008) argued that under certain circumstances numerical [I think Hamblin means “parametric”] calculations are acceptable. The scale should be at least 5 points, which is what this survey used.” Readers may want to read this statement against what Grace-Markin actually recommends:

At the very least, insist that the item have at least 5 points (7 is better), that the underlying concept be continuous, and that there be some indication that the intervals between points are approximately equal. Make sure the other assumptions (normality & equal variance of residuals, etc.) be met.

That is to say, Grace-Markin suggests that by one can use Likert scales data in parametric calculations. This, however, requires that the data meet at least five criteria (multitude and equidistance of points, construct validity, normality and equal variance of residuals). Of these, Hamblin ignores the final four and re-interprets the one that remains to fit her research design.

So, what is one to do?

So, what is one to do when they find out their work distorted through careless reading and ‘refuted’ though selective and creative recourse to the literature? From an affective perspective, the variability of what counts as doctoral work across the world causes some frustration.

All one can do to process it is repeat Alan Greenspan’s quote: “I know you think you understand what you thought I said, but I’m not sure you realise that what you heard is not what I meant”.

Update (2025)

When I write the post, ten years ago, I was clearly very angry and frustrated. I still believe I have good reason to do so. And I believe that all academic work should be rigorous, and doctoral dissertations should represent a standard of scholarship that was not met in this instance. I also believe that cutting corners in methods leads to producing poor results, which in turn feeds into the perception, among practicing teachers, that academic research is of low value.

In the time since I originally posted this, I have tried to approach this problem more productively. A large strand of my work involves empowering teachers research literacy, as we do in our Research Literacy of Teachers project. I think that such work can help teachers become better at both reading research and making their own useful contributions to our collective professional knowledge. At minimum, it should help them identify and stay clear of poor scholarship.

If you’ve ever felt frustrated by how research gets misquoted or misunderstood, join the conversation below. How do you approach accuracy and fairness when citing others?