Achilleas Kostoulas

Applied Linguistics & Language Teacher Education

Two scientific pocket calculators on a desk.

On being misquoted

Although I am not a statistician, through some quirk of Google’s search algorithm, it appears that I have become promoted to the status of a go-to internet expert on Likert scales. This is sometimes awkward, especially when a less-than-perfect blog post is cited in a peer-reviewed publication, but I can live with that. On the other hand, I tend to be somewhat more frustrated when my writings are misunderstood and misquoted – and the purpose of this post is to set the record straight after one of these instances.

Phelps et al. (2015) Pairwise Comparison Versus Likert Scale for Biomedical Image Assessment. American Journal of Roentgenology 204(1), 8-14.

So what’s the problem?

It was recently brought to my attention that my views on Likert scaling have been cited by Dr Carolyn J. Hamblin in her PhD thesis (or dissertation, to go by US usage). In the methodology chapter, Hamblin states that “[s]ome scholars, such as Kostoulas (2015), asserted that any numerical calculation applied to the data [produced by Likert scales] are [sic] invalid in all cases.” (p. 57). After a “comparison of medians and interquartile ranges (Kostoulas, 2015) with means and standard deviations” (p. 58), Hamblin concludes that it’s quite safe to ignore my recommendations, since her calculations (mean and standard deviation) produced similar results with mine (median and interquartile range) most of the time.

From minor mistakes…

Before engaging with Hamblin’s argument in a more substantive manner, I want to correct a minor point. The in-text citations to ‘Kostoulas (2015)’ are, as far as I can tell, in reference to two distinct blog posts I wrote in 2013 and 2014. Of these, only one is listed in the bibliography, with an incorrect year of publication and URL.

Hamblin, C. J. (2015) How Arizona Community College Teachers Go About Learning to Teach. Unpublished PhD Thesis, Utah State University. (p. 118)

…to disingeniousness

Moving on to a less trivial issue: I never stated that Likert scale data cannot be subjected to any kind of numerical calculation. I have emphatically claimed that “ordinal data cannot yield mean values”, which is, I should think, an uncontroversial thing to say. I have stated that, in my opinion, Likert-type items produce ordinal data, but I have also written that Likert scales (which are composites of several items) allow for more flexibilityElsewhere, I have explained that:

Under certain circumstances, a Likert scale (i.e., a collection of Likert items) can produce data that are suitable for calculating means, or running statistical tests that rely on the mean. These can be called ‘ordinal approximations of continuous data’.

In all, I think that the selective presentation of my writings in Hamblin’s thesis does little justice to either my views or her research.

This is not the only instance where Hamblin is being disingenuous. Further in the same paragraph, she writes that: “Grace-Markin (2008) argued that under certain circumstances numerical [I think Hamblin means “parametric”] calculations are acceptable. The scale should be at least 5 points, which is what this survey used.” Readers may want to read this statement against what Grace-Markin actually recommends:

At the very least, insist that the item have at least 5 points (7 is better), that the underlying concept be continuous, and that there be some indication that the intervals between points are approximately equal. Make sure the other assumptions (normality & equal variance of residuals, etc.) be met.

That is to say, Grace-Markin suggests that the data produced by Likert scales can be used in parametric calculations, as long as at least five criteria are met (multitude and equidistance of points, construct validity, normality and equal variance of residuals). Of these, Hamblin ignores the final four and re-interprets the one that remains to fit her research design.

So, then?

So, what is one to do when they find out that their work has been distorted through careless reading and ‘refuted’ though selective and creative recourse to the literature? At minimum, one can always repeat Alan Greenspan’s quote: “I know you think you understand what you thought I said, but I’m not sure you realise that what you heard is not what I meant”. In addition to that, one feels compelled to register profound frustration at the variability of what is considered to be doctoral work across the world.

Featured Image by Michael Kwan [CC BY-NC-ND]





6 responses to “On being misquoted”

  1. Mariam Attia avatar

    Hi Achilleas,
    It must be very frustrating to be misquoted. How about contacting the author herself? This might serve three purposes: a) Hamblin will learn that misqoutation (intentional or unintentional) do not go unnoticed; b) She will check correct information about Likert scales – in case it was a mis-reading of your work; and c) You will (hopefully) prevent this from re-appearing in future publications.
    .. and prominent scholars are sometimes misquoted :-)

    1. Achilleas avatar

      Excellent points as always, Mariam! There is indeed value in what you are suggesting; but in addition to that, I think that it is important to correct the public record, since her thesis is a publicly available document.

      There was an article recently about a professor who has been living with the repercussions of a hoax article citing him, for years ( Though this case is not nearly as dramatic, the principle still applies: one cannot be too careful about the way in which they are represented online.

  2. musingfrommanchester avatar

    Hmm. In a quandary now: I feel sure I have interpreted your writing on this topic correctly – going beyond what is usually expected in my field by even *considering* the issue of interval data/Likerts – but now I feel I shouldn’t cite you. Victim of your own success? It is one of the better and clearer explanations available to those of us not immersed in the minutiae of statistical assumptions…

    1. Achilleas avatar

      Thanks for the kind words! I am, of course happy to be cited, and even happier if I have been of help. I am also very fine with being cited if you disagree – I am perhaps less comfortable when people alter what I say.

      Perhaps your advisor might be able to help you decide whether a citation is useful, or if there is a more appropriate source to consult; or, if you prefer, I am also happy to give an opinion :)

      1. musingfrommanchester avatar

        Thanks, Achilleas. I am slightly scared now ;) but may send something along after checking it all over again…

        Are you still based at UoM? I work in the Library there.

      2. Achilleas avatar

        I’ve moved on to the University of Graz in Austria, but I used to love working in the Library :)

        As for the scales, there’s really nothing to be scared of, and it has certainly not been my intention to undermine your confidence! Do let me know what I can do to undo the damage :)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.