ARTICLES

 

Addressing five common weaknesses in qualitative research: Sticking feathers together in the hope of producing a duck

 

 

Philippa Kerr

Psychology Institute. University of Oslo. Forskningsveien 3a. 0373 Oslo. Norway. philippalouisekerr@gmail.com

 

 

---

ABSTRACT

This paper identifies a number of common conceptual and methodological weaknesses that crop up in qualitative social science research articles and theses. These weaknesses are: (1) conceptual frameworks with no implications; (2) conceptual frameworks which dominate findings; (3) generic technical jargon in methods sections instead of a transparent account of how the research and analytical decisions actually proceeded; (4) superficial and/or anecdotal results sections; and (5) an overuse of social science jargon that sometimes does not mean very much. Suggestions for improving on these weaknesses are made. It is argued that the validity of a piece of qualitative research is established through coherence among all sections of a paper or thesis - concepts, methods, and findings. The metaphor in the paper's title conveys the point that simply including the right-sounding terminology or sections in a qualitative research article or thesis in the hope that this will, in and of itself, produce good social science is a strategy about as likely to succeed as sticking feathers together in the hope of eventually producing a duck!

Keywords: qualitative research, qualitative methods, methodology, theoretical framework, academic writing, validity

---

 

 

This paper is about validity and coherence in qualitative social science research. The idea of writing it came about while I was a postdoctoral fellow, after being asked to review some papers for journals in social psychology and other fields, supervising and examining postgraduate research, and reading a large number of published research articles in South African education journals for a review chapter (Kerr & Luescher, 2018). In a number of these theses and articles, some similar weaknesses cropped up. It occurred to me that there may be value in offering an overview of common problems in qualitative research, and revisiting some of the qualitative methodological literature to provide suggestions for how to improve on them. In my own masters and PhD research, I also made some of these mistakes and was set right by my supervisor or journal reviewers. Consequently, the aim of this paper is a pedagogical one: to highlight common problems in qualitative research so that postgraduate students and new academics can identify these in their own work and avoid or improve on them. It collects together a number of useful insights already made in the methodological literature - including a number from David Silverman - as well as some examples of critical feedback I have received on my own work, in order to suggest improvements.

The paper's observations are mainly a response to qualitative research in social psychology and higher education studies, as these are two fields in which I have worked; but, they are likely to be more widely applicable. They are also mostly about qualitative research based on thematic analysis of interview or focus group data. This is certainly not the only or best way of doing qualitative research, but it appears to be the most common one (Silverman, 2013, 2017a), and it is this kind of research the article speaks to. The paper addresses five weaknesses in approximately the same order they would appear in a paper or thesis: weaknesses in conceptual frameworks, methodologies, and findings/results sections. These weaknesses are: (1) conceptual frameworks with no implications; (2) heavy-handed conceptual frameworks which dominate findings; (3) generic technical terms in methods sections instead of a transparent account of how the research and analysis actually proceeded; (4) superficial or anecdotal results sections which lack focus; and (5) overuse of social science jargon, which sounds sophisticated but often does not say very much. Following Silverman (2005), the overall case made is that the validity of a piece of qualitative research is established through coherence among all of these sections (not just by the reporting of some procedures under a section headed 'validity', although such procedures can also be useful). The metaphor in the paper's title, which is borrowed - somewhat out of context - from a meditation by Ian McCrorie (2012), conveys this point: that simply including the right-sounding terminology or section headings in a qualitative research article or thesis in the hope that this will, in and of itself, produce good social science is a strategy about as likely to succeed as sticking feathers together in the hope of eventually producing a duck. This paper remains reasonably agnostic about the merits and pitfalls of specific conceptual and methodological approaches. It is more about how, irrespective of paradigm or technique, authors themselves can create an incoherent or unconvincing piece of academic writing by failing to connect all its sections meaningfully - and thus the paper is partly about mastering the genre of writing academic journal articles and theses.

 

Conceptual frameworks with no implications

It is not uncommon to find conceptual frameworks in qualitative research articles and theses that make reference to particular philosophical terms or approaches but whose authors do not engage with the implications of these concepts for what to make of their empirical materials. For example, an author might claim that their research is based in the 'social constructionist paradigm' or uses 'hermeneutic phenomenology', but on further inspection, it turns out that there is little about the way the empirical findings are analysed or interpreted that differentiates them from interpretations that would have been forthcoming if any other kind of theoretical approach (or none) had been invoked (see Silverman, 2011: 41). Supposedly widely differing conceptual frameworks are often then followed by the same kind of generic thematic analysis of the data, where it is simply announced that 'four themes emerged from the data'. In other words, while they may spell out well-known paradigmatic terms and claims which are not incorrect, many conceptual framework sections ultimately do not add any value because they cannot show how these particular concepts mean something for this particular data.

In an article about the difficulties education scholars often have with writing conceptual frameworks for their dissertations and first journal articles, Casanave and Li similarly observe that

A...problem for novice scholars is that their theoretical framework [often] appears only in the introductory sections of a dissertation or article, without much further discussion at all. In other words, it seems to serve no purpose beyond display...[E]ven if a theoretical or conceptual framework chapter early in the work is both appropriate and quite strong, the discussion chapter often begins by merely repeating and summarizing the findings. (2015: 110)

Casanave and Li also recall their own early experiences of encountering jargon-filled social science literatures and 'wonder[ing] if authors themselves understood well the theoretical foundations of their research or if [we were] seeing a mass epidemic of lip-service' (2015: 105). For example, I recently reviewed a psychology article which claimed it would pay attention to the 'historical and social embeddedness' of the research participants' words, and would analyse their 'functions and effects'. This is good if it can be done, and matches with a broadly discursive approach in psychology (Billig, 1996; Edwards & Potter, 1992). But in practise, no historical or rhetorical context was provided in the rest of the paper, so it was not possible to analyse the historical embeddedness, functions or effects of what the participants were saying, and the data analysis section consisted mainly of simply repeating and paraphrasing the participants' words (cf Antaki, Billig, Edwards & Potter, 2003). In other words, the analysis did not live up to the claims made in the conceptual framework.

 

Conceptual frameworks that dominate empirical findings

An opposite problem to that described above happens when imported theoretical frameworks dominate the empirical findings, such that the results section of a thesis or paper becomes an exercise in confirming a previous theory or in simply 'finding' instances of concepts or phenomena which were already identified in the literature. The danger is that this can lead to what Potter and Wetherell (1987: 42) called 'selective reading', where the analyst focuses on those aspects of the empirical material which 'simply mirror [their] prior expectation. In this situation the data can be used to simply buttress the favoured analytic story rather than being used to critically evaluate it'. As an example, an author might draw on the Capabilities Approach, and spell out its main concepts in their theoretical framework (e.g. 'capabilities', 'functionings', 'conversion factors'). The empirical section of the paper might then be organized according to these same concepts, and consist of identifying instances in the data which can be read as providing illustrations or evidence for each of them.

Legitimate questions about the plausibility of the paper's empirical claims may be raised at that point (Silverman, 2005). The reader may wonder about what other possible findings were blotted out or overlooked by the imposition of a predetermined set of concepts onto a body of empirical materials which may have been generated by participants attending to other sets of concerns (Potter & Wetherell, 1987). We might ask whether the analysis could have been just as well, or better, organized according to the terms and concepts used by the participants themselves, rather than terms and concerns imported by the author; or even according to a totally different set of theoretically-derived concepts (see Wetherell, 1998). If at least some attempt is not made to engage seriously with qualitative materials in their own terms, or to be reflective about the way one is applying theoretical perspectives from the literature, a too heavy-handed use of theory can sometimes make qualitative research into a kind of anti-empirical exercise in conceptual paper-shuffling. Though it may seem unproblematic to the writer who is committed to a particular paradigm, to the reader, empirical materials presented as examples or confirmations of frameworks we already know begs the question of what the point of the research was, and whether it has actually taught us anything new.

 

Improving conceptual frameworks

How to improve on these weaknesses in the use of theoretical or conceptual frameworks? A first step is obviously to figure out the implications of the paradigm and/or specific theory one is claiming, and how these differentiate themselves from others. What actually distinguishes hermeneutic phenomenology, say, from discourse analysis, or grounded theory, or indigenous approaches? Especially, how do they differently treat matters of empiricism and interpretation? There are many methods textbooks and papers which address this; some useful ones include Silverman (2011, 2016, 2017), Denzin and Lincoln (2011) and Terre Blanche, Durrheim and Painter (2006) (but see Barnes, 2018, for a sceptical view on how paradigms have proliferated). As one example, consider different approaches to interview research. A realist view of the interview is one which treats it as a neutral conduit for extracting information that the participant already held, whereas a constructionist view prefers seeing interviews as interactions in which speakers are performing various activities which 'inherently - not incidentally - shape the form and content of what is said' (Holstein & Gubrium, 2016: 16; see also Baker, 2003). Which of these positions one adopts has significant implications for how one analyses the resulting data: either as a straightforward window onto the life and experiences of the participant, or as something oriented to a wider rhetorical and ideological context (Billig, 1996). Sometimes authors who claim a constructionist or interpretive approach nevertheless end up reverting to a common-sense realism in the way they handle interview text.

A second question to ask oneself is whether a separate section specifically headed 'theoretical framework' is even necessary (cf Silverman, 2014). Casanave and Li (2015: 105) worry that a conceptual framework chapter or section following immediately after the introduction is a 'strict and ubiquitous structural requirement' in academic theses, but many papers and theses do not have a separate section with this heading - the concept work can be done in discussing the relevant literature, methods and findings (for a sociology thesis that does this see Wilderman, 2014). This doesn't cancel out the need for theoretical thinking - indeed, all approaches, whether explicitly or not, are adhering to some underlying theoretical assumptions (Silverman, 2011). However, it may help to avoid theory sections that are seemingly divorced from the rest of the work, and thus often quite boring for readers!

Thirdly, whether or not one has a specific section called 'theoretical framework', a helpful way of thinking about theory is as something that can shed light on a phenomenon which the participants' own 'insider' perspectives are not able to provide (Kelly, 2006). A piece of qualitative research should ideally be a critical conversation between theory and data. Theory - broadly speaking, what has been written on the topic before, or perspectives from social science more generally - can give us suggestions of what to look out for in our data, and can provide insights into that data which the speakers themselves cannot give (Kelly, 2006). But this theory does not preclude, and is even revisable in the light of, original insights and discoveries from the empirical materials. As Kelly (2006: 353) reminds us, "good interpretive research should neither impose theoretical understandings on phenomena nor simply reproduce the phenomena uncritically". That is, we do not necessarily have to take research participants' own views as the final word on whatever issue we are studying.

How we engage with these views might be informed by a particular set of "theoretical" concerns. Either way, how we use theory is a pragmatic issue: "[t]he degree to which we rely on the disclosive power of theory...to reveal aspects of a context which are not (and possibly cannot be) known in that context, varies. It varies according to the purposes of the research, and there is no best approach" (353). A theoretical framework can thus be evaluated according to whether it is useful - whether it actually sheds light on, or teaches us something more about, the phenomenon we are studying than what the participant's own perspectives - or even previous academic perspectives - can tell us. If a theory section does not really do this, then we may ask whether it is necessary.

 

Generic technical language instead of transparency in methods sections

A common weakness in the methodology sections of many journal articles and theses is the use of generic technical methods terminology instead of a clear and transparent description of what actually happened in the research process. This is particularly evident in data analysis sections of methodologies. Here, procedures like "thematic analysis" and/or "coding" (including complex-sounding variations like "open coding", "focused coding", "axial coding" and so on) are often described at length, but without any reference to the author's actual data. This may include statements about how thematic analysis proceeds in the abstract, e.g. that codes or labels are attached to sections of data; that relationships between codes are sought; and that there are different levels of coding for different levels of generality/specificity.

The problem arises when there is no connection between these generic process descriptions and the actual data that is being analysed - in other words, when the description of coding could apply to almost any qualitative research project. Such generic descriptions are problematic for a number of reasons. First, they misrepresent qualitative data analysis as a kind of sausage-machine process (cf Gergen, 1985) by which a procedure does the analysis for the author and produces and validates the results. Gergen was speaking of science generally, not qualitative research specifically, when he claimed that empirical methodology is often treated like "some form of meat-grinder from which truth could be turned out like so many sausages" (1985: 273). But the metaphor aptly captures the way that qualitative researchers sometimes seem to believe that coding will produce its own results for them. These generic accounts give the reader no sense of how the author's own thinking about this particular data evolved, such that we ended up at these particular findings (rather than any others). Since no piece of data is self-evidently about any particular topic - the same extract could be grouped or analysed in a number of ways, depending on what it is being compared with, or on what our theory leads us to look out for in it - it isn't enough to simply report that coding or some other analysis procedure was applied, and that this produced X number of codes/themes. Relatedly, authors often mention that they did coding but without saying anything about where the codes were derived from. This may suggest a lack of clarity about the difference between what the data itself says, and what the analyst brings to that data. Indeed, unless one is working with a highly-structured, pre-decided coding scheme, "codes" are arguably the outcome of, not the prerequisite for, a process of qualitative data analysis.

Generic methodologies sometimes also include promises to the reader that some procedure for ensuring validity was undertaken behind the scenes (like "triangulation"), but without giving any evidence for this in the way the empirical materials are actually presented and discussed. If one is going to attempt triangulation, say, arguably this should be made evident to the reader by actually showing (not just telling) us how different perspectives or different data sources converged. Generic references to coding and validity processes do not help establish the plausibility of the research findings, because they do not establish any link between the authors' process of analytical decision-making, and the particular findings that were the outcome of this process. These generic references also do not show how potential challenges to these particular findings and interpretations have actually been addressed. A lack of any account of the author's process of deciding which issues to focus on can also have implications later on in the paper, in the form of a superficial or vague results section which lacks a central empirical claim.

 

How to improve on generic methodologies?

Silverman is a great advocate of transparently-written methodologies which have "openness and clarity about what actually happened during your research...a bland account in the passive voice is an entirely inappropriate format for your methodology chapter" (2005: 303). A transparent methodology section enables readers to evaluate whether there is coherence between the author's knowledge claims and the procedures by which they arrived at these claims. From a realist social science perspective, this is important in establishing whether the research has internal validity: that is, whether the findings and interpretations follow in a direct and unproblematic way from the methods that were used (Tredoux & Smith, 2006). Conversely, "Findings can be said to be internally invalid because...the interpretation of the data by the researcher is not clearly supportable" (Seliger & Shohamy, 1989: 95, emphases added). In many qualitative research articles there is little or no link between what is in the methods section and how this relates to the findings, and so a lot of qualitative research is perhaps being published whose validity or soundness is basically unknowable.

From a constructionist perspective, questions of validity and transparency have a slightly different resonance. From this view, one does not simply "discover" things in qualitative research, because the "findings" are entangled with the researcher's point of view and partly produced by them (Silverman, 2011: 38-40). In their book Helping doctoral students write, Kamler and Thomson adopt this position when they argue for dissolving the distinction between what is "research" and what is the "writing up" of that research. They argue especially that the phrase 'writing up'

obscures the fact that doctoral writing is not transparent. ...Facts are not already there, waiting for the researcher to discover and grab. What writing creates is a particular representation of reality. Data is produced in writing, not found. And the data and subsequent texts that are written are shaped and crafted by the researcher through a multitude of selections about what to include and exclude, foreground and background, cite and not cite. (2006: 4)

Kamler and Thomson thus remind us that we are the authors who must decide about what empirical conclusions we come to (not leave this to "coding"). And we must also communicate something of this decision-making process to the reader. For a doctoral thesis with a reasonably transparent methodology, see Kerr (2017).

The prevalence of generic coding language in theses and articles suggests that many authors have difficulty narrating their own processes of selection and crafting. Speculatively, this could be because the author has never worked out what it is they actually want to focus on; or because the author does know this, but does not think it appropriate to include their own personal processes and reflections in the methodology. Silverman argues that this is entirely appropriate, however: "Treat your methodology chapter...as a set of cautious answers to questions that another researcher might have asked you about your work (e.g. why did you use these methods; how did you come to these conclusions?)" (2005: 302). In their helpful, clearly-worked example of a thematic analysis, Braun and Clark (2006) similarly note that "[t]hematic analysis involves a number of choices which are often not made explicit (or are certainly typically not discussed in the method section of papers), but which need explicitly to be considered and discussed" (81-82). In an early draft of my own masters thesis methodology, before I had decided what I was going to say in the findings, I wrote that I had "grouped extracts according to the issues of interest". My supervisor circled this and pointed out that I had not explained what those issues of interest were or how I had arrived at them. This lack of focus also showed up in the vagueness of the findings chapter, which made few sustained empirical claims and had to be reworked substantially.

Another way of understanding the purposes of a 'transparent' methodological account is thus as a retrospectively-written rhetorical account which works more to convince the reader that the writer themselves knows or has arrived at a focus for what they want to say about their data than by allowing the reader to independently verify whether the findings do in fact flow in a direct and unproblematic way from the methods. This is because qualitative research very often does not proceed according to a set design decided on at the outset (Silverman, 2011). Findings are often not answers to pre-decided questions; new issues of interest often appear in the course of generating or collecting the data, reading literatures, trawling the dataset, and so on (Potter, 2012). Some critics argue that it is therefore probably not possible to be entirely transparent about qualitative research analytical procedures. There is potentially a 'limitless' number of analytical decisions taken throughout the course of a research project which it may be impossible to fully record and communicate (Holstein & Staples, 1992: 32), so that 'the actual ways that findings are arrived at remain largely obscure' (Jahoda, 2012: 341). In this view, 'transparent' accounts of the researcher's thought trail may be considered as rhetorical devices which help convince the readers that the author knows what they want to say - which, indeed, often turns out to be very important for the strength and focus of the empirical claims that are eventually made (for further examples of such transparent methodological accounts see Silverman, 2005; Potter, 2012).

Examples of questions about the data analysis process which might help an author to narrate how they arrived at their particular focus could include: How did your own thinking about your empirical materials develop over time, from the beginning of the project until you arrived at this particular way of presenting the findings? What (if anything) did you initially expect to find in your data, and did you actually find that? Did you have an initial way of thinking about your data that was at some point superseded by a better way? Since any piece of text can always be treated as an instance of more than one thing, why is it particularly plausible or useful to say that these four themes (as opposed to some other ones) emerged in the data (if you are presenting your findings as themes)? Do these 'themes' correspond to the initial questions you actually asked participants (if you did interviews), or did new issues of interest occur to you once all the transcripts and texts had been done and collected? Did you read any literature which helped direct or shape your thinking about your empirical materials? Did you pursue any promising themes in your data which turned out to be dead-ends? Answering such questions can help an author articulate why this is a plausible reading - which is ideally the job of a methodology section.

 

Anecdotalism and lack of focus in results sections

This section concerns weaknesses in results sections, which to some extent follow on from the problems in methodologies identified above. Producing credible qualitative research findings is a balancing act between good and bad kinds of selectivity. All qualitative research involves being selective about what one chooses to reproduce and analyse as 'findings', because it is not usually possible (or interesting) to reproduce all the empirical materials for the reader. However, the difference between warranted, explained selectivity and loose, unwarranted or unexplained selectivity can mean the difference between a rigorous and tightly focused results section, versus a results section which suffers from 'anecdotalism' (Silverman, 2005) or superficiality. It is not uncommon to find results sections in journal articles which consist of a few brief quotes cut from a much larger body of empirical materials, with no attempt to explain the extent to which these reflect any broader pattern or phenomenon in the dataset as a whole. Sometimes results sections are general but shallow, as they skip over a number of disparate features in the data without building a sustained case about any of them. Sometimes there is also no attempt to show whether there were any 'deviant cases' (Silverman, 2011) -instances where the data may say something different to the general empirical claim the author is making. For Silverman, this is "anecdotalism" - one of the main threats to validity in qualitative research. "How", he asks, "are [qualitative researchers] to convince themselves (and their audience) that their 'findings' are genuinely based on critical investigation of all their data and do not depend on a few well-chosen 'examples'?" (Silverman, 2005: 211). Arguably, these issues are related to the methodological problems described above, mainly the lack of a specific account of how these particular findings were decided on. If it is simply announced that a particular set of findings 'emerged from the data', without explaining why these particular findings are a good, useful, or interesting thing to focus on (out of other things the author could have chosen to focus on, but did not), the impression given is that the author has not decided on the main empirical claim that they wish to make.

 

How to improve on anecdotalism and unfocussed results sections? Know thy data

Perhaps the most important things when writing a findings section are, firstly, to know your data, and then secondly, to decide what you want to focus on and say about that data. One of the best pieces of qualitative methods advice I have ever read is this statement from Silverman: "When planning the topics of your data chapters, do not assume that you must tell the 'whole story'. There is no 'whole story', there is only the story that you want to tell" (2017b: 485). In saying this, Silverman is encouraging us to focus. It can be useful to ask yourself, or have someone else ask you, what the one or two key empirical claims are that you want to make in a paper or section, and what evidence you have for and against these claims. Neither the data itself, nor 'coding', can tell you what these claims should be.

Silverman also insists that we must resist the temptation "to jump to easy conclusions just because there is some evidence that seems to lead in an interesting direction. Instead, we must subject this evidence to every possible test" (2005, p. 213). When working with large quantities of qualitative empirical materials, getting to know the data can be overwhelming at first. But instead of rushing too quickly to do 'coding', it may help to read other examples of good qualitative analysis - on one's own research topic as well as different topics. This will obviously not give you a 'recipe' for how to do analysis, but instead it can stimulate your own thinking about what, by comparison, you want to show in your own findings section. Sometimes, what ends up being the focus in a results section is not the same thing as what the author initially thought they were setting out to research.

A concern about anecdotalism was raised by a reviewer of a journal article I submitted for publication. The reviewer said: "To develop this into a fully-fledged empirical paper would require being much more transparent about the research methods and explaining much more systematically how the selected quotes refer to the material as a whole". The reviewer went on to ask, "How has a material of such impressive diversity been reduced to four extracts? How did the authors select their quotes? Were they chosen because they are outstanding cases of a rare phenomenon or, on the contrary, because they represent easily exchangeable examples of a recurrent pattern? Both cases can be interesting to analyse, but it really changes the interpretation whether the quotes shall be read as examples of common sense or as marginal/radical statements." Essentially this was a charge of anecdotalism, and I had to make revisions to better establish the representativity (or not) of the interview extracts I had chosen. I went back to my transcripts and found that there were actually more deviant cases than I had first thought. This was important for nuancing the overall empirical claims I was making.

The extent to which it is possible to illustrate, rather than simply report, the extent of the empirical claim an author is making may depend on how much space there is to reproduce lengthy extracts of text. In a thesis, one can take advantage of the greater space allowance and try to show as many examples of the phenomenon of interest as possible, as well as any cases which contradict this. In a journal article where words are more limited, one may have to choose between presenting fewer longer extracts, or a greater number of shorter ones; though tables summarising elements of the extracts can also be useful (Silverman, 2017b). One may also simply have to tell (rather than show) the reader how prevalent or marginal the issue of interest was in one's empirical materials, and whether there were any cases which contradicted this general tendency. However, one cannot escape the fact that, in order to do either of these, one has to know one's data very well.

 

Jargon

Although perhaps the language in which a piece of qualitative research is written is more a matter of style than of validity, arguably it warrants inclusion in a paper on qualitative research weaknesses. Sometimes authors use sophisticated-sounding philosophical or methodological terms, presumably in the hope that it makes their work sound more impressive; but often this language turns out to be basically empty, not saying anything, or saying it in a more convoluted way than necessary. This kind of jargon in academic writing is problematic insofar as it can sometimes cover up for a lack of substance and understanding (Billig, 2013). Like Casanave and Li (2015), who wondered whether they were witnessing a "mass epidemic of lip-service" to social science theory, Billig recounts his attempts as a young academic to translate technical and complicated-sounding social science ideas into simple language. Occasionally,

when I'd finished the translation, the ideas and the sense seemed to dribble away, leaving truisms and little else. Then, I would be perplexed. Was it my failure to understand or was it that a writer, who had actually been published, really had so little to say? (2013: 2).

If you think your own writing suffers from too much jargon and too little transparency, it may help to try writing a particular section, e.g. methods, without using any technical terms or generic descriptions of coding (or any other procedure). Simply tell us how your own thinking about your topic and empirical materials developed, how you arrived at this particular way of presenting your findings, and why this way is better than other possible ways you considered. Forcing oneself to write more plainly is a good way of asking "what do I actually have to say?". If the answer is "not that much", then the exercise has been useful because now you know where you need to be clearer, gain focus, or, alternatively, where you can cut out sections that don't add anything. (This can be cathartic.)

 

Caveats and conclusions

The observations in this paper are mainly a response to qualitative research articles and theses which follow a similar pattern: thematic analysis of data generated in individual or group interviews, presented in a format where a literature/theory section is followed by methods and then results. The paper does not presume that this is the only or the best way of doing qualitative research, but it is one very common - perhaps the most common - way. To be clear, the article has not engaged with, and is not an implicit critique of, other ways which deviate in principle from these patterns. There is a debate about the use (and overuse) of interviews as a default data collection method in qualitative research (Potter & Hepburn, 2008; Silverman, 2017a; Atkinson & Sampson, 2019), and about the merits of using alternative kinds of data which were not generated by the researcher (Potter, 2002, 2012; for examples of research which use qualitative data other than interviews, see Durrheim, Greener & Whitehead, 2015; Cornell, Ratele & Kessi, 2016; Luescher, Loader & Mugume, 2017). Different disciplines and journals also have different norms and requirements for methodological detail and transparency.

Nevertheless, this paper has argued that doing qualitative research is a matter of more than including the standard sections (conceptual framework, methodology, results) and mentioning the expected methodological and philosophical language (coding, thematic analysis, social constructionism). Instead, the paper has argued that without coherence among theory, method and findings sections, writing qualitative research can be an empty, pseudo-technical exercise akin to sticking feathers together in the hope of eventually producing a live bird. This position is similar to that of Kamler and Thomson (2006), whose objection to the phrase "writing up" when talking about doctoral thesis writing is that it suggests writing is something we do separately from the research and analysis. It obscures

the fact that doctoral writing is thinking. We write to work out what we think. It's not that we do the research and then know. It's that we write our way to understanding through analysis (p. 4, emphasis in original).

Not only do we write our way to our own understanding, we also write our way to convincing our audience, and thus, to defensible knowledge claims. In this sense, questions of validity in research methods cannot easily be separated from questions about the form and genre(s) of academic writing.

Perhaps the proliferation of generic 'coding' language and references to the agency of technical-sounding procedures in qualitative research also happen because many students think doing research is a process of following a recipe that someone else knows better than they do. So they claim to have stuck closely to a prescribed set of steps for doing their analysis, such as the steps for thematic analysis outlined by Braun and Clarke (2006, 2012). They do not dare to take an authoritative position of their own that might involve modifying or questioning these steps. Perhaps this is understandable for postgraduates who are still learning the ropes¹. However, the main aim of qualitative research is not ultimately to follow procedures for their own sake, but to discover something we did not know before. Procedures and steps, such as line-by-line coding, can direct one towards a close and careful reading of empirical materials, and to that extent they are helpful. But sometimes these steps outlast their own usefulness - for example, when students feel obligated to stick to a coding process that they began even once they start to see emergent findings which make detailed coding in that original way no longer useful. Indeed, according to Rapley (2016), "The practises of good or even adequate qualitative data analysis can never be summed up...by a list of specific steps that have been undertaken. Above all, you need to develop a hands-on knowledge of analysis" (p. 332). Silverman (2011: 64) borrows the metaphor of a ladder from Wittgenstein, to describe how rules and procedures for data analysis give you initial steps and direction for where you are going. But, "[o]nce you start to write up [sic] your research and become confident in what you are arguing, you can throw the ladder away". Throwing the ladder away involves recognizing one's own authority and responsibility as an author - an authority which many writers appear to think they are required to surrender, when they imply that "coding" did the analysis for them, or that findings "emerged" by themselves. This paper has aimed to encourage qualitative researchers to re-claim that responsibility by leading the conversation rather than thinking of themselves as obligated to simply repeat the conventions of the field.

 

References

Antaki, C, Billig, M, Edwards, D & Potter, J (2003) Discourse analysis means doing analysis: A critique of six analytic shortcomings. Discourse analysis online, 1 (1). https://pdfs.semanticscholar.org/0278/161e36ba3e3b15270379399f6b463a295066.pdf        [ Links ]

Atkinson, P & Sampson, C (2019) Narrative stability in interview accounts. International Journal of Social Research Methodology, 22 (1), 55-66.         [ Links ]

Barnes, B (2018) Decolonising research methodologies: Opportunities and caution. South African Journal of Psychology, 48 (3), 379-387.         [ Links ]

Baker, C (2003) Ethnomethodological analyses of interviews. In Holstein, JA & Gubrium, JF (eds), Inside interviewing: New lenses, new concerns. Thousand Oaks: Sage.         [ Links ]

Billig, M (1996) Arguing and thinking: A rhetorical approach to social psychology. Cambridge: CUP (1987).         [ Links ]

Billig, M (2013) Learn to write badly: How to succeed in the social sciences. Cambridge: CUP.         [ Links ]

Braun, V & Clarke, V (2006) Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101.         [ Links ]

Braun, V & Clarke, V (2012) Thematic analysis, In Cooper, H (ed), APA handbook of research methods in psychology: Vol. 2. Research designs: Quantitative, qualitative, neuropsychological and biological. Washington: American Psychological Association.         [ Links ]

Casanave, C P & Li, Y (2015) Novices' struggles with conceptual and theoretical framing in writing dissertations and papers for publication. Publications, 3, 104-119.         [ Links ]

Cornell, J, Ratele, K & Kessi, S (2016) Race, gender and sexuality in student experiences of violence and resistances on a university campus. Perspectives in Education, 34 (2), 97-119        [ Links ]

Denzin, N & Lincoln, Y (2011) The SAGE handbook of qualitative research. Thousand Oaks, CA: Sage.         [ Links ]

Durrheim, K, Greener, R & Whitehead, K (2015) Race trouble: Attending to race and racism in online interaction. British Journal of Social Psychology, 54 (1), 84-99.         [ Links ]

Edwards, D & Potter, J (1992) Discursive psychology. London: Sage.         [ Links ]

Gergen, K (1985) The social constructionist movement in modern psychology. American Psychologist, 40 (3), 266-275.         [ Links ]

Harley, A (2017) Alienating academic work. Education as Change, 21 (3), 1-14.         [ Links ]

Jahoda, G (2012) Critical comments on experimental, discursive, and general social psychology. Journal for the Theory of Social Behaviour, 43 (3), 341-360.         [ Links ]

Kamler, B & Thomson, P (2006) Helping doctoral student write: Pedagogies for supervision. London: Routledge.         [ Links ]

Kelly, K (2006) Lived experience and interpretation: The balancing act in qualitative analysis, In Terreblanche, M, Durrheim, K & Painter, D (eds) (2006), Research in practise: Applied methods for the social sciences. Cape Town: UCT Press.         [ Links ]

Kerr, P (2017) Xenophobia, social change and social continuity: Changing configurations of intergroup allegiance and division among farm workers and farmers in De Doorns, 2009-2013. Unpublished PhD thesis, University of KwaZulu-Natal.         [ Links ]

Kerr, P & Luescher, T (2018) Students' experiences of university life beyond the curriculum. In P Ashwin & J Case (eds), Higher education pathways: South African undergraduate education and the public good. Somerset West: African Minds.         [ Links ]

Kessi, S & Boonzaier, F (2018) Centre/ing decolonial feminist psychology in Africa. South African Journal of Psychology, 48 (3), 299-309        [ Links ]

Luescher, T, Loader, L & Mugume, T (2017) #FeesMustFall: An internet-age student movement in South Africa and the case of the University of the Free State. Politikon, 44 (2), 231-245.         [ Links ]

McCrorie, I (2012) Children of silence and slow time: More reflections of the Dhamma. Onalaska, Washington: Pariyatti Press.         [ Links ]

Mohamedbhai, G (2014) Massification in higher education institutions in Africa: Causes, consequences and responses. International Journal of African Higher Education, 1 (1), 59-83.         [ Links ]

Muller, S (2018) The university is not a factory. Mail & Guardian. https://mg.co.za/article/2018-10-12-00-the-university-is-not-a-factory/        [ Links ]

Nwoye, A (2015) What is African Psychology the psychology of? Theory & Psychology, 25 (1), 96-116.         [ Links ]

Portnoi, L (2015) Pushing a stone up a hill: A case study of the working conditions of South African academics. Research in Comparative and International Education, 10 (2), 257-274.         [ Links ]

Potter, J (2002) Two kinds of natural. Discourse Studies, 4 (4), 539-542.         [ Links ]

Potter, J (2012) Discourse analysis and discursive psychology. In H Cooper (ed) (2012) APA handbook of research methods in psychology: Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological. Washington: American Psychological Association Press.         [ Links ]

Potter, J & Hepburn, A (2005) Qualitative interviews in psychology: Problems and possibilities. Qualitative Research in Psychology, 2, 281-307.         [ Links ]

Potter, J & Wetherell, M (1987) Discourse and social psychology. London: Sage.         [ Links ]

Rapley, T (2016) Some pragmatics of qualitative data analysis, In Silverman, D (ed) (2016) Qualitative Research. London: Sage.         [ Links ]

Ratele, K (2017) Four (African) psychologies. Theory & Psychology, 27 (3), 313-327.         [ Links ]

Segalo & Cakata (2017) A psychology in our own language: Redefining psychology in an African context. Psychology in Society, 54, 29-41.         [ Links ]

Seliger, H & Shohamey, E (1989) Second language research methods. Oxford: Oxford University Press.         [ Links ]

Silverman, D (2005) Doing qualitative research. London: Sage (2000).         [ Links ]

Silverman, D (2011) Interpreting qualitative data. London: Sage (1993).         [ Links ]

Silverman, D (2013) What counts as qualitative research? Some cautionary comments. Qualitative Sociology Review, 9 (2), 48-55.         [ Links ]

Silverman, D (2014) Taking theory too far: A commentary on Avison and Malaurent. Journal of Information Technology, 29, 353-355.         [ Links ]

Silverman, D (2017a) Doing qualitative research. London: Sage (2000).         [ Links ]

Silverman, D (2017b) 'How was it for you?'. The interview society and the irresistible rise of the [poorly analysed] interview. Qualitative Research, 17 (2), 144-158.         [ Links ]

Terre Blanche, M, Durrheim, K & Painter, D (2006). Research in practise: Applied methods for the social sciences. Cape Town: UCT Press.         [ Links ]

Tomaselli, K G (2018) Perverse incentives and the political economy of South African academic journal publishing. South African Journal of Science, 114 (11-12), 1-6.         [ Links ]

Tredoux, C & Smith, M (2006) Evaluating research design. In M Terreblanche, K Durrheim & D Painter (eds), Research in practise: Applied methods for the social sciences. Cape Town: UCT Press.         [ Links ]

Wilderman, J (2014) Farm worker uprising in the Western Cape: A case study of protest, organising and collective action. Unpublished master's thesis, University of the Witwatersrand.         [ Links ]

Wetherell, M (1998) Positioning and interpretative repertoires: Conversation analysis and poststructuralism in dialogue. Discourse and Society, 9 (3), 387-412.         [ Links ]

 

 

1 To some extent, the problems identified in this paper can also be seen as a consequence of large structural forces acting on universities that cannot simply be reduced to the aptitudes and interests of individual academics and postgraduates. Overwork from increased postgraduate and undergraduate class sizes and the instrumentalising of academic research through the current publishing incentives regime, among other things, have arguably helped to encourage the production of mediocre research in South African universities (Mohamedbhai, 2014; Portnoi, 2015; Harley, 2017; Tomaselli, 2018; Muller, 2018). So far, psychology has not engaged extensively with this issue, as the recent decolonization and African psychology critiques have focused on the epistemic foundations of the discipline (Nwoye, 2015; Kessi & Boonzaier, 2018; Ratele, 2017) but without connecting this to the political economy of research and publishing within which all psychology academics must now operate (cf Barnes, 2018; Segalo & Cakata, 2017).