REVIEW ARTICLE

 

Postgraduate research methodological flaws detected at final examination stage: Who is to blame?

 

 

Aceme Nyika

Faculty of Health Sciences, University of the Witwatersrand Medical School, Johannesburg, South Africa

Correspondence

 

 

---

ABSTRACT

In the wake of globalisation, most universities are intensifying efforts to improve their overall performance in order to attract students and enhance chances of securing competitive funding from various sources. As part of these efforts, universities are striving to ensure that their postgraduate programmes meet nationally and internationally acceptable standards. Research projects conducted by students form a critical component of most postgraduate programmes and universities have put in place procedures meant to ensure that postgraduate research meets acceptable minimum standards. The procedures include setting minimum entry educational qualifications, supervision by qualified members of university academic staff, institutional evaluation of research proposals before the proposed research is embarked on, submission of progress reports by postgraduate students during the course of their programmes, and final examination of students' theses, dissertations or research reports by internal as well as external examiners. In some instances, the examiners recommend outright rejection of the student's write-up if they consider the methodology used to be inappropriate to answer research questions of the project conducted. The implications of research methodological shortcomings which are identified at the final examination stage, even if the research proposals were evaluated and approved by appropriate university structures before commencement of the research projects, are discussed. As postgraduate programmes are meant to nurture a competent and resourceful workforce and future researchers, universities need to pay attention to the issue of research methodology and internal evaluation systems in order to minimise chances of compromising the quality of their postgraduate degree programmes.

Keywords: postgraduate research; research proposal; proposal evaluation; final examination; methodological flaws

---

 

 

Introduction

Globally, universities aim to provide quality education which is demand driven and is relevant to dynamic needs of countries, continents and the world. In light of these aspirations, various systems of evaluating and ranking universities have emerged, with some using indicators based on the main activities of universities which are teaching, research and service provision^1-3 while others are based on cybermetric indicators.^4-7 Regardless of the system of ranking used, universities in Africa have very low scores in the global ranking system, with the top 100 universities being in developed countries. The best African universities according to the global ranking systems are in South Africa. According to The Times Higher Education World University Rankings, the best African universities are the University of Cape Town (UCT), the University of the Witwatersrand (Wits), Stellenbosch University (SU) and the University of Kwazulu-Natal (UKZN), which were at position 113, in category 226-250, in category 251-275, and in category 351-400, respectively, in the global ranking system for the year 2012.¹

Some countries have developed their own national ranking systems which are to a large extent based on the global ranking systems. For instance, South Africa has its own ranking system which refers to the category of the best universities in the country as the 'red cluster', which in 2012 was composed of five universities: Wits, UKZN, SU, UCT and the University of Pretoria (UP).⁸ With the exception of UP these are the same African universities ranked highly in the global ranking system relative to other universities on the African continent.

It has been observed that, generally, after colonisation most universities in African countries did not develop at the same rate as universities in developed countries.⁹ In some African countries, universities were initially established as part of a main parent university in the respective colonising countries, but the established African universities gradually became autonomous and independent. In addition, governments and other players such as churches and the private sector have become increasingly involved in the establishment of universities in most African countries. The slow development of the African universities could arguably be attributed to poor infrastructure¹⁰ and inadequate resources as a result of a combination of limited injection of resources by the respective African governments and a shift in focus of foreign aid from tertiary education to primary and secondary education.¹¹

Regardless of positions occupied in national or international ranking systems, universities have developed internal systems which are meant to ensure that the quality of education and training they provide meets certain minimum acceptable standards. One of the most critical roles of universities is to run postgraduate programmes, which to a large extent enhance the productivity of universities in terms of research outputs. Consequently, universities have put in place institutional procedures aimed at ensuring that they produce postgraduate students of the highest possible quality. The internal procedures vary from one university to another, but in general the first step is the screening of potential students at the admission stage so as to enroll candidates with the necessary educational background. After enrolment, the postgraduate students have to register annually and the registration is subject to satisfactory progress in the preceding year. If the postgraduate degree is by course work only, then progress is in terms of any courses that the student is required to take. For postgraduate programmes that require research projects to be done either in full or partial fulfillment of the degree requirements, progress in the research project is assessed from the proposal stage through the conduction of the research to the stage of writing up the results from the research for final examination.

A postgraduate student in a research-based programme has to submit a thesis, dissertation or research report to the postgraduate department for the final examination. The type of write-up to be submitted depends on the degree programme and the policy of the particular university. The four main possible outcomes of the examination process are (1) pass, (2) conditional pass subject to minor revision, (3) major revisions to be followed by re-examination, and (4) fail, which is an outright rejection of the write-up. Revisions which are to do with mistakes in the scientific writing or presentation of the findings could be corrected within a reasonable period of time. However, failure or rejection as a result of fundamental research methodological shortcomings would require the original research proposal to be corrected in light of the examiners' reports and the research project to be repeated by the student using the corrected methodology.

Currently there is a paucity of literature addressing pertinent issues surrounding postgraduate research and examination processes in universities. In this paper, the issues surrounding failure at the final examination stage because of methodological flaws in postgraduate research projects that were evaluated and approved by the university concerned in the first place, are discussed. Firstly, an overview is provided of university processes aimed at ensuring that research methodologies used by students are appropriate for their research questions and meet internationally acceptable standards. A discussion follows which covers various possible implications of rejection of a thesis, dissertation or research report by examiners at the final examination stage because research methods used are deemed to be inappropriate. Research is the main factor which strengthens the other complementary factors such as teaching, provision of services and ability to secure funding. Globally, postgraduate research programmes make significant contributions towards the research outputs of universities.

 

Postgraduate research proposal

A postgraduate student in a programme which involves a research project has to develop a research proposal under the guidance of at least one supervisor appointed by the university. The proposal has to explain in detail the research project to be conducted by the student. In general, the proposal has to have a title that captures the main aim of the intended study. An introduction has to put the study into context, explaining the background and the research question or hypothesis to be addressed by the study. A literature review should show what is already known about the research issue and the nature of any gap in knowledge which the study intends to address.

The introduction and literature review should be followed by the main objective as well as specific objectives of the study. It is the specific objectives that form the basis for the research methods which will be used. The research methodology should be explained in detail to enable readers to assess whether or not it is appropriate for the research questions to be answered. The overall research design has to be explained. The proposal should explain whether it will be a retrospective or prospective study and whether it will be qualitative or quantitative. Details such as the targeted population, sampling technique, inclusion and exclusion criteria, variables to be measured, laboratory tests to be done and statistical analyses to be done should be fully explained in the proposal if applicable. Also to be included in the research proposal are pertinent ethical issues, the work plan, budget and references.

Institutional procedures for postgraduate research

Postgraduate supervision

University students enrolled in postgraduate programmes that include research as partial or full fulfillment of the requirements of their respective postgraduate degrees must have supervisors appointed by the relevant postgraduate department in order to guide the students during the course of their programmes. Each university has its own procedures for the nomination and appointment of supervisors for postgraduate students. However, in order to foster effective supervision, universities generally require supervisors to have relevant and adequate educational qualifications and experience which are commensurate with the level of the postgraduate programme to be supervised. This requirement is intended to ensure that the supervisor has the knowledge and experience needed to be able to provide effective leadership to the student. More than one supervisor may be appointed to lead a postgraduate student if it is deemed necessary.

Institutional procedures to ensure an acceptable quality of postgraduate research start with guidance provided by the supervisor during the development of the proposal. Postgraduate students are supervised throughout their degree programme from start to finish and most universities stipulate the minimum face-to-face contact hours between student and supervisor which depend on the level of the postgraduate degree.

Postgraduate proposal

The next step in the procedure to ensure the minimum acceptable quality is the evaluation of the research proposal through an internal system involving assessors appointed by the relevant postgraduate department of the university. The evaluation of students' proposals is to ascertain the soundness of the research methodology, originality, the adequacy of the proposed work for the postgraduate degree for which the student is registered and the feasibility of the proposed study in terms of the time and funds needed. In addition to checking the scientific merits of the proposal, it is also submitted to the relevant ethics committee of the university for ethical clearance, depending on whether the project deals with humans, animals or the environment in general. The student can only start the proposed research project after obtaining clearance from the postgraduate department and from the relevant ethics committee. However, some universities exempt certain types of research projects in fields such as engineering, geology and mathematics from the requirement to undergo ethical clearance.

Progress reports

Most universities require progress reports to be submitted periodically (every semester or annually) so as to monitor the progress of the student. If progress is deemed to be unsatisfactory, appropriate remedial measures may be taken by the postgraduate department in consultation with the supervisor(s). The reports also enable the university to ensure that there is no deviation from the approved proposal. Any major methodological problems should be detected during the course of the programme as the progress reports should cover results obtained and methods used to gather the data.

Final examination

After completing the research project, the student writes a thesis, dissertation or research report to be submitted to the postgraduate department for final examination. The exact submission procedure and requirements depend on specific policies of the particular universities. The submitted postgraduate write-up is then examined by internal examiners plus external examiners appointed by the postgraduate department. The majority of universities provide institutional guidelines to be used by the postgraduate examiners, who should not have any conflict of interest. In general, the guidelines cover aspects of the research such as originality, introduction, literature review, research methodology, research results, discussion and conclusion to be assessed. Also to be assessed are the overall flow of ideas and presentation of findings in the write-up, grammar, typography and spelling. Based on the final write-up, the examination process should aim to assess the level of intellectual grasp of the concept of research and how it should be conducted, which makes research methodology a very critical aspect of postgraduate research programmes.

The examiners write reports which give their recommendations regarding the thesis, dissertation or research report which they evaluated. There are generally four possible examination outcomes that could be given by the examiners, namely (1) pass without any revisions, (2) conditional pass subject to minor revisions, (3) major revisions followed by re-examination and (4) fail without any opportunity for revision. Some universities conduct oral examinations, which are generally referred to as 'vivas',¹² after examination of the write-up. The examiners explain the basis of their decisions in their reports which are then made available to the supervisor(s) and student in order to enable the student to do any revisions which may have been deemed necessary.

 

Implications of research methodological flaws

Whereas other types of shortcomings could be considered to be 'peripheral', research methodology is arguably the most important aspect of research which postgraduate students, or any researcher for that matter, should understand before embarking on any research project. It is mainly the research methodology which determines whether or not the research question(s) can be answered. Hence most universities have systems in place which evaluate research proposals before postgraduate students can begin their research projects.

It follows therefore that having a postgraduate research thesis, dissertation or research report being condemned by examiners on the basis of flawed research methodology raises several issues and questions which need to be addressed if the quality of postgraduate research in universities is to be enhanced. The scenario of research methodological flaws being detected at the final examination stage has four main implications, all of which reflect badly on the university concerned.

First implication: The proposal was flawed

The first main implication is that the research proposal which was evaluated by the university postgraduate system in the first place was flawed but the internal evaluation system did not detect the research methodological shortcomings. The internal system includes the supervisor(s) who guided the student during proposal development and the postgraduate structures, such as the departmental and faculty higher degrees committees, which were responsible for evaluating and approving the postgraduate research proposal. The possibility that flawed proposals can be approved and research projects completed before shortcomings are detected at the last stage of the postgraduate programme, brings into question the effectiveness of the whole evaluation system in ensuring high-quality postgraduate research. Such a scenario implies that the steps in the internal system are carried out as mere formalities without rigorous evaluation of research proposals.

Second implication: The proposal was inadequate

The second implication is that the format of the proposal does not enable adequate evaluation of the research methodology. This situation would exist if the proposal was so abridged that insufficient detail about the research methodology was included. Hence such a proposal could sail through the internal evaluation system without any methodological flaws being detected, and the flaws would become evident when the detailed thesis, dissertation or research report is submitted for examination. Some universities require that the postgraduate student presents and 'defends' the proposal in the postgraduate department so that proposal assessors can ask for details and explanations which may not be clear in the proposal.

Third implication: The proposal was deviated from

The third main implication is that the postgraduate student deviated from the assessed and approved research proposal during the course of the programme and ended up with data collected through inappropriate research methods. Consequently, the collected data, the analyses, interpretation of findings and conclusions contained in the thesis, dissertation or research report would not be appropriate to answer the research question(s) as explained in the approved proposal. However, such a deviation would imply that the checks and balances put in place by the university to ensure that postgraduate students are guided in the correct path of research are not effective. It would mean that the supervision process either could not detect the deviation as it was occurring, or the deviation was detected and deemed to be appropriate.

If the deviation was not detected, it implies that several qualified and experienced members of the academic staff of the university concerned would have failed to pick up the major methodological shortcomings which were eventually detected by examiners at the final examination stage of the programme. If the deviation was actually allowed by the supervisors and it is eventually criticised by examiners as being inappropriate for the research, then the question arises as to who is correct - the supervisors or the examiners. It would be unfair, unethical and unprofessional to blame and penalise the postgraduate student who is an unqualified and inexperienced researcher enrolled in the postgraduate programme to gain the requisite qualification and experience.

Fourth implication: The examiners are wrong

The fourth main implication is that the examiners are wrong. In other words, there are no research methodological shortcomings in the final thesis, dissertation or research report but the examiners incorrectly think that there are shortcomings. If all the examiners independently and separately, but incorrectly, criticise the research methodology, then one implication is that the criteria of selecting and appointing examiners is not effective. If there are discordant reports submitted by examiners, then some of them may be wrong while others may be right, but it becomes a challenge to determine who is wrong and who is right. Another possible implication of this scenario is that the guidelines developed by the university to guide the examiners are not effectively providing the intended guidance and marking framework, which could lead to some examiners making flawed judgements.

 

Recommendations to consider

A major recommendation to consider is for universities to review their postgraduate evaluation systems to ensure that they effectively serve the purpose for which they were put in place. If there are cases of thesis, dissertations or research reports criticised at the final examination stage as a consequence of methodological problems, the university should investigate the cases starting with re-evaluation of the original proposals to determine if flaws slipped through without detection. The supervision process, including progress reports submitted and minutes of meetings between supervisor and students, should be assessed, preferably by assessors independent of the university. Finally, the condemned thesis, dissertation or research report should be re-examined by independent examiners with relevant qualifications and experience. The re-examination reports should then be compared with the 'controversial' examiners' reports. Such a thorough and comprehensive investigation would enable the university to assess how issues of research methodology could crop up at the final examination stage after the research proposals and research processes had been subjected to internal evaluation systems. Any loopholes which may be identified in the system should then be addressed and the system should be assessed regularly thereafter.

Another recommendation to consider is to improve (if already in existence) or put in place support mechanisms aimed at strengthening postgraduate supervision in the university. Elementary and advanced training workshops could be developed to cater for emerging and experienced supervisors, respectively. As for the examination process, it would be important to assess the guidelines for examiners to be sure that they are up to date and clearly conform with the policy of the university regarding postgraduate programmes that involve research. It would also be helpful to increase the pool of internal and external examiners from which to choose appropriate examiners for specific types of research. The increase should be both in terms of the number of examiners and the diversity of areas of specialisation. One possible method to increase the pool of examiners is to offer reasonable honoraria in order to attract and retain desired examiners. Universities could also consider developing and running appropriate training workshops for examiners so as to ensure uniform institutional marking schemes and standards pertaining to postgraduate research. In addition, the issue of weighting of the internal versus external examiners' comments and marks should be dealt with and clarified in the university's policy or guideline documents.

 

Conclusion

Postgraduate research programmes contribute significantly towards the overall productivity of universities and the quality of their products. The advancement of technology in various fields depends to a large extent on the continuous availability of resourceful and innovative researchers for the future. Universities across the world strive to collectively produce such researchers through postgraduate research programmes. It is therefore critical that universities take measures to ensure that their postgraduate research programmes effectively impart the required profound intellectual comprehension of research methodologies.

Research methodology queries arising at the final examination stage of postgraduate programmes raises questions about the quality and appropriateness of the evaluation systems of the universities concerned. It arguably implies ineffectiveness on the part of the student, the postgraduate supervisors, postgraduate proposal assessors and or examiners. Regardless of where the blame actually lies, emergence of research methodological shortcomings at the final examination stage of a postgraduate thesis, dissertation or research report gives a bad impression of the university concerned. Overall, universities demonstrating high productivity and internationally acceptable standards attract the best students, the best workforce and sustainable funding opportunities.

 

References

1. Times Higher Education World University Rankings 2012-2013 [homepage on the Internet]. No date [cited 2013 Aug 19]. Available from: http://www.timeshighereducation.co.uk/world-university-rankings/2012-13/world-ranking

2. Academic ranking of world universities [homepage on the Internet]. No date [cited 2013 Aug 20]. Available from: http://www.arwu.org/        [ Links ]

3. QS university ranking methodology [homepage on the Internet]. No date [cited 2013 Aug 20]. Available from: http://www.iu.qs.com/university-rankings/world-university-rankings/        [ Links ]

4. Aguillo IF, Granadino B, Ortega JL, Prieto JA. Scientific research activity and communication measured with cybermetrics indicators. J Am Soc Inf Sci Technol. 2006;57(10):296-302. http://dx.doi.org/10.1002/asi.20433        [ Links ]

5. Ortega JL, Aguillo IF. Mapping world-class universities on the web. Inform Process Manag. 2009;45:272-279. http://dx.doi.org/10.1016/j.ipm.2008.10.001        [ Links ]

6. Aguillo I. Measuring the institution's footprint in the web. Library Hi Tech. 2009;27(4):540-556. http://dx.doi.org/10.1108/073788309        [ Links ]

7. Ranking Web of universities [homepage on the Internet]. No date [cited 2013 Aug 20]. Available from: http://www.webometrics.info/en/Methodology        [ Links ]

8. Rankings of universities in South Africa [homepage on the Internet]. No date [cited 2013 Aug 22]. Available from: http://en.wikipedia.org/wiki/Rankings_of_universities_in_South_Africa        [ Links ]

9. Gukas I. Global paradigm shift in medical education: Issues of concern for Africa. Med Teach. 2007;29:887-892. http://dx.doi.org/10.1080/01421590701814286        [ Links ]

10. Burdick W. Challenges and issues in health professions education in Africa. Med Teach. 2007;29:882-886. http://dx.doi.org/10.1080/01421590701821695        [ Links ]

11. Bloom D, Canning D, Chan K. Higher education and economic development in Africa [document on the Internet]. c2006 [cited 2013 Sep 10]. Available from: http://www.sciencedev.net/Docs/Higher%20Education%20and%20economic%20developmnet.pdf        [ Links ]

12. Carter B, Whittaker K. Examining the British PhD viva: Opening new doors or scarring for life? Contemp Nurse. 2009;32:169-178. http://dx.doi.org/10.5172/conu.32.1-2.169        [ Links ]

 

 

Correspondence:
Aceme Nyika
Faculty of Health Sciences, University of the Witwatersrand Medical School
7 York Road, Parktown 2193, South Africa
Aceme.Nyika@wits.ac.za

Received: 11 Sep. 2013
Revised: 31 Oct. 2013
Accepted: 13 Nov. 2013