GENERAL ARTICLES

 

Bibliometrics to assess the productivity and impact of medical research

 

 

M. Ndwandwe^I; D. G. Bishop^II; R. Wise^III; R. Rodseth^IV

^IDepartment of Anaesthetics (UKZN) University of KwaZulu-Natal Durban, South Africa https://orcid.org/0000-0002-5819-3621
^IIDistrict Clinical Specialist (KZN DOH). *Department of Anaesthetics (UKZN) University of KwaZulu-Natal Durban, South Africa https://orcid.org/0000-0001-9861-3646
^IIIAdult Intensive Care Unit, John Radcliffe Hospital Oxford University Hospitals NHS Foundation Trust, United Kingdom, Department of Anaesthetics (UKZN) University of KwaZulu-Natal Durban, South Africa https://orcid.org/0000-0001-5237-5582
^IVAnaesthesiologist Rodseth and Partners, Department of Anaesthetics (UKZN) University of KwaZulu-Natal Durban, South Africa https://orcid.org/0000-0002-3779-7805

 

 

---

ABSTRACT

BACKGROUND: Bibliometrics is the use of statistical and mathematical analysis to assess research production and quality. These metrics provide important insights into the quality and impact of research by applying standardised metrics. However, there are inherent limitations in their application.
OBJECTIVE: We aimed to review existing bibliometric indices and assess their comparative utility in the assessment of medical researchers. We specifically aimed to evaluate the utility of the h-index in identifying young or developing medical researchers with future research potential.
METHOD: We conducted a focussed literature review on commonly used bibliometrics. To explore the utility of these metrics we then used them to evaluate a sample of researchers from a South African medical school faculty. Researchers were ranked with the following metrics: number of publications; h-index; citations per paper; citations per paper per year; and m-index. The h-index, citations and publication counts were drawn from ResearchGate and, if not available, from Google Scholar. The top 20 researchers, based on publication count, were then analysed further.
RESULTS: We identified 145 researchers for analysis of which 37 were excluded due to an inability to obtain additional information. Higher time-dependent metrics (publication count, citation count, h-index) were directly proportional to years since first publication. Indices that corrected for time, such as the m-index, provided more insight and better discrimination in identifying younger researchers with greater research potential.
CONCLUSION: Bibliometrics have utility as part of the assessment of academic output but may be subject to time-dependent bias. Research quality is best measured using the h-index, g-index and m-index. The h-index is limited by being time dependent and field specific and overlooks highly cited papers. Bibliometrics that account for time, such as the m-index, should be considered in the early identification of young researchers, ideally accompanied by critical peer review.

Keywords: bibliometrics, clinical research, h-index, m-index.

---

 

 

BACKGROUND

Research production has become a driving force in academic medicine, fuelled by a culture of "publish or perish", which demands continuous, quantifiable research output (Carpenter, Cone, and Sarli 2014, 1160). Medical academics are required to maintain and improve their clinical practice, but are also expected to participate in the ongoing training of junior staff while actively participating in research (Carpenter, Cone, and Sarli 2014, 1160). In South Africa this has been compounded by the need to oversee master's degree students as a requirement for the registrar training program and registration with the Health Professions Council of South Africa as a specialist (Biccard et al. 2017, 4). Monetary incentives for publications are thought to have increased research output at universities but with concerns regarding a reduction in research quality (Tomaselli 2018, 1). In South Africa, the Department of Higher Education and Training (DHET) funds universities for research outputs published in accredited journals locally or internationally (Lee and Simon 2018, 1). One South African university saw a 14 per cent increase in output over a four-year period.

It is still uncertain how best to evaluate the research quality of any one particular researcher. Being able to accurately answer this question allows academic institutions the opportunity to: 1) appropriately recognise and reward high quality research output; 2) channel resources to those researchers that will maximise them; and 3) identify young researchers of potentially high calibre that can be mentored and developed.

A variety of different approaches and methods have been designed to answer these questions. In this article we examine a few of the metrics most commonly used in clinical medicine and evaluate their utility in identifying high quality researchers and young researchers of great research potential. To demonstrate the utility and limitations of these metrics we apply them to an anonymised sample of clinical medicine researchers from a South African medical university.

 

BIBLIOMETRICS

Bibliometrics is the use of statistical and mathematical analysis to assess research production and quality. These metrics aim to quantify the quality and impact of research by using an objective standardized measure. As many as 108 bibliometrics have been studied but no single metric exists that fully encompasses the ideal measure of quality.

Raw output

Quantifying research output is of questionable importance when output is simply defined as the number of projects, abstracts, papers or presentations produced. All that is required is to set minimum criteria against which to measure the output and then to determine how many of these outputs a single researcher has produced. This method of quantification, also known as research productivity, is commonly used in many South African universities. Researchers are awarded productivity points based on the number of outputs (be that publications, letters, editorials, or abstracts) produced in a specific year. These points are then used to evaluate and reward research performance and are incorporated into the criteria for academic promotion. Points are divided between authors involved in the research activity. This incentivises aiming for more publications with fewer authors. However, publication quantity has almost no bearing on publication quality. Solely measuring quantity becomes almost meaningless without assigning a measure of quality to the output. The task of quantifying research quality is significantly more challenging.

Publication quality

A simple method to determine quality is to only recognise those papers published in journals indexed in databases such as Medline or EMBASE, or those that are recognised by credentialing organisations such as the South African Department of Higher Education and Training (DHET). The limitation of this method is that it does not differentiate between novel high impact papers and those of lesser importance or impact. In South Africa, with a small academic community and a multiplicity of accredited local journals, low impact papers are more likely to be accepted. This erodes the value of this metric in evaluating quality.

Citations per paper

The number of citations that a paper receives has commonly been used to evaluate research quality. The greater the impact of the paper the more it will be cited by other researchers active in the field. However, the number of citations garnered by a publication is directly related to the time the paper has been in the public domain. Therefore, older papers of lower quality may have a greater number of citations as compared to newer papers of a higher quality. Using this metric benefits older, more established researchers and disadvantages newer, less established researchers.

H-index

The h-index is an attempt to address some of the limitations inherent in raw citation analysis (Hirsch 2005, 16569). A researcher's h-index is the number (n) of published papers that have been cited at least n times by other papers. For example, if a researcher has published 23 papers of which 16 have been cited at least 16 times, then their h-index is 16; or if they have published 100 papers of which 5 have been cited at least 5 times, then their h-index is 5. The advantage of this metric is that it measures both productivity (papers published) as well as the impact of the output (number of citations). Academics who produce a large number of publications receiving an above-average number of citations will have a higher h-index.

As with all metrics there are several limitations inherent to the h-index. The first is that h-indices differ substantially depending on the research field; in general, researchers in the life sciences have significantly higher h-indices as compared to physics researchers (Hirsch 2005, 16572). As a result certain groups have attempted to determine h-index benchmarks within specific specialties to allow meaningful comparison between researchers (MacMaster, Swansburg, and Rittenbach 2017, 452; Svider et al. 2013, 884). Despite this, most of these studies have found a strong correlation between academic rank and h-index, (Pagel and Hudetz 2011, 1085; Rad et al. 2010, 817) suggesting that incorporation of the h-index into promotion criteria may be of value (Tschudy et al. 2016, 272).

The second limitation is that the h-index is in part determined by the length of time that a researcher has been publishing - older researchers will have had longer to accumulate citations which then translates to a higher h-index. The close correlation between the number of citations and the h-index has been described by Yong in his rule of thumb as 0.54 X square root of (Natations), which again highlights the impact of total citations in driving the h-index (Yong 2014, 1042). The third limitation is that the h-index does not capture the impact of very highly cited papers. If a researcher has an h-index of 10 but has a paper that has been cited 10 000 times, his h-index will remain 10. To address this limitation, and to increase the sensitivity of the metric to highly cited papers, the g-index has been proposed (Egghe 2006, 131). In a group of articles ranked in decreasing order of citations received, the g-index is defined as the largest number (g) so that the combination of the citations from the top g articles add up to at least g² or more citations. A g-index of 10 means that the researcher has published at least 10 articles that combined have received at least 100 citations. The g-index allows discrimination between two researchers with the same h-index by identifying the one with the greater number of high impact papers. The definitions for bibliometric indices are shown in Table 1.

Time adjusted metrics

As shown, the time that a publication has been in circulation is an important factor in the number of citations it receives. This is problematic when trying to identify high potential researchers early in their careers. This problem can partially be overcome by adjusting for the time a publication has been available. Adjusting the (citations per paper) by the (number of years since first publication) creates a new metric - (citations per paper per year). This gives an index of the average number of citations that the average paper receives per year.

Similarly, h-index can be modified to create the m-index. The m-index is defined as the h-index, divided by the number of years since the researcher's first published paper. This normalisation allows comparison between researchers irrespective of their career stage. It is the m-index which holds the greatest promise in identifying early-stage high potential researchers. A suggested rule of thumb for interpreting the index is that an m-index of <1 is average; 1-2 is above average; and >2 is excellent.

Alternatives to bibliometrics

In many institutions the use of bibliometrics has been integrated into the process of academic review. However, critics believe peer review provides a more credible measure of research quality as opposed to the blind application of a mathematical formula. It also allows for institutions to correct for time-dependent indices that may be unfair to certain groups (e.g. researchers who have paused careers for child-care or researchers overcoming systemic educational disadvantages). Further, rewarding output based on publication metrics results in authors gaming the system to produce higher rankings as opposed to producing work that adds value to the discipline (Van den Besselaar and Sandström 2019). Bibliometrics also do not take author contribution into account, meaning co-authorship can be used to boost individual metrics.

In an editorial titled "Watch out for cheats in citation game" Mario Biagioli states "All metrics of scientific evaluation are bound to be abused". This formulation of Goodhart's law (named after the British economist who may have been the first to announce it) states that "when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it" (Biagioli 2016, 201). More succinctly - when a measure becomes a target, it ceases to be a good measure. In view of this the Leiden Manifesto was published with ten principles that were identified to assist in holding reviewers and the bibliometrics accountable. These included not using metrics in isolation but aligning them with expert assessments in each field; using metrics to compare similar fields with each other; and developing and using specific measurement tools for specific research fields (Hicks et al. 2015, 429). Metric use often overlooks publications that are not published in English and these publications may go unnoticed despite the value they have in addressing regional issues (Amano, González-Varo, and Sutherland 2016, 1; Hicks et al. 2015, 429).

 

METHODOLOGY

We aimed to evaluate the research productivity, as measured by bibliometric indices, of the faculty at a South African medical university, with particular focus on the h-index. We conducted a retrospective, observational analysis of medical researchers identified through the annual university research report booklet of a South African university. The study was granted exemption from review by the local ethics review committee.

Participants

Using the annual university research report booklet and research database of a South African university, we identified the authors in medicine with the highest number of research outputs as well as those who were identified as upcoming researchers. This included authors from the following disciplines: Microbiology, Paediatrics, General Surgery, Trauma Surgery, Internal Medicine, Rheumatology, Obstetrics and Gynaecology, Cardiology, Endocrinology, Psychiatry, Public Health, Anaesthesia, Plastic Surgery and Orthopaedics. We included all faculty members in the medical school mentioned in the research booklet for the year under review. Researchers whose work was not registered on Google Scholar and/or ResearchGate were not included, as their scores could not be calculated.

Management and data collection

Individual indices were drawn from ResearchGate and if not available, from Google Scholar. We used researcher's names as recorded in the published booklet for correct identification. For eligible researchers the following data were extracted: date of first publication, number of publications, total number of citations and the h-index score. Using these data, we calculated: citation per paper, citation per paper per year and the m-index. A citation was defined as a reference to a published source. Citation per paper was defined as the number of citations divided by papers published. Citation per paper per year was defined as citation/paper/year since first publication. H-index was taken directly from the source database and m-index was calculated using the h-index over the author active publication years. Researchers were then ranked according to each metric, a comparison was done for each metric and any change in ranking was noted.

We initially used the h-index to identify and rank university author productivity. We hypothesised that the use of the h-index may better identify researchers producing high-quality research as well as guide appropriate resource allocation. We then examined the m-index to correct for time-dependence to allow for the early identification of researchers with higher potential.

 

RESULTS

We identified 145 researchers for analysis of which 37 were excluded due to an inability to obtain additional information. We assigned a letter value to researchers to ensure anonymity for publication. Rankings (top 20) according to each of the metrics (number of publications; h-index; citations per paper; citations per paper per year; m-index) can be found in Table 2. The citation counts with increasing duration of research are shown in Figure 1, and the change in h-index with increasing citations is illustrated in Figure 2.

The change in ranking when different metrics were applied provides insight into the utility of each of these metrics. Researchers A, D, and I (all with more than 100 lifetime publications) ranked 1^st, 4^th, and 9^th respectively on number of publications, but fell to 6^th, 18^th and 20^th respectively on h-index ranking. Similarly, researchers N, P and Q (all with less than 50 publications) and initially ranked in the top 20 on number of publications, were not ranked in the top 20 for the h-index. There was little substantial difference between the h-index ranking and citations per paper ranking other than seeing researchers A, D, and I exit the top 20.

Interestingly, A, D, I, N, P and Q were not ranked in the top 20 when using citation per paper or citations per paper per year, despite being in the top 20 for publications. This suggest that these individuals generated a high number of publications with low numbers of citations. More substantial change can be seen when ranking according to citations per paper per year. The top four h-index researchers remained in the top twenty, but the 5^th and 6^th h-index rated researchers (A and C) dropped out of the rankings. Y, BB, CC, and CC newly entered the rankings. Across the four metrics of publications, h-index, citations per paper, and citations per paper per year there are a core of individuals whose ranking remains largely stable - B, E, F, G, H, J, K, L, M, O, S, and T.

The m-index ranking showed the greatest change and the greatest number of unique new entries. One researcher (F) was rated excellent (m-index >2), nine (EE, B, BB, T, G, E, S, A, EE) were rated as above average (m-index 1-2) with the remaining being classified as average. Of note were the new entries EE, BB, and EE who had a rating of 1 or more thereby identifying individuals of significant potential. In the case of these three researchers, the m-index may be identifying individuals with significant scientific research potential. We further examined the metrics of one of the researchers, who was awarded the university's top research award in the previous year. The value of the h-index and m-index are shown in Figure 3. Using a "flag" of m-index >1.5; this researcher would have been identified in 2010 as a young researcher with potential.

 

DISCUSSION

Bibliometrics remain a useful tool for assessing the quality and impact of a researchers work, but with several limitations. Total number of publications and total number of citations are indices that provide good measures of quantity impact but place new researchers at a relative disadvantage to researchers who have published over a longer period. Research quality is best measured through indices such as the h-index, g-index and m-index. While these indices can be used to identify high-performing researchers, some degree of peer review is advisable, as all bibliometrics lack the ability to identify high impact, novel work in major journals.

Academic institutions aiming to improve the quality of the research need to consider alternate incentive schemes. Studies such as the ASOS study, published in the Lancet and involving a collaborative effort of almost 2000 researchers, will earn almost no productivity points for the lead authors on the paper (Biccard et al. 2018, 1589). In contrast, publishing a single author case study in a local journal will earn maximum reward. This lower impact, local research, is nevertheless important and should not be discouraged; particularly as junior researchers and master's students begin their research careers. However, universities need to be wary of incentivising this work over higher impact research, especially in light of the current Masters requirement for registrars, which often consumes a disproportionate amount of time for supervisors and often produces low impact work (Rodseth, Wise, and Bishop 2017, 5).

A key component in research programmes is the early identification, mentoring and support of young researchers with significant potential. Commonly used bibliometrics are disproportionately influenced by time in the field, resulting in difficulty flagging these young researchers. Initial screening systems aiming to identify this group should consider using metrics that account for time, such as the m-index. In our analysis, the research prize-winner would have been identified six years earlier, using a threshold of 1.5. At this stage, the researcher had an h-index of 3, a level that would not normally draw attention. Academic institutions may be able to provide better support to these researchers when using screening tools such as this, coupled to peer review of this group. It would also enable appropriate identification of older (but junior) researchers who have recently entered the field and are behind their colleagues in academic ranking. This group would include those that have paused their careers due to child-care (predominantly women) and those that have had to overcome educational disadvantages.

Implications of this analysis for higher education

Our analysis suggests that the use of bibliometrics may form a key part of recognising, rewarding and developing research excellence. There are several Department of Higher Education and Training (DHET) programmes that seek to identify high-performance individuals. These include the New Generation of Academics Programme (nGAP), which "recruits new permanent university academics, while at the same time improving staff demographic profiles and addressing the ratio of permanent to temporary staff members" (Department: Higher Education and Training 2019, 4. Additionally, the Nurturing Emerging Scholars Programme seeks to support and encourage the pursuit of an academic career from high-achieving postgraduate students. The mechanism for achieving these goals includes the University Capacity Development Grant, which aims for "the creation of an academic development pipeline that enables the recruitment of adequate numbers of new academics in ways that transform the academic workforce and that provides for quality research development and teaching development opportunities for academic and professional staff along the full career trajectory from recruitment to retirement" (Department: Higher Education and Training 2019, 1-56). This grant operates through a number of programmes (such as the Existing Academic Staff Capacity Enhancement Programme), and aims to assist with the recruitment and retention of existing staff and the development of research capacity within institutions. Systems that aim to identify the individuals that should benefit from these programmes could incorporate bibliometric analysis as screening tools. Additionally, the application of time-independent metrics will potentially identify those who are overcoming educational disadvantages by recognising excellence over a shorter time-frame. Appropriate research support, seed funding and academic promotion may then be allocated following a process of peer review.

 

STUDY LIMITATIONS

Our analysis was performed using authors from one published university report and may therefore have limited generalisability. However, the principles behind the analysis should be relevant to all academic institutions. It is also possible that data for researchers was incomplete on Research Gate or Google Scholar, as it relies on some degree of author maintenance.

 

CONCLUSIONS

Bibliometrics have utility as part of the assessment of academic output but may be subject to time-dependent bias. While research quality is best measured through indices such as the h-index, g-index and m-index; the commonly used h-index is limited by being time dependent and field specific and overlooks highly cited papers. Bibliometrics that account for time, such as the m-index, should be considered in the early identification of young researchers, ideally accompanied by critical peer review.

 

REFERENCES

Amano, Tatsuya, Juan P. González-Varo, and William J. Sutherland. 2016. "Languages are still a major barrier to global science." PLoS Biology 14(12): 1-8.         [ Links ]

Biagioli, M. 2016. "Watch out for cheats in citation game." Nature 535(7611): 201. doi: 10.1038/ 535201a.         [ Links ]

Biccard, B. M., R. A. Dyer, J. L. Swanevelder, J. F. Coetzee, and S. L. Shafer. 2017. "Is the HPCSA requirement for a research dissertation for specialist registration the best option?" Southern African Journal of Anaesthesia and Analgesia 23(4): 4-6.         [ Links ]

Biccard, B. M., T. E. Madiba, H. L. Kluyts, D. M. Munlemvo, F. D. Madzimbamuto, A. Basenero, C. S. Gordon, C. Youssouf, S. R. Rakotoarison, V. Gobin, et al. 2018. "Perioperative patient outcomes in the African Surgical Outcomes Study: A 7-day prospective observational cohort study." Lancet 391(10130):1 589-1598. doi: 10.1016/S0140-6736(18)30001-1.         [ Links ]

Carpenter, Christopher R., David C. Cone, and Cathy C. Sarli. 2014. "Using publication metrics to highlight academic productivity and research impact." Academic Emergency Medicine 21(10): 1160-1172.         [ Links ]

Department of Higher Education and Training. 2017. "Ministerial Statement on the Implementation of the University Capacity Development Programme through Effective Management and Utilisation of the University Capacity Development Grant 2018-2020." South Africa. https://www.dhet.gov.za/UCD%20Policies/Ministerial%20Statement%20on%20the%20Impleme ntation%20of%20the%20University%20Capacity%20Development%20Programme.pdf.         [ Links ]

Department of Higher Education and Training. 2019. "Strategic Plan 2020-2025": 1-56. South Africa. https://www.dhet.gov.za/SiteAssets/Strategic%20Plans/DHET%20Amended%20SP%202020-2025.pdf        [ Links ]

Egghe, Leo. 2006. "Theory and practise of the g-index." Scientometrics 69(1): 131-152. doi: 10.1007/s11192-006-0144-7.         [ Links ]

Hicks, Diana, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015. "Bibliometrics: The Leiden Manifesto for research metrics." Nature 520(7548): 429-431.         [ Links ]

Hirsch, J. E. 2005. "An index to quantify an individual's scientific research output." Proceedings of the National Academy of Sciences of the United States of America 102(46): 16569-16572. doi: 10.1073/pnas.0507655102.         [ Links ]

Lee, Alan T. K. and Carol A. Simon. 2018. "Publication incentives based on journal rankings disadvantage local publications." South African Journal of Science 114(9-10): 1-3.         [ Links ]

MacMaster, F. P., R. Swansburg, and K. Rittenbach. 2017. "Academic Productivity in Psychiatry: Benchmarks for the H-Index." Academic Psychiatry 41(4): 452-454. doi: 10.1007/s40596-016-0656-2.         [ Links ]

Pagel, P. S. and J. A. Hudetz. 2011. "H-index is a sensitive indicator of academic activity in highly productive anaesthesiologists: Results of a bibliometric analysis." Acta Anaesthesiol Scand 55(9): 1085-1089. doi: 10.1111/j .1399-6576.2011.02508.x.         [ Links ]

Rad, A. E., W. Brinjikji, H. J. Cloft, and D. F. Kallmes. 2010. "The H-index in academic radiology." Academic Radiology 17(7): 817-21. doi: 10.1016/j.acra.2010.03.011.         [ Links ]

Rodseth, R. N., R. D. Wise, and D. Bishop. 2017. "Polluting the well." Southern African Journal of Anaesthesia and Analgesia 23(6).         [ Links ]

Svider, P. F., A. A. Pashkova, Z. Choudhry, N. Agarwal, O. Kovalerchik, S. Baredes, J. K. Liu, and J. A. Eloy. 2013. "Comparison of scholarly impact among surgical specialties: an examination of 2429 academic surgeons." Laryngoscope 123(4): 884-889. doi: 10.1002/lary.23951.         [ Links ]

Tomaselli, Keyan G. 2018. "Perverse incentives and the political economy of South African academic journal publishing." South African Journal of Science 114(11-12): 1-6.         [ Links ]

Tschudy, M. M., T. L. Rowe, G. J. Dover, and T. L. Cheng. 2016. "Pediatric Academic Productivity: Pediatric Benchmarks for the h- and g-Indices." The Journal of Pediatrics 169: 272-276. doi: 10.1016/j.jpeds.2015.10.030.         [ Links ]

Van den Besselaar, Peter and Ulf Sandström. 2019. "Measuring researcher independence using bibliometric data: A proposal for a new performance indicator." PloS one 14(3).         [ Links ]

Yong, Alexander. 2014. "Critique of Hirsch's citation index: A combinatorial Fermi problem." Notices of the AMS 61(9): 1040-1050.         [ Links ]