SciELO - Scientific Electronic Library Online

vol.14 issue1Weighting of items in a tutorial performance evaluation instrument: Statistical analysis and resultsA qualitative investigation of south african cigarette smokers’ perceptions of fear appeal messages in anti-smoking advertising author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand



Related links

  • On index processCited by Google
  • On index processSimilars in Google


Health SA Gesondheid (Online)

On-line version ISSN 2071-9736
Print version ISSN 1025-9848

Health SA Gesondheid (Online) vol.14 n.1 Cape Town  2009




Using subjective judgement to determine the validity of a tutorial performance evaluation instrument



Judith C. Bruce; Melanie L. Lack

Department of Nursing Education, University of the Witwatersrand, South Africa





Evaluating students' learning performance is dependent on assessment criteria from which valid inferences can be made about student learning. An existing 36-item instrument used to evaluate baccalaureate nursing students' performance in problem-based learning tutorials was presented to experts in nursing for their subjective judgement of item validity. Quantitative analysis of data sets from experts' judgements was used to construct a valid measurement scale for evaluating students' tutorial performance. The objectives of the study were to determine the content validity of items in a tutorial performance evaluation (TPE) instrument and to determine the construct validity of items through paired comparison of main and sub-items in the instrument. Academic experts (n = 8) from two South African universities were selected by means of purposive, maximum variation sampling. Data were collected in three rounds of the Delphi technique, which incorporated the Subjective Judgement Model for paired comparison of instrument items. Experts' ratings were captured on a visual analogue scale for each item. Relative item weights were determined using paired comparisons. Statistical analysis resulted in ratio scale data, each item being assigned a ratio relative to its weight. It was concluded that quantitative analysis of subjective judgements is useful to determine the construct validity of items through paired comparison of items in a TPE instrument. This article presents the methodological perspectives of subjective judgement to establish instrument validity.

Keywords: problem-based learning; quantitative analysis; subjective judgement; tutorial performance; validity; nursing


Die evaluering van studente se leervermoë is afhanklik van die waardebepalingskriteria waarvan geldige afleidings betreffende die student se leerervaring gemaak kan word. 'n Bestaande instrument met 36 items waarmee baccalaureus-verpleegkundestudente se prestasie in die probleemgebaseerde leertutoriale geëvalueer is, is aan kundiges in verpleegkunde gegee vir subjektiewe beoordeling van die geldigheid van die items. 'n Geldige meetinstrument vir die evaluering van studente se tutoriale prestasie is ontwerp deur van die kwantitatiewe ontleding van die datastelle op grond van die kundiges se oordeel gebruik te maak. Die doelwitte van die studie was om die inhoudsgeldigheid van items in 'n evalueringsinstrument van tutoriale prestasie te bepaal en om die konstrukgeldigheid van items te bepaal deur die gepaarde vergelyking van hoof- en sub-items in die instrument. Akademiese kundiges (n = 8) van twee Suid-Afrikaanse universiteite is deur middel van doelgerigte, maksimale variasie-steekproeftrekking geselekteer. Data is deur middel van drie rondtes van die Delphi-tegniek ingesamel, wat die subjektiewe oordeelmodel vir gepaarde vergelyking van die instrumentitems ingesluit het. Die kundiges se beoordeling is op 'n visueleanaloë-skaal vir elke item weergegee. Relatiewe itemgewigte is deur middel van gepaarde vergelyking bepaal. Statistiese ontleding het verhoudingskaaldata tot gevolg gehad, en elke item is van 'n verhouding relatief tot die gewig voorsien. Daar is bevind dat kwantitatiewe ontleding van subjektiewe beoordeling bruikbaar is om die geldigheid van 'n konstruk deur gepaarde vergelyking van items in 'n evalueringsinstrument van tutoriale prestasie te bepaal. Hierdie artikel bied die metodologiese perspektiewe van die subjektiewe beoordeling aan om die geldigheid van die instrument te bepaal.



“Full text available only in PDF format”




Burns, N. & Grove, S.K., 2005, The practice of nursing research conduct: Critique and utilization, WB Saunders, New York.         [ Links ]

Crawford, G. & Williams, C., 1985a, The analysis of subjective judgement matrices. A project AIR FORCE report, Rand, Santa Monica.         [ Links ]

Crawford, G. & Williams, C., 1985b, 'A note on the analysis of subjective judgement matrices', Journal of Mathematical Psychology 29, 387-405.         [ Links ]

David, H.A., 1963, The method of paired comparison, Charles Griffin, London.         [ Links ]

Dornan, T., Boshuizen, H., Cordingley, L., Hider, S., Hadfield, J. & Scherpbier, A., 2004, 'Evaluation of self-directed clinical education: Validation of an instrument', Medical Education 38, 670-678.         [ Links ]

Downing, S.M., Tekian, A. & Yudkowsky, R., 2006, 'Procedures for establishing defensible absolute passing scores on performance examinations in health professions education', Teaching and Learning in Medicine 18(1), 50-57        [ Links ]

Lederer, A. & Prasad, J., 1992, 'Nine management guidelines for better cost estimating', Communication ACM 35(2), 51-59.         [ Links ]

Lynn, M.R., 1986, 'Determination and quantification of content validity', Nursing Research 35(6), 382-385.         [ Links ]

McCarthy, M.A., Keith, D., Tietjen, J.,Burgman, M.A., Maunder, M., Master, L., Brook, B.W., Mace, G., Possingham, H.P., Medellin, R., Andelman, S., Regan, H., Regan, T. & Ruckelshaus, M., 2004, 'Comparing predictions of extinction risks using models and subjective judgement', International Journal of Ecology 26, 76-74.         [ Links ]

Mindtools E-books, 2006, Paired comparison analysis. Working out the relative importance of different options, Mindtools, viewed 17 June 2006, from         [ Links ]

Miranda, E., 2001, Improving subjective estimates using paired comparisons, IEEE Software, Ericsson Research Canada, Mississauga, Ontario.         [ Links ]

Niemenin, J., Saure, P. & Lonka, K., 2006, 'On the relationship between group functioning and study success in problem-based learning', Medical Education 40, 64-71.         [ Links ]

Oermann, M.H. & Gaberson, K.B., 2006, Evaluation and testing in nursing education, Springer Publishing Company, New York.         [ Links ]

Patton, M.Q., 2002, Qualitative research and evaluation methods, Sage, Thousand Oaks.         [ Links ]

Polit, D.F. & Beck, C.T., 2004, Nursing research principles and methods, Lippincott Williams and Wilkins, Philadelphia.         [ Links ]

Rideout, E., 1999, Transforming nursing education through problem based learning, Jones and Bartlett Publishers International, London.         [ Links ]

Savin-Baden, M., 2000, Problem-based learning in higher education: Untold stories, viewed 7 July 2006, from:         [ Links ]

Smith, A.J., Thurkettle, M. & De la Cruz, F.A., 2004, 'Use of intuition by nursing students: Instrument development and testing', Methodological Issues in Nursing Research 47(6), 614-622.         [ Links ]

Wewers, M.E. & Lowe, N.K., 1990, 'A critical review of visual analogue scales in the measurement of clinical phenomena', Research in Nursing and Health 13(4), 227-236.         [ Links ]



Judith Bruce
Department of Nursing Education
Faculty of Health Sciences
7 York Road, Parktown
Johannesburg, 2193
South Africa

Received: 15 Oct. 2007
Accepted: 21 Nov. 2008
Published: 25 May 2009

Creative Commons License All the contents of this journal, except where otherwise noted, is licensed under a Creative Commons Attribution License