RESEARCH ARTICLE

 

The Development and Refinement of a Questionnaire on the Investigation of Students' Experience of First-Year Chemistry Practicals

 

 

Liezel Retief^I; Marietjie Lutz^I; Marietjie Potgieter^II

^IDepartment of Chemistry and Polymer Science, Stellenbosch University, Private BagXI, Matieland, 7602, South Africa
^IIDepartment of Chemistry, University of Pretoria, Private Bag X20, Hatfield, 0028, South Africa

Correspondence

 

 

---

ABSTRACT

This article reports on the development and refinement of a questionnaire for first-year Chemistry students. Three subscales were probed, namely affective attitudes, perceptions towards tutor effectiveness and outcomes achieved.

Keywords: Attitudes, perceptions, chemistry, practical training.

---

 

 

Laboratory practicals play an important role in the teaching and learning of Chemistry. The current first-year Chemistry practical curriculum at the university at which the study was conducted has no defined outcomes, aims and objectives for the practical sessions. Reid and Shah have argued that chemistry practical curricula should be rethought in order to develop practical activ-ities that are much better suited for the outcomes we wish the students to attain.¹ Literature also suggests that the students' perceptions of the usefulness and applicability of a subject, method or technique impacts their success rate in that subject and that laboratory work has an impact on enhancing student attitudes towards science; stimulating interest, enjoyment and motivation of science learning.^2-5 It is therefore of interest to us to investigate what the first-year students' perceptions and attitudes are towards Chemistry practical training and how these change throughout the year. These include their affective attitudes, their perceptions regarding the effectiveness of the tutors, and their perceptions of the outcomes that they achieve through the practical training. We are also interested in the investigation of the perceptions of the lecturers and tutors of the outcomes achieved for Chemistry practicals and how these correlate with those of the students.

Sproul suggests that a questionnaire is an appropriate research tool when attempting to collect information about people's 'atti-tudes, values, beliefs, or self-reports...'.⁶ DeVellis states that when a construct cannot be measured directly, then a question-naire that contains scale items which represent the desired con-struct can be a useful means of measure.⁷ A survey was therefore identified as the most practical and effective method available to measure students' attitudes and perceptions. Although literature indicates the presence of several attitude studies using question-naires as test instruments, Blalock et. al. have reported in a review article that the majority of science attitude test instru-ments show an absence of psychometric evidence and have sev-eral methodological deficiencies including the absence of infor-mation on instrument reliability, validity and a disregard for missing data.⁸ Therefore, we felt the need to develop our own test instrument which would be tailor-made for our context and would not suffer from these deficiencies. In order to address issues of instrument reliability and validity we decided to use the Rasch model for instrument analysis and refinement. The Rasch Latent Trait model has been shown to be highly suitable for this purpose.⁹ It is a probabilistic model that describes the interaction of persons with given abilities with items of given difficulties to produce data that comprehensively describe person perfor-mance on a single latent trait. Raw scores collected by means of a test instrument such as a questionnaire are non-linear in nature but are transformed into linear measures by means of the Rasch model.^9,10

An exploratory sequential mixed-method approach was used. A Likert scale questionnaire was developed where the items and response options were qualitatively generated based on informal small group and individual interviews with students, student written feedback and e-mail questions sent to lecturers, as well as information obtained from literature. Using the students' and lecturers' own wording in the questionnaire ensured both face and construct validity. The questionnaire involved the use of a five point Likert scale with options 'strongly disagree, disagree slightly, neutral, agree slightly, strongly agree' applied to the three subscales probing student perceptions and affective atti-tudes regarding the laboratory experience, the effectiveness of the tutors/demonstrators, and the outcomes expected from the laboratory training.

The questionnaire was administered three times during the year: at the start of the first semester, at the end of the first semester and again at the end of the year. The purpose of data collection at the beginning of the year was to probe the expectations of the students. However, the questionnaires were only administered at the end of the first practical session in order to give the students some exposure to what the Chemistry practical training entails. This was followed up by data collection at the end of the last practical session of the respective semesters with the purpose of probing the students' actual experience of the Chemistry practi-cals. A separate questionnaire was distributed to the lecturers and tutors, with regard to what they believed the outcomes achieved should be and what outcomes they believed were achieved.

This study started in 2011 and the last two years have been spent on piloting and refining the questionnaire using Rasch Latent Trait Theory. In a previous publication we illustrated and explained the various aspects of the Rasch analysis process as applied to the data obtained for one of the subscales, namely those questions that dealt with tutor effectiveness.¹¹ In this paper we report on the specific aspects of the subscales, where Rasch analysis indicated that the questionnaire required refinement. We report on the relevant changes that were made based on this analysis process and describe the way forward.

Rasch analysis indicated a few problems with the Likert scale which is shown in Table 1 for the subscale on tutor effectiveness. Firstly the neutral category was found to be problematic, skewing statistical analysis of the data. This is most likely due to an ambi-guity in interpretation of the category. Some respondents may perceive this option to fit the endorsement scale as a middle option while others might find a specific statement to not be applicable in their context and thus choose the neutral option for such indication. We therefore removed the neutral category from the Likert scale, thus reducing the scale to only four catego-ries and a 'not applicable' option was added to the side of the Likert scale. The response frequencies obtained for this 'not applicable' option will serve as an indicator of whether a specific item is perceived to be not relevant to the experience of students within the context of their practical training. The finding in our study that the neutral option was problematic is not new. Sev-eral researchers have observed similar complications and have made recommendations for removing the ambiguities intro-duced by such a middle option in the rating scale.¹²

The second problem that Rasch analysis pointed out is that of misfitting items. Rasch analyses generate a set of fit statistics which can be used to identify items that are not functioning optimally. These items must then be analysed qualitatively in order to find possible reasons for misfit. For example, in the subscale on tutor effectiveness (Table 1), items 6.iv ('made mistakes often') and 6.viii ('confused you regularly') showed unacceptable misfit and upon inspection it was clear that they were phrased ambiguously.' The words 'often' and 'regularly' were thus removed for the refined questionnaire. The wording of the Likert scale itself was also changed. The words 'slightly' in the 'slightly disagree' and 'slightly agree' categories were furthermore removed to avoid possible complications between the adjacent categories, 'slightly agree' and slightly disagree'. In the refined version the Likert scale has only four categories 'strongly disagree, disagree, agree, strongly agree' with the addi-tional option 'not applicable' on the side.

Lastly, Rasch analysis indicated four other items from the subscale that dealt with affective attitudes (Table 2) that were problematic. Three of these items were investigated qualita-tively and showed potential duplication with other items on the questionnaire. These items and their duplicates included items 5.iii and 5.iv ('lost' and 'confused'), items 5.vi and 5.vii ('scared' and 'nervous'), items 5.x and 5.xi ('enthusiastic' and 'excited'). Upon refinement of the questionnaire, one item of each of these duplicate sets were removed, namely items 5.iii, 5.vi, and 5.xi. The last item that was flagged as misfitting was item 5.xiv ('to keep you busy'). Qualitative investigation of this item suggested it did not fit the latent trait (affective attitudes) that was being measured and was thus removed from the questionnaire.

Two additional South African universities are joining the project in 2013. Data obtained from the questionnaires will be compared between the different cohorts and results will be used to inform decisions regarding improvement of practical training with particular attention to affective responses of students and perceived tutor effectiveness seeing that such factors are important for student dedication and motivation. The aim of such a large intra-institutional study is to exchange information on best practises between the Chemistry departments of the various cohorts in order to increase the effectiveness of the first-year Chemistry practical curriculum and the way it is implemented. Results from this phase of the project will be communicated later.

Acknowledgements

We thank the National Research Foundation of South Africa, the Centre for Teaching and Learning and Stellenbosch Univer-sity for financial support. We also thank David Andrich and Ida Marais from the University of Western Australia for technical advice on using Rasch analysis for the refinement of the scale.

 

References

1 N. Reid and I. Shah, Chem. Educ. Res. Pract, 2007, 8(2), 172-185.         [ Links ]

2 A.H. Johnstone, Chem. Educ. Res. Pract. Eur., 2000,1(1), 9-15.         [ Links ]

3 J. Henderleiter and D.L. Pringle, J. Chem. Educ, 1999, 76(1), 100-106.         [ Links ]

4 A. Hofstein and V.N. Lunetta, Sci. Educ, 2004, 88, 28-54.         [ Links ]

5 L.D. Shulman and P. Tamir, Research on teaching the natural sciences, in Second handbook of research on teaching, (R.M.W Travers, ed.), Rand McNally, Chicago, USA, 1973, pp. 339-345.         [ Links ]

6 N.L. Sproul, Handbook of Research Methods: A Guide for Practitioners and Students in the Social Sciences, 1st edn., The Scarecrow Press, New York, USA, 1988.         [ Links ]

7 R.F. DeVellis, Scale Development, 2nd edn., SAGE Publications, Califor-nia, USA, 2003.         [ Links ]

8 C.L. Blalock, M.J. Lichtenstein, S. Owen, L. Pruski, C. Marshall and M. Toepperwein, Int. J. Sci. Educ., 2008, 30(7), 961-977.         [ Links ]

9 W. Boone and J. Rogan, AJRMSTE, 2005, 9(1), 25-38.         [ Links ]

10 J.P. Ryan, Introduction to latent trait analysis and item response theory, in Testing in the Schools. New Directions for Testing and Measure-ment, (W.E. Hathaway, ed.), Jossey-Bass, San Francisco, USA, 1983, pp. 49-65.         [ Links ]

11 L. Retief, M. Potgieter and M. Lutz, AJRMSTE, 2013,17(1&2), 126-138.         [ Links ]

12 Andrich, D., de Jong, J.H.A.L., Sheridan, B.E., Diagnostic opportuni-ties with the Rasch model for ordered response categories, in Applica-tions of Latent Trait and Latent Class Models in the Social Sciences, (J.R.a.R. Langeheine, ed.), Waxmann Verlag GMBH, Munster and New York, Germany and USA, 1997, pp. 58-68.         [ Links ]

 

 

Correspondence:
E-mail: liz@sun.ac.za

Received 11 April 2013
Revised 30 August 2013
Accepted 1 September 2013