RESEARCH ARTICLES

 

Perspectives of first-year Business Studies students on the Certificate of Information Literacy: a case study of the Cape Peninsula University of Technology

 

 

Yunus Omar^I; Zulaiga Davids^II

^ISenior Librarian at the Cape Peninsula University of Technology, South Africa omary@cput.ac.za ORCID: 0000-0003-2701-1781
^IIFaculty Librarian: Business at the Cape Peninsula University of Technology, South Africa davidsz@cput.ac.za ORCID: 0000-0003-3625-6334

 

 

---

ABSTRACT

This paper presents the results of a survey on the perspectives of students and lecturers after students had attended a course on information literacy (IL) training. A student self-assessment survey aimed to explore what IL skills students are applying and how they feel about completing the IL course. A second survey gained feedback from lecturers to verify whether students had applied IL skills within their subjects after taking the IL course. The population studied was Business Studies students at first year and foundation level, drawing a sample from Communication and English courses. Findings indicate that lecturers have seen an improvement in the academic work of students after their IL training. Students perceived an improvement in their skills to evaluate sources - a skill most applied in their academic work. The value of the study lies in its focus on how students view their IL skills. The study finds that students see IL training as beneficial, not only in Communication and English, but in their other subjects as well. They also see the benefit it will have for all students at a university. Literature suggests that soliciting students' perspectives on IL is under-researched. This case study adds to research in that area.

Keywords: Information literacy certificate, student perspective, self-assessment, academic perspective, Kirkpatrick Four Level of Learning Evaluation Model

---

 

 

1 Introduction

This paper brings to a close a case study of the implementation of an outcomes-based short course, the Certificate of Information Literacy (CIL), at the Cape Peninsula University of Technology (CPUT). The course is based on the Association of College and Research Libraries (ACRL) Information Literacy (IL) Competency Standards (American Library Association [ALA] 2000). IL training has been identified as a need at CPUT as many of its students had disadvantaged schooling experiences. Lanning and Mallek (2017: 448) reported that first-year and foundation-level students enter university without the necessary IL skills to meet the demands of research-related tasks. In our previous article (Davids and Omar 2018), we discussed lecturers' and librarians' assessment of the CIL course; lecturers agreed that the implementation of the course had been successful. Students had not yet assessed the course because assessment could only be done over time, after they had had several opportunities to apply their IL skills. As librarians with extensive experience in IL, the authors of this paper recognise that IL is complex and requires repeated application and analysis before proficiency can be reached.

Students at first year and foundation level in CPUT's Business Faculty's Extended Curriculum Programme (ECP) attended IL training as part of their Communication or English courses. Training consisted of a series of five sessions (one per ACRL standard) of 90 minutes each during the Communication or English lecturer's class time. Various in-class formative assessments tested students' short-term understanding and application of IL skills. Thereafter, students completed a multiple-choice test as a summative assessment task. Test scores contributed towards students' course mark. After the training, students should have been able to apply the learning outcomes that were set for each module and to understand the nature, creation and dissemination of information, use scholarly encyclopaedias, dictionaries and thesauri to find background information, search the library catalogue and databases to find relevant books and peer-reviewed journal articles, differentiate between scholarly and popular sources, apply criteria to evaluate a variety of sources, adhere to copyright, avoid plagiarism, and accurately create in-text references and bibliographies. Librarians and lecturers used a compulsory assignment to assess whether learning outcomes had been achieved and to ensure that students had practical experience of applying IL skills in the context of their academic studies.

The focus of this paper is a self-assessment questionnaire which determined how students rated their own IL skills and to what extent they applied IL skills in their academic work after attending IL training. Lecturers completed a questionnaire to validate students' responses. Both questionnaires were analysed using Kirkpatrick's Four Levels of Learning Evaluation model (hereafter referred to as the KP model). The KP model is a framework which offers a practical approach to assess instruction. Levels 1 to 3 relate to students, while Level 4 is useful to librarians and lecturers. Level 1 does not assess what students have learned, but rather the extent to which new students were satisfied with the training activity offered, in other words, to what extent they experienced a supportive learning environment and a positive, beneficial learning experience. Level 2 assesses the degree to which students grasped the skills taught to them and whether they had opportunities to practice them, resulting in their gaining new knowledge. Level 3 assesses whether there is empirical evidence that students had transferred the skills learnt in Level 2 to their research assignments. For example, did the students apply skills in keyword searching, searching the library's databases, or citing and referencing accurately? This application would demonstrate a change in behaviour of students, indicating an increase in IL skills. To gauge the accuracy of the students' responses, lecturers were asked to validate them. Level 4 indicates to the librarians whether the IL instruction has been effective for new students and whether training needs to be improved (Schwandt 1998, Turnbow 2015).

The suitability of the KP model to the CIL is that the model also focuses on the attainment of outcomes of educational programmes. The model is used widely in educational settings, is suitable for informal or formal training, and will be used in this study to measure lower and cognitive levels of IL skills of students within four scaffolded levels (as per Bates 2005: 221) as well as to measure the quality of IL instruction delivered. The four levels are reaction (did students like the course?), learning (did students understand the content?), behaviour (have students applied the content?) and results (did students show an improvement in academic performance?) (Kirkpatrick 1998: 19). This paper uses the KP model to articulate the views of students and lecturers on the application of IL skills in students' academi c work subsequent to their attending the CIL.

 

2 Research problem

Various formative assessments indicated that students understood and were able to apply the IL concepts taught during the CIL. Most students were successful in the summative assessment test. CIL instructors were hopeful that students would retain the IL skills and apply them in their academic work after the training. However, as pointed out by Fielding et al. (2013), skills are not always applied when, for example, instead of using the skills they have been taught, it is so easy for students to revert to the internet as their main search tool for information sources. We needed to establish whether the self-assessment by the students and the lecturers' responses would confirm what students know and are able to do in terms of IL and whether their information behaviour changed as a result of the CIL training. In this follow-up case study, the researchers focus on the application of IL skills. The main goals were to investigate, firstly, the level of IL applied in the students' academic work, and, secondly, whether lecturers can confirm that students have applied the IL skills and that they can see improvement in their academic work. We posed four research questions:

• Q1: Has transfer of IL skills taken place?

• Q2: Have students applied IL skills in their academic studies?

• Q3: Have lecturers observed the application of IL in the academic work of students?

• Q4: Have lecturers seen any difference in students' academic work?

The questions posed would provide us with insight into the scope of IL awareness that students possess after attending IL training.

 

3 Literature review

The value of IL is evident in studies that link IL to enriching student learning. Yevelson-Shorsher and Bronstein (2018: 535) found that IL training was unsuccessful in their particular study and that students still struggled with IL. Students, lecturers and librarians were surveyed on their views of problems associated with student acquisition of IL skills. The survey responses would be used to develop an ideal IL programme that would meet the needs of the students, as voiced by the students (Yevelson-Shorsher and Bronstein 2018: 542). Despite the problems, students who attended IL training said they knew the importance of IL skills and would apply self-learning methods to further develop them (Yevelson-Shorsher and Bronstein 2018: 542). The study recommended that lecturers, librarians and students should collaborate to develop current IL training into an ideal IL training programme to which students would attach value (Yevelson-Shorsher and Bronstein: 541). Students need to be constantly engaged with IL in order to gain increased value and benefit from it. Kim and Shumaker (2015: 453) surveyed students, lecturers and librarians on the importance and impact of IL teaching, and about the IL skills that students gained in two courses. Findings indicated that "students' grades were positively correlated with their self-ratings of IL skills for both courses" and lecturers saw improvement in the academic work of students (Kim and Shumaker 2015: 454). Students rated the importance of IL higher in the course where IL was substantially integrated into assignments than the course with less IL integration (Kim and Shumaker 2015: 455). However, students attached less importance to IL than lecturers and librarians, highlighting the need for closer collaboration between lecturers and librarians to ensure that students are taught why IL skills are important and learn various methods of how to apply IL skills to their work (Kim and Shumaker 2015: 456). Persistent engagement with IL is needed for IL to be of value and of benefit to students. Fielding et al. (2013: 119) reported on studies linking IL to improved student outcomes: that even within a single course over a six-week period, students still showed positive IL outcomes compared to multiple IL exposure in different subject areas. Detlor et al. (2012: 147) indicated that, during the learning process, students who are engaged in IL apply critical thinking skills which have a lasting effect on student learning outcomes. Studies also reveal that improvement in IL skills correlate positively with student writing scores and final grades (Shao & Purpur 2016: 670).

In an earlier paper (Davids & Omar 2018), we reported that we shifted our pedagogical approach in the CIL course from traditional to active learning methods, such as blended learning and problem solving where students fully participated in IL activities. Previous IL assessment results indicated that students benefited from the active learning approach (Davids & Omar 2018). Assessment is fundamental to teaching and learning at universities. Libraries, too, need to demonstrate a culture of assessment by using recognised measurable outcomes to provide reliable evidence about "what students have actually learned and what they are able to do following instruction" (Chan 2016: 51). The works of Oakleaf (2008: 233) and Erlinger (2018: 443) asked questions about the assessment methods used by librarians. Self-assessment is defined by Butler (2018: 2) as assessment in which students "evaluate their own performance or knowledge based on some criteria". Assessment in this context means assessing students' strengths and weaknesses in each of the IL skill areas. The purpose of assessment is to collect information about students' acquired skills. IL instruction and assessment are well documented in the library and education literature. Over the past half-century, researchers have focused on self-assessment or self-evaluation (Falchikov and Boud 1989: 395). Researchers report that, since 2000, assessment has been a key theme in the Reference Services Review journal after the ACRL IL Competency Standards were published (Reynolds et al. 2016: 537, Johnson et al. 2018), A prevailing theme in the literature is for libraries to measure the impact of IL instruction to demonstrate value (Detmering et al. 2015: 534). According to Stevenson (2012: 81), the KP model is the most widely used typology to assess IL training in educational environments and is a valuable approach to IL assessment. The levels signify how affective processes (Levels 1 and 2) can lead to cognitive outcomes (Levels 3 and 4). The model is progressive: as the levels increase, more valuable information is gathered. Erlinger (2018: 446) explained that the KP model classifies assessment activities based on what researchers wish to find out about their students. Each level has a distinct assessment goal posed as a question and uses suitable assessment methods to obtain a picture of the perception of an IL programme. Erlinger (2018: 446) explained the levels as follows:

• Level 1: Reaction - Elicits whether students had positive reactions (thoughts, feelings and satisfaction) towards the IL training.

• Level 2: Learning - Examines changes in students' attitudes by assessing whether students gained new knowledge and increased their skills.

• Level 3: Behaviour - Monitors changes in students' behaviour by observing whether transfer of skills has taken place.

• Level 4: Results - Considers whether IL training resulted in improved student performance (applying IL concepts, reduction in plagiarism, and improvement in referencing).

We will use the KP model to reflect on the qualitative data about students' perspectives on the value that they place on the IL training and whether it has led to improved student performance. The lecturers' responses will be used to substantiate students' responses.

De Saulles (2015: 124) asserted that students are not aware of the importance of IL skills and that librarians and educators should make it their duty to assist and develop students' IL skills. In the study by Hartman (2001: 111), poor attendance rates of IL training by students, together with their overestimation of their IL skills, prompted a re-evaluation of the design of an IL programme. Students had limited IL skills and relied heavily on non-authoritative information sources, such as Google and Wikipedia (Detlor et al. 2012: 148, De Saulles 2015: 124). In a study by Fielding et al. (2013: 116), students stated that Google was the first place they looked for information. Maurer, Schloegl and Dreisiebner (2017: 314) reported that students from the Faculty of Business, Economics and Social Sciences at the University of Graz (Austria) had the lowest level of IL of all academic disciplines tested, and scored the lowest grades. Students use mainly one-word searches and seldom use Boolean operators (Seamans 2002: 116). Hartman (2001), Seamans (2002) and Saunders (2012) found that students are generally uncritical when using information sources. Taylor and Dalal (2017: 92) stated that college students choose internet sources over library sources and most of them are unaware of sound evaluation criteria which need to be applied to find quality information. However, according to Seamans (2002:119), students are aware of plagiarism and avoid committing it in their academic work. Fielding et al. (2013) and Hartman (2001) reported a positive improvement in evaluation of sources, even though students do not always apply this skill. In hindsight, students think IL should be made compulsory (Hartman 2001:117).

There is a perceived gap between self-assessed levels of IL skills and actual IL skills. Mahmood's (2016: 200) review found that the majority of students displayed the Dunning-Kruger effect in which students rate their IL skills much higher than what they truly are. Maurer, Schloegl and Dreisiebner (2017: 317) reported that students rated their search skills as "very good"; however, a significant percentage of students did not search the library catalogue or they made minimal use of traditional information sources (books and journals), while some students were not aware of library databases. Even at graduate level, students showed underdeveloped IL skills while overestimating their IL capabilities (Oakleaf, Millet and Kraus 2011: 832). Our questionnaire to lecturers shed light on the Dunning-Kruger theory relating to our student sample. Our study investigated how high or low our students rated their IL skills.

Various methods of assessing IL programmes are reported in the literature, including formative and summative assessments. Objective methods, such as standardised tests with high reliability of results, are commonly used methods. However, Rosman, Mayer and Krampen (2015: 750) and Falchikov and Boud (1989: 395) stated that IL is a multifaceted concept and there is no definitive method to assess it. Self-assessments offer a different perspective. Their qualitative nature enables reflection on student learning and the quality of IL instruction which shows evidence of transferral of IL skills and improvement of student's application of IL in their coursework. Students perceive that their IL skills are higher than their true IL skills which points to the subjective nature of self-assessments. This subjectivity casts doubt on the validity and reliability of self-assessments (Butler 2018: 2). Self-assessment should never be the only testing instrument; rather it should be used in conjunction with other types of reliable assessments after objective/formal assessments (Oakleaf 2008: 283). The pedagogical approach used in IL instruction will influence the impact of student learning.

3.1 The students' voice

The opinions of all three stakeholders involved in IL instruction (librarians, lecturers and students) are seldom solicited and meaningful student feedback on their feelings about IL training tends to be overlooked (Kim and Shumaker 2015: 449, Yevelson-Shorsher and Bronstein 2018: 356). Research by the ACRL (2015) as well as by Kim and Shumaker (2015: 454) indicated that students who engaged in a course which includes an IL-related assignment rated their own IL skills as high and course benefits were shown in improved assignment grades. Students recognised the value of IL as they rated highly the effectiveness, importance and impact of IL instruction. According to Cook and Klipfel (2015: 3), "learning is a change in learners' knowledge". In order to understand and use concepts taught, the students must be able to remember them (retention) and thereafter, by combining what they have learnt with their prior knowledge, students must be able to apply these concepts in new situations or new subject content (transfer). Cook and Klipfel (2015: 12) stated that, when there is active classroom engagement with the subject matter, the process of IL application takes place. For transfer of learning to take place in our CIL programme, the librarians worked with a subject-related assignment and rubric. The assignment was relevant, and students could relate to content while working with the IL concepts, apply it within the subject context as well as in doing the in-class exercises. As IL is a life skill, students were given scenarios and had to demonstrate how to apply the IL skills in different environments and how to find appropriate solutions. Practice and application of IL concepts in new environments demonstrate retention, transfer and learning taking place.

Yevelson-Shorsher and Bronstein (2018) found a need for a comprehensive IL programme that exposes students not only to a variety of sources but also appropriate information seeking behaviour. Their findings highlighted students' perspectives and the difficulty in "finding the right term" to search for information, concurring with our views that students have difficulty with identifying keywords and related terms to guide them in finding relevant information. There is an underlying weakness in students' ability to compile a strong search strategy and this weakness has repercussions in the retrieval of relevant and reliable information and application in their work. The authors of this paper have observed this weakness of students over many years and over different study levels. The same is posited in the work of Hartman (2001), Seamans (2002), Fielding et al. (2013) and Davids and Omar (2018). Yevelson-Shorsher & Bronstein's (2018) study of three population groups reported that students' challenges are due to insufficient training and lack of awareness of library resources. Faculty members perceived that students would gain IL skills during their studies. Due to the students' insufficient IL knowledge, librarians developed a programme to improve the student IL dilemma. Students indicated that navigating through and engaging with the vast variety of information sources was challenging, particularly the starting point of searching. Challenges were due to the absence of a well-formulated search strategy as well as students' knowledge deficiency about reputable sources available to them and how to access them.

'Googling' everything made students not only impatient but also reluctant to invest time in searching for quality sources. Lecturers expect students to know how to find information, but they are not provided with the relevant training to do so. Lecturers believe that students who lack IL skills are overwhelmed by the unrestricted access to and the variety of information that is available (Yevelson-Shorsher and Bronstein 2018: 541). These authors also quote faculty members remarking that it is easy to use Google to access information for superficial questions, but that Google should not be used to answer complex questions. Lecturers agree that IL is critical to their students' learning, and that IL instruction and support are important to develop information literate students (Wakimoto et al. 2016: 12). IL helps to develop critical thinking and should be applied to assist students to move beyond Google. In today's world of immediate gratification and the fast pace of technology, students are unable to or fail to be persistent and persevere in the process of searching for relevant and current information (DaCosta 2016: 43).

 

4 Research methodology

A mixed methods approach was chosen for this study, which was mostly quantitative but included some qualitative elements. The qualitative method allowed the researchers to understand "the phenomena" within its natural context (Maree 2016: 54). The study was designed to provide an in-depth understanding of students' views of the IL programme. Data were collected from another source (seven lecturers) to validate the trustworthiness of the information (Maree 2016: 310). This study is an illustrative case study using a student self-assessment survey to help librarians understand students' experiences of IL training. When doing assessments, to provide for reliability and validity, it is best to match quantitative and qualitative methods appropriate to the scenario (Grassian and Kaplowitz cited in Oakleaf and Kaske 2009: 280). The population studied was first year and foundation level students from the Business Faculty drawing a sample from Communication and English courses. Purposive sampling was used based on the researchers' knowledge of the population (Babbie, 2007: 182) and because we had access to students and lecturers. Purposive sampling is defined as a "procedure in which elements are selected from the target population on the basis of their fit with the purposes of the study" (Daniel 2012: 7). In addition, "each participant will provide unique and rich information of value to the study" (Etikan, Musa and Alkassim 2016). The researchers requested lecturers to distribute the questionnaires during classes well after the summative assessment. Questionnaire respondents were anonymous, and principles of ethical research were observed.

Data were collected from two questionnaires, one from students (eighteen questions; Appendix A) and one from lecturers (twenty-two questions; Appendix B) to gather quantitative and qualitative data. The surveys included open-ended and closed, multiple choice, and rating questions. Open-ended questions enabled us to gather qualitative data. The student survey was a self-assessment on the five modules of IL and about how they applied IL within their academic subjects. The purpose of the lecturer survey was to validate IL application in students' academic work. Data were collected from both students and lecturers to support the "credibility, dependability and confirmability of the study" (Seamans 2002: 114). Lecturers returned 386 out of 450 student questionnaires (86% response rate) and all seven of the lecturer questionnaires (100% response rate). The university statistician performed data analysis using SPSS. Four open-ended questions were categorised manually according to themes. The questionnaire design was based on the IL Competency Standards for Higher Education (ALA, 2000) which are to:

• determine the extent of information needed;

• access needed information effectively and efficiently;

• evaluate information and its sources critically;

• use information effectively to accomplish a specific purpose; and

• use information ethically and legally, knowing the economic, legal, and social issues relating to the use of information.

The quantitative data were captured in graphs and the students' responses to open-ended questions were thematically analysed. The responses and validation by the lecturers were subsequently assigned to the four levels of the KP model showing where low level and higher-level IL skills were applied.

 

5 Results

The majority of respondents were first years (83%) and 17% were ECP students. Hereafter, the paper makes no distinction between these groups. Usually, lecturers recognise the need for IL intervention early in the first semester in order for students to have the research skills to complete academic assignments. Unfortunately, this was not possible due to several factors: large classes, overlapping class timetables and unavailability of training venues. These challenges resulted in poor levels of attendance for Modules 1 to 3 of the five modules. Training occurred in two semesters with attendance rates of 74% in the first semester and 24% in the second semester.

5.1 Student attendance, value attached to modules and IL concepts applied

Figure 1 shows the correlation between attendance of the modules, the value attached to the modules by students and the IL concepts that students applied the most. The attendance for Modules 1 and 2 (Search Strategy and Information Sources and Tools) was very low but those who attended reported that the sessions added meaningful value to their academic work. Possible reasons for low attendance were that students were not yet registered, that they were not familiar with the library as a venue, or that the CIL offered no credits (therefore they did not feel an obligation to attend). The low attendance can further be attributed to: lecturers having covered aspects of topic analysis in class, an online IL course being available on Blackboard (the online learning management system of CPUT) and the PowerPoint presentations for class having been uploaded to Blackboard for pre-reading. Students placed highest importance on Module 3 (Evaluation of Information Sources), even though their attendance was low at 9.4%. The topics of plagiarism (Module 4) and referencing (Module 5) had been emphasised by lecturers in class and by librarians during IL training. These modules had the highest attendance, but the values attached to these modules were low and fell well below the expectations of the researchers of this paper.

In their self-evaluation, 63,9% of students reported applying the IL concepts of Module 1 to the analysis of their assignment topic. Nearly a third (31.5%) of students indicated using keywords as a technique to guide them in finding the relevant and current information for an assignment; 20.5% used search strings to formulate a search strategy; and 11.9% made use of Boolean operators. Our observations based on our experience and the low percentages above show that students find it difficult to grasp the use of keywords and Boolean operators to create an effective search strategy. (It is not expected that students should use all the operators, but they should be able to apply the 'AND' operator and know the relevance of results that this operator will retrieve.) While students attached 19.8% value to Module 4 (Plagiarism), data extracted from Question 4 of the student questionnaire, an open-ended question about how the IL classes benefitted their academic work, indicated that only 7.5% applied plagiarism concepts in their academic work, which shows that students still struggle with the concept of plagiarism. The data shows that 98% of the respondents had attended the IL classes -three quarters of them in semester one and one quarter in semester two.

5.2 Information sources and evaluation criteria

Although students were trained in using various information tools, namely the catalogue and databases, 70.5% of students still used the open internet as their main source of information. In our study, and as shown in Figure 2, 29.5% indicated that they use library resources such as peer-reviewed journal articles. More than half of the students (58.6%) indicated that they know the importance of using current resources published within the last five years. A very high percentage (96.8%) indicated that they find relevant and accurate information, although not always from library resources. Most of the students (93.7%) responded that they apply evaluation criteria before using information for their assignments. More than half of the students (68.8%) did revision using self-study materials on Blackboard, while 31.2% did not see the need for it.

5.3 Importance of IL in academic work

Students were asked whether they would recommend the CIL to other students, and a sizable core of 98.4% felt strongly that all students should attend the course. In a follow-up, open-ended question students were asked, "How would you promote IL to your peers?". They commented positively on the value of the course, the knowledge and skills that could be gained, such as more accurate referencing, finding quality information sources, reduced plagiarism and improvement in their marks for academic tasks. Student responses included:

• I will promote it by telling them how good my marks are after going to those classes

• I will tell them to go and attend because it will not help them only for their school years, but it will help in future

• I would advise my peers to attend the IL class because it helps a lot with referencing and finding proper information

• IL gives students several ways of doing academic work

• Not sure

• I won't

5.4 Lecturers' perspectives

A second questionnaire was distributed to the lecturers of these students and a 100% response rate was achieved. Lecturers were asked to rate the levels of students' awareness of and a corresponding reduction in plagiarism, their increase in the use and variety of library sources in bibliography lists, the improvement in their in-text citations and referencing, and their work's closer alignment with standards (Figure 3). All lecturers indicated that students were aware of plagiarism and its negative implications for their work and the institution. Furthermore, 85.7% of lecturers replied that students have applied techniques to avoid plagiarism in their work after attending the IL programme. Over half (57.1%) of the lecturers indicated that students use library resources, evidenced in the students' assignment bibliographies. From the bibliography in students' assignments, lecturers indicated that 28.6% used library sources and 28.6% cited their sources, while only 14.3% cited intext sources accurately (Figure 4).

 

 

Even though lecturers reported that students use techniques to avoid plagiarism (Figures 3 and 4), the data shows that students have still not realised the importance of in-text referencing and how it relates to plagiarism. Lecturers required students to use current sources, published within the past ten years, and 72% of lecturers indicated that students met the criteria on currency of sources. While all lecturers expected students to use library sources after IL training, they reported that students still use mostly sources found on the wider internet in their assignments. More than half of the lecturers (57.1%) indicated that students use a variety of sources, while many of them (85.7%) stated that sources are relevant to the topic. Less than half (42.9%) of the lecturers reported that students use sources which give different perspectives on a topic (see Figure 5).

 

 

Figure 6 shows that students have a better grasp of the IL concepts through their engagement with a wider variety of quality information sources. Lecturers (83%) responded that they have seen improvement in the application of IL in the academic work of the students. Figure 6 also corroborates students' responses that they can find relevant and accurate information, and that they can apply evaluation criteria before using the information in their assignments.

 

 

6. Assessment using the four evaluation levels of the Kirkpatrick model

The responses from the students' self-assessment survey and lecturer survey were analysed according to each level of the KP model. The model espouses that affective events should lead to cognitive end results. Levels 1 and 2 focus on the affective experiences, feelings and value of IL training expressed by students. Levels 3 and 4 focus on the extent of cognitive learning retained by students. In the sections that follow, lecturers' responses under each level will be used to verify the students' responses.

6.1 Level 1: Reaction

The objective of this level is to evaluate students' reactions to IL training by asking them what passing the CIL assessment meant to them, whether the training venue had been suitable and how they rated the usefulness of the IL training to their work. The pass rate of first-year students was 83% and of ECP students, 71%. Passing the assessment meant the following to students: they felt that it was a big achievement; they will use the certificate to apply for student employment; they feel confident in using IL skills; they saw improvement in their performance; and it was the first certificate that they had achieved in their studies. While almost all students (97%) indicated that they had understood the content of the classes, only 43,5% confirmed that they had understood and applied the IL concepts, and seen corresponding improvement in their academic work. According to the students' responses, the library training venue was conducive to learning. The library venue enabled active learning: it offered access to computers for hands-on training, and classrooms for interactive group work and blended learning activities. A reflection of this is that 95% of students liked the library venue more than their classrooms. All lecturers approved of the blended learning approach and commented that it engaged students by breaking the monotony of formal lectures. We were able to gauge further positive reactions from lecturers' comments. Students enjoyed hands-on use of computers for searching for information and liked the combination of online and face-to-face interaction. Lecturers remarked that students could see the relevance of IL in more than one subject that they are studying as well as seeing how IL is relevant for all studies going forward.

6.2 Level 2: Learning

The aim of this level is to determine whether students recognise the new knowledge, skills and changed mindset they have developed after the IL training and what they found most beneficial from it. Data from Question 4 of the student questionnaire indicated that the overwhelming majority of students (80%) felt that IL helped them in their academic work. The helpfulness of the modules was rated as follows: the module on Information Tools, Catalogue and Databases was rated most helpful (40%), followed by the module on Referencing (30%) and the module on Information Sources and Evaluation (17%). Interestingly, 69% returned to the IL exercises after training. They had access to presentations and IL exercises on Blackboard. This behaviour is significant because students do not usually engage with IL content on their own after IL training. In response to the question, "How did the IL classes help you with your academic work?", themes extracted from students' responses were about plagiarism, in-text referencing and bibliography. A third of students indicated that IL was helpful to their tasks with responses such as: It helped me to reference my work and to use keywords.

The significance of these responses to the KP model is that learning has taken place: students have recognised, understood and a fair proportion were applying the concepts learnt during IL training. Another theme that emerged from the students' responses related to cognitive skills. It is the perception of 87% of students that they h ave a good understanding of how to do their work as they applied search skills and keywords in their assignments as well as in other subjects as illustrated in the comment that IL skills help [me] to understand the work and apply [IL] to other subjects.

Lecturers responded that all students are aware of plagiarism. While 67% said that students listed more library sources than before in their bibliography, they conceded that fewer than one-third of these sources were referenced accurately. The majority of lecturers (86%) were dissatisfied with in-text referencing; only 14% were satisfied that in-text sources were cited accurately. From our experience, students struggle with in-text referencing. We have observed that students approach librarians to have their references checked for accuracy. In addition, lecturers have requested refresher referencing training for students.

6.3 Level 3: Change in behaviour

This level observes changes in student behaviour by looking at how students have applied their IL skills. Such behaviour would include whether students had transferred their skills to other subjects, whether they had applied IL concepts in their assignments, whether they applied evaluation criteria to sources before using them, whether they used the most recent sources, and whether they used library sources over other sources. Students (79%) reported that they applied IL skills in subjects other than English or Communication, their host subjects. As shown in Figure 7, 15% applied IL in five other subjects, 32% in two other subjects and 32% in one other subject.

 

 

Students reported that they applied the IL concepts of Keywords (32%), Related Terms (17%), Boolean Operators (12%), Search Strings (21%) and Phrase Searching (19%) (Table 1). Only 30% of students indicated that they use sources obtained through the library's catalogue and electronic databases. Most students (94%) indicated that they evaluate their sources according to CRAAP (currency, relevance, authority, accuracy, purpose) and 59% of students used the most current sources published within the last five years. This level of the KP model assesses a change of behaviour in students and indicates that transfer of skills has taken place. About one-third (32%) of students applied IL concepts to their assignments which shows that some students had retained the skills learnt in training. Nearly one-third of students (30%) no longer relied on Google as their main source to find quality information; instead, students used library resources.

6.4 Level 4: Results

This level examines the extent of students' learning. The aim is to determine whether there is an overall improvement in the quality of the students' work as mentioned by students and lecturers given that students had limited IL skills before attending IL training. While 97% of students reported that they can find relevant sources when searching for information, over two-thirds of students (70%) used sources from the internet, and 30% used library resources. Furthermore, the majority (98%) felt that they would encourage other students to attend IL training as it helped them in their academic work, and they benefitted from attending the training. Lecturers (83%) reported improvements in the content of the students' assignments as a result of attending IL training. Improvements included less plagiarism, more credible and better-quality sources used by students in their bibliographies, and sources were more current and relevant to the assignment topic. Lecturers had seen improved academic performance by students as a result of IL training, and remarked that IL training had a positive impact on students and their academic work. Level 4 of the KP model looks for tangible results of learning and improvement. The statistics from students and lecturers show that student learning has taken place and that students showed improvement in their academic work. This result is validated by most lecturers. The survey was conducted a few months after IL training and responses showed that students had retained their IL skills and applied them and that transfer of skills took place. Lecturers observed improvement in students' work. In this light, our survey results show the IL programme was successful.

 

7 Discussion

The IL training discussed in this case study is at an introductory level and students are not expected to master all the IL skills taught in the CIL. The results indicate that students have made improvement in areas across all IL modules. This is confirmed by most lecturers. Students and lecturers found the active learning approach beneficial. Students and lecturers approved of the blended learning approach. It engaged students in learning activities such as hands-on training on computers, online learning, face-to-face contact and interactive group work which enhanced the relevance of IL to their work. A substantial number of students felt strongly that all students should attend the course, and that their peers and new students would benefit from IL training.

This study provided deeper insight into the views of our IL programme from the perspectives of students and lecturers. It has also highlighted contradictory viewpoints. Responses by students and the observations of lecturers confirm the Dunning-Kruger theory that students over-estimated their skills and that lecturers did not validate all of the students' assessments. Evaluating sources was noted by most students as an area in which they had improved most. However, their perception was not validated by the lecturers who stated that using a variety of sources does not mean that students evaluated the sources. Some lecturers agreed that students used sources that gave different points of view. Interestingly, lecturers stated that all students have been made aware of plagiarism and that they saw a reduction in plagiarism. However, lecturers indicated that relatively few students do in-text referencing accurately.

The survey revealed that students understood the different IL concepts and applied these concepts. When building a search string, students needed to apply a combination of keywords, related terms and phrase searching with a relevant Boolean operator. One-fifth of the students were able to build a search strategy, which shows that IL concepts were applied, and that retention of IL skills has taken place. The assignment used throughout the IL training assisted in consolidating deeper learning, linking new knowledge with students' prior knowledge and application in an assignment-based environment. Transfer of IL skills has taken place as suggested by responses on the application of IL in different subjects: Most (79%) students applied IL in subjects other than the host subject. Our experience and this survey point out that, in order to consolidate retention and application, IL training should not only happen in the first year or in one semester, but as a cumulative process over the period of the students' university years. Only when this happens, will we have fully information literate students.

 

8 Conclusion

The objective of this case study was to gain insight into students' perspectives on their experiences of the IL training in the CIL programme. It follows from our previous paper (Davids & Omar 2018) in which lecturers and librarians assessed the CIL programme. The two papers together give us a holistic view of the perspectives of all the stakeholders on the CIL. The value of this study is that it combines student perspectives on IL with validation by lecturers across seven departments in the Business Faculty at CPUT. This study adds to the body of knowledge on IL already produced by CPUT. Furthermore, this paper contributes to research on IL training from the perspective of students, an under-researched area as identified by Yevelson-Shorsher and Bronstein (2018) and Kim and Shumaker (2015). The students' voice allowed us to understand how they perceive IL and how IL skills have altered their behaviour through equipping them with skills fit for academic purposes. The students' voice also highlighted that the Dunning-Kruger theory discussed by several authors (Mahmood 2016, Maurer, Schloegl and Dreisiebner 2017, Oakleaf, Millet and Kraus 2011) holds true for these students. Students inflated their IL skills while their actual skills were rated lower by their lecturers. The results of this study indicate that students and lecturers confirmed an increase in application of IL in students' work and noted improvement in the quality of their work. The usefulness of this study is that it shows that IL skills can be transferred to other subjects and environments in succeeding years of study and thereafter in the workplace. This study indicates that the CIL has generated positive learning experiences for students which improved their beginner level of understanding of research and serves as a foundation to prepare them for research in the future. IL development needs to be ongoing and not presented in the first year only.

 

9 Recommendations

The CIL is a stand-alone course. To benefit all students, IL should be embedded into the curriculum. Students gave their views on how IL training would be of benefit to them, but they were not asked about the problems they experienced, or how the IL training programme could be improved. Their responses would enable us to structure the training to be more suitable to their training needs. It would be helpful if lecturers share the weaknesses of students within the IL rubric. This would help the librarians to address the difficulties students encounter with IL. To complement classroom training, we need to introduce technology-based learning materials for self-study, such as videos, podcasts and infographics, to cater for diverse learning styles. This study highlights the need for closer collaboration between students, lecturers and librarians towards developing a more student-focused, embedded IL training programme for inclusion in the curriculum.

 

References

American Library Association. 2000. Information literacy competency standards for higher education. [Online]. http://www.ala.org/acrl/standards/informationliteracycompetency (6 December 2018).

Association of College and Research Libraries (ACRL). 2015. Academic library contributions to student success: documented practices from the field. Chicago: Association of College and Research Libraries. [Online]. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/issues/value/contributions_report.pdf (21 January 2019).         [ Links ]

Babbie, E. 2007. The practice of social research. 11^th ed. Belmont, CA: Thomson Wadsworth.         [ Links ]

Bates R. 2005. Kirkpatrick four-level evaluation model. In Encyclopedia of Evaluation. S. Mathison, Ed. Thousand Oaks, CA: Sage Publications. 221 -222.         [ Links ]

Butler, Y.G. 2018. Student self-assessment. In The SAGE encyclopedia of educational research, measurement, and evaluation. B. Frey, Ed. Thousand Oaks: Sage Publishing. DOI:10.4135/9781506326139.         [ Links ]

Chan, C.P. 2016. Institutional assessment of student information literacy ability: a case study. Communications in Information Literacy, 10(1): 50-61. DOI:10.15760/comminfolit.2016.10.1.14.         [ Links ]

Cook, D.B. and Klipfel, K. M. 2015. How do our students learn? An outline of a cognitive psychological model for information literacy instructions. Reference & User Service Quarterly, 55(1). [Online]. https://escholarship.org/uc/item/6964s53p (12 November 2018).         [ Links ]

DaCosta, J.W. 2016. Information literacy in the digital environment. In University libraries and digital learning environments. M. Holland, J. Beard and P. Dale, Eds. UK: Routledge. 59-74.         [ Links ]

Daniel, J. 2012. Choosing the type of nonprobability sampling. In Sampling essentials: practical guidelines for making sampling choices. Thousand Oaks, CA: SAGE Publications. DOI:10.4135/9781452272047. 81-124.         [ Links ]

Davids, Z. and Omar, Y. 2018. Implementing a certificate of information literacy programme and engaging with faculty: a case study of the Cape Peninsula University of Technology. South African Journal of Libraries and Information Science, 84(1): 1-10. DOI:10.7553/84-1-1716.         [ Links ]

De Saulles, M. 2015. Information 2.0: new models of information production, distributions and consumption. 2^nd ed. London: Facet.         [ Links ]

Detlor, B., Booker, L., Serenko, A. and Julien, H. 2012. Student perceptions of information literacy instruction: the importance of active learning. Education for Information, 29: 147-161.         [ Links ]

Detmering, R., Johnson, A., Sproles, C., McClellan, S. and Linares, R. 2015. Library instruction and information literacy 2014. Reference Services Review, 43(4): 533-642. DOI:10.1108/RSR-07-2015-0037.         [ Links ]

Erlinger, A. 2018. Outcomes assessment in undergraduate information literacy instruction: a systematic review. College & Research Libraries, 79(4): 442-479. DOI:10.5860/crl.79.4.442.         [ Links ]

Etikan, I., Musa, S. A. and Alkassim, R. S. 2016. Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics. 5(1): 1-4. DOI:10.11648/j.ajtas.20160501.11.         [ Links ]

Falchikov, N and Boud, D. 1989. Student self-assessment in higher education: a meta-analysis. Review of Educational Research, 59(4): 395-430.         [ Links ]

Fielding, J., Hans, J., Mabee, F., Tracy, K., Consalvo, A. and Craig, L. 2013. Integrated information literacy and student outcomes in foundational first-year writing. Journal of Assessment and Institutional Effectiveness, 3(2): 106-139.         [ Links ]

Hartman, E. 2001. Understandings of information literacy: the perceptions of first year undergraduate students at the University of Ballarat. Australian Academic & Research Libraries, 32(2): 110-122.         [ Links ]

Johnson, A., Willenborg, A., Heckman, C., Whitacre, J., Reynolds, L., Sterner, E., Harmon, L., Lunsford, S. and Drerup, S. 2018. Library instruction and information literacy 2017. Reference Services Review, 46(4): 628-734. DOI:10.1108/RSR-07-2018-0061.         [ Links ]

Kim, S.U. and Shumaker, D. 2015. Student, librarian, and instructor perceptions of information literacy instruction and skills in a first year experience program: a case study. The Journal of Academic Librarianship, 41 (4): 449-456. DOI:10.1016/j.acalib.2015.04.005.         [ Links ]

Kirkpatrick, D. L. 1998. Evaluating training programs: the four levels. 2^nd ed. San Francisco: Berrett-Koehler Publishers.         [ Links ]

Lanning, S and Mallek, J. 2017. Factors influencing information literacy competency of college students. The Journal of Academic Librarianship, 43(5): 443-450. DOI:10.1016/j.acalib.2017.07.005.         [ Links ]

Mahmood, K. 2016. Do people overestimate their information literacy skills? A systematic review of empirical evidence on the Dunning-Kruger Effect. Communications in Information Literacy, 10(2): 199-213. DOI:10.15760/comminfolit.2016.10.2.24.         [ Links ]

Maree, K. 2016. First steps in research. 2^nd ed. Van Schaik: Pretoria.

Maurer, A., Schloegl, C. and Dreisiebner, S. 2017. Comparing information literacy of student beginners among different branches of study. Libellarium: Journal for the research of writing, books, and cultural heritage institutions, 9(2): 309-319. DOI:10.15291/libellarium.v9i2.280.         [ Links ]

Oakleaf, M., 2008. Dangers and opportunities: a conceptual map of information literacy assessment approaches. portal: Libraries and the Academy, 8(3): 233-253.         [ Links ]

Oakleaf, M. and Kaske, N. 2009. Guiding questions for assessing information literacy in higher education, portal: Libraries and the Academy, 9(2): 273-286.         [ Links ]

Oakleaf, M., Millet, M. S. and Kraus, L. 2011. All together now: getting faculty, administrators, and staff engaged in information literacy assessment. portal: Libraries and the Academy, 11 (3): 832-851. DOI:10.1353/pla.2011.0035.         [ Links ]

Reynolds, L., McClellan, S., Finley, S., Martinez, G. and Linares, R. 2016. Library instruction and information literacy 2015. Reference Services Review, 44(4): 436-543. DOI:10.1108/RSR-08-2016-0051.         [ Links ]

Rosman, T, Mayer, A-K. and Krampen, G. 2015. Combining self-assessments and achievement tests in information literacy assessment: empirical results and recommendations for practice. Assessment & Evaluation in Higher Education, 40(5): 740-754. DOI:10.1080/02602938.2014.950554.         [ Links ]

Saunders, L. 2012. Faculty perspectives on information literacy as a student learning outcome. The Journal of Academic Librarianship. 38(4): 226-236. DOI:10.1016/j.acalib.2012.06.001.         [ Links ]

Schwandt, T. A. 1998. How we think about evaluation practice. Paper presented at the American Evaluation Association Conference. 4-7 November. Chicago, IL.

Seamans, N. 2002. Student perceptions of information literacy: insights for librarians. Reference Services Review, 30(2): 112-123. DOI:10.1108/00907320210428679.         [ Links ]

Shao, X. and Purpur, G. 2016. Effects of information literacy skills on student writing and course performance. The Journal of Academic Librarianship, 42(6): 670-678. DOI:10.1016/j.acalib.2016.08.006.         [ Links ]

Stevenson, P. 2012. Evaluating educational interventions for information literacy. Health Information & Libraries Journal, 29(1): 81-86. DOI:10.1111/j.1471-1842.2011.00976.x.         [ Links ]

Taylor, A. and Dalal, H. 2017. Gender and information literacy: evaluation of gender differences in a student survey of information sources. College & Research Libraries, 78(1): 90. DOI:10.5860/crl.78.1.90.         [ Links ]

Turnbow, D. 2015. Instruction by design: strategies for creating effective learning moments. [Online]. https://learningservicesucsd.wordpress.com/2014/11/05/does-your-instruction-get-two-thumbs-up/.

Wakimoto, D. K., Alexander, S., Bussman, J. D., Winkelman, P. and Guo, J. 2016. Campus-wide information literacy assessment: an opportunity for library leadership. Library Leadership & Managament, 31(1): 1-19. DOI:10.5860/llm.v31i1.7184.         [ Links ]

Yevelson-Shorsher, A. and Bronstein, J. 2018. Three perspectives on information literacy in academia: talking to librarians, faculty, and students. College and Research Libraries, 79(4): 535-553. [Online]. https://crl.acrl.org/index.php/crl/article/view/16728/18235 (29 November 2018).         [ Links ]

 

 

Received: 30 September 2019
Accepted: 12 October 2020

 

 

Appendix A Student Questionnaire

 

Click to enlarge

 

Appendix B Lecturers' Questionnaire

 

Click to enlarge