versão On-line ISSN 2076-3433
versão impressa ISSN 0256-0100
S. Afr. j. educ. vol.33 no.3 Pretoria Jan. 2013
Anass BayagaI; Newman WadesangoII
IFaculty of Education, University of Fort Hare, East London, South Africa firstname.lastname@example.org
IICentre for Learning and Teaching, Walter Sisulu University, East London
The current study was an exploration of how to develop assessment resources and processes via in-depth interviews with 30 teachers. The focus was on how teachers use and apply different assessment situations. The methodology, which was a predominately qualitative approach and adopted case study design, sought to use a set of criteria based on constructs from literature reviews to evaluate assessments. Thus these characteristics guided the study which included: a brief description of assessment and moderation; assessment materials/resources; assessment objectives; assessment activities; assessment/re-evaluation; and alignment/consistency. The case (one site) and 30 respondents were selected purposively. The study revealed that assessors need to use different methods of assessment depending on the socio-cultural setting of learners' environment and resources, if applicable. We argue that teachers should note the socialisation within their domain as well as the culture of their domain and domain-specific ways of talking, acting, and seeing the world.
Keywords: academic; assessment; consistency; continuous planned process moderation; evaluation; feedback; standards of assessment; surface learning and resources
The current study was an exploration of how to develop assessment resources and processes. The focus was on how teachers use and apply different assessment situations. Following the focus and according to the Department of Basic Education (2010: 101), "assessment is a continuous planned process of identifying, gathering and interpreting information about the performance of learners, using various forms of assessment. It involves four steps: generating and collecting evidence of achievement; evaluating this evidence; recording the findings; and using this information to understand and thereby assist the learners' development in order to improve the process of learning and teaching". Reflecting on a variety of influences, national attention over the years has increasingly shifted to a greater emphasis on assuring high-quality education. This focus on improving assessment and subsequently student learning outcomes has resulted in increased emphasis on assessment in many academic institutions, including our faculty. In particular, the ability to assess well, in addition to having research skills, is a valuable factor in any faculty (faculty of education). For example, with regard to research, some universities request that academics discuss assessment rubric and strategies and philosophies with students (Krause & Coates 2008; Yorke, 2007). Some have also begun to request a statement of assessment philosophy, sample course materials, or an assessment demonstration as part of the application process, which all impact on the quality assurance process. It is in this all-important direction our study of assessment is developed to reflect our assessment career practices and beliefs.
This study has been conducted to explain how to develop assessment resources and processes via in-depth interviews with teachers in Mathematics, Science, Technology and Research Methods. Although we teach different year groups and courses depending on the faculty's demand, for the purpose of this study, it suffices to state that we teach the aforementioned courses.
Due to the focus of the study as noted in the background and based on the different courses that are taught, the current study was situated in the constructivist theory; which was geared towards exploring the process of designing assessment strategies (Reason, Terenzini & Domingo, 2007). The strategies include but are not limited to the following listed points:
- The purpose of assessment is clearly defined
- The assessment approach is described
- Use naturally - occurring evidence
- Ensure triangularity of evidence (knowledge, process, product)
- Ensure that assessment activities focus on the outcome
- Ensure sufficient assessment activities to gather sufficient evidence according to the assessment criteria
- Ensure adhering to principles of good assessment
- Involve the candidate to suggest ways to gather evidence
- Ensure cost-effective assessments
- Provision is made for the collection of evidence
- Ensure the time allocated for assessment is realistic
- Is the context clearly defined and described?
- All role-players are defined and responsibilities described
- Appeals procedures are described
It is important to also mention that identifying challenges in achievement of student learning helps in assessment. A few of the challenges are (1) building confidence and student self esteem; (2) instilling qualities of a good teacher using the educators' norms and standards; (3) instilling in students the importance of assessment in the form of projects, tests, exams etc.; and (4) how to prepare for exams. Some researchers argue that this is often discussed with students while additionally encouraging them to customise such approaches used (Nyaumwe & Mtetwa, 2011). However, what is not clear in general practice is whether it is important to recognise that assessment can serve a number of purposes simultaneously. Also whether if assessors are responsible for designing assessment, does that imply that they need to be quite clear, before they begin, what educational purpose(s) assessment will be serving? Although the 100 point is obvious as the total mark, educators often seem to forget that it is the purpose and function of an instrument that determines its design. Assessors tend to give little thought to the purpose and function and begin explicit planning of assessment by jumping in at given points (Lane, Wehby & Cooley, 2006). This has ramifications on learning, most particularly surface learning. This can sometimes be likely to be motivated primarily by fear of failure. One interesting study (Lane et al., 2006) has suggested that efforts by teachers to convey that what they want is deep learning only succeeds in getting surface learners to engage in ever more complex contextualising exercises, trying to use surface strategies to reproduce the features of the deep approach.
Different stakeholders require different things of the assessment process Assessing learners provides explanatory tools to translate assessment into practical terms. Further, to provide a sense of direction for decision makers, policies, regulation systems and performance and assessment tools are used to determine how successful efforts are at meeting targets and objectives. Assessment of different stakeholders is not easy to achieve, especially with different learners from different socio-economic backgrounds (Hlalele, 2012). This calls for sustainable assessments that provide an opportunity not only to assess a baseline of current status of learners, but also to identify progress and promote desirable behaviour, taking into account the context of learners (Lane et al., 2006; Ramnarain, 2011). What is also unclear is the cultural and social variations between programmes and courses and thus assessment may vary from one leaner to another, even when the same criteria are applied. Therefore a flexible assessment system is required to allow the user to consider spatial boundaries, while retaining an understanding of what is being changed and why.
Assessment instrument usefulness and possible limitations
Different kinds of tests are given to determine student achievement. Standardized achievement tests for instance are often used, but have many limitations. They may be useful in determining the content students should learn and its relevance for future learning and the curriculum. Criterion referenced tests eliminate a major weakness of standardized tests in that there are accompanying objectives that teachers can use in teaching students. When a constructivist approach is taken to assessment, evaluation is based on contextual situations. The study represents a constructivist approach that can sample student achievement. Computer-based tests have many advantages, especially in the immediacy of feedback to students, but they also have limitations such as a lack of opportunity for students to raise questions or have input into the curriculum. Many problems remain to be resolved with regard to assessing student attainment, but new ways to assess student achievement are certainly needed.
Transformation and barriers to change
Almost everything assessors do in performance improvement and training requires change. Thus learning in itself is a process of change. One reason is that where we are now is known. For example, we may find ourselves in a bad situation - say students not learning, but it is nevertheless a known situation - and many learners prefer the known to the unknown. Nevertheless, there is limited research on how learners initiate and sustain change and also transform in the educational settings. Therefore it is imperative to determine which barriers or which combination of barriers is most likely to keep educators, and thus the students, from executing the change that educators want to make. Based on the aforementioned contestations and discourses which still lack established and confirmed results, the current study proposes research questions in anticipation of established results.
- To determine the impact of diversifying assessments
- Identify alternative type of assessment which may be useful
- To identify assessment and moderation processes and change management requirements
The data collection instrument was an in-depth interview schedule; normally considered to be one of a range of data collection in qualitative research (Cohen, Manion & Morr, 2000). In-depth interviews are those interviews that encourage the capturing of respondent's perception in their own words. In this study, a determined set of criteria were used to for assessment interview. They enabled further probing by the interviewer in order to get detail and rich information from the respondents. The characteristics that guided the study were in six components: (a) brief description of assessment and moderation; (b) assessment materials/resources; (c) assessment objectives; (d) assessment activities; (e) assessment/re-evaluation; and (f) alignment/ consistency. They (respondents) were also asked to write at least one question related to every sub-category in the assessment process. Moreover, during the case study, respondents (30) were exposed to an open-ended interview on the themes related to the background of the study. Participants, who were selected prudently, explained how planning, implementation and moderation of assessment were conducted. They unpacked their assessment and moderation processes so that it could be clear if viability and sustainability of the assessment and moderation is possible. Validity of the instrument was achieved by peers of researchers ensuring that the content was geared towards the objectives of the research. Since the study was a predominately qualitative approach, data were analysed according to the themes that emerged from the interview. Some responses were reported word-verbatim in order to capture expression, phrases and sentences as presented by the participants (Creswell, 2003).
Firstly, participation in the study was on a voluntary basis, thus no one was forced to participate in the study. Secondly, respondents were assured of anonymity and confidentiality (Creswell, 2003). Therefore, every effort was made to ensure that the identity of anyone who participated in the study was kept confidential.
Given the objectives of the research as noted, it was important to bear in mind, when evaluating assessment, that the research questions guided and assisted the researchers to know what worked: What purposes and whose interests prevailed? For instance respondent A noted that questions such as: "... are the interests of my students of primary importance? Is it the interests of all students or only certain groups of students?"
Because the purposes of assessment often remain vague and implicit, there is a danger that the different purposes, i.e. summative, formative, or diagnostic, become confused and conflated. As a consequence, in support of respondent A's assertion, it is argued that assessment often fails to play a truly educational role. It was also re-echoed by respondent R that:
... in my case, questions during the development streamlined the assessment process. Although, on one occasion, for example, an over-stretched lecture may set a diagnostic purpose resulting to lack of time and energy in a technology class, in which case I decide to use a project instead.
Not only is this kind of practice unfair to students, but it also undermines the developmental potential of assessment. Students are entitled to be informed beforehand how their assessment results will be used. It is rather recommended to view them as two ends of a continuum (Jones & Jones, 2007).
Comment on the arguments in favour of diversifying assessments If assessment is to achieve any of the functions listed in the previous section, it is vital that it has validity. Whether it is intended as feedback to students on their progress or as part of the grading process for a final qualification, common standards must be applied. However, this is easier said than done; assessment is by no means a pure science (Young, 2006). Nevertheless, marking (of essays and exams in particular) is something virtually all academics have to do and most rapidly gain a good deal of experience. In support of this, respondent G affirmed that that the reason why teachers in higher education vary in the marks they award to essays, for example, is that "they are using different criteria or, if they are using the same criteria, they are giving different weightings to them in terms of importance." This exercise underlines the importance of working to explicit criteria. There are good reasons for making criteria explicit; respondent X claimed (1) to be fair to the student: When I set a task or assignment, make clear the criteria on which it will be assessed, so the student can tailor her or his response to my requirements. (2) To avoid 'academic drift': This is the term used to describe student responses where the criteria are unclear; they assume the key criterion to be an emphasis on content. (3) To encourage staff to adopt common standards in marking: If a number of different staff members are involved in marking a piece of work, clearly stated criteria should help to bring closer the standards to be applied. It also gives a clearer basis for discussion of any disagreements either between staff or between students and staff.
Identify an alternative type of assessment which may be useful
Assessment is all about making judgements. A major argument for involving students in self and peer-assessment is that it helps them to develop the ability to make judgements, in particular about themselves and their work. This is an important life-skill as well as an academic one. Research in Australia (Jones & Jones, 2007) showed that recent graduates rated the ability to assess their own performance among the most important skills used in their jobs, but one that their degree courses had almost totally ignored. It follows, then, that we should consider the ways in which these skills can be developed within existing courses. Table 1 shows a number of ways in which respondents are involved in assessment on their courses.
Many lecturers automatically think of marking when they hear the word assessment; note that only the last two methods suggested here involve students in the formal marking process. There are many ways in which students can be involved in assessment and develop their judgment without involving them in marking their own work or that of others.
Moderation processes and changes that may be needed in the light of requirements
Despite the use of guidelines such as level descriptors, it is important to acknowledge that differences among assessors can never be entirely resolved although this is often tactfully ignored (Yorke, 2007), and for this reason, the concept of moderation has been developed. Moderation attempts to mediate between different assessor interpretations of student performance or simply specify the 'rules' of the marking game (Yorke, 2007). A respondent X alluded that:
There are a number of different methods of moderation which are distinguishable from one another in terms of whether they are part of a quality promotion process (one which is formative and aims to promote quality), or whether they are part of a quality control mechanism (one which is summative and makes judgments about quality).
It is useful to consider these different forms of moderation when determining institutional models of external and internal 'examining'. The interpretation of student performance also needs to report on the variability of student work and to track it over time. This is often done via student externally examined papers.
Yorke (2007) argues that good practice suggests that learners should be encouraged to gather naturally occurring evidence or their key skills in terms of ability to investigate, communicate, evaluate and estimate. This is applicable in various courses that are taught "especially in technology during times of modelling and designing of simple and compound machines" (Respondent B). Where this is not feasible, learners are sometimes set projects or assignments to assist in evidence gathering. It is important for us to have these assignments reviewed before they are delivered to learners. It offers prompts that help to consider whether the assignment or project (1) is set in a relevant context; (2) identifies opportunities for demonstrating the key skills; (3) is clear in its description of the task or activity; (4) meets the requirements of key skills standards; (5) offers opportunities for learners to apply their skills and knowledge. Having reviewed an assignment or project, respondents make suggestions which should provide detailed feedback to the learners so that they know what ways may improve their learning the next time.
Summative assessment of a key skills study in lessons requires judgement of the learners against the standards set out in the national curriculum statement. Respondent M said that moderation ensures that the standards of assessment in subjects are consistent both across the institution and with national standards.
Feedback often suggests variety of thoughts either from us both or moderators. However, principal purpose of the feedback is to 'health check' assessment, which most often is a part of a requirement of an institution and for purposes of best practice. Although, as respondent G claims, "I have not encountered this before, dealing with disagreements over moderation decisions is something I have thought of for some time now."
Northedge (2003:169) argues that: "The teacher, as subject expert, has three key roles to play in enabling learning: lending the capacity to participate in meaning, designing well planned excursions into unfamiliar discursive terrain and coaching students in speaking the academic discourse."
This proposal put forward by Northedge (2003) elaborates on communication challenges many experts face (in this case teachers). This underlines the importance of getting over assumptions about language or medium of instruction of assessment.
In this case, it is imperative to consider the contexts that shape patterns of expert communication, which Northedge (2003:169) notes as "... lending the capacity to participate in meaning ...". We argue for the importance of attending to student learning in the classroom. Discourse is becoming increasingly more valued in the learning and teaching of mathematics. Theoretical discussions of learning that once focused only on the mental constructions of individuals are now acknowledging the vital role that discourse plays in those constructions. Furthermore, the very act of assessing one's thinking can also transform one's understanding (Jones & Jones, 2007; Green, Condy & Chigona, 2012). Consequently, it is difficult to conceive of knowledge formation in the absence of discourse of assessment.
Indeed, some assessors (Jones & Jones, 2007:7-15) have come to see the discursive processes, carried on in sociocultural contexts, as better describing the "understanding" of mathematics than more cognitively focused studies. It is through participation in a discourse community that the individual learns how to engage in meaningful mathematical activity (Jones & Jones, 2007). The structured nature of assessment practice in a discourse community points to the presence of important norms (Jones & Jones, 2007) or meta-discursive rules (Jones & Jones, 2007) that guide accepted practice. While such underlying norms and meta-rules can be made the subject of explicit discussion and negotiation (Jones & Jones, 2007), it is often difficult for participants to engage in such a discussion due to the tacit nature of the norms and meta-rules (Jones & Jones, 2007). Thus, these norms and meta-rules are often taught and learned unknowingly by interlocutors through participation in the discourse community. These learned norms and meta-rules are often seen as being appropriated to form the mental structures and habits of mind that we traditionally view as an understanding of mathematics.
Some researchers argue that social practice does not bring about learning; rather, participation in social practice is constitutive of learning (Jones & Jones, 2007). The researchers view is that it makes no sense phenomenologically to ignore or devalue the existence of an inner mental life by reducing everything to practice or to participation in a discourse community. Clearly, learners do have a sense of "understanding" Jones and Jones (2007:7-15) ideas, of remembering and executing procedures. For the above reasons, we agree with Jones and Jones (2007) that when studying the learning and teaching subjects it is necessary to attend to both individuals' understanding and social contexts and practices.
The study addressed practices in assessment and moderation and explored various processes and methods used in assessment. It indicates that a focus on assessment in practice or discourse is often unsatisfactory as the sole lens for viewing learning, because knowing then becomes a property of the particular social context in which it took place, and thus there is no room for the notion of transfer. The study revealed that assessors need to use different methods of assessment dependent on the socio-cultural setting of learners' environment and resources if applicable. We argue that teachers ought to note the socialisation within their domain as well as the culture of their domain and domain-specific ways of talking, acting, and seeing the world. This not only defies learners' sense that they carry something away with them from our social interactions, but also violates the underlying purpose of education, namely, to enable people to participate appropriately in similar practices in subsequent contexts that will unavoidably be different from the context in which the initial practice took place.
We argue that teachers ought to note the socialisation within their domain as well as the culture of their domain and domain-specific ways of talking, acting, seeing the world. This may include but not limited to teachers' knowledge being organised as schemas, the teachers possessing deep tacit knowledge of their domain that can make articulation difficult.
Cohen L, Manion L & Morr K 2000. Research Methods in Education (5th ed). London: Routledge. [ Links ]
Creswell JW 2003. Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.) Thousands Oaks, CA: SAGE Publications. [ Links ]
Department of Basic Education 2010. Curriculum and Assessment Policy Statements (CAPS). Pretoria: Department of Basic Education. [ Links ]
Green L, Condy J & Chigona A 2012. Developing the language of thinking within a classroom community of inquiry: pre-service teachers' experiences. South African Journal of Education, 32:319-330. Available at http://www.scielo.org.za/pdf/saje/v32n3/08.pdf. Accessed 24 June 2013. [ Links ]
Hlalele D 2012. Exploring rural high school learners' experience of mathematics anxiety in academic settings. South African Journal of Education, 32:267-278. Available at http://www.scielo.org.za/pdf/saje/v32n3/04.pdf. Accessed 24 June 2013. [ Links ]
Jones VF & Jones LS 2007. Comprehensive classroom management: Creating communities of support and solving problems (8th ed). Boston, MA: Allyn & Bacon. [ Links ]
Krause KL & Coates H 2008. Student's engagement in first-year university. Assessment and Evaluation in Higher Education, 33:493-505. doi: 10.1080/02602930701698892 [ Links ]
Lane KL, Wehby JH & Cooley C 2006. Teacher expectations of students' classroom behaviour across the grade span: Which social skills are necessary for success? Exceptional Children, 72:153-167. [ Links ]
Northedge A 2003. Enabling participation in academic discourse. Teaching in Higher Education, 8:169-180. doi: 10.1080/1356251032000052429 [ Links ]
Nyaumwe LJ & Mtetwa DK 2011. Developing a cognitive theory from student teachers' post-lesson reflective dialogues on secondary school mathematics. South African Journal of Education, 31:145-159. Available at http://www.scielo.org.za/pdf/saje/v31n1/v31n1a11.pdf. Accessed 24 June 2013. [ Links ]
Ramnarain U 2011. Teachers' use of questioning in supporting learners doing science investigations. South African Journal of Education, 31:91-101. Available at http://www.scielo.org.za/pdf/saje/v31n1/v31n1a07.pdf. Accessed 24 June 2013. [ Links ]
Reason RD, Terenzini PT & Domingo RJ 2007. Developing social and personal competence in the first year of college. The Review of Higher Education, 30:271-299. [ Links ]
Yorke M 2007. Assessment, especially in the first higher education: old principles in new wrapping? (Keynote paper at on-line REAP conference: Assessment design for learner responsibility, 29-31 May). Available at http://www.reap.ac.uk/reap/reap07/Portals/CSL/keynotes/ mantz%20yorke/Assessmentold_principles_new_wrapping.pdf. Accessed 24 June 2013. [ Links ]
Young K 2006. The Art of Youth Work (2nd ed). Lyme Regis: Russell House Publishing. Available at http://www.russellhouse.co.uk/pdfs/Artofyouthwork.pdf. Accessed 24 June 2013. [ Links ]