RESEARCH ARTICLES

 

Designing an English language and literacies knowledge and skills test for Initial Teacher Education students in South Africa

 

 

Nicky Roberts^I; Thelma Mort^II

^IKelello Consulting, Johannesburg, South Africa. nicky@kelello.org; https://orcid.org/0000-0002-1910-0162
^IIDepartment of Language, Arts and Culture, University of South Africa, Pretoria, South Africa mortkb@unisa.ac.za / thelmamort@gmail.com (corresponding); https://orcid.org/0000-0002-8775-0618

 

 

---

ABSTRACT

Since 2016 the Primary Teacher Education (PrimTEd) project has assessed first and fourth year student teachers' English language and literacies knowledge and skills with a view to optimising what is offered in teacher education courses. In 2021, after a critical review of these English language assessments, a process of test redesigning that also provides professional development opportunities for language teacher educators, was initiated.
In this paper, we report on this process of modifying an existing test and discuss a new design that is still a work in progress, in order to make it more responsive to the English language needs of the student teachers. We argue that the emerging test has the potential to contribute to improved English language teacher education at universities and to improved teaching and learning of languages and literacies in South African primary schools.

Keywords: assessment, PrimTEd English language project, test item development, Core Academic Language Skills, Initial Teacher Education (ITE), professional development of language teacher educators

---

 

 

Introduction

This special issue calls for research that contributes to answering the research question, "How can teacher educators prepare student teachers to be responsive to varying contextual demands, ethical imperatives, diverse learning needs of learners, new modes of delivery, specific professional expectations, transformation priorities, amongst others?" This question is both timely and highly relevant to the Initial Teacher Education (ITE) sector in South Africa. It is timely since Covid-19 disrupted ways of working in ITE and this led to both opportunities and challenges with regard to what is taught, how it taught, and how it is assessed. It is highly relevant since it invites alternatives to what Moloi and Kanjee (2018) describe as an increasing focus on performativity over learning in both the formal schooling system, and in the ITE sector.

We identify two critical features of this call: first, it expects a professional teacher education community that makes explicit its choices about how best to support student teachers, and second, it identifies responsiveness as the core hoped-for attribute of the student teacher. By focusing on how teacher educators can better prepare student teachers to be responsive, the editors of this special issue place the teacher educator at the centre and consider how they engender responsiveness in the student teachers given that this is considered the most valuable skill of a teacher.

This is a provocative claim in a context in which teachers in schools are being de-professionalised and expected to follow slavishly a highly prescriptive Annual Teaching Plan (ATP), and in which they are policed in relation to pacing and curriculum coverage. A responsive student teacher responds to varying contextual demands, ethical imperatives, diverse learning needs, new modes of delivery, and transformation priorities. By necessity, the responsive student teacher cannot follow a recipe but must act as a professional in making professional judgements about the best course of action given the particular circumstances. The professionalisation of the teaching profession that will lead to teachers being trusted to make such professional judgements in relation to varying contextual demands is sorely needed.

We hold that for teachers to make such professional judgements and be responsive to their learners' needs they require deep subject matter knowledge. We concur with Robinson and Rusznyak (2020) that the practice of teaching is structured by the principles of pedagogy as well as by those of the subject domain. In the end, it is "the degree of connectivity" (Liakopoulou 2011 p. 69) between the different types of teacher knowledge that will ensure teaching effectiveness Therefore student teachers need strong foundational subject knowledge.

A focus on English language and literacies indicates that there is abundant evidence that in South Africa, student teachers' (and teachers') foundational knowledge is not yet secure. Taylor (2014), reporting on findings from the Initial Teacher Education Research Project (ITERP), noted that "low levels of English proficiency among both teachers and learners [place] a fundamental limit on academic progress, since English is the medium of teaching and learning in around 90% of schools" (p. 6). For Maja (2015), in South Africa the majority of "teachers themselves are not native speakers of the English language [so] South African learners are challenged by being taught by teachers whose own English proficiency is limited" (p. 6). Wildsmith-Cromarty and Balfour (2019) noted that

the majority of teachers, especially in the under-resourced schools, are not L1 [first language] speakers of [English] and neither do they have adequate competence in it sufficient to facilitate deep understanding and learning of concepts for their pupils. (p. 313)

In South Africa, the failure of many teachers to draw optimally on learners' linguistic repertoires, together with their own limited competence in speaking and teaching English, contributes to a spiral of academic failure for learners.

According to Madikiza et al. (2018), many teachers are unable to teach reading and "have an underdeveloped understanding of teaching literacy, reading and writing" (p. 2). Mpiti (2015) concluded that "more work still needs to be done in this area, especially in training teachers to be in a position to assist learners in the development of EFAL [English First Additional Language] writing" (p. 222). It has also been argued that teachers need much more pedagogical knowledge in order to help EFAL learners since "learner proficiency in medium of instruction largely determines academic success" (Manyike & Lemmer, 2014, p. 1).

The Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ) Report of 2011 found that teachers' literacy knowledge and skills were below an acceptable level. The 2014 National Education Evaluation Development Unit (NEEDU) report advised that if we are improve the educational outcomes of primary school pupils, teacher quality had to be addressed since "the quality of teaching and learning cannot rise above the ceiling imposed by teachers' capacity to teach" (p. 61). Wildsmith-Cromarty and Balfour (2019) asserted that attention needs to be given to

creating teacher education courses, both for initial and in-service teacher education and training, that focus on teaching reading specifically at primary level. This needs to be coupled with a principled basis for determining professional competence in additional language teaching. (p. 316)

Given the above, there is clearly a national need to identify the levels and kinds of competencies of entering and exiting student teachers with a view to enabling high level exit competence in English language and literacies. In this paper, we report briefly on the Primary Teacher Education (PrimTEd) project's work in developing standards to be achieved by student teachers in ITE programmes and, more extensively, on the design and redesign of tests to measure student teachers' English language and literacies knowledge and skills on entering and on exiting the degree programme. This provides an example of how the teacher education curricula in the particular subject domain of English language and literacies knowledge for primary school teachers can be collaboratively developed by a professional community of teacher educators. Further, it makes explicit the ways in which English language and literacies knowledge for primary school teachers is being assessed, at entry and exit levels, through a standardised PrimTEd English assessment. Tests of teacher knowledge are common in North America, as Bertram et al. (2015) have remarked, but South Africa has not the same history. It is precisely this that PrimTEd wishes to accomplish, believing that this testing of student teachers' English language and literacies knowledge and skills offers the possibility of turning around South Africa's poor school results in the long term. This process of designing the English first additional language and literacies tests is our concern in this paper.

 

The problem

We trace the selection and use of English tests by teacher educators involved in the PrimTEd project from the inception of the assessment strand of this project in 2016 to its second and on-going iteration in 2021. In these two phases of the project, a continual refining, testing, and redesigning of the overall test and of individual test items has resulted in changes that aim to create an improved assessment measure. The research question addressed in this paper therefore is, "How does one design a test and improve it through ongoing reconceptualisation and redesign in order to effectively and fairly assess student teachers' English language competence on entry and at graduation?"

 

Conceptual framing

This paper is framed by two types of discourse arising from first, local research into the necessity of improving additional language teaching and teacher education, and second, the discourses around English additional language teaching assessments, discussed below in relation to the PrimTEd English language tests.

 

Discussion

The context

When the first democratically elected government came into power in 1994, the provision of education was unequal at every level. In teacher education, there were no commonly agreed upon standards of proficiency for graduates from ITE and, a decade later, there was widespread concern about low standards of primary school teaching (Deacon, 2012; Spaull, 2013). The National Education Evaluation and Development Unit (NEEDU) was established in 2009, to see how inequality could be reduced in schooling. Its 2014 report showed that although the South African school system now provided more access to education, "for a very high proportion of children of school-going age, the overriding problem is systemic underperformance" (2014, p. 5). The report noted, "[T]o make matters worse, the quality of schooling is inequitably distributed, with the poorer 80% of the population generally receiving schooling of significantly inferior quality to that enjoyed by the most affluent 20%" (p. 2). NEEDU valuably offered specific guidelines, the first being about "strengthening recruitment and promotion procedures" but PrimTEd's (2014) work responds to the second and third of their three recommendations. The second of these is to work with a "scientific, evidence-based trajectory, through allocating adequate resources for evaluation, research and development, and acting on research results" (p. 61). PrimTEd aims to use assessment in a rigorous way. Perhaps the need for rigour has been put most strongly by Soudien (2008) who, lamenting the poor state of literacy and numeracy in the country, stated,

We do not . . . have a body of empirical work and theoretical engagement that is able to speak to this situation . . . after almost 50 years of serious research into teaching and learning, we cannot say, without qualification, what works and what does not (p. 7).

The third NEEDU recommendation was to "improve the quality of ITE" (2014, p. 61). The necessity of improving the quality of teacher education with a view to improving pupil learning outcomes led to the development of the Minimum Requirements of Teacher Education Qualifications by the national Department of Higher Education. The PrimTEd project began in 2016, as a Department of Higher Education and Training project, with funding from the Teaching and Learning Development Capacity Improvement Programme and the European Union. The intention was for PrimTEd to focus on improving standards in mathematics and English language education, the two greatest areas of systemic weakness. It was important to focus on ITE since this four-year programme is the link between matriculants exiting Grade 12, on one hand, and newly qualified teachers leaving university to take up posts in schools, on the other. It is a key cog in the school system, and a strong case can be made for its primacy as a lever, if we mix the metaphors here, for effecting systemic change (NEEDU, 2014).

Findings from ITERP (2012-2014) indicated that the curricula and pedagogies of ITE programmes at South African universities differed in significant ways from one another and also varied greatly in terms of resource provision, including the provision of well-qualified teacher educators in sufficient numbers to meet the needs of student teachers. One of the aims of PrimTEd was to build a professional community of teacher educators in and between the historically uneven and fragmented teacher education programmes offered by South African universities. To this end, teams of teacher educators drawn from across South African universities worked to develop a teaching standards framework for ITE.

PrimTEd teacher standards

In 2018, teams of teacher educators acting on the Integrated Strategic Planning Framework for Teacher Education and Development in South Africa 2011-2025, (Departments of Basic Education and Higher Education and Training, 2011, pp. 76-77) recommended that

teacher knowledge and practice standards in each learning area be developed to assist teachers to identify their specific development needs focused on their core functions (such as classroom teaching of a specific subject or in a specific phase, or for school leadership and management purposes etc.). (Aitchison et al., 2018, p. 5)

The team that developed standards for primary school teachers of languages consulted widely with teacher educators and representatives of education NGOs. It is important to note that the standards developed are not generic since they "include only standards that apply specifically to language and literacy or to generic type standards as specifically applied to language and literacy" (Aitchison et al. 2018, p. 6).

The standards that teachers should meet were conceptualised in terms of the following: "Knowledge (graduate teachers have knowledge of language and literacy and how to teach learners to read and write); Practice (graduate teachers can organise systematic language and literacies instruction with a focus on reading and writing, guided by the requirements of the curriculum); English First Additional Language (EFAL) (Graduate teachers demonstrate that they understand the knowledge, skills, and processes required to teach English as a First Additional Language); and, First Additional Language (FAL): Graduate teachers demonstrate that they understand the knowledge, skills, and processes required to teach African languages or Afrikaans as First Additional Languages" (PrimTEd Primer, 2021, p. 4, bold type in original).

The standards documents include notes on teaching learners to read, different types of literacies, a desirable environment for promoting literacies and, importantly, very specific lists of what language and literacy teachers need to know and be able to do ((PrimTEd Primer, 2021, p. 7). With regard to language learning, reference is made to the importance of using all the languages and varieties of languages that learners know as classroom resources. As will be explained below, these standards became the reference point for redesigning the English tests used for diagnostic purposes in ITE programmes at several universities from 2016 onwards.

The first language test used in the PrimTEd project

The designers of the language tests aimed to develop an instrument that could be used twice-at the entrance to and exit from the Bachelor of Education degree. The test aimed to measure first, student teachers' entry level language and literacies competence at the start of the first semester of the B.Ed degree and, second, to compare their results on this test with those obtained after a second writing of the same test late in their fourth and final year of study. First year students' answers to test questions could be used by language and literacies teacher educators to redesign aspects of their curricula and to reconsider their pedagogic approaches in ways that would both fill knowledge and skills gaps and enhance and extend students' knowledge and skills. Mathipa and Manyike, (2016, p. 382) described the important role of diagnostic assessment in assisting teacher educators to "design intervention programmes and strategies." The hope is that when final year student teachers retake the test there will be results-based evidence of considerable improvement. Ultimately, all those involved in the PrimTEd project would like graduating teachers to have the knowledge and skills required to teach learners how to read, write, and present texts of various kinds, including multimodal texts, and to use subject discourses effectively for learning across the curriculum.

It is important to describe and discuss the level of the PrimTEd test. While the Core Academic Language Assessment (CALS)-1 test (Uccelli, Barr and Phillip Galloway, 2014) was designed for pupils in Grades 4-8 (Uccelli & Galloway, 2017), the PrimTEd test was designed for university students. This is because South Africa's EFAL results in the Grade 12 public examination have been determined to be comparable with those of middle year pupils internationally. While this might be seen as the acceptance of a very low level of proficiency, it was a pragmatic decision. As Pretorius and Currin (2010) noted,

It is important that targets are modest and pragmatic . . . If reading levels are initially very low, then literacy changes do not come about easily or quickly. Reading skills improve slowly and reading gaps between weak and more able readers can persist in such contexts. Long-term interventions of at least four to five years are needed to bring about effective changes. Stakeholders involved in such interventions need to be aware of this and avoid the allure of short-term or 'quick fix' interventions. (p. 75)

It was therefore decided that the level of the test needed to be suitably modest if it was to minimise floor effects and provide maximum information about student teachers' levels in terms of disaggregated results on a test consisting of the two components: Persuasive Language and Authentic Texts.

While the test is intended for all primary school student teachers it is weighted more towards assessing the knowledge and skills required for teaching in the Foundation Phase (Grade R-3), because "it is there that initial literacy skills are learned." However, Intermediate Phase teachers also need the knowledge and skills central to Foundation Phase teaching because "current evidence is that a majority of South African Intermediate Phase learners are not yet competent readers or writers in any language and Intermediate Phase teachers have to remediate this" (Aitchison et al., 2018, p. 7). The two components of the first PrimTEd English test, in its first iteration, were Authentic texts (English)-this part of the test assesses comprehension skills and is an abridged version of the Tea Test, so named because of its content being about tea,¹ and Persuasive Language (English). This is a reduced and adapted version of CALS.

The Authentic texts (English), adapted from the Academic Language Skills test, was a comprehension test that had been designed by Alan Cliff (2014) at the University of Cape Town (UCT)² and used by academics from Cape Peninsula University of Technology (CPUT) almost a decade ago. The intention was for it to be used diagnostically at CPUT to first assess and then to strengthen the English language knowledge and skills of student teachers. The test assesses vocabulary, separating the essential from the non-essential, inference, metaphorical expression and text genre, and editing and understanding the communicative function of sentences.

The PrimTEd assessment team reduced the length of the Tea Test and renamed it Authentic Texts in recognition of the need for assessments in African languages to make use of texts and passages originating in the language being assessed (rather than being translations from English). The concept of a test using authentic passages on a common theme was considered valuable. The underlying constructs being assessed could then be applied when the test was re-versioned into particular African languages. The version of the Tea Test used at CPUT initially included a composition-writing component, but when the test was moved online (to save the labour of physical marking), this component was dropped.

The Persuasive Language (English) test was adapted from the CALS-1 test that was designed initially for pupils in North America to cater to a multilingual school population with a range of home languages. For these pupils, English was an additional language with which they experienced challenges in the classroom. It was found that what was holding these learners back substantially was a lack of core academic language skills (Gee, 2004; Snow, 1986). According to Gee, a core feature of "academic languages is that they demand that learners learn how to produce lots of explicit language around one focused topic" (2004, pp. 24-25). Thus, to succeed in the classroom, learners require strong language knowledge and skills as well as the ability to use the discourse and terminology specific to the discrete content and processes of certain subjects. To promote equity in education it is necessary to focus on academic language skills in teaching and in testing because these are crucial for "supporting students' independent learning from text in school and beyond" (Uccelli and Phillips Galloway, 2017, p. 1).

Uccelli et al. (2013) recognised that high frequency or commonly used words and terms are required as a foundation for the development of the academic language knowledge and skills that are core to progression and success not only in English but also across other subjects. Similarly, the PrimTEd team, in developing the first iteration of an English test to be used with South African student teachers from 2016 to 2020, recognised that while all primary school teachers need to be able to communicate in English, they should also have the language and literacies knowledge and skills to teach all subjects across the curriculum. Uccelli and Phillips Galloway maintained that "academic texts across content areas exhibit some recurrent language patterns" (2017 p. 2). In other words, specific language features appear in academic texts repeatedly, and if students are not proficient in these core academic language skills they are precluded from understanding texts, from using that academic language, and thus, from performing optimally at school.

The CALS-1 test, on which the initial PrimTEd language test was based, is described as being

a group-administered instrument designed to assess core academic language skills (CALS) in grades 4 to 8. Each CALS-I form consists of a 50-minute paper-and-pencil test that includes eight tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentences, Identifying Definitions, Interpreting Epistemic Stance Markers, and Understanding Metalinguistic Vocabulary (Uccelli, Barr & Phillips Galloway, 2016, p. 1, emphasis in original)

However, of necessity the test needed to be redesigned for the South African academy.

Redesigning the test initially used by the PrimTEd project

In 2021 the next iteration or cycle³ of PrimTEd work began, based on the intention to develop further items so that these could be rotated, Africanising the items, and aligning them more to the PrimTEd teacher standards developed by Aitchison et al. in 2018. By 2021, one of the interests of the assessment team in the PrimTEd project was the match or mismatch between the first test used and the PrimTEd teacher standards. After the completion of a mapping exercise undertaken by several members of the assessment team, a new core item writing team of six English language and literacies lecturers from six national universities was constituted. This group received substantial input and was involved in ongoing dialogue about Africanising⁴ the test items, and using vignettes and multimedia sources in test items. The group went beyond CALS and looked at other sources such as the Global Proficiency Framework tests (United Nations Educational, Scientific and Cultural Organization) (UNESCO) (2019), examining these too, for examples of excellence.

Moloi (2021) advised the group on designing multiple choice question (MCQ) items and on how to identify examples of well-designed MCQs. Towards the end of 2021 each item writer was tasked with writing 30 new items in 2022 that would be put into a randomised new test in 2023, after review and if necessary, redesign. This new test will be piloted in five universities and data from it will be captured. Designing so many new items (as opposed to using the same 30 items repeatedly in the previous version of the Persuasive Language part of the test) creates an item bank that enables the use of new combinations of items each time the test is administered. It is hoped that eventually none of the PrimTEd test items will give floor or ceiling results, but will give, rather, specific, accurate and rigorous information about each candidate's knowledge and skills.

Designing the second test

Much work went into the consideration of items from the following perspectives: accuracy; fairness; validity and reliability; and accessibility (Moloi, 2021).⁵ These criteria are also a consideration in the interpretation and the application of the test results, and we discuss these below.

In terms of accuracy, it was decided that the most recently used (2017-2021) test needed to be improved in the following ways: first, the current test has ceiling results, evident when many students achieve extremely high marks on some questions. The new test needed to include more difficult questions to give more diversity and information in the results. Diversity is needed in that when scores are disaggregated per university cohort, they reveal language areas that are weak. This gives the teaching staff at universities information that enables them to attend to these weaker areas in terms of remediation and course design, to effect improvement. This is done through data analysis and is where the PrimTEd project's specialised data analytic skills are used.

On the subject of data, it is imperative that in the PrimTEd work involving assessment, there is a focus on quantitative data analysis since results involve basic statistics. We calculate the mean and standard deviation for each year group and compare the phase specialisations (IP and FP) at first year and fourth year level. This is done to establish whether the observed differences in means is significant or not and we conduct a t-test for unpaired data. Where significance is found we quantify this difference by calculating the effect size using Cohen's D. We also report five-point summaries, to offer another indicator of the distribution of the results for each group and reflect on their similarities and differences.

To obtain more detailed information at item level, the facility scores for each item are calculated, and areas of strength and weakness in the test are noted. Being able to assess constructs through different items is a key aspect of the assessments. Measuring performance on these constructs has an impact on the iterative process of designing further testing items, enabling PrimTEd to refine them and allowing for increasingly disaggregated results.

Examining the test results over time, and construct by construct, explicates a process of improvement, showing how specific standards of engagement with Authentic Texts in English can be assessed, and shows how such rigorous work can potentially improve standards in English language education. It is also hoped that this provides standard setting processes meeting the minimum requirements for teacher education (MRTEQ).⁶ The results also give universities the opportunity to define and design their English language courses, giving them the tools to attend to language issues in more depth in ITE.

Having understood the key features of the assessment data, academics in teacher education can then give attention to their current university course offerings and design in relation to English language. This is indeed one of the ways in which responsivity can be enabled by making use of the test data; one of the exciting features of the testing is that it offers the opportunity for real and efficient changes to be made to curricula on ITE courses that should improve language teacher standards in ITE, and ultimately standards in schools.

With reference to fairness, the team decided that this aspect would be promoted if items were well written and culturally accessible to students from diverse backgrounds. For example, in the case of the former criterion, each aspect of the item (from the instruction to the stem, the stimulus, and the distractors) had to be unambiguous, well written, and purposeful. The distractors had to be plausible answers rather than obviously incorrect. The appearance of the key (correct answer) and distractors had to be visually similar, so as not to make the correct answer too apparent. There could be no repetition in the distractors, so that the test candidates' time would not be wasted with unnecessary reading. With reference to the latter criterion, that of cultural accessibility, the content of prompts-either written texts and/or images-needed to be accessible to students from widely varying socio-cultural backgrounds, and to express southern African diction and experience.

In terms of reliability and validity, the first test could not be closely linked to PrimTEd Teacher standards since its selection, modification, and use pre-dated the availability of these standards. The second iteration is linked to these standards, with the aim of increasing its reliability. To make the test more robust and give the whole test instrument further validity, the number of question items had to be increased so that there could be an unpredictable rotation of like-for-like items.

The original Persuasive Language (English) test had been based on a test for pupils. It was decided to make the new, redesigned test relate more to the experience of beginner teachers than to children. (The CALS-1 test, being designed for children from Grades 4 -8, as discussed earlier, is related more to children than to teachers, and it therefore includes questions that relate to a child's experience of the classroom, not to a teacher's experience.) While it is particularly challenging in a written MCQ language test to assess pedagogical content knowledge, it was decided to make the test more about teaching and to do so by creating a sense of the test being for future teachers. To do this, the questions include in their stimulus examples from classrooms and language teachers' decisions including the kind of knowledge and skills that language and literacies teachers should know and be able to enact. This aspect of the item writing really reflects the complexity of teacher knowledge as Bertram et al. (2015) have noted, including "knowing that" [and] "knowing how" (p. 170). It was hoped that by doing this the student teachers who sat the test would be showing not only the PrimTEd team what they knew but how that knowledge could be applied in the classroom, and also that the test might even extend their understanding.

Finally, with reference to another important aspect of reliability and validity, the process of item writing has been a rigorous one. After participating in professional development workshops, item writers (all academics from national universities, and English language specialists) signed a non-disclosure agreement to protect the test, and attended an item writing retreat, during which they produced 30 draft items, in the style of the Persuasive Language component of the 2017-2021 test. All these were discussed at the retreat and subsequently refined, blind-reviewed, and critiqued by pairs of item writers. A list of review criteria was supplied to each pair: is the item clear, well written, appropriate in context, and level? Is it coded⁷ correctly? The feedback of each pair was sent to the item writing coordinator for return to the original writer. The writers were required to engage with this feedback to further improve the items before all items designed to date are taken to a round table discussion and another round of review. This rigour is part of the test's accuracy. At the conclusion of this process selected items will be packaged into a pilot version of the test to be used in 2023.

In terms of accessibility, it was decided to further Southern Africanise the test. This meant finding topics and texts (both written and visual) that reflect a Southern African experience and going beyond what had happened in the first test that involved Southern Africanising the language of items like, for example, changing American words such as yard to the South African equivalent that would be garden or plot, and American style school terms like recess to breaktime. In the new wave of item development, this meant going further, beyond words, in also paying attention to the context and the type of world view and experience reflected in the stimuli or prompts for test items.⁸ Given the current importance of visual literacy, identified in both the UNESCO Global Proficiency Framework Standards tests and in the new Teacher Standards, it was decided to include diverse stimuli such as graphs/tables, paragraphs, photographs, pictures/cartoons, maps, vignettes, formulae, and recipes. In another way these stimuli also became more meaningful in that they were related to the African context, to teaching and learning, and/or to content in current age-appropriate text⁹ books.

Another important aspect of the development of the current test and test items is that writers were encouraged to design questions at the three levels of easy, medium, and difficult. This will also increase the accuracy of the results and the information given by the results by establishing levels of difficulty and achievement within the test. It is hoped that creating test items at each of these levels will produce results with disaggregated information that can indicate to language teacher educators what student teachers know and where they need support.

Going forward

Before we conclude this paper, we think it useful to note the new directions in which the work and impetus of the project's current iteration has grown.

The first one to be noted relates to technology. This concerns the relationship of the item design to the programme or platform used for administering and marking the test online. Taking the test online provides both constraints and opportunities. It is intended that item writers will participate in Jumptrak¹⁰ training, or an introduction to it, so as to be better acquainted with the platform on which the tests will be located, and the way this affects, and creates possibilities for, the construction of test items.

New items are being developed for piloting. When the new test is piloted, because the new items are similar in form to those used in the Persuasive Language component of the existing test, it is hoped that these new data sets will be used for longitudinal cross-sectional data comparisons.

Since this process began, another exciting development has occurred in that a PrimTEd African languages development unit has been initiated. From early in 2021 teacher educators, whose focus is the teaching of African languages in teacher education and in schools, have participated in online and face to face meetings with the English item writing team. Their contributions have enriched the English test development process through their raising of important issues in relation to teachers' and learners' linguistic repertoires, together with the complexity of teaching both English and African languages in diverse classroom contexts. Through this engagement with colleagues across the language teaching spectrum, PrimTEd hopes to contribute to the improvement of all language teaching and learning in South African teacher education programmes and schools, and the strengthening of African languages' place in schools, and teacher training institutions, as well as the promotion of multiliterate education in South Africa.

 

Conclusion

In this paper, we have reported on PrimTEd's considered and multi-faceted strategy to contribute to improved English language teaching knowledge and skills through the development and use of a standardised assessment of first and fourth year B.Ed students' English language and literacies knowledge and skills in Southern African ITE programmes.

An important feature of the PrimTEd assessment team's work is that it leads to strong data sets being captured over time, which is, as Manyike and Lemmer (2014) have pointed out, a necessary component of research in improving South African language in education over time. And, as Taylor (2019) observed, "[W]ithout very significantly improving teacher disciplinary knowledge, and pedagogic proficiency, all other efforts aimed at improving the quality of South African schooling are likely to come up against low ceiling effects" (p. 279).

The testing of student teachers' English language and literacies knowledge and skills and the using of test results to enhance the quality of teacher education programmes offers the possibility of turning around South Africa's poor school results in the long term. By working collaboratively with teacher educators who specialise in African languages and in English as an additional language, participants in the assessment strand of the wider PrimTEd project hope to see the teaching of English as a First Additional Language take place in the context of African languages, and not in a vacuum.

Collaboration across South African languages and the resulting academic reciprocity strengthens the bi- and multi-lingual teaching that is required to increase responsivity.

If teacher educators to support student teachers to be responsive when teaching English and when teaching in English, we contend, first, that the teacher educators themselves must be responsive. Teacher education curricula need to be responsive to student teachers' entry-levels of subject knowledge, and to consider student teachers' own biographies, as their personal experiences at school and their particular social conditions affect how they engage with the process of learning.

Second, we contend that teacher educators should act as professionals, setting and assessing their own subject knowledge curriculum standards and making explicit their subsequent course design choices. In this way, they would indicate their being responsive to their students and to the need for a national intervention. We think that it is incumbent on teacher educators to assess continually both the entry-level and exit-level subject matter knowledge. Without collecting and engaging with this data it is not possible to be responsive to the student teachers in a particular cohort, nor is it possible to reflect on, nor adequately improve, the efficacy of the programme designed to improve their foundational knowledge.

 

References

Aitchison, J., Dixon, K., Pretorius, E., Reed, Y., Verbeek, C., & Beisman-Simons, C. (2018) Towards competency standards for language and literacy teachers. PrimTEd. https://www.jet.org.za/clearinghouse/projects/primted/standards/literacy-teacher-standards/towards-competency-standards-for-language-and-literacy-teachers.pdf

Bertram, C., Christiansen, I., & Mukeredzi, T. (2015). Exploring the complexities of describing foundation phase teachers' professional knowledge base. South African Journal of Childhood Education, 5(1), 169-190. Retrieved February 12, 2023, from http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S2223-76822015000100011&lng=en&tlng=en        [ Links ]

Cliff, A. (2014) Entry-level students' reading abilities and what these abilities might mean for academic readiness. Language Matters, 45(3). https://doi.org/10.1080/10228195.2014.958519        [ Links ]

Cliff, A., & Hanslo, M. (2009) The design and use of 'alternate' assessments of academic literacy as selection mechanisms in higher education. Southern African Linguistics and Applied Language Studies, 27(3): 265-276. https://doi.org/10.2989/SALALS.2009.27.3.5.939        [ Links ]

Deacon, R. (2012). Initial teacher education project: Institutional case studies composite report. JET Education Services. https://www.jet.org.za/resources/deacon-iterp-final-composite-report.pdf

Gee, J. P. (2004) Situated language and learning: A critique of traditional schooling. Routledge.

Liakopoulou, Maria. (2011), The professional competence of teachers: Which qualities, attitudes, skills and knowledge contribute to a teacher's effectiveness. International Journal of Humanities and Social Science, 1(21). https://www.researchgate.net/publication/285775518_The_professional_competence_of_teachers_which_qualities_attitudes_skills_and_knowledge_contribute_to_a_teacher%27s_effectiveness        [ Links ]

Madikiza, N., Cekiso, M. P., Tshotsho, B. P., & Landa, N. (2018). Analysing English first additional language teachers' understanding and implementation of reading strategies. Reading & Writing Journal of the Reading Association of South Africa, 9(1), 1-10. https://doi.org/10.4102/rw.v9i1.170        [ Links ]

Manyike, T., & Lemmer, E. (2014). Research in language education in South Africa: Problems and prospects. Mediterranean Journal of Social Sciences, 5(8), 251-258. https://doi.org/10.5901/mjss.2014.v5n8p251        [ Links ]

Mathipa, E. R., & Manyike, T. V. (2016, September 19-21). Teacher development and student support: The two opposite sides of the same coin. Towards excellence in educational practices. [Conference session]. South Africa International Conference on Education. Pretoria, RSA.

Maja, M. M. (2015). Classroom interaction in teaching English first additional language learners in the intermediate phase [Doctoral dissertation, UNISA, RSA]. http://hdl.handle.net/10500/20219        [ Links ]

Moloi, M., & Kanjee, A. (2018). Beyond test scores: A framework for reporting mathematics assessment results to enhance teaching and learning. Pythagoras, 39(1), a393. https://doi.org/10.4102/pythagoras.v39i1.393        [ Links ]

Moloi, Q. (2021, October 10). How to write MCQ [Internal Training]. PrimTEd.

Mpiti, T. (2015). Teachers' and learners' experiences of learners' writing in English first additional language: A case study of isiXhosa and Afrikaans learners [Unpublished Doctoral dissertation, University of Fort Hare, RSA]. https://www.semanticscholar.org/paper/Teachers%27-and-learners%27-experiences-of-learners%27-in-Mpiti/50977^1ad19242f798ab8791cff849efafd827b7        [ Links ]

National Education Evaluation and Development Unit. (2014). The quality of learning outcomes: Reducing the inequalities at the higher levels of schooling in South Africa. [Report]. https://search.yahoo.com/search?fr=mcafee&type=E210US105G0&p=National+Education+Evaluation+and+Development+Unit

Pretorius, E., & Currin, S. (2010). Do the rich get richer and the poor poorer?: The effects of an intervention programme on reading in the home and school language in a high poverty multilingual context. International Journal of Educational Development, 30(1), 67-76. http://dx.doi.org/10.1016/jijedudev.2009.06.001        [ Links ]

Pretorius, E., & Murray, S. (2019) Teaching reading comprehension. Oxford.

PrimTEd Project Management Office. (2021, July 1). A Primer: What are the PrimTEd standards and how do they relate to MRTEQ? [Presentation]. University of Johannesburg.

PrimTEd Project Management Office (2021, September 15). PrimTEd Assessment Theory of Change. [Presentation]. PrimTEd resources.

Robinson, M., & Rusznyak, L. (2020) Learning to teach without school-based experience: Conundrums and possibilities in a South African context. Journal of Education for Teaching, 46(4), 517-527. https://doi.org/10.1080/02607476.2020.1800408        [ Links ]

Southern and Eastern Consortium for Educational Quality. (2011). Characteristics of Grade 6 teachers. http://www.sacmeq.org/?q=publications

Snow, C. (1986). Conversations with children. In P. Fletcher and M. Garman (Eds.), Language acquisition (2nd ed.) (pp. 69-89.) Cambridge University Press.

Soudien, C. (2008). The implications of the crisis in numeracy and literacy in South Africa for teacher education. [Keynote address]. Teacher Education Research and Development Programme Conference 2008. https://d1wqtxts1xzle7.cloudfront.net/46394305/Managing_to_learn_Instructional_leadersh20160611-32161-1 da6ecv-libre.pdf

Spaull, N. (2013). South Africa ' s education crisis: The quality of education in South Africa 1994-2011. Centre for Development & Enterprise. https://section27.org.za/wp-content/uploads/2013/10/Spaull-2013 -CDE-report-South-Africas-Education-Crisis.pdf

Taylor, N. (2014) The Initial Teacher Education Research Project: An examination of aspects of initial teacher education curricula at five higher education institutions. [Summary Report.] JET Education Services. admin.jet.org.za > resources > taylor-iterp-summary

Taylor, N. (2019) Teacher knowledge in SA. In J. Jansen & N. Spaul (Eds.), The enigma of inequality (pp. 263-282). Springer. https://doi.org/10.1007/978-3-030-18811-5

Uccelli, P., Dobbs, C. L. & Scott, J. (2013) Mastering academic language: Organization and stance in the persuasive writing of high school students. Written Communication, 30(1), 36-62. https://doi.org/10.1177/0741088312469013        [ Links ]

Uccelli, P., Barr, C, & Phillips Galloway, E. (2014) Core Academic Language Skills Instrument (CALS-I) [Final Report]. Harvard Graduate School of Education. http://nrs.harvard.edu/urn-3:HUL.InstRepos:11380186

Uccelli, P., Barr, C., & Phillips Galloway, E. (2016) Core Academic Language Skills Instrument (CALS-I). JET Educational Services. https://www.jet.org.za/clearinghouse/projects/primted/resources/language-and-literacy-resources-repository/uccelli_2016_ies_cals_report-1.pdf

Uccelli, P. & Phillips Galloway, E. (2017) Academic language across content areas: Lessons from an innovative assessment and from students' reflections about language. Journal of Adolescent & Adult Literacy, 60(4) 395-404.         [ Links ]

United Nations Educational, Scientific and Cultural Organization (2019) Global proficiency framework for reading and mathematics for grades 1-6. https://gaml.uis.unesco.org/wp-content/uploads/sites/2/2019/05/GAML6-REF-16-GLOBAL-PROFICIENCY-FRAMEWORK.pdf https://www.edu-links.org/resources/global-proficiency-framework-reading-and- mathematics

Wildsmith-Cromarty, R., & Balfour, R. J. (2019) Language learning and teaching in South African primary schools. Language Teaching, 52. https://doi.org/10.1017/S0261444819000181        [ Links ]

 

 

Received: 15 July 2022
Accepted: 17 February 2023

 

 

1 To some extent the subject matter, because of its being local and about a locally produced tea, Rooibos, could be said to be familiar to Southern Africans, thus meeting one of PrimTEd's criteria to produce a test that speaks to a Southern African experience and understanding.
2 Cliff subsequently worked on the National Benchmarking Test design, focusing on English academic literacy assessments for selection into Higher Education in South Africa. See Cliff and Hanslo (2009).
3 These iterations relate to funding cycles, work streams and responding to previous data - where ceiling results are found new items which seek to show disaggregated data, or a more detailed result are being developed.
4 This involved writing for a southern African audience and reflecting Southern African experiences, language, and context.
5 Moloi (2021) drew on Black and Wiliam (2011). Since we direct this paper at teacher educators, and not assessment experts, we describe the concepts in non-technical terms, thus capturing the perspectives and interpretations of their application by the PrimTEd English language and literacy team.
6 As promulgated by the Department of Higher Education
7 All new items are coded in terms of language difficulty level for ease of filing, reference, comparison, and discussion, as well as area or type of core academic language covered. Also among the codes were expressions of teacher standards relating to knowledge as well as to teaching standards or pedagogic skills.
8 Given the confidentiality of the test item design, examples cannot be shared. Breaching this would, understandably, compromise the test data.
9 One of the locally written books that contains many good resources and comprehension sources is Teaching Reading Comprehension by Pretorius and Murray (2019).
10 Jumptrak is an online educational assessment and tracking programme and platform, on which PrimTEd tests are taken. It is also very important to PrimTEd's work that there is a reciprocal relationship between the structure of the platform and the PrimTEd assessment questions since it is a repository of all the test items.