RESEARCH ARTICLES

 

Adult education and training system evaluation constructs aimed at performance excellence: A South African perspective

 

 

CG Joubert^{I, *}; Z Sosibo^II

^ICape Peninsula University of Technology, Faculty of Education and Social Sciences ianj@atns.co.za ORCID nr: https://orcid.org/0000-0002-7574-3998
^IICape Peninsula University of Technology, Faculty of Education and Social Sciences SosiboL@cput.ac.za ORCID nr: https://orcid.org/0000-0002-2424-7337

 

 

---

ABSTRACT

In the South African context, existing adult education and training (AET) evaluation models are lacking in comprehensiveness and fail to offer tools which guide organisational system evaluations. An understanding of the interaction and relationship between the various dimensions of a holistic system for the management and evaluation of AET is required. This study was aimed at identifying and describing evaluation system constructs which can provide performance-based evidence for AET organisations in South Africa. Constructs are abstract theoretical expressions regarding phenomena of interest. A qualitative research design was used. Data were collected from documents of the South African regulatory authorities and professional bodies that deal with policies, procedures and processes of AET. This article describes five constructs that can be used in the South African context to facilitate and enhance AET system evaluation, namely: learning organisational culture; organisational intent; organisational system enablers; AET system drivers; and learning organisation performance outcomes. These constructs offer a performance-based approach that is aligned to legislative goals and which has the potential to support learning organisations towards accomplishing their strategic intents and objectives._

Key phrases: Adult education and training; evaluation constructs; evaluation system; training evaluation and performance excellence

---

 

 

1. INTRODUCTION

There is a growing demand for South African adult education and training (AET) organisations to review existing systemic levers of change in order to ensure the effectiveness of teaching and learning and to meet their educational and organisational goals (Shay 2017). In existing literature, AET providers emphasise training/learning programme evaluation (Gandomkar 2018; Khan 2016; Mertens & Wilson 2019; Tshilongamulenzhe, Coetzee & Masenge 2013). In the existing models, evaluation practices tend to lack comprehensive system evaluation constructs that are supportive of organisational performance (Iftikhar & Sirajud 2009; Sitzmann & Weinhardt 2019). Griffin (2010) assert that most organisations merely calculate the impact and return of their investment in AET by evaluating learning, without considering the improvement of organisational performance. Topno (2012) Pineda (2010) acknowledge that although AET has been examined from diverse viewpoints, hardly any of these examinations evaluate extensively, due to the scarcity of effective instruments and practical models. Aziz (2015) and Pineda (2010) mention that traditional evaluation methods focus narrowly on using the assessment processes to advance training delivery. They argue that information obtained from such assessments should also determine whether AET is assisting an organisation towards improving its performance. Marock, Hazell and Akoobhai (2016) contend that it could be problematic to interpret the results of a system evaluation without any basis or theory to assist comprehension, guide implementation and measure organisational change. Chyung (2019), Langmann and Thomas (2019) and Ritzmann, Hagemann and Kluge (2014) call for introduction of a practical, integrated, systematic and credible evaluation method aimed at continuous improvement which can adapt to the organisational context.

 

2. RESEARCH PURPOSE AND OBJECTIVES

The South African Council on Higher Education (CHE 2009) advocates for the optimisation of all organisational system elements that could facilitate transformation and innovation in AET, as directed by the national skills development strategy (Walters 2006). The Department of Higher Education (DHET 2015) found that the collection, compilation and provision of AET monitoring and evaluation data are unsystematic, inconsistent and unrelated. Thus, systemic concerns cannot be identified with the intention of refining performance and encouraging continuous improvement aimed at overall organisational development and advancement. The DHET (2015) advised that a system evaluation approach has to be introduced, which considers all management processes, standards, strategies, plans, indicators, information system, reporting lines and accountability specifications. A common, holistic, systematic, thematic, standardised and comprehensive approach to system evaluation is thus desired (DHET 2016). However, the CHE does not prescribe detailed core system constructs and fundamental relationships, including external factors surrounding AET organisations which affect service delivery performance. Without specified constructs, it is not easy to delineate a domain of AET attributes that can be operationalised (Gioia, Corley & Hamilton 2012). Efforts aimed at evaluating the operational effectiveness and efficiency of system performance may be problematic when prescribed constructs that serve as focus areas are not detailed. The Quality Council for Trades and Occupations (QCTO 2019) stresses a need for such detail, because annual evaluation reports have to be submitted to the QCTO by AET organisations. In view of this requirement, AET system evaluations have to report on expected and achieved accomplishments and examine the results of processes and contextual factors in order to understand overall organisational performance (DHET 2015 & 2016). Evidently, there is a need to identify system evaluation constructs for South Africa's AET organisations that are flexible to different AET contexts and which are aimed at holistic organisational performance excellence. The purpose of this research was to identify, describe and present AET system evaluation constructs which are supportive of organisational performance. The objective of this study was thus to identify and describe system evaluation constructs, as derived from an in-depth understanding of policies, procedures, processes and associated activities, which are relevant to AET organisations in South Africa. This study was guided by the following research question: Which effective system evaluation constructs are appropriate for South Africa's AET organisations to enhance performance excellence?

 

3. LITERATURE REVIEW

It was necessary to find and describe AET system enablers, processes and associated results, understand training system evaluation methodologies and determine how education excellence in support of professional organisational performance should manifest. A critical review of literature aimed at identifying existing AET systems-thinking, system-based evaluation and performance excellence was thus required. The purpose of the literature review was thus to gain an in-depth understanding of the existing knowledge and to identify gaps in said knowledge which is relevant to applied systems-thinking of AET system evaluation in the South African context.

3.1 Systems-thinking

A framework for AET evaluation that underpins this study is systems-thinking. Systems-thinking provides a theoretical foundation which guides deliberations, insights and explanations of systems (Adams, Hester, Bradley, Meyers & Keating 2014). Systems-thinking as a theoretical orientation provides for a perspective that recognises systems as collections of components that are interrelated and whose inter-relationships are as important as the components themselves (Monat & Gannon 2015). Reaching out to the future by creating new meaning and by bringing together parts that are not yet together is the very essence and benefit of system-thinking (Grisold & Peschl 2017). A need exists to specifically address AET evaluation from a systemic perspective within the South African context. The aim is to identify and describe interrelated AET system components that direct, enable and support the performance results of learning organisations. Monat and Gannon (2015) support such systems-thinking because it contemplates the functions of a system's parts, based on their relations with one another and within the system's larger context.

Systems-thinking facilitates the construction of knowledge which relies on the notion of functional fitness and the idea of knowledge being organised and presented in its perceived context (Grisold & Peschl 2017). Cabrera and Cabrera (2015) suggest that AET advancement may benefit from systems-thinking which provides knowledge about system dynamics in order to formulate conceptual themes and designs. In the context of this study an understanding of AET system constructs was advanced by means of qualitative information derived from document analysis. Results obtained were used to identify and describe system constructs and the contexts in which they are embedded. Creating new meaning and realities were thus based on knowledge, embedded in existing AET regulatory frameworks of reference (Grisold & Peschl 2017).

3.2 The need for systemic AET evaluation

In South Africa and elsewhere, AET is a major priority for the government and the private and public sectors (Meyer, Bushney, Mey, Joubert & Van der Merwe 2010). Not only is AET critical to achieving organisational and national transformation goals, but it also constitutes a key element in the pursuit of the achievement of business objectives (Meyer et al. 2010). Desired AET results have to be strategically directed at interventions that can serve as systemic levers of change (Shay 2017) in support of the overall business objectives of the organisation (Meyer et al. 2010). In the current literature, specific systemic evaluation aimed at determining value and benefit by means of organisational-specific conventions for South Africa's AET organisations is not offered.

According to the DHET (2012), evaluation data from AET institutions is not always accurate, comprehensive or organised as part of an integrated system. The CHE (2009) suggests optimisation of all system elements in order to facilitate continuous change and proposes the use of a theory of change and associated logical model. However, in the South African context, there is no prescribed theory of change and associated logical model which facilitates holistic, detailed and continuous systemic review and evaluation of AET organisations. Introducing a specific theory of change and associated logical model may not serve as a viable solution due to its lack of comprehensiveness and effectiveness. Hummelbrunner (2010) states that a logical model/framework is frequently too simple and omits vital aspects of a process for it tend to be fixed and not updated, thus hindering learning and adaptation. De Bruijn (2007) found that monitoring and evaluation systems have become increasingly complex, less practical and inefficient. Ward and Rosenberg (2019) discovered that monitoring and evaluation frameworks in the South African Sector Education and Training Authority environment are often not strategically aligned and inadequately managed. These frameworks tend to narrowly emphasise performance monitoring and compliance initiatives (Ward & Rosenberg 2019). Furthermore, evaluation efforts tend to be isolated and ignore the wider organisational system (Ward & Rosenberg 2019). Williams (2010) suggests the use of system modelling as an alternative to a theory of change and logical modelling. The use of system modelling provides a more comprehensive and effective alternative because all the organisational system enablers which influence AET practices have to be identified and described (Meyer et al. 2010). Hussein, Omar, Noordin and Ishak (2016) propose the development of such a system evaluation model, which is specific to an AET organisation. Such a system evaluation requires an in-depth understanding of aspects to be considered when determining and measuring the efficiency of AET organisations (Meyer et al. 2010). Evaluating the efficiency and effectiveness of an AET system is thus indispensable (Bottyán 2004). Therefore, a need exists for pertinent system-based evaluation constructs within a framework which encompasses specific institutional performance foci, exclusively aimed at AET organisations.

3.3 Conceptualising system-based elements within a learning organisation context

The CHE (2016) emphasises a need to ensure and evaluate AET performance excellence in South Africa as an integral part of the agenda for the transformation of higher education. Extending the evaluation foci to include learning organisation characteristics is a step towards performance excellence (Mohd-Zainal, Yusof & Goodyear 2016). A learning organisation views all its processes as knowledge processes (Dragomir 2017) and have strategies, systems and plans in place to transform learning into action in order to adapt more quickly to internal and external demands which it faces (Sarder 2016). Fourie (2014) acknowledges that a better understanding and implementation of learning organisations in a South African context is needed. However, existing literature does not provide a detailed account of an AET system that encompasses the learning organisation intent for the South African context.

The notion of a learning organisation embraces a culture of learning (Gibbons 2014). A learning culture increases the competitive advantage and is quick to respond to change, while encouraging learning in an organisation (Hussein et al. 2016). The CHE (2004b) acknowledges that a learning culture is indispensable when implementing a system that enables optimal individual, team and organisation innovation and advancement. Mrisha, Ibua and Kingi (2017) found that when a learning culture is cultivated in an organisation, the organisation realises performance improvement. However, details explaining the functional fit of a learning organisation culture within an AET system in the South African context are not disclosed in the literature reviewed.

3.4 The need to promote performance excellence

AET system elements and measures have to be context-specific and aligned to performance excellence in order to support a learning organisation intent (Brown 2014). The CHE (2004f) encourages AET organisations to strive for excellence in terms of system outputs. Organisational learning can enhance the effectiveness of an organisation's drive towards performance excellence (Heydari & Davoodi 2013). Performance excellence is the product of interactions of different sub-systems and enablers in the organisation (Stankard 2002). The South African Qualifications Authority (SAQA 2001) and CHE (2012) recommend the use of a quality award model or excellence model which will facilitate the evaluation of AET accomplishments and organisational performance. These models (for example, the 4P Model, the Vanguard Method, European Foundation for Quality Management Business Excellence Model, and the South African Business Excellence Foundation Model have unique characteristics which measure the rate of achievement in system improvement and performance (Mohajer & Peykani 2016). Such models encourage the implementation of best practices and tools, which allow for the realisation of a strategy of quality, benchmarking of best practices, self-assessment and unremitting improvement (Todorut 2013). Asif and Gouthier (2014) state that these models may be restrictive due to the specificity of services within a pre-defined context and associated pre-defined essential constructs which are frequently understated. Anderson (2014) found that there is a lack of structured processes offered by these models in order to create a learning strategy linked to business objectives and measurement of effectiveness.

The CHE (2006) and South African Board for Personnel Practice (SABPP 2014) urges AET organisations to develop an integrated AET measurement and reporting framework which is linked to organisational strategies and performance. According to the CHE (2001), AET evaluation frameworks should follow a systems-approach. The SAQA (2006) argues that system variables may be measured in terms of outputs (what is produced or delivered through an activity), outcomes (the immediate, short-term impact achieved through the results) and objectives (the long-range consequences of the outcomes). The purpose of such an approach is to analyse the effectiveness of the whole system and to enhance the interfaces between the sub-systems in order to increase the effectiveness of the entire system (Singh 2013). Nonetheless, the SABPP and CHE do not advance details of such an evaluation framework and agreed metrics which support performance excellence criteria. Therefore, an AET schemata which specifically links learning organisation features and organisational successes is needed because it is contemplated as an important prerequisite for organisational excellence.

 

4. RESEARCH DESIGN

This inquiry intended to identify and describe collections of components that are interrelated and whose inter-relationships could be grouped to form AET system evaluation constructs. Following systems-thinking, the researchers tried to understand and explain a phenomenon in terms of interlinked events, activities and flow. Core system constructs and fundamental relationships, including external factors applicable to South Africa's AET organisations were thus sought. A qualitative interpretive research approach using an inductive strategy (Merriam 2002) was used to identify, describe and present AET system evaluation constructs which are supportive of organisational performance. This approach provided an in-depth understanding of policies, procedures, processes and associated activities, which are relevant to AET organisations in South Africa. The purpose of this qualitative approach was to guide the generation of new concepts and relationships, which were transferable to other settings (Corley & Gioia 2011). This design facilitated an in-depth understanding of AET processes for it allowed the researchers to collect, structure and analyse data from an interpretivist position using a thematic coding structure.

 

5. SAMPLING

Bowen (2009) suggests that researchers need to have access to a wide array of documents in order to provide a preponderance of evidence, which have to be used for data analysis and verification. The focus of this research was restricted to AET evaluation information sourced from documents provided by governing and professional organisations. The researchers decided to draw a sample which would provide important and meaningful contributions concerning the phenomenon under investigation. The sample drawn consisted of 36 web-based AET documents from the CHE; DHET; SAQA; SABPP; QCTO; Department of Labour (DOL) and Council of Higher Education Quality Committee (CHEQC). These South African regulatory authorities and professional organisations are responsible to ensure sustainable effectiveness, efficiency and continuous improvement of AET. Sustainability rely on the nurturing of performance improvement which implies consistent evaluation of improvement (Van Dyk & Pretorius 2014). Consistent evaluations of the performance of system elements help to determine the outcome of the system as a whole (Van Dyk & Pretorius 2014). Understanding system evaluation considerations inferred and/or detailed by these authorities and organisations was required. Therefore, documents which reported on AET evaluation concepts, enablers, results and performance excellence as applicable to South Africa's AET organisations, were selected.

 

6. DATA COLLECTION

Documents represent the official perspective on a topic, issue or process (Passer 2014). Selected documents presented complete, authentic, credible and accurate data, which were analysed in order to determine its contribution to the AET issue being explored. Data collection was thus aimed at gathering information about AET evaluation focus areas from documents (including legislation, bulletins, policies, reviews, procedures, frameworks and guides). Corbetta (2003) states that an advantage of document analysis is that the information presented in each document is not subject to a possible distortion, as a result of the interaction between the researcher and another contributor. A further benefit of this analysis is that it negates the difficulties associated with encouraging involvement and participation by users of the documents (Bowen 2009). The chosen documents provided the researchers with text that detailed facts and conditions which are relevant to system evaluation within the AET organisational context. During data collection and analysis, the researchers paid specific attention to the original purpose of each document, its context and the intended audience. By adhering to this process, the researchers were able to investigate the construction of the meaning expressed in each document. The researchers thus scrutinised each document and the language used, in order to identify and describe AET evaluation necessities. Use of this method ensured that the researchers drew their conclusions from the factual evidence at hand, as opposed to their possible predispositions.

 

7. DATA ANALYSIS

Documents used in qualitative research rely on a systematic procedure for review and analysis (Bowen 2009). Each document was analysed to identify and describe the priority given to AET system evaluation, specific AET system evaluation factors, the inputs, processes and results required for AET system evaluation and information on how AET system evaluation must be implemented. Braun and Clarke's (2006) thematic framework was used in this study. A thematic analysis is a form of pattern recognition within the data, with emerging themes becoming the categories for analysis (Maguire & Delahunt 2017). The purpose of the thematic analysis was to identify themes, which were important for this study and to use these themes to address the research question. Braun and Clarke's thematic framework allowed the researchers to become familiar with data, generate initial codes, search for themes, review and define themes and record all findings. In addition to this framework, Atlas.Ti was used to code and classify data as codes, coded families (data categories) and coded super-families (data themes). Word-processed data were directly imported into the Atlas.ti. project file. Data analysis commenced by creating a project (hermeneutic unit) which was meant to enclose data, findings, codes and structures under a single name.

In accordance with Braun and Clarke's framework, the researchers first ensured familiarity with the data by reading through all the documents. The data collected, emphasised the critical role and importance of AET organisational performance standards and operational system requirements. Thereafter, during the second step, initial codes were generated which identified system evaluation considerations that incorporated AET and organisational enablers, processes and results. Data were grouped to form code families during the third step. Consideration was thus afforded to determine how codes relate to one another. During the fourth step, the systemic characteristics and relationships of identified AET process elements were summarised. Throughout the fifth step the researchers considered and explained how the code families relate to each other by grouping these into coherent super-families which identified system evaluation constructs. Finally, all the findings which supported the research results, as well as recommendations were formulated.

 

8. RESULTS

In all the documents the importance, necessity and role of AET evaluation were mentioned. The CHE (2004f:7) explains that a "culture of evaluation needs to define what to evaluate, why to evaluate and, finally, how to evaluate". Therefore, "an evaluation system should be able to account for the paths to change and explanations about success, deviation, and failure of policy and policy implementation" (CHE 2004f:89). Specific AET system evaluation focus areas organised within a specific process were not offered. Instead, the use of quality management and assurance processes was advanced. The CHE (2004b:18) asserts that "effective structures and processes that quality assure and monitor education are required". However, criteria only indicate the requirements for institutional policies, systems, strategies and resources for assuring, developing and monitoring the quality of teaching and learning, research and community engagement (CHE 2004b). Inputs, processes and results required for AET system evaluation were listed and explained but not integrated within a holistic structure. According to the CHE (2004b:1), "institutional quality management covers aspects of input and process as well as outcomes" but "most systems operate on a fitness for purpose premise, evaluating quality arrangements on the basis of whether they support and give effect to self-defined institutional missions and goals" (CHE 2006:71). The documents explain that AET system evaluation is performed as part of a quality assurance and monitoring process. For that reason, "institutional arrangements for quality assurance and quality monitoring include the policies, systems, strategies and resources used by the institution to monitor, evaluate and act on quality issues" (CHE 2004b:1).

The document analysis codes were consolidated into 13 code families, being organisational leadership; organisational strategy, policy and objectives; organisational AET processes; organisational resources; quality management system processes; instructional design; instructional development; implementation appraisal; AET evaluation results; stakeholder relations; organisational capabilities; innovation and excellence; and learning culture.

The study yielded five AET evaluation constructs that are flexible to different AET contexts and which are aimed at holistic organisational performance excellence. These are learning organisational culture that highlight features of a conducive AET and learning environment; organisational intent which emphasises context-specific purpose, vision and strategies; organisational system enablers which deal with supporting arrangements, required to achieve strategic goals; organisational AET system drivers which describe critical processes needed to design and develop products and services; and learning organisation performance results that call attention to the structuring and measuring of outcomes related to an organisation's strategy. The AET system evaluation constructs that were obtained from the results are depicted in Figure 1 and discussed below.

 

 

8.1 Learning organisational culture

Within the South African AET context, a learning organisational culture was found to be an over-arching construct that influences and drives organisational performance and excellence. AET organisations have to produce a vibrant learning culture within the institution (CHE 2004e) because "a learning culture and environment enables optimal individual, team and organisation learning and growth in both competencies and behaviour" (SABPP 2014:60). The "institutional context is of significance in so far as it should create an environment within which good quality higher education programmes can be offered" (CHE 2009:1). The researchers identified the learning organisation culture as the main theme, for it was frequently mentioned as a driver of organisational performance and excellence. The CHE (2004b:3) acknowledges that a learning culture can "act as an incubator of new ideas and cutting-edge knowledge as part of system innovation". Thus, within an education and training organisation, a specific need exists to "create a learning culture and environment" (SABPP 2014:60). Developing an organisation by building strong systems and values which are integrated in the learning organisational culture is thus an important step towards desired performance and innovation.

8.2 Organisational intent

Expressed purpose and intent of an AET organisation have to be stated in its strategies, policies and objectives. AET organisations must "have a clear sense of mission and purpose" (CHE 2006:24) which "articulates well with the institutional mission and strategic goals" (CHE 2004a:8) in order to "guide policy and strategy formulation, implementation, monitoring and evaluation; and support the planning of capacity building in institutions" (DHET 2012:82). These strategies, policies and objectives have to "encourage higher education institutions to engage in systematic and continuous improvement appropriate to their context as well as to their mission and strategic goals" (CHE 2004c:9). Organisational intent thus refers to the strategic direction and goal of the organisation, as well as what it would like to achieve or accomplish in the future. The researchers found that in the documents consulted, the roles and responsibilities associated with organisational intent was leadership which incorporates the strategy, policy and objectives of an organisation. The CHE (2016:108) acknowledges that "leadership is concerned with establishing and promoting the direction of the system or individual institutions of higher education, and the formulation of priorities, policy and strategy in relation to established rules". Thus, "leadership must be capable of proposing a vision that unifies purpose and galvanises different internal stakeholders into action, becomes essential for the development of higher education" (CHE 2016:12).

8.3 Organisational system enablers

In the documents analysed, the researchers found that key organisational enablers consist of organisational AET processes, resources and quality management system processes. Organisational enablers and processes typically include "student recruitment, marketing of the programme, admission and selection policies and practices, staff competence and effectiveness, programme quality, strategies for teaching and learning, ways to monitor, evaluate and effect improvement in teaching and learning, suitable and adequate infrastructure and resources, programme administrative services" (CHE 2004e:5), "budgetary allocations" (CHE 2004e:4) and "financial planning" (CHE 2004b:5). It is essential that "human, financial and infrastructural resources are available to give effect to the goals and priorities" (CHE 2004b:6) of AET organisations. Quality management is critical for "planning, resource allocation and monitoring at institutions" (CHE 2004d:4). Organisational links between "planning, strategic choices, resource allocation and quality management" (CHE 2004b:7) are therefore required. Organisational system enablers thus rely on processes which are described and executed in accordance with a quality management system. What is needed is "an enabling framework, impetus and resources that will lead to on-going improvements in South African higher education institutions" (CHE 2014a:21).

8.4 Organisational AET system drivers

Organisational system drivers within an AET organisation comprise instructional design, development and implementation appraisals. System drivers produce product and/or service quality that is central to an organisation's strategic direction. Instructional design requirements include "occupational curricula which specifies learning inputs required to achieve occupational competence" (QCTO 2008:11) that is necessary to "ensure that training is designed to address industry requirements" (QCTO 2008:12). Instructional development stresses the "arrangements for institutional planning and management of academic programmes" (CHE 2004a:10) by means of "clear and efficient systems and procedures for the design and approval of new programmes, courses and modules" (CHE 2004b:11). Implementation appraisals refer to "policies and procedures that ensure academic and professional standards in the design, approval, implementation and review of assessment strategies for programmes" (CHE 2004a: 14). "Maintaining and applying academic and educational standards, both in the sense of expectations and requirements that should be complied with and in the sense of ideals of excellence that should be aimed for" is critical "with a view to improvement, renewal or progress" (CHE 2004f:16). Organisational AET system drivers afford vocation-directed, purposeful, valuable and outcomes-based AET because they support learning, teaching, assessment and curricula management processes.

8.5 Learning organisation performance outcomes

In the documents consulted, the researchers found four performance-based focus areas that were linked to learning organisation performance outcomes. These learning organisation performance outcomes which are linked to an AET evaluation system, include stakeholder relations, organisational capabilities, and innovation and excellence. An AET evaluation system should be able to "account for the paths to change and explanations about success, deviation and failure of policy and policy implementation" (CHE 2004f:8). An evaluation system is thus "focused on the process of transformation of higher education as well as on the analysis of the modalities and pace at which change takes place at both the systemic and the institutional level" (CHE 2004f:8). At both these levels, stakeholder relations include "collaboration between and among institutions on a regional basis, between public and private provider sectors, between universities and universities of technology, between higher education institutions and the business sector, and between institutions across national borders" (CHE 2004e:2). "Gathering and analysing information from different stakeholders are also important instruments in evaluating the effectiveness of institutions" (CHE 2004b:19). Organisational capabilities and effectiveness depend on "an institution's capacity for quality management of its academic activities in a manner that meets its specified mission, goals and objectives, and engages appropriately with the expectations and needs of various internal and external constituencies" (CHE 2014a:11). A systems-approach used for AET evaluation aims to "improve the processes and structures so that they will be more effective and efficient" (QCTO 2008:10). These organisational business performance focus areas allow AET organisations to benefit from "a degree of competition that rewards merit and performance and promotes innovation and quality" (CHE 2003:19). It also ensures that "the degree of excellence specified is achieved" (SAQA 2001:22) or at least the "sense of ideals of excellence that should be aimed for" were pursued (CHE 2004f:16).

 

9. DISCUSSION

The intention of current AET evaluation efforts is aimed at programme and learning results. The CHE (2009) and CHE (2014b) acknowledge that quality enhancement and continuous improvement need to become a mind-set for AET organisations. The purpose of the research is to describe considerations that can be drawn on by AET organisations to develop system evaluation constructs with the intention of complementing and enhancing AET within the South African context.

Evidence from the documents surveyed identified a systems perspective aimed at organisational performance excellence of AET evaluation. Thus, the entire system has to deal with both performance directed and system evaluation constructs. This is different from merely appraising training and/or learning outcomes. The findings revealed a more practical approach aimed at optimising AET entities' performance which is flexible for different AET contexts. Organisational context is also acknowledged because of its potential impact on AET evaluation focal points, which include system enablers, drivers and results associated with organisational performance. From an institutional perspective, AET evaluation systems must be action-oriented, informed by organisational strategies and influenced by learning culture. AET evaluation systems considerations can thus be identified in terms of performance directed constructs that include the organisation's intent, enablers, drivers and outcomes. Construct details are however considered as being context-specific.

The importance of having structure, competence and capacity which facilitate the design, development and implementation of a system-based organisational focus, in pursuit of AET excellence, is also accentuated by the CHE (2001) and CHE (2016). Instructional design, curricula development, learner assessments as well as implementation of learning emerged as specific AET core functions/drivers. This addition serves an important insight and system contribution because these system drivers are not mentioned in existing excellence models (Sampaio, Saraiva & Monteiro 2012).

Organisational-specific evaluation construct details are not advanced by the researchers because these are context-dependant. Each organisation needs to determine which learning endeavours are suitable and measurable, or which knowledge should be managed or advanced to satisfy organisational performance ambitions. System evaluation construct measures can then be determined by considering how organisational drivers, enablers, intent and learning culture contribute to the actual organisational performance outcomes.

 

10. RECOMMENDATIONS

The researchers recommend that AET organisations should use a more comprehensive AET system evaluation approach that is aimed at measuring performance excellence (favouring a sustainable approach to continuous improvement). Such an AET system evaluation should take cognisance of a learning organisation, quality management/assurance system, as well as business performance influences. The researchers acknowledge that these are organisation-specific, and the setting will influence the construct contents, including the associated measures. Therefore, unique situational aspects, for example national transformation, equity, decolonisation of curricula and social redress priorities within the South African context, have to be determined, addressed and consistently evaluated. Such a context-specific effort thus needs to bring together performance directed considerations and system evaluation constructs. System constructs, features and measures, consisting of organisational intent, organisational system enablers, organisational AET system drivers, learning organisation performance results and learning organisational culture, should be quantifiable and qualifiable to allow for a comprehensive evaluation. Moreover, system-based results must be presented as trustworthy and factual evidence of accomplishment aimed at overall institutional performance to be meaningful.

Designing, developing and introducing such a context-specific AET system evaluation may be risky and difficult. However, Hamel (2006) proposes a pragmatic approach, which could help organisations to review existing management processes by answering specific questions. The following questions, based on Hamel's approach, have been aligned to AET organisational settings and could be posed and answered as an initial orientation, for each relevant AET system construct:

• Which subject matter has to be included in each construct process and sub-processes to ensure compliance and conformance within regulatory and professional necessities?

• In which manner could each construct be defined in terms of learning organisation principles and standards?

• Who owns each construct process and sub-processes?

• Who has the power to change the construct's process and sub-processes?

• What is the success or performance metrics for each construct process and sub- processes?

• Who are the stakeholders for each construct process and sub-processes?

• Who could participate in change management?

• What is the data or information inputs for each construct?

• Which analytical tools were used to ensure trustworthy and factual evidence of accomplishment aimed at overall performance excellence for each construct process and sub-processes?

• How does each construct process and sub-processes link to other constructs?

Implementation of an AET system depends on organisational actions, which will ensure system monitoring, reacting to internal and external influences and refining processes and structures. These actions have to be aimed at improving both current activities and future planning, programming and decision making in order to facilitate organisational transition towards performance excellence. Evidence of such a transition relies upon the collection of evaluation information to identify, understand and appreciate the integrated, interdependent and dynamic constructs within an AET system. Thus, systems-thinking can identify and consider dynamic patterns rather than distinct events which are often found in organisational silos.

 

11. CONCLUSION

The need for a more holistic systemic approach for evaluation purposes is contemplated in the South African context which has to be guided by an evaluation of organisational enablers, processes and results. In the absence of such a holistic approach, evaluation problems such as inappropriate focus, unsuitable criteria, predisposed findings, unjustified conclusions, superfluous recommendations and inadequate interpretations may become apparent (Stufflebeam 2000). Consequently, the interactions on and the relationships between the different parts of the whole (or system) need to be understood, as these influence an entity's organisation, functioning and results. Findings from this study show that system standards which are used to evaluate AET and determine successes need to be defined and framed in terms of organisational effectiveness. Most important is that this systems-based practice can be used to optimise context-specific organisational performance and continuous improvement initiatives. The purpose and value of such an interpretation is that organisations can internalise the concept of training system excellence in their respective internal quality management systems and learning cultures. Centres and organisations that offer AET can use these constructs to evaluate their performance excellence. However, specific criteria associated with these constructs should be determined and aligned to the organisational context to ensure coherence and validity of measures. Information from key role players in the South African AET context should be obtained in future to determine which criteria and measures within the system constructs were identified, designed, implemented and evaluated. Future research may thus benefit from investigations aimed at determining the impact of context-specific AET system evaluations.

 

REFERENCES

ADAMS KM, HESTER PT, BRADLEY JM, MEYERS TJ & KEATING CB. 2014. Systems Theory as the foundation for understanding systems. Systems Engineering 17(1):112-122. (DOI:https://doi.org/10.1002/sys.21255)        [ Links ]

ANDERSON C. 2014. Bad measurement affects training impact. Chief Learning Office. (May 2014.) [Internet: https://www.chieflearningofficer.com/2014/04/23/bad-measurement-affects-training-impact/; downloaded on 12 August 2018.         [ Links ]]

ASIF M & GOUTHIER MHJ. 2014. What service excellence can learn from business excellence models. Total Quality Management & Business Excellence 25(5):511-531. (DOI:https://doi.org/10.1080/14783363.2013.839348.         [ Links ])

AZIZ SFA. 2015. Developing General Training Effectiveness Scale for the Malaysian Workplace Learning. Mediterranean Journal of Social Sciences 6(4:1):47-56. (DOI:http://dx.doi.org/10.5901/mjss.2015.v6n4s1p47.         [ Links ])

BOTTYÁN S. 2004. Leadership and organisational climate. Gardenview: Zytek Publishing.         [ Links ]

BOWEN GA. 2009. Document Analysis as a Qualitative Research Method. Qualitative Research Journal 9 (2):27-40. (DOI : https://doi.org/10.3316/QRJ0902027.         [ Links ])

BRAUN V & CLARKE V. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3:77-101. (DOI:http://dx.doi.org/10.1191/1478088706qp063oa.         [ Links ])

BROWN A. 2014. Organisational paradigms and sustainability in excellence: From mechanistic approaches to learning and innovation. International Journal of Quality and Service Sciences 6(2/3):181-190. (DOI:https://doi.org/10.1108/IJQSS-02-2014-0020.         [ Links ])

CABRERA D & CABRERA L. 2015. Systems Thinking made simple: New hope for solving wicked problems. Ithaca, NY: Odyssean Press.         [ Links ]

CHYUNG SY. 2019. 10-step evaluation for training and performance improvement. Thousand Oaks: Sage Publications.         [ Links ]

CORBETTA P. 2003. Social Research: Theory, Methods and Techniques. Thousand Oaks: Sage Publications.         [ Links ]

CORLEY KG & GIOIA DA. 2011. Building Theory about Theory Building: What constitutes a Theoretical Contribution? Academy of Management Review 36 (1):12-32. (DOI:https://doi.org/10.5465/amr.2009.0486.         [ Links ])

COUNCIL OF HIGHER EDUCATION (CHE). 2001. A New Academic Policy for Programmes and Qualifications in Higher Education. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2003. Good governance in higher education; reflections on cooperative governance in South African higher education - Kagisano Issue Number 2. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/ downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004a. Criteria for Institutional Audits - April 2004 version. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004b. Criteria for Institutional Audits - June 2004 version. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004c. Framework for Institutional Audits. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004d. Improving Quality in Higher Education: Who's Responsibility? Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004e. The Criteria for Programme Accreditation. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2004f. Towards a Framework for the Monitoring and Evaluation of South African Higher Education. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2006. Kagisano Issue Number 4 (Winter 2006). Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2009. The Guide for Evaluators: Accreditation and re-accreditation of programmes submitted to the Higher Education Quality Committee. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2012. Teaching Excellence Awards in South Africa: A National Study. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2014a. Framework for Institutional Quality Enhancement in the Second Period of Quality Assurance. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2014b. Quality Enhancement Project - The Process for Public Higher Education Institutions. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 14 October 2016.         [ Links ]]

COUNCIL OF HIGHER EDUCATION (CHE). 2016. South African higher education reviewed: Two decades of democracy. Pretoria: Council of Higher Education. [Internet:https://www.che.ac.za/; downloaded on 08 January 2018.         [ Links ]]

DE BRUIJN H. 2007. Managing performance in the public sector. Oxford: Routledge.         [ Links ]

DEPARTMENT OF HIGHER EDUCATION AND TRAINING (DHET). 2012. Green Paper for Post-School Education and Training. Pretoria: Department of Higher Education and Training. [Internet:http://www.dhet.gov.za/; downloaded on 14 October 2016.         [ Links ]]

DEPARTMENT OF HIGHER EDUCATION AND TRAINING (DHET). 2015. Monitoring and evaluation and reporting framework for technical and vocational education and training college performance. Pretoria: Department of Higher Education and Training. [Internet:http://www.dhet.gov.za/; downloaded on 14 October 2016.         [ Links ]]

DEPARTMENT OF HIGHER EDUCATION AND TRAINING (DHET). 2016. Draft National Policy for the monitoring and evaluation of Community Education and Training Colleges. Pretoria: Department of Higher Education and Training. [Internet:http://www.dhet.gov.za/; downloaded on 09 January 2018.         [ Links ]]

DRAGOMIR C. 2017. Interrelation between total quality management and learning organization. Review of General Management 25(1):31-37. [Internet:http://www.managementgeneral.ro/pdf/1_2017_3.pdf; downloaded on 23 May 2018.         [ Links ]]

FOURIE S. 2014. Learning organisations in a South African context. Randburg: Knowledge Resources.         [ Links ]

GANDOMKAR R. 2018. Comparing Kirkpatrick's original and new model with CIPP evaluation model. Journal of Advances in Medical Education and Professionalism 6(2):94-95. [Internet:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5856911/; downloaded 14 August 2019.         [ Links ]]

GIBBONS AS. 2014. An architectural approach to instructional design. New York: Routledge.         [ Links ]

GIOIA DA, CORLEY KG & HAMILTON AL. 2012. Seeking qualitative rigor in inductive research: Notes on the Gioia Methodology. Organizational Research Methods 16(1):15-31. (DOI:https://doi.org/10.1177%2F1094428112452151.         [ Links ])

GRIFFIN RP. 2010. Means and ends: effective training evaluation. Industrial and Commercial Training 42(4):220-225. (DOI:https://doi.org/10.1108/00197851011048582.         [ Links ])

GRISOLD T & PESCHL MF. 2017. Why a systems-thinking perspective on cognition matters for innovation and knowledge creation. A framework towards leaving behind our projections from the past for creating new futures. Systems Research and Behavioral Science 34:335-353. (DOI:https://doi.org/10.1002/sres.2456.         [ Links ])

HAMEL G. 2006. The why, what and how of management innovation. Harvard Business Review 84(2):72-84. [Internet:https://hbr.org/2006/02/the-why-what-and-how-of-management-innovation; downloaded on 22 May 2018.         [ Links ]]

HEYDARI H & DAVOODI SMR. 2013. A Study of the Relationship between Organizational Learning and EFQM Excellence Model in University of Tehran. European Online Journal of Natural and Social Sciences 2(3s):1987-1991. [Internet:http://european-science.com/eojnss_proc/article/view/3909; downloaded on 07 December 2017.         [ Links ]]

HUMMELBRUNNER R. 2010. Beyond logframe: Critique, Variations and Alternatives. In Fujita N. Ed. Beyond Logframe; Using Systems Concepts in Evaluation. Tokyo: Foundation for Advanced Studies on International Development. (pp. 1-33.) [Internet:https://www.scirp.org/(S(351jmbntvnsjt1aadkposzje))/reference/ReferencesPapers.aspx?ReferenceID= 1207386; downloaded on 28 June 2019.         [ Links ]]

HUSSEIN N, OMAR S, NOORDIN F & ISHAK NA. 2016. Learning Organization Culture, Organizational Performance and Organizational Innovativeness in a Public Institution of Higher Education in Malaysia: A Preliminary Study. Procedia Economics and Finance 37:512-519. (DOI:https://doi.org/10.1016/S2212-5671(16)30159-9.         [ Links ])

IFTIKHAR A & SIRAJUD D. 2009. Evaluating training and development. Gomal Journal of Medical Sciences 7:165-166. [Internet:https://www.researchgate.net/publication/234101726; downloaded on 23 May 2017.         [ Links ]]

KHAN MS. 2016. Human resource development through training and development: A case study of HAL Lko. International Journal of Applied Research 2(3):353-359. [Internet:http://www.allresearchjournal.com/archives/2016/vol2issue3/PartF/2-2-66-814.pdf; downloaded on 11 March 2018.         [ Links ]]

LANGMANN S & THOMAS SR. 2019. Rethinking training evaluation: omnidirectional evaluation, range of audience and intentional change. The International Journal of Human Resource Management 30(12):1902-1930. (DOI:https://doi.org/10.1080/09585192.2017.1303782.         [ Links ])

MAGUIRE M & DELAHUNT B. 2017. Doing a Thematic Analysis: A practical, step-by-step guide for learning and teaching scholars. All Ireland Journal of Teaching and Learning in Higher Education 8(3):3351-33514. Internet:http://ojs.aishe.org/index.php/aishe-j/article/view/335; downloaded 23 November 2019.         [ Links ]]

MAROCK C, HAZELL E & AKOOBHAI B. 2016. Chapter 6 - What will it take to turn TVET colleges around? Evaluation of a large-scale college improvement programme. In Kraak A, Paterson A & Boka K. Eds. Change management in TVET colleges: Lessons learnt from the field of practice. Johannesburg: JET Education Services: (pp. 103-119.         [ Links ])

MERRIAM SB. 2002. Qualitative research in practice: Examples for discussion and analysis. San Francisco, CA: Jossey-Bass.         [ Links ]

MERTENS DM & WILSON AT. 2019. Program evaluation and theory. 2nd ed. New York: The Guilford Press.         [ Links ]

MEYER M, BUSHNEY MJ, MEY M, JOUBERT P & VAN DER MERWE J. 2010. 8th Annual American Society for Training and Development (ASTD) State of the South African Learning and Development Industry Report, ASTD Global Network South Africa in association with the South African Board for People Practices. [Internet:http://hdl.handle.net/10500/22973; downloaded on 12 August 2018.         [ Links ]]

MOHAJER H & PEYKANI MH. 2016. Comparing the Human Resource Development based on the level of organizational excellence. International Business Management 10(9):1642-1648. [Internet:http://docsdrive.com/pdfs/medwelljournals/ibm/2016/1642-1648.pdf; downloaded on 22 November 2018.         [ Links ]]

MOHD-ZAINAL A, YUSOF SM & GOODYER J. 2016. Validity and reliability study of learning organisation in New Zealand manufacturing companies. Jurnal Mekanikal 39:25-38. [Internet:https://jurnalmekanikal.utm.my/index.php/jurnalmekanikal/article/view/10; downloaded on 22 November 2018.         [ Links ]]

MONAT JP & GANNON TF. 2015. What is Systems Thinking? A Review of selected literature plus recommendations. American Journal of Systems Science 4(1):11-26. (DOI:10.5923/j.ajss.20150401.02.         [ Links ])

MRISHA G, IBUA M & KINGI W. 2017. Effect of Learning Organization Culture on Organizational Performance among Logistics Firms in Mombasa County. Journal of Human Resource Management 5(2):32-38. (DOI:10.11648/j.jhrm.20170502.11.         [ Links ])

PASSER MW. 2014. Research Methods: Concepts and Connections. New York: Worth Publishers, Macmillan Higher Education.         [ Links ]

PINEDA P. 2010. Evaluation of training in organisations: a proposal for an integrated model. Journal of European Industrial Training 3(7):673-693. (DOI:http://dx.doi.org/10.1108/03090591011070789.         [ Links ])

QUALITY COUNCIL FOR TRADES AND OCCUPATIONS (QCTO). 2008. Introduction to the Quality Council for Trades and Occupations (QCTO). Pretoria: Quality Council for Trades and Occupations. [Internet:https://www.qcto.org.za/; downloaded on 14 October 2016.         [ Links ]]

QUALITY COUNCIL FOR TRADES AND OCCUPATIONS (QCTO). 2019. Policy on accreditation of Skills Development Providers. Pretoria: Quality Council for Trades and Occupations. [Internet:https://www.qcto.org.za/; downloaded on 27 November 2019.         [ Links ]]

RITZMANN S, HAGEMANN V & KLUGE A. 2014. The Training Evaluation Inventory (TEI) - Evaluation of training design and measurement of training outcomes for predicting training success. Vocations and Learning 7(1):41-73. (DOI:https://doi.org/10.1007/s12186-013-9106-4.         [ Links ])

SAMPAIO P, SARAIVA P & MONTEIRO A. 2012. A comparison and usage overview of business excellence models. The TQM Journal 24(2):181-200. (DOI:https://doi.org/10.1108/17542731211215125.         [ Links ])

SARDER R. 2016. Building an innovative learning organisation: A framework to build a smarter workforce, adapt to change and drive growth. Hoboken, New Jersey: Wiley.         [ Links ]

SHAY S. 2017. Educational investment towards the ideal future: South Africa's strategic choices. South African Journal of Science 113(1/2):10-15. (DOI:https://doi.org/10.17159/sajs.2017/20160227.         [ Links ])

SINGH M. 2013. Training Evaluation: Various Approaches and Applications. The IUP Journal of Soft Skills 7(1):27-43. [Internet:https://search.proquest.com/openview/5d300db5611d800cd863c5788048411f/1?pq-origsite=gscholar&cbl=2029989; downloaded on 22 November 2018.         [ Links ]]

SITZMANN T & WEINHARDT JM. 2019. Approaching evaluation from a multilevel perspective: A comprehensive analysis of the indicators of training effectiveness. Human Resource Management Review 29(2):253-269. (DOI:https://doi.org/10.1016/j.hrmr.2017.04.001.         [ Links ])

SOUTH AFRICAN BOARD FOR PERSONNEL PRACTICE (SABPP). 2014. SABPP HR Management System Standards Model. Johannesburg: South African Board for Personnel Practice. [Internet: https://sabpp.co.za/; downloaded on 14 October 2016.         [ Links ]]

SOUTH AFRICAN QUALIFICATIONS AUTHORITY (SAQA). 2001. Quality Management Systems for Education and Training Providers. Pretoria: South African Qualifications Authority. [Internet: http://www.saqa.org.za/; downloaded on 14 October 2016.         [ Links ]]

SOUTH AFRICAN QUALIFICATIONS AUTHORITY (SAQA). 2006. A proposed Tool for conducting a Performance Audit of ETQA's. Draft document. August 2006. Pretoria: South African Qualifications Authority. [Internet: http://www.saqa.org.za/; downloaded on 14 October 2016.         [ Links ]]

STANKARD MF. 2002. Management Systems and Organizational Performance: The Search for Excellence Beyond ISO9000. Westport, CT: Greenwood Publishing Group.         [ Links ]

STUFFLEBEAM DL. 2000. The Methodology of Metaevaluation as reflected in Metaevaluations by the Western Michigan University Evaluation Center. Journal of Personnel Evaluation in Education 14(1):95-125. (DOI:https://doi.org/10.1023/A:1008198315521.         [ Links ])

TODORUT AV. 2013. The need of Total Quality Management in higher education. Procedia - Social and Behavioral Sciences 83:1105-1110. (DOI:https://doi.org/10.1016/j.sbspro.2013.06.207.         [ Links ])

TOPNO H. 2012. Evaluation of Training and Development: An Analysis of Various Models. IOSR Journal of Business and Management 5(2):16-22. (DOI:http://dx.doi.org/10.9790/487X-0521622.         [ Links ])

TSHILONGAMULENZHE MC, COETZEE M & MASENGE A. 2013. Development of the learning programme management and evaluation scale for the South African skills development context. SA Journal of Industrial Psychology/SA Tydskrif vir Bedryfsielkunde 39(1):1-14. (DOI:http://dx.doi.org/10.4102/sajip.v39i1.1080.         [ Links ])

VAN DYK DJ & PRETORIUS L. 2014. A systems thinking approach to the sustainability of quality improvement programmes. South African Journal of Industrial Engineering 25(1):71-84. (DOI:https://doi.org/10.7166/25-1-696.         [ Links ])

WALTERS S. 2006. Adult learning within lifelong learning: A different lens, a different light. Journal of Education 39:7-26. [Internet:https://core.ac.uk/download/pdf/62635851.pdf; downloaded on 23 November 2018.         [ Links ]]

WARD M & ROSENBERG E. 2019. Monitoring & Evaluation in a SETA Environment: Scoping Report. Grahamstown: Rhodes University. [Internet:https://www.ru.ac.za/media/rhodesuniversity/content/elrc/documents/2019/setadeliverables/ Project_1_Scoping_Report_31_Jan_2019_(edit).pdf; downloaded on 27 November 2019.         [ Links ]]

WILLIAMS B. 2010. Systems Thinking and Capacity Development in the International Arena. In Fujita N. Ed. Beyond Logframe; Using Systems Concepts in Evaluation. Tokyo: Foundation for Advanced Studies on International Development. (pp. 34-53.) [Internet:https://www.fasid.or.jp/english/_files/web_library/h21-3.pdf; downloaded on 22 February 2017.         [ Links ]]

 

 

* corresponding author