versión On-line ISSN 2413-3221
versión impresa ISSN 0301-603X
S Afr. Jnl. Agric. Ext. vol.41 no.1 Pretoria ene. 2013
Model for the integrated and transversal monitoring and evaluation of rural development programmes implemented by government departments1
General Manager: Strategic Management, Limpopo Department of Agriculture, RSA
Literature indicates that there is a lack of an integrated approach to monitoring and evaluation (M&E) of interventions focusing on rural development, resulting in policy development and analyses not being supported, transparency and accountability on rural spending not enhanced and organisational learning not encouraged. The study looked into the institutionalisation of M&E in government. The research was conducted in three phases: Comparative Case Studies (Canada, Chile, Brazil, Uganda and the United Nations Development Programme), a single Case Study (Limpopo Provincial Government, South Africa) and the development of a model for the integrated and transversal M&E of rural development programmes implemented by government departments (being the general objective of this empirical study). Qualitative data were obtained through document analysis and desk based survey of existing information from various sources. M&E and rural development specialists and agriculturalists were sampled and interviewed using a semi-structured interview schedule. To support the three phases of the research, the data collection method of experience surveying was also utilised.
The study found that M&E is not adequately institutionalised, particularly in the rural development and agricultural sector. The study therefore recommends capacity building on the M&E skills of professionals in the agricultural extension and advisory sector.
Keywords: Monitoring and evaluation, rural development, integrated, transversal, South Africa, Limpopo Provincial Government
In its renewed strategy for rural development titled "Reaching the Rural Poof", the World Bank (2003, p. v) states that three out of every four of the world's poor live in rural areas. The International Fund for Agricultural Development (IFAD) 2011 Rural Poverty Report projects that over the next two to three decades rural deprivation will persist (2010:16). Monitoring and evaluation (M&E) has been promoted as an important concept to improve the quality and impact of rural development efforts (Touwen 2001). Regular and objective monitoring of rural development indicators will assist in governments' ability to formulate and implement rural development policies, effectively assessing progress and demonstrating accountability.
M&E is seen as a priority on the agenda of most developed and developing countries, within the framework of the Millennium Development Goals (MDGs) of the United Nations (UN) (Kusek & Rist 2004:19). The development and sustaining of a comprehensive results-based M&E system at various levels will be key to measuring and monitoring achievement of the MDGs, which set out a range of specific goals and targets to be reached by 2015 (Kusek, Rist & White 2005:8).
According to Sangweni (2004:1) a culture of accountability is also growing in Africa, as evidenced by the African Peer Review Mechanism (APRM), established in 2003, under the initiative of the 2001 New Economic Partnership for African Development (NEPAD). Poverty Reduction Strategy Papers, which have become important vehicles for development in the poorest countries, require M&E systems to measure progress achieved (Ellis & Biggs 2001:444). International donor funding agencies solidified the importance of M&E with the 2005 Paris Declaration on Aid Effectiveness with donors stepping in to ensure the necessary assistance for developing countries to implement M&E systems. The Paris Declaration was further accelerated and implementation deepened through the 2006 Windhoek Declaration and the Accra Agenda for Action in Ghana in 2008 (OECD, Final Country Evaluation Report 2011).
South Africa has nine provinces, three of which have the largest population living in rural areas. That is Limpopo at 86.7%, Eastern Cape 61.2% and Mpumalanga 58.7% (Stats SA 2001:8). Policy on rural development has been changing from the social and political goals implicit of the Reconstruction and Development Programme (RDP) in 1994, the spatial concepts of nodes, corridors and infrastructure strategies contained in the Integrated Sustainable Rural Development Strategy (ISRDS) of 2000 to the extension of quality government services to rural areas in the Comprehensive Rural Development Programme (CRDP) of 2009. Rural development is one of the ten priority areas identified in the South African Medium Term Strategic Framework (MTSF) 2009-2014 (MTSF 2009:2-3). Besides the MTSF the growth strategies supporting rural development from 1996 until 2012 are the Growth, Employment and Redistribution Strategy (GEAR) in 1996, National Outcomes (specifically Outcome 7) in 2010, the National Growth Plan (NGP) in 2011 and both the National Development Plan (NDP) and the Strategic Infrastructure Projects (SIP) in 2012.
The South African Government aims to address rural development through a cross-sectoral and multi-occupational diversity of programmes (ANC 1996). Government needs to respond to constitutional and legislative imperatives to ensure that implementation of (rural development) strategies and programmes can be monitored. During 2005 the Cabinet approved a process to develop an M&E system for the use across government, termed the Government Wide Monitoring and Evaluation System (GWMES). Goldman, Engela, Akhalwaya, Gasa, Leon, Mohamed & Phillips (2012:1) state that despite positive developments regarding the GWMES "significant challenges remain in ensuring the coherence of reform initiatives conducted by central government departments, improving administrative data quality, and establishing M&E as a core role of management".
2. PROBLEM STATEMENT
The literature reflects that there is a growing emphasis on an integrated and comprehensive approach to M&E. Coupled with this is the view that evaluation should not be an event that occurs at the end of a programme, but rather an ongoing process which assists decision makers to better understand the programme, how it is impacting on those involved (PSC 2005:133) and how it is being influenced by external factors (Mackay 2007:60).
According to Csaki (2001:572) international role-players in rural development are attempting to, in conjunction with the implementation of strategies, improve the monitoring of regional and global progress in rural development. However, no standard framework for measuring a country's performance exists as few of the governments of developing countries take a sufficiently cross- or multi-sectoral view of the M&E of rural development efforts (Ellis and Biggs 2001:445). As a result, it is a challenge to develop a common approach to M&E progress within a country and across several countries involved in rural development as a theme which cut across implementation boundaries.
The integrated and transversal M&E of rural development is challenged by the design of M&E systems, particularly the absence of indicators that can be monitored, and a lack of ownership and participation by the stakeholders in rural development. The description of project objectives, components and implementation arrangements are not clearly articulated in the M&E system. Delays in conducting baseline surveys and impact evaluations, and in operationalising the M&E system, are challenges encountered during project implementation (Muller-Praefcke 2010:32).
Detailed knowledge, both across and within sectors, is required, and interactions between planning, budgeting and implementation are key to successful and efficient M&E. Entities developing M&E systems often miss the complexities and needs of the sector context in which the system is to be institutionalised (Kusek & Rist 2004:23 ). M&E is often hampered by separation of powers, legal and regulatory requirements, overlapping information demands from key stakeholders and organisation cultures and capacities (Wholey 2003:9). Inadequate cognisance is taken of the political, organisational, technological and development dimensions, within which the system is to be developed, function and be sustained (World Bank 2008). According to Kennerly & Neely (2002:1243) these dimensions can be drivers of, or barriers to, M&E systems. Literature highlights the necessity of M&E policies reflecting on, amongst others, the function, objectives and roles of rural development M&E. The reality is that there is a lack of sound M&E policies, roles are not defined in the policy and M&E objectives are not internalized (Mebrathu 2002:504).
The elements of the problem as identified above translates into the lack of an integrated approach to M&E of the broader development impact of multi-programmes focusing on rural development, resulting in policy development and analyses not being supported, transparency and accountability on rural spending not enhanced and organisational learning not encouraged.
3. OBJECTIVES OF THE STUDY
The general objective of this empirical study was to develop a model for the integrated and transversal M&E of rural development programmes implemented by departments in the Limpopo Provincial Government in South Africa. The specific objectives of the study can be summarised as follows:
- Identify and study the components and concepts of the model
- Develop the model through:
a) Determining the linkages and interrelationships between components,
b) Defining concepts, and
c) Reflecting processes and flows between components.
4. RESEARCH METHODOLOGY
The research was conducted in three phases. To provide context and substance to the phases the following critical matters were first interrogated: Evolution of M&E in development internationally, Evolution of M&E in South Africa, Context of rural development in South Africa and M&E of rural development in South Africa.
4.1 Phase One: Comparative Case Studies
Through comparative case studies as an empirical enquiry, theory was studied to extract best practices and lessons acquired by other role-players involved in M&E and rural development. Comparative case studies were conducted of the M&E systems of Canada, Chile, Brazil, Uganda and the UNDP. These are a number of countries and an institution deemed to provide evidence of good or promising practice. Specifically Brazil made significant progress in meeting the MDG Goal of reducing poverty by end of 2009, six years ahead of the 2015 deadline (Roche, Burlandy & Maluf 2012: 519).
Qualitative data were obtained through document analysis and desk based survey of existing information from various sources. To guide the document analysis and desk based survey a Framework for the Comparative Case Studies was developed. The contents of the Framework were based on the research questions and the theoretical framework of the study and covered the following concepts: Context of M&E, Foundation of the M&E model, Level of M&E, Organisational framework underpinning the M&E model, Architecture of the M&E model, Role of management information systems (MIS) and geographic information system (GIS) in M&E, and Institutionalising of evaluation. A conceptual exposition was given and a number of key lessons that have emanated from implementing M&E were reflected, discussed and analyzed.
There was found to be a growing body of experience with the institutionalisation of M&E systems internationally.
4.2 Phase Two: Single Case Study
A single case study was also conducted to document the implementation of the GWMES in the Limpopo Provincial Government as a model for M&E of rural development. Qualitative data were obtained through document analysis and desk based survey of existing information from various sources in relation to the subject matter. Data was collected on implementation of the (2005-2008) GMWES model and additional information regarding the 2009 and 2011 enhancements of the model.
4.3 Phase Three: Development of Model
Based on the empirical data collected through the single case study, gaps and limitations were identified with regards to the GWMES as appropriate model for the M&E of rural development. Using the findings of the comparative case studies suggestions were made with regard to changes to the GWMES as model. The changes were tested through interviews with key informants as respondents. The key informants were sampled by applying purposive sampling as sample design and expert sampling as the sampling technique. The sample size was thirty, including M&E specialists, rural development specialist and agriculturalists as key informants. The interviews were conducted face-to face and telephonically guided by a semi-structured interview schedule.
Thereafter the proposed model was developed as stated in the general and specific objectives of the research. The foundation of the model is an integrated and transversal monitoring and evaluation of rural development and the model is based on the interpretation of the qualitative and quantitative data collected through the research process and the resultant findings.
To support the three phases of the research, the data collection method of experience surveying was also utilised. The researcher attended local conferences, workshops and training and other learning opportunities which were used to do experience surveying by interacting with specialists on the research subject matter. Data collected at these forums covered the spectrum of the research.
5. RESULTS AND DISCUSSIONS
5.1 Government Wide Monitoring and Evaluation Model
Prior to 1994 limited information existed on the institutionalisation of M&E systems to monitor progress made with developmental goals. During 2005 the South African Cabinet approved a process to develop an M&E system for use across government, termed the Government Wide Monitoring and Evaluation System (GWMES). Approval was preceded and followed up with the publication by the Presidency of a "Proposal and Implementation Plan for a Government-Wide Monitoring" during 2005 and a "Policy Framework for the Government-wide Monitoring and Evaluation System" during 2007. The GWMES is the model against which M&E is institutionalised in South Africa. The GWMES evolved since its inception in 2005 when in 2009 the Outcomes based approached was introduced and in 2011 when the National Evaluation Policy Framework was developed. Figure 1 reflects the GWMES model in terms of the data terrains, policy frameworks and main stakeholders of the model.
5.2 I mplementation of the GWMES in the Limpopo Provincial Government
The enactment of the GWMES placed an obligation on all nine provinces in South Africa to set up their M&E system. The Limpopo Provincial Government (LPG) has similarly steered the focus towards improving the provincial government's M&E approach with the objective of assessing the impact of its programmes more efficiently and effectively.
However, the findings of the research have indicated that the implementation of the GWMES as a model for M&E in the LPG has not resulted in the integrated and transversal monitoring of rural development. M&E is approached in an ad hoc manner and there are limitations inherent in such an approach. These limitations manifest at different levels where M&E are supposed to be implemented.
The political support for the GWMES is strong, based on various pronouncements on M&E since 2005. However, there are no legislative or regulatory instruments, beyond the Constitution, which enforces the institutionalization of government wide M&E. Some policy documents lack comprehensive consideration of lessons to be learnt from international experience on developing M&E systems, as well as adequate consideration of the context (e.g. provincial) in which the system is to be developed.
The implementation of the GWMES as model has not succeeded in creating M&E practices across the three spheres of government and streamlining and aligning existing reporting structures. Data and information is not shared between the three spheres of government, with the result that different versions of the same data exists and the baseline information utilised is not similar between departments. Data and information are the very life blood of M&E but the quality of these data sets is often unknown and widely variable. There is poor comparability of statistics across isolated producers, and no common quality standards.
At programme level development interventions are not M&E in an integrated and transversal manner and indicator development is problematic. One of the values of indicators is that it can be used to make comparisons across programmes at both provincial and national level. Such comparability requires indicators to be defined and utilised across programmes. However, indicator data is often available in a fragmented and incoherent form and there is a need for such data to be integrated into a coherent and meaningful form, a need which is not being addressed by the GWMES.
At the systems level the extent of reporting requirements and M&E systems in government, especially at provincial level, are not known. There is a tendency to over emphasize the role of information technology in M&E without the resultant benefits being evident. The term "system"" tends not to be understood as an enabler, but rather as an electronic system. The GWMES approach to M&E is silent on the methodology on how to sustain the M&E system.
Currently evaluations are not institutionalised in spite of evaluations being particularly useful in a country such as South Africa where development needs are so great. The National Evaluation Policy Framework attempted to address the challenge of institutionalising of evaluation; however the application of the framework across government will be progressive. In addition the capacity required for evaluations to be undertaken, still needs to be developed.
Capacity building and learning opportunities around M&E are not realized and the shortage of M&E has resulted in failure to systematically record and share lessons which could add value to evidence based decision making.
6. MODEL FOR THE INTEGRATED AND TRANSVERSAL M&E OF RURAL DEVELOPMENT PROGRAMMES IMPLEMENTED BY GOVERNMENT DEPARTMENTS
The model consists of a Strategic, Institutional and Operational dimension, of which the latter dimension is reflected in Figure 2.
The provincial stakeholders (which include the Premier, Legislature and the Members of the Executive Council) require reliable and accurate information on provincial government's performance on rural development to guide evidence based decision making, integrated planning and resource allocation. For the purpose of the article the Operational Dimension is touched on as it is here that the contribution of agricultural extension officers becomes paramount. Extension officers play a role as they function at the coalface of agricultural development and are critical role-players involved in the cornerstones of the Operational Dimension, being indicator development, providers of data, contributors to M&E products and recipients of credible M&E findings.
7. CONCLUSIONS AND RECOMMENDATIONS
Two forward-looking South African policy documents, the NGP and NDP, confirm that agriculture as a sector contributes to rural development in instances such as food security, agrarian reform and rural infrastructure. The agricultural extension and advisory sector has a role to play in the monitoring and evaluation of progress made with agriculture towards the vision that by 2030 South Africa's rural communities must be empowered to participate fully in the socio-economic domain of the country. To strengthen this role it is recommended that the National Policy on Extension and Advisory Services to Agriculture, Forestry and Fisheries that is currently being consulted on by the Department of Agriculture, Forestry and Fisheries, be improved with a Capacity Building Framework. The Framework should include how the monitoring and evaluation skills of extension officers can be enhanced.
The role of Agricultural Colleges in education and training is being solidified by the National Agricultural Training Institute Bill, 2012. The Colleges can play a central role in the enactment of such a Capacity Building Framework in that it manages curricula development and compilation of unit standards, learning programmes and course manuals in line with accreditation bodies' requirements. In the College environment skills development in monitoring and evaluation of agriculture can then benefit from on campus training (lecture room) and off campus site training (practical and hands on).
Such capacity building would also contribute to the Re-skilling and Reorientation Pillar of the Extension Recovery Plan in that it can ensure that officials in the extension and advisory sector continues to contribute to evidence based policy and decision making concerning rural development.
CSAKI, C. 2001, Reaching the rural poor. Development Policy Review, vol. 19 no.4, pp. 563573. [ Links ]
ELLIS, F. & BIGGS S. 2001, Evolving Themes in Rural Development 1950s-2000s. Development Policy Review, vol.19, no.4, pp. 437- 448. [ Links ]
GOLDMAN I., ENGELA R., AKHALWAYA I., GASA N., LEON B., MOHAMED H., & PHILIPS S. 2012. Establishing a National M&E System in South Africa. The World Bank Prem Notes, September 2012, No 21. [ Links ]
INTERNATIONAL FUND FOR AGRICULTURAL DEVELOPMENT (IFAD). 2010. Rural Poverty Report 2011. International Fund for Agricultural Development, Rome. [ Links ]
KENNERLY, M. & NEELY, A. 2002. A framework of the factors affecting the evolution of performance measurement systems. International Journal of Operations and Production Management, vol. 22, no 11, pp. 1222-1245. [ Links ]
KUSEK, J. Z. & RIST C. R. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. The World Bank, Washington D.C. [ Links ]
KUSEK, J. Z, RIST, R. C. & WHITE, E. M. 2005, How will we know the Millennium Development Goals results when we see them? Building a Results-based Monitoring and Evaluation System to give us the answers. Evaluation, 2005, vol. 11, no 1, pp. 7-26. [ Links ]
MACKAY, K. 2007. How to build M&E systems to support better government. The World Bank, Washington DC. [ Links ]
MEBRATHU, E. 2002. Perceptions and practices of monitoring and evaluation: international NGO experiences in Ethiopia. Development in Practice, vol 12, no 3&4, August 2002. [ Links ]
MULLER-PRAEFCKE, D. 2008. The use of monitoring and evaluation in agriculture and rural development projects. FAO\World Bank Cooperative Programme, Washington, D.C. [ Links ]
OECD. 2011. Phase Two evaluation of the Implementation of the Paris Declaration and Accra Agenda for Action in South Africa. Final Country Report February 2011. WYG International Limited. [ Links ]
PUBLIC SERVICE COMMISSION. 2005. Monitoring and evaluation literature review. Arcadia, Pretoria [ Links ]
THE PRESIDENCY. 2005. Proposal and Implementation Plan for a Government-wide Monitoring and Evaluation System. Pretoria: Government Printers. [ Links ]
THE PRESIDENCY. 2007. Policy Framework for the Government-wide Monitoring and Evaluation System. Pretoria: Government Printers. [ Links ]
THE PRESIDENCY. 2009. Medium Term Strategic Framework 2009-2014. Pretoria: Government Printers. [ Links ]
ROCHA, C., BURLANDY, L. & MALUF R. 2012. Small farms and sustainable rural development for food security: The Brazilian experience. Development Southern Africa Vol.29, No. 4, October 2012. (Pp.519-529). [ Links ]
SANGWENI, S. 2004. Monitoring and evaluation as a key instrument in the Renaissance of Africa and good governance, in The 3rd African Evaluation Association Conference, 1 December, Cape Town, South Africa. [ Links ]
TOUWEN, A. 2001. Handbook for projects: Development Management and Fundraising. International Federation of University Women, Geneva, Switzerland. [ Links ]
WHOLEY, J. S. 2003. Using policy evaluation to support decision making and improve program performance. The Japanese Journal of Evaluation Studies, vol. 3, no. 2, pp 6-20. [ Links ]
WORLD BANK. 2003. Reaching the Rural Poor A Renewed Strategy for Rural Development. Washington D.C. [ Links ]
WORLD BANK. 2008. Tracking results in agriculture and rural development in less-than-ideal conditions. A sourcebook of indicators for monitoring and evaluation. Washington D.C. [ Links ]
STATS SA. 2001. Investigation into appropriate definitions of urban and rural areas for South Africa. Discussion document. Statistics South Africa, Pretoria. [ Links ]
1 Extract from a thesis submitted to the University of Pretoria as fulfilment for the degree PhD in Rural Development Planning.