GENERAL ARTICLES

 

Towards a framework to guide the evaluation of inclusive innovation systems

 

 

L. Botha^{I, II}; S. Grobbelaar^{I, *}; W. Bam^I

^IDepartment of Industrial Engineering, University of Stellenbosch, South Africa
^IIDST-NRF CoE in Scientometrics and Science, Technology and Innovation Policy, Stellenbosch University, South Africa

 

 

---

ABSTRACT

Innovation for inclusive development aims not only to produce products and services for 'base of the pyramid' (BoP) markets, but also to improve innovative capacity and empower those that form part of the BoP. Although a wide range of actors aim to support innovation for inclusive development, they lack proper methods to assess the impact that these projects have. This paper sets out to identify an appropriate method or approach for evaluating systems change brought about by innovation for inclusive development projects. Based on a systematic literature review, a process-level approach is proposed. This approach is focused on the components within a system and the changes that occur in the system functions.

---

OPSOMMING

Inklusiewe innovasie beoog nie net om produkte en dienste aan die basis van die piramide mark te verskaf nie, maar ook om hul te bemagtig en kapasiteit vir innovasie te verhoog. Innovasie rolspelers streef daarna om inklusiewe innovasie te bevorder, maar daar is 'n tekort aan voldoende metodes wat gebruik kan word om die impak van inklusiewe innovasie projekte te evalueer. Die doel van die studie is om 'n toepaslike metode vir die evaluasie van stelsel veranderinge, wat deur inklusiewe innovasie projekte meegebring is, te identifiseer. 'n Proses-vlak metode, wat fokus op die komponente in 'n stelsel en die veranderinge wat plaasvind in die stelsel se funksies, was voorgestel op grond van 'n sistematiese literatuurstudie.

---

 

 

1 INTRODUCTION AND PROBLEM STATEMENT

Hammond & Prahalad [1] defined the four billion poorest people who earn $1 or $2 per day as the 'base of the pyramid' (BoP) population. Alleviating poverty is a challenge that various organisations and institutions continuously try to address. Innovation is seen as a key driver of economic growth - it induces the development of products and services for consumers and increases industrial economic development. Although economic growth has played a vital role in decreasing absolute poverty, there are growing concerns that innovation has also contributed to inequality and exclusion. Although a thriving body of knowledge is developing, the economic periphery such as informal microenterprises, affordable goods and services, and innovation to solve wider social challenges, has traditionally been neglected in innovation policy thinking and practice [2].

Several authors have stated that alternative or non-traditional innovation models could be used to stimulate economic inclusion and growth for those in the BoP [2],[3]. The literature highlights two main types of innovation models that can be applied towards these goals. The first type are models where products and services are specifically developed for the benefit of lower income markets. Here the BoP is viewed as a potential customer, and the goal is mainly to decrease costs and produce more affordable products and services. These innovation modes are often referred to as 'appropriate technology' [4], 'frugal innovation', 'pro-poor innovation', and 'BoP innovation' [5]. The second type of innovation model takes a more developmental approach. Examples of this type include 'inclusive innovation', 'innovation for inclusive growth' [6], and 'innovation for inclusive development' [7]. These are terms used to describe innovations that "create or enhance opportunities to improve the wellbeing of those at the BoP" [6]. Here the aim is that BoP individuals be regarded not only as potential customers but also as business partners who are included in the innovation process, so that, as solutions are conceptualised and developed and goods are manufactured, they can benefit economically from such innovations. For the purpose of this paper, the authors use the term 'innovation for inclusive development'.

Linear (predominantly black-box) evaluation methods fail to provide insights into critical institutional lessons about how to improve innovation as a process [8]. Biggs and Underwood [9] identified the need to challenge the current methods for monitoring direct poverty impacts at the project/programme level. They argue that a more holistic approach than that of the status quo can be achieved through the use of the innovation systems (IS) and inclusive innovation systems (IIS) approaches, as these shift the emphasis from a framework of problem-solving to one of learning [9]. Inclusive innovation systems are made up of interconnected components that work together to develop and diffuse innovations aimed at resource-poor individuals or groups, in order to make a positive contribution to improved life conditions and upward mobility [6],[10].

An initial review of the multidisciplinary literature on IIS revealed a lack of methods or approaches to evaluate IIS. [11]. Several authors have emphasised the potential of innovation system concepts for understanding innovation for marginalised communities [12], but as yet there has been little actual primary or field data-based application of this idea [11]. Very often, IIS stakeholders are interested in the impact of innovations for inclusive development. Such impacts need to be determined by means of 'deeper and/or longer-term' evaluation approaches [11], yet the literature lacks both tools for and examples of such evaluations. There is thus a need for more studies to develop such evaluation approaches and to apply such approaches to empirical cases.

Based on these findings from the literature, the IS and IIS approaches were used as a lens through which to view the evaluation of innovation for inclusive development projects. Foster and Heeks [11] argue that the structural representation of IIS needs to be supplemented by a process representation. Such a process level approach to the assessment of the outcomes and impact of inclusive innovation enables the identification of measures that can track institutional changes. This process representation opens the 'black box' of innovation - i.e., looking at inclusions in the innovation process where the inclusive innovation process can be conceptualised to include the following stages: invention, design, development, production, distribution, adoption, and use [13]. In agreement with Heeks et al. [13], we propose that, by opening the 'black box', it is possible to develop a list of impact classification measures that are applicable on both the project- and the systems-level. Using a process focused on both the components within a system and the changes that occur in the system functions will enable the translation of these impacts back to a systems view. We therefore decided to investigate the most appropriate approach by which to open the 'black box' of processes within IIS, thereby paving the way for future empirical studies to develop frameworks for the evaluation of IIS. To do so, the key aim of this systematic review is to review the extant literature to identify methods or approaches that could be used to evaluate IIS.

 

2 METHODOLOGY

In order to avoid creating an ad hoc list of randomly-collected publications, we conducted a systematic literature review. This section describes the methodology followed. The review process followed the established three-phase process used by authors such as Kitchenam and Charters [14] and Fink [15]. The three main stages are planning, conducting the review, and reporting the review. The authors used the first two stages of the methodology, but added a different third stage: assessment of the evaluation categories. Table 1 describes the methodology followed in this paper.

 

3 TOWARDS A FRAMEWORK FOR EVALUATING IIS

3.1 Stage 1: Planning the review

This section focuses on Stage 1 of the methodology, and is structured as follows: (1) Step 1.1 describes the problem statement and rationale for the review; (2) Step 1.2 describes specific search terms used and the databases searched; and (3) Step 1.3 provides an overview of the pre-defined analytical framework according to which the methods or approaches will be evaluated.

3.1.1 Step 1.1: Rationale for our study

An initial review of the IIS literature revealed a lack of methods or approaches to evaluate IIS systematically. There were also very few studies that aimed to determine the success factors and understanding of IIS [11]. In order to address this gap in the literature, the key aim of this article is to review the extant literature in related fields to identify methods or approaches that could be adapted to evaluate the impact of IIS.

3.1.2 Step 1.2: Specify the review

As previously mentioned, we aimed to identify an appropriate method or approach to guide the evaluation of IIS. The review research questions were therefore:

1. Which methods or approaches are used to evaluate IS and/or IIS?

2. How, and in which context, have these methods or approaches been applied?

Step 1.2 also describes the search strategy used to identify primary studies, including the resources and databases searched and the specific search terms that were used. Digital bibliographic databases, reference lists, and conference proceedings were systematically searched for relevant documents.

Because the literature yields very few available studies of the evaluation of IIS, it was decided to cast a wider net and look at the evaluation of traditional IS as well. Studies were identified through searches of the Web of Science, Scopus, Academic Search Premier, Emerald, Science Direct, and ProQuest Research Library databases. Title, abstract, and keyword information was searched using predefined search terms. A list of search terms was constructed, based on the knowledge gained from the initial literature and the field knowledge of the review team. The terms fall into three broad categories: approaches or methods, innovation systems, and the purpose of the paper. These three categories were specifically chosen to ensure that the studies included in the review described the approach or method used to analyse or evaluate the innovation system in question. To ensure that multi-disciplinary studies were included, results were not filtered by discipline. The main search terms are shown in Table 2. Both peer-reviewed and grey literature were included. Only articles in English were considered.

3.1.3 Step 1.3: Analytical framework to analyse methods identified from the review

The evaluation of innovation systems focuses on how well the system functions rather than on the performance of the system. This implies that system evaluations should emphasise the assessment of knowledge creation, the functions (processes) within the system, and the transparency of the system [16].

Rose and Winter [102] emphasise that there is no 'optimal state' according to which innovation systems can be compared, since each is dependent on the context in which it operates. Thus it is important to identify the areas in which the system fails (gap-analysis). Once the area that hinders innovative performance has been identified, the analysis approach can be specified. In order to evaluate IS/IIS, an integrated analysis must be performed that can provide insight into a system's innovation capacity and functioning. To compare and draw insight from the different methods or approaches in the literature, we identified two sets of criteria against which the methods were analysed. The first set assesses each method or approach according to its functionality as an evaluation methodology. These criteria were based on those defined by Schut et al. [8] and by various other authors from the review. To summarise, these criteria are:

1. Methods or approaches should produce mixed data - both qualitative and quantitative. Qualitative data may provide insights into implicit causes and the historical development of constraints to innovation capacity. It also enables the identification of the different dimensions of complex IS problems and structural conditions that induce or block innovative capacity. Quantitative data analysis complements qualitative data by providing descriptive trends and statistical data [8].

2. Methods or approaches should enable both internal and external analysis. Internal analysis is conducted by stakeholders, but is insufficient on its own, given the lack of a broader view. So it should be accompanied by an external analysis conducted by independent researchers [18].

3. Methods or approaches should be applicable to various actors across different levels [8].

4. Methods or approaches should enable an integrated analysis of complex problems, innovative capacity, and the functioning of the system in question [19].

5. Methods or approaches must enable the identification of system failures or 'gaps' in order to understand which part of the system is dysfunctional or hinders effective innovation [17].

The second set of criteria describes the requirements that a method or approach must meet in order to be suitable for the evaluation specifically of IIS. Traditional views of innovation regard development as generalised economic growth. Inclusive innovation, however, "explicitly conceives development in terms of active inclusion of those who are excluded from the mainstream of development" [20]. Differing in its foundational view of development, inclusive innovation therefore refers to the inclusion, within some aspect of innovation, of groups who are currently marginalised [2]. Given these definitions, we deduced that a method or approach to evaluate the impact of IIS will have to be actor-orientated and focus on the complex relationships between the actors that contribute to the emergence of inclusive innovations. Van der Hilst [16] noted that, for the purpose of identifying opportunities for improved interventions, an approach should be used that enables rich qualitative data to be gathered and retained. Several authors have noted that IIS aims constantly to develop and diffuse knowledge and skills; so an evaluation method should be able to analyse processes or functions within the system [21], [22]. Finally, although some empirical studies evaluate Is, these are focused on empirical results from developed countries [23]. It remains uncertain, therefore, how suitable these approaches are for the evaluation of IIS of less-developed countries. Thus examples were sought where each approach was applied to the IS of a less-developed country whose demographics include people who can be regarded as part of the BoP population. For the purpose of this study, the criteria that a method or approach must meet in order to be suitable for the evaluation of IIS are summarised below:

1. Be actor-orientated - i.e., the method must be able to identify actors within the systems, their roles, and 'who' from the excluded group is to be included. IIS consists of demand-side actors, supply-side actors, and a rich network of innovation intermediaries; and the method must enable the identification of each group [13].

2. Focus on the complex relations between the actors - i.e., the method or approach must be able to identify the "informal, loose but socialised relations" that are present in IIS [11]. It must focus on how these relationships foster learning and capability transfer, diffusion, and retention to the BoP [13], [24].

3. The method or approach must be able to take the context of the system into consideration - it therefore must be used in such a way that rich qualitative data is gathered and retained [16].

4. The method or approach must also be able to analyse the processes within the system [21], [22].

3.2 Stage 2: Conducting the review

The 117 studies included in our review were individually coded by the authors. This was done in order to standardise the information contained in the studies and to help to identify trends in the literature. The main aspects of the articles that were coded were:

1. The type of paper and the publication type. This was extracted in order to identify the target audience of the study in question, and to gain broad insights into the dimensions of the studies.

2. The method or approach used in the article. As previously mentioned, the main objective of this review was to identify methods and approaches used to analyse or evaluate innovation systems.

3. The type of innovation system that was analysed or evaluated, and its context or discipline. This was extracted in order to connect the method or approach used with the type of innovation system to which it was applied, in order to identify patterns.

4. The goal of the analysis or evaluation. This again served to provide insights into the types of methods or approaches that are used for specific goals.

3.2.1 Step 2.1 and 2.2: Identification of research and selection of primary studies

The first part of Step 2.1 begins with descriptive statistics of the 117 studies reviewed, including the type of publications in which the studies were published and the chronology of the studies. The second part identifies the main methods or approaches used in the studies.

The total yield of the combined searches, using the databases and search terms described in Stage 1, once duplicates had been removed, was 247 publications. Publications were discarded if they were written in a foreign language. In Step 2.2 the authors reviewed the titles and abstracts, and discarded publications that either were not related to the evaluation of IS/IIS or just mentioned it in passing. This resulted in 117 studies. Table 3 below shows the different types of publications from which the 117 included studies were collected.

 

 

Figure 1 is a timeline that shows the number of studies (included in this review) that were published per year, in chronological order. From this figure it is clear that there has been an increase in studies on IS since about 2004, and it also depicts the relatively small number of emerging studies in the area of IIS.

3.2.2 Step 2.3: Categories of evaluation methods

The 117 papers were analysed with the use of Atlas.ti software. This enabled the authors to code different methods or approaches. The first iteration served as a scoping opportunity. The identification of methods or approaches was an iterative process that was conducted in a 'learning-by-doing' manner. It became evident that many approaches are based on the same fundamental characteristics. All the methodologies described in the review were categorised into nine main methods or approaches. These methods or approaches are briefly described below:

(1) Macro-economic level approach (Comparative approach): The innovation system performance of geographically different innovation systems is compared in order to guide policy development. It considers the stages of development and the evolution of the national innovation systems, and distinguishes between policies as either supply-side or demand-side in order to create technological dynamism within the country by means of a comparative perspective between two or more countries. It is a comparative framework used to evaluate innovation policy. The framework starts by identifying activities across sectors where a country could build comparative advantages. Next, the innovation chains consisting of both technical and economic interfaces are identified. This evaluation is then used to generate new 'innovation policy' that aims to strengthen the techno-economic network as a whole, rather than supporting specific activities [25].

(2) Regime (macro, meso, micro) approach: This approach divides the NIS into three levels: macro, micro, and meso. The IS actors and functions are categorised into these levels, and evaluated based on the requirements of each level [26], [27].

(3) Triple-helix model: This model focuses on three main actors within the IS: academy, industry, and government [89].

(4) Systems dynamics (SD) approach: System dynamics aims to describe the system through both qualitative and quantitative models. This method entails the construction of causal loop diagrams that describe the causal assumptions of the system in question, and often also include quantitative modelling through a stock and low diagrams. The key concept behind these diagrams is to conceptualise complex behaviours in systems and to model the expected behaviour of systems due to their non-linear nature. These models capture feedback loops and causal thinking, and are hugely useful in improving understanding of the nonlinear behaviours of such systems [28].

(5) Component-based (structural) approach: The structural approach entails the identification of system elements such as all the actors, institutions and organisations within the system, and understanding the relations between these elements [50].

(6) Function-based approach (also called TIS approach): The functional approach was identified as one of those most often used in the literature that forms part of this review. This approach involves the analysis of the system by means of certain activities commonly referred to as 'functions'. Different authors orientate the approach to different focuses, and integrate it with other approaches [21], [29].

(7) Component-function-based approach: This approach is an integration of the component-based and functional approaches. Lamprinopoulou et al. [30] note the argument in the literature that, since structures and functions are mutually dependent, it is better to follow an integrated approach where the component-based and functional approaches are used together.

(8) Data envelopment analysis (DEA): It is one of the most mature efficiency evaluations. This method uses mathematical techniques that can deploy a range of variables and constraints to evaluate the relative efficiency of decision-making units (DMUs). The advantage of the DEA method is that the analysis of the effect of an innovation system on the system outputs can provide managers with useful decision information; and DEA is very objective [31].

(9) Analytical hierarchy process (AHP) method: This method first entails the construction of evaluation indexes for the overall goals and sub-objectives of a specific programme or system, and then it is used to determine the weight of these indexes. This method is an effective way of simplifying complexity in order to determine the relative importance of the objectives of each process within a system [32]..

Table 4 shows which studies applied each of the above-mentioned approaches; and Figure 2 shows the frequency of studies that applied each method or approach.

3.3 Stage 3: Criteria assessment of evaluation categories

To compare and gain insight into the different methods or approaches in the literature, we identified two sets of criteria according to which the methods were analysed. This section comprises the analysis of the main methods or approaches identified from the literature. A framework for analysing the methods was defined in Step 1.3 of Stage 1.

3.3.1 Step 3.1: Evaluation of each of the identified methods or approaches

Appreciating that no single optimal evaluation method applies to all evaluations, the authors constructed a database of different approaches, the context in which they were applied, and the purpose of each. To be able to compare the functioning and the applicability of each of the methods or approaches to the current study, each was assessed based on whether it meets the criteria for effective evaluation, mentioned in Section 3. The result of this evaluation, and the applicability of each method or approach to the evaluation of IIS, is shown in Table 5, with examples of the application.

3.3.2 Step

3.2: Selection of the component-function-based approach as the most appropriate approach.

The analysis in Table 5 revealed that the approaches meeting most of the pre-defined criteria are the functional approach and the component-function-based approach. Each of these is applicable to various actors, and both use functions of innovation to analyse complex problems, innovative capacity, and the functioning of a system. Both identify system failures by identifying problems that hamper the effective functioning of the system - i.e., that hamper innovation in the system under study. The analysis in Table 5 was also used to determine which of these two approaches would be most applicable to the evaluation of IIS. As previously mentioned, we determined that such an approach should (1) be actor-oriented, (2) analyse complex relationships between actors; (3) use rich qualitative data; and (4) conduct a process-focused analysis.

The study conducted by Van der Hilst [16] suggests that, if the functions are slightly altered, they can apply to the evaluation of IIS. The component-function-based approach meets all four requirements. The combination of the component- and function-based approach ensures that it is actor-orientated and seeks to identify complex relations between actors. The approach does require rich qualitative data to be gathered in order to identify how well system functions are performed. The system processes are analysed by means of system functions. Finally, the review identified a study in which the component-function-based approach was successfully used to analyse an IIS. Van der Hilst [16] developed a tool based on the component-function-based approach to determine the performance of innovation intermediaries within the agricultural IIS of Vietnam. It was developed by altering the method proposed by Wieczorek and Hekkert [114] to be suitable for the evaluation of IIS.

Adopting the component-function-based approach described by Van der Hilst [16], in combination with a process view of innovation suggested by Heeks et al. [11], we have constructed the broad outlines of a framework to guide evaluation. This is an advantageous combination, as the process view provides structure (invention, design, development, production, distribution, adoption, and use), and the component-function-based approach enables the evaluation of the functioning of the process. The broad outlines for constructing such a framework are shown in Figure 3.

 

4 CONCLUSIONS AND RECCOMMENDATIONS FOR FUTURE STUDIES

The goal of this article was to identify a method or approach that could serve as a basis on which to develop a framework to evaluate IIS. To that end the authors conducted a review of the extant IS evaluation literature. This review identified nine methods that are most frequently used throughout the literature. Each method was evaluated according to pre-defined criteria. The results described in Step 3.2 show that the component-function-based approach is the most appropriate one from which to develop an evaluation framework. It is an actor-oriented approach that enables the identification of the different actors within the system - demand-side, supply-side, and intermediaries. It focuses on the complex relations between actors in order to evaluate knowledge and capability transfer. The combined structural and functional approach is best for the evaluation of IIS, as the functions of a system can only be improved by altering one or more components. Every function is evaluated from the perspective of the components [16].

In the course of this review we have found that more research is needed into aspects of the component-function-based approach, such as indicators, how to assess functionality, and impact classification measures. We propose that further studies apply a systematic learning process in order to improve our understanding of the practical application of such an approach. The component-function-based approach needs to be applied using primary data in order to start revising this approach and constructing practical evaluation frameworks. This could help unpack the component-function-based approach in order to develop a list of impact classification measures that are applicable on both the project- and the systems-level. Previous research shows that, by opening the 'black box' - i.e., looking at inclusions in the process - it is possible to develop such a list [11]. For this we have begun to move towards the construction of a broad framework for understanding and evaluating innovation for inclusive development projects (Figure 3).

As is the case with any study, this paper has limitations. Although the predefined systematic literature review methodology reduces bias in the review process, it only provides specific answers to specific questions [116], [117]. The review only included studies that are written in English, thus excluding several studies. The search was also restricted to studies that specifically refer to the search terms listed in Table 2. While this approach was consistent with the goal of identifying a suitable method or approach, it also excluded several research streams that attempt to evaluate IIS.

 

REFERENCES

[1] C. K. Prahalad and A. Hammond, "Serving the world's poor, profitably," Harv. Bus. Rev., vol. 80, no. 9, pp. 4-11, 2002.         [ Links ]

[2] C. Foster and R. Heeks, "Analyzing policy for inclusive innovation: the mobile sector and base-of-the-pyramid markets in Kenya," Innov. Dev., vol. 3, pp. 103-119, 2013.         [ Links ]

[3] C. K. Prahalad, "Bottom of the Pyramid as a Source of Breakthrough Innovations," J. Prod. Innov. Manag., vol. 29, no. 1, pp. 6-12, Jan. 2012.         [ Links ]

[4] R. Kaplinsky and M. Morris, A Handbook for Value Chain Research. Ottawa: International Development Research Centre, 2001.         [ Links ]

[5] S. E. Cozzens and R. Kaplinsky, "Innovation, poverty and inequality: cause, coincidence, or co-evolution?," in Handbook of Innovation Systems and Developing Countries, B.-Å. Lundvall, K. J. Joseph, C. Chaminade, and J. Vang, Eds. Massachussetts: Edward Elgar Publishing, Inc., 2009, pp. 57 - 82.         [ Links ]

[6] G. George, a. M. A. M. Mcgahan, and J. Prabhu, "Innovation for Inclusive Growth: Towards a Theoretical Framework and a Research Agenda," J. Manag. Stud., vol. 49, no. 4, pp. 661-683, 2012.         [ Links ]

[7] C. Paunov, "Innovation and Inclusive Development," OECD Publ., p. 67, 2013.         [ Links ]

[8] M. Schut, L. Klerkx, J. Rodenburg, J. Kayeke, M. Raboanarielina, P. Y. Adegbola, A. Van Ast, L. Bastiaans, and L. C. Hinnou, "RAAIS: Rapid Appraisal of Agricultural Innovation Systems (Part I). A diagnostic tool for integrated analysis of complex problems and innovation capacity," Agric. Syst., vol. 132, pp. 1-11, Oct. 2014.         [ Links ]

[9] S. D. Biggs and M. P. Underwood, "Review of Crop Post-Harvest programme," 2001.         [ Links ]

[10] W. J. Blanchard, Benjamin S. Fabrycky, "Systems Engineering andAnalysis," UpperSaddle River, NJ Prentice-Hall. Probl. solving a prelude to Quant. Anal. IEEE Trans. Syst. man Cybern. , vol. 23, pp. 746765, 1998.         [ Links ]

[11] C. Foster and R. Heeks, "Conceptualising Inclusive Innovation: Modifying Systems of Innovation Frameworks to Understand Diffusion of New Technology to Low-Income Consumers," Eur. J. Dev. Res., vol. 25, no. 3, pp. 333-355, 2013.         [ Links ]

[12] E. Kraemer-mbula and W. Wamae, "Adapting the Innovation Systems Framework to Sub-Saharan Africa," Innov. Dev. agenda, pp. 65-90, 2010.         [ Links ]

[13] R. Heeks, M. Amalia, R. Kintu, and N. Shah, "Inclusive Innovation: Definition , Conceptualisation and Future," Centre for Development Informatics, 2013.         [ Links ]

[14] B. Kitchenham and S. Charters, "Guidelines for performing Systematic Literature Reviews in Software Engineering," Engineering, vol. 2, p. 1051, 2007.         [ Links ]

[15] A. Fink, Conducting Research Literature Reviews. From the Internet to Paper. Sage Publication, Inc., 2005.         [ Links ]

[16] B. Van der Hilst, "How innovation intermediaries can strengthen the innovation system: a case study of Vietnam," Thesis Attain. Master degree Sci. Innov. Manag. Univ. Utrecht., no. August, 2012.         [ Links ]

[17] J. Rose and K. Winter, "A gap analysis of the South African innovation system for water," Water SA, vol. 41, no. 3, p. 406, Apr. 2015.         [ Links ]

[18] B. Van Mierlo and M. Arkesteijn, Transitions. The Netherlands: Wageningen Academic Publishers, 2009.         [ Links ]

[19] World Bank, Agricultural Innovation Systems: An Investment Sourcebook. 2012.         [ Links ]

[20] R. Heeks, C. Foster, and Y. Nugroho, "New models of inclusive innovation for development," Innov. Dev., vol. 4, no. 2, pp. 175-185, 2014.         [ Links ]

[21] A. Bergek, S. Jacobsson, B. Carlsson, S. Lindmark, and A. Rickne, "Analyzing the functional dynamics of technological innovation systems: A scheme of analysis," Res. Policy, vol. 37, no. 3, p. 407, Apr. 2008.         [ Links ]

[22] P. G. Foster-Fishman, B. Nowell, and H. Yang, "Putting the system back into systems change: A framework for understanding and changing organizational and community systems," Am. J. Community Psychol., vol. 39, no. 3-4, pp. 197-215, 2007.         [ Links ]

[23] M. P. Hekkert, R. A. A. Suurs, S. O. Negro, S. Kuhlmann, and R. E. H. M. Smits, "Functions of innovation systems: A new approach for analysing technological change," Technol. Forecast. Soc. Change, vol. 74, no. 4, pp. 413-432, May 2007.         [ Links ]

[24] S. Ansari, K. Munir, and T. Gregg, "Impact at the 'Bottom of the Pyramid': The Role of Social Capital in Capability Development and Community Empowerment," J. Manag. Stud., vol. 49, no. 4, pp. 813-842, 2012.         [ Links ]

[25] A. Aggarwal, "Technology policies and acquisition of technological capabilities in the industrial sector: A comparative analysis of the Indian and Korean experiences," Sci. Technol. Soc., vol. 6, no. 2, pp. 255-304, 2001.         [ Links ]

[26] A. R. Asa, N. S. Prasad, and M. M. Htay, "Evaluation of National Innovation System in Developing Economies: A Namibian Perspective," IOSR J. Econ. Financ., vol. 1, no. 2, pp. 8-12, 2013.         [ Links ]

[27] P. Van der Duin and H. Hermeler, "Innovating in a Government Context: An Evaluation of a Dutch Water Innovation Program Using the Cyclic Innovation Model," Int. J. Innov. Technol. Manag., vol. 11, no. 03, p. 1440008, Jun. 2014.         [ Links ]

[28] A. Muhammad Iqbal, A. Shahid Khan, F. Bashir, and A. Amat Senin, "Evaluating National Innovation System of Malaysia Based on University-industry Research Collaboration: A System Thinking Approach," Asian Soc. Sci. , vol. 11, no. 13, pp. 45-60, 2015.         [ Links ]

[29] M. Alam Hossain Mondal, L. M. Kamp, and N. I. Pachova, "Drivers, barriers, and strategies for implementation of renewable energy technologies in rural areas in Bangladesh-An innovation system analysis," Energy Policy, vol. 38, no. 8, pp. 4626-4634, Aug. 2010.         [ Links ]

[30] C. Lamprinopoulou, A. Renwick, L. Klerkx, F. Hermans, and D. Roep, "Application of an integrated systemic framework for analysing agricultural innovation systems and informing innovation policies: Comparing the Dutch and Scottish agrifood sectors," Agric. Syst., vol. 129, pp. 40-54, 2014.         [ Links ]

[31] S. Guan and W. Zhu, "Efficiency Evaluation on Knowledge Creation in Regions of China," in 2009 International Conference on Management and Service Science, 2009, no. 200702, pp. 1-4.         [ Links ]

[32] Z. Chen and R. Xiao-hong, "Research on the Innovative Capabily Evaluation of Innovative Enterprises: Taking National Innovative (Pilot) Enterprises in Anhui Province as Examples," in 2011 International Conference on Information Management, Innovation Management and Industrial Engineering, 2011, vol. 3, pp. 173-176.         [ Links ]

[33] P.-L. Chang and H.-Y. Shih, "The innovation systems of Taiwan and China: a comparative analysis," Technovation, vol. 24, no. 7, pp. 529-539, Jul. 2004.         [ Links ]

[34] Y.-W. Chuang, L.-C. Lee, W.-C. Hung, and P.-H. Lin, "Forging Into the Innovation Lead - a Comparative Analysis of Scientific Capacity," Int. J. Innov. Manag., vol. 14, no. 03, pp. 511-529, 2010.         [ Links ]

[35] K.-J. Lundquist and M. Trippl, "Distance, Proximity and Types of Cross-border Innovation Systems: A Conceptual Analysis," Reg. Stud., vol. 47, no. 3, pp. 450-460, Mar. 2013.         [ Links ]

[36] B. M. Rodriguez and M. V. M. José, "The region's intellectual capital benchmarking system: enabling economic growth through evaluation," J. Knowl. Manag., vol. 10, no. 5, pp. 41-54, 2006.         [ Links ]

[37] K. Miyazaki and N. Islam, "Nanotechnology systems of innovation-An analysis of industry and academia research activities," Technovation, vol. 27, no. 11, pp. 661-675, Nov. 2007.         [ Links ]

[38] X. Xuguang, Z. Zaixu, and Z. Jing, "Based on the analysis of foreign oil company the construction of Sinopec technology innovation system," in 2011 International Conference on Product Innovation Management (ICPIM 2011), 2011, pp. 574-578.         [ Links ]

[39] D. Yuri, S. Irina, and R. Zivota, "National Innovation Systems: The Fundemental Approaches to Definition and Evaluation," Int. J. Econ. Law, vol. 1, no. 2, pp. 23-30, 2011.         [ Links ]

[40] N. U. Blum, C. R. Bening, and T. S. Schmidt, "An analysis of remote electric mini-grids in Laos using the Technological Innovation Systems approach," Technol. Forecast. Soc. Change, vol. 95, pp. 218-233, Feb. 2015.         [ Links ]

[41] S. Breukers, M. Hisschemöller, E. Cuppen, R. Suurs, and M. Hisschemoller, "Analysing the past and exploring the future of sustainable biomass. Participatory stakeholder dialogue and technological innovation systems research," Technol. Forecast. Soc. Change, vol. 81, no. 1, pp. 227-235, Jan. 2014.         [ Links ]

[42] L. M. Kamp, "Socio-technical analysis of the introduction of wind power in the Netherlands and Denmark," Int. J. Environ. Technol. Manag., vol. 9, no. 2/3, p. 276, 2008.         [ Links ]

[43] J. Köhler, W. Schade, G. Leduc, T. Wiesenthal, B. Schade, and L. Tercero Espinoza, "Leaving fossil fuels behind? An innovation system analysis of low carbon cars," J. Clean. Prod., vol. 48, pp. 176-186, Jun. 2013.         [ Links ]

[44] S. Li, "Analysis of the complex system of technological innovation based on IDFE0 method," in 2009 International Conference on Management Science and Engineering, 2009, no. 07, pp. 1602-1607.         [ Links ]

[45] A. Mohtarami, S. H. K. Hosseini, and H. Kandjani, "Rethinking the national innovation system functions based on viable system model: A theoretical discussion and a comparative analysis," Middle East J. Sci. Res., vol. 16, no. 10, pp. 1383-1392, 2013.         [ Links ]

[46] B. Praetorius, M. Martiskainen, R. Sauter, and J. Watson, "Technological innovation systems for microgeneration in the UK and Germany - a functional analysis," Technol. Anal. Strateg. Manag., vol. 22, no. 6, pp. 745-764, Aug. 2010.         [ Links ]

[47] G. Southon, "IT, change and evaluation: an overview of the role of evaluation in health services.," Int. J. Med. Inform., vol. 56, no. 1-3, pp. 125-33, 1999.         [ Links ]

[48] K. van Alphen, J. van Ruijven, S. Kasa, M. Hekkert, and W. Turkenburg, "The performance of the Norwegian carbon dioxide, capture and storage innovation system," Energy Policy, vol. 37, no. 1, pp. 4355, 2009.         [ Links ]

[49] X. J. Lai, "Actors-Network Analysis on Carbon Capture and Storage Technological Innovation System in China and the U.S," Appl. Mech. Mater., vol. 248, pp. 331-336, Dec. 2012.         [ Links ]

[50] M. P. Hekkert and S. O. Negro, "Understanding technological change: explanation of different perspectives on innovation and technological change," 2011.         [ Links ]

[51] World Bank, "Public Disclosure Authorized NICARAGUA: RESPONDING TO THE NEEDS OF FARMERS USING AGRICULTURAL TECHNOLOGY ," 2001.         [ Links ]

[52] E. Autio, "Evaluation of RTD in regional systems of innovation," Eur. Plan. Stud., vol. 6, no. 2, pp. 131140, 1998.         [ Links ]

[53] M. Bellandi and A. Caloffi, "An Analysis of Regional Policies Promoting Networks for Innovation," Eur. Plan. Stud., vol. 18, no. 1, pp. 67-82, Jan. 2010.         [ Links ]

[54] M. Busse, W. Schwerdtner, R. Siebert, A. Doernberg, A. Kuntosch, B. König, and W. Bokelmann, "Analysis of animal monitoring technologies in Germany from an innovation system perspective," Agric. Syst. , vol. 138, no. August, pp. 55-65, Sep. 2015.         [ Links ]

[55] P.-L. Chang and H.-Y. Shih, "Comparing patterns of intersectoral innovation diffusion in Taiwan and China: A network analysis," Technovation, vol. 25, no. 2, pp. 155-169, Feb. 2005.         [ Links ]

[56] V. Bikar, H. Capron, M. Cincera, V. Bikar, and H. Capron, "An integrated evaluation scheme of innovation systems from an institutional perspective," IDEAS Work. Pap. Ser. from RePEc, no. August 2015, 2006.         [ Links ]

[57] U. Seidel, L. Müller, G. Meier zu Köcker, and G. De Araújo Filho, "A new approach for analysing national innovation systems in emerging and developing countries," Ind. High. Educ., vol. 27, no. 4, pp. 279-285, 2013.         [ Links ]

[58] I. Feller, "Mapping the frontiers of evaluation of public-sector R{&}D programs," Sci. Public Policy, vol. 34, no. 10, pp. 681-690, Dec. 2007.         [ Links ]

[59] B. B. Fischer and J. M. Zayas, "Towards a Taxonomy of Firms Engaged in International R&D Networks: an Evaluation of the Spanish Participation in Eureka," J. Technol. Manag. Innov., vol. 7, no. 3, pp. 121-134, 2012.         [ Links ]

[60] W. Fu and C. Han, "Evaluation system for the government's role of regional independent innovation system," in The 2nd International Conference on Information Science and Engineering, 2010, pp. 2840 2843.         [ Links ]

[61] S.-C. Hung, "Institutions and systems of innovation: an empirical analysis of Taiwan's personal computer competitiveness," Technol. Soc., vol. 22, no. 2, pp. 175-187, Apr. 2000.         [ Links ]

[62] G. Intxaurburu and J. Olaskoaga, "The evolving role of the university in Basque technology policy: A system of innovation analysis," Eur. Plan. Stud., vol. 7, no. 6, pp. 759-774, 1999.         [ Links ]

[63] B. König, A. Kuntosch, W. Bokelmann, A. Doernberg, W. Schwerdtner, M. Busse, R. Siebert, K. Koschatzky, and T. Stahlecker, "Analysing agricultural innovation systems: a multilevel mixed methods approach," IDEAS Work. Pap. Ser. from RePEc, no. C, pp. 270-280, 2012.         [ Links ]

[64] B.-Á. Lundvall and J. L. Christensen, "Extending and Deepening the Analysis of Innovation Systems - with Empirical Illustrations from the DISKO Project," DRUID's Summer Conf. Natl. Innov. Syst. Ind. Dyn. Innov. Policy, no. Johnson 1988, 1999.         [ Links ]

[65] E. Millstone, P. Van Zwanenberg, and F. Marshall, "Monitoring and Evaluating Agricultural Science and Technology Projects: Theories, Practices and Problems," IDS Bull., vol. 41, no. 6, pp. 75-87, Nov. 2010.         [ Links ]

[66] M. Russo and F. Rossi, "Cooperation Networks and Innovation: A Complex Systems Perspective to the Analysis and Evaluation of a Regional Innovation Policy Programme," Evaluation, vol. 15, no. 1, pp. 75-99, 2009.         [ Links ]

[67] J. Svarc, J. Perkovic, and J. Laznjak, "Unintended consequences of innovation policy programmes: Social evaluation of technological projects programme in Croatia," Innov. Manag. Policy Pract., vol. 13, no. 1, pp. 77-94, 2011.         [ Links ]

[68] T. Temel, W. Janssen, and F. Karimov, "Systems analysis by graph theoretical techniques: assessment of the agricultural innovation system of Azerbaijan," Agric. Syst., vol. 77, no. 2, pp. 91-116, 2003.         [ Links ]

[69] B. van Mierlo, M. Arkesteijn, and C. Leeuwis, "Enhancing the Reflexivity of System Innovation Projects With System Analyses," Am. J. Eval., vol. 31, no. 2, pp. 143-161, May 2010.         [ Links ]

[70] A. Wang and S. Li, "Analysis of technology innovation diffusion effect in regional innovation system by unascertained measurement," in 2009 International Conference on Management Science and Engineering, 2009, pp. 1873-1878.         [ Links ]

[71] S. L. L. L. Zhao, L. Cacciolatti, S. H. H. H. Lee, and W. Song, "Regional collaborations and indigenous innovation capabilities in China: A multivariate method for the analysis of regional innovation systems," Technol. Forecast. Soc. Change, vol. 94, pp. 202-220, 2015.         [ Links ]

[72] OECD, "Regional Innovation Strategies and Foresight," in OECD Innovation Policy Handbook, no. November, OECD, 2010.         [ Links ]

[73] T. de Noronha Vaz, P. V. Galindo, E. de Noronha Vaz, and P. Nijkamp, "Innovative firms behind the regions: Analysis of regional innovation performance in Portugal by external logistic biplots," Eur. Urban Reg. Stud. , vol. 22, no. 3, pp. 329-344, Apr. 2013.         [ Links ]

[74] M. A. Diez, "The Evaluation of Regional Innovation and Cluster Policies: Towards a Participatory Approach," Eur. Plan. Stud., vol. 9, no. 7, pp. 907-923, Oct. 2001.         [ Links ]

[75] T. Foran, J. R. A. A. Butler, L. J. Williams, W. J. Wanjura, A. Hall, L. Carter, and P. S. Carberry, "Taking Complexity in Food Systems Seriously: An Interdisciplinary Analysis," World Dev., vol. 61, pp. 85-101, Sep. 2014.         [ Links ]

[76] D. Horton and R. Mackay, "Using evaluation to enhance institutional learning and change: recent experiences with agricultural research and development," Agric. Syst., vol. 78, no. 2, pp. 127-142, Nov. 2003.         [ Links ]

[77] J. Markard and B. Truffer, "Actor-oriented analysis of innovation systems: exploring micro-meso level linkages in the case of stationary fuel cells," Technol. Anal. Strateg. Manag., vol. 20, no. 4, pp. 443-464, Jul. 2008.         [ Links ]

[78] K. Matatkova and J. Stejkal, "DESCRIPTIVE ANALYSIS OF THE REGIONAL INNOVATION SYSTEM - NOVEL METHOD FOR PUBLIC Katerina MATATKOVA Jan STEJSKAL," Transylvanian Rev. Adm. Sci., vol. 39, no. E, pp. 91-107, 2013.         [ Links ]

[79] A. Mostafavi, D. M. Abraham, D. Delaurentis, and J. Sinfield, "Exploring the Dimensions of Systems of Innovation Analysis: A System of Systems Framework," IEEE Syst. J., vol. 5, no. 2, pp. 256-265, Jun. 2011.         [ Links ]

[80] E. Rametsteiner and G. Weiss, "Assessing policies from a systems perspecitve - Experiences with applied innovation systems analysis and implications for policy evaluation," For. Policy Econ., vol. 8, no. 5, pp. 564-576, Jul. 2006.         [ Links ]

[81] F. Randelli and B. Rocchi, "Analysing the role of consumers within Technological Innovation Systems towards sustainability: the case of Alternative Food Networks," IDEAS Working Paper Series from RePEc. Federal Reserve Bank of St Louis, St. Louis, 2015.         [ Links ]

[82] C. Edquist, "The Systems of Innovation Approach and Innovation Policy: An account of the state of the art," DRUID Conf. Aalborg, pp. 12-15, 2001.         [ Links ]

[83] A. Hall and K. Dorai, "Thematic Note 5: Evaluating Agricultural Innovation System Interventions," France, 2012.         [ Links ]

[84] World Bank, Enhancing agricultural innovation. 2007.         [ Links ]

[85] World Bank, "Turkey National Innovation and Technology System," 2009.         [ Links ]

[86] Z. Bajmócy, M. Lukovics, and Z. Vas, "A subregional analysis of universities' contribution to economic and innovation performance," Transit. Stud. Rev., vol. 17, no. 1, pp. 134-150, 2010.         [ Links ]

[87] Y. S. Duman, "Regional Innovation Policy: An Analysis of Turkey's Aegean, Marmara, East Anatolia and Southeast Anatolia Regions," Altern. Turkish J. Int. Relations, vol. 10, no. 1, pp. 37-57, 2011.         [ Links ]

[88] R. Naghizadeh, S. Elahi, M. Manteghi, S. Ghazinoory, and M. Ranga, "Through the magnifying glass: an analysis of regional innovation models based on co-word and meta-synthesis methods," Qual. Quant., vol. 49, no. 6, pp. 2481-2505, Nov. 2014.         [ Links ]

[89] M. Schut, J. Rodenburg, L. Klerkx, J. Kayeke, A. van Ast, and L. Bastiaans, "RAAIS: Rapid Appraisal of Agricultural Innovation Systems (Part II). Integrated analysis of parasitic weed problems in rice in Tanzania," Agric. Syst., vol. 132, no. August, pp. 12-24, Jan. 2015.         [ Links ]

[90] Y. Wang, W. Vanhaverbeke, and N. Roijakkers, "Exploring the impact of open innovation on national systems of innovation - A theoretical analysis," Technol. Forecast. Soc. Change, vol. 79, no. 3, pp. 419428, Mar. 2012.         [ Links ]

[91] J. Kim, S. J. Bae, and J.-S. Yang, "Government roles in evaluation and arrangement of R&D consortia," Technol. Forecast. Soc. Change, vol. 88, pp. 202-215, Oct. 2014.         [ Links ]

[92] B. Pérez-Astray and N. C. Babío, "Analysis of the interface systems as mediating agents in university/industry relations. Proposal of the 'relationship promoter' as a strategic role in the R&D transference," Eur. Res. Stud. J., vol. 14, no. 1, pp. 55-74, 2011.         [ Links ]

[93] K. Amankwah, L. Klerkx, S. J. Oosting, O. Sakyi-Dawson, A. J. van der Zijpp, and D. Millar, "Diagnosing constraints to market participation of small ruminant producers in northern Ghana: An innovation systems analysis," NJAS - Wageningen J. Life Sci., vol. 60-63, pp. 37-47, Dec. 2012.         [ Links ]

[94] U. Cantner and H. Graf, "The Network of Innovators in Jena: An Application of Social Network Analysis," IDEAS Working Paper Series from RePEc. Federal Reserve Bank of St Louis, St. Louis, 2004.         [ Links ]

[95] F. Castellacci and J. M. Natera, "The dynamics of national innovation systems: A panel cointegration analysis of the coevolution between innovative capability and absorptive capacity," Res. Policy, vol. 42, no. 3, pp. 579-594, 2013.         [ Links ]

[96] C. W. W. Choi, J. S. S. Shin, B. G. G. Yoon, W. Y. Y. Lee, and Y. T. T. Park, "On the linkage between industries and technologies: Patent citation analysis," in IEEE International Engineering Management Conference, 2004, vol. 2, pp. 576-580.         [ Links ]

[97] A. M. Iqbal, A. S. Khan, F. Bashir, A. A. Senin, A. Muhammad Iqbal, A. Shahid Khan, F. Bashir, and A. Amat Senin, "Evaluating National Innovation System of Malaysia Based on University-industry Research Collaboration: A System Thinking Approach," Asian Soc. Sci., vol. 11, no. 13, pp. 45-60, Jun. 2015.         [ Links ]

[98] S. Krätke, "OF KNOWLEDGE RESOURCES," vol. 17, no. 1, pp. 83-97, 2010.         [ Links ]

[99] L. Jun, "A Dynamic Analysis of Triple Helix of Industry-University-Research Institution: The Case of China," in 2008 4th International Conference on Wireless Communications, Networking and Mobile Computing, 2008, pp. 1-6.         [ Links ]

[100] T. L. Lee, "An alternative approach to technology policy assessment: dynamic simulation analysis of Taiwan's IC industry," Int. J. Technol. Policy Manag., vol. 6, no. 2, p. 121, 2006.         [ Links ]

[101] E. Samara, P. Georgiadis, and I. Bakouros, "The impact of innovation policies on the performance of national innovation systems: A system dynamics analysis," Technovation, vol. 32, no. 11, pp. 624-638, Nov. 2012.         [ Links ]

[102] H.-Y. Y. Shih and P.-L. L. Chang, "Industrial innovation networks in Taiwan and China: A comparative analysis," Technol. Soc., vol. 31, no. 2, pp. 176-186, May 2009.         [ Links ]

[103] B. A. Wood, H. T. Blair, D. I. Gray, P. D. Kemp, P. R. Kenyon, S. T. Morris, and A. M. Sewell, "Agricultural science in the wild: a social network analysis of farmer knowledge exchange.," PLoS One, vol. 9, no. 8, p. e105203, Jan. 2014.         [ Links ]

[104] B. Parsons, "Using Complexity Science Concepts When Designing Systems Interventions and Evaluations," Ft. Collins, CO InSites, no. 1996, 2001.         [ Links ]

[105] J. Cao, M. Shen, and D. C. Fan, "Research on performance evaluation of the regional IUR cooperative technological innovation system," in 2011 2nd International Conference on Artificial Intelligence, Management Science and Electronic Commerce (AIMSEC), 2011, no. 200802171023, pp. 5116-5119.         [ Links ]

[106] K. Chen and J. Guan, "Measuring the Efficiency of China's Regional Innovation Systems: Application of Network Data Envelopment Analysis (DEA)," Reg. Stud., vol. 46, no. 3, pp. 355-377, Mar. 2012.         [ Links ]

[107] S. Dezhong, "Regional Intellectual Capital Integration Performance Evaluation Based on Two-Phase Model," 2014 Sixth Int. Conf. Meas. Technol. Mechatronics Autom., pp. 467-471, 2014.         [ Links ]

[108] S. Kai, J. Xiao-feng, L. Yu-hua, K. Sun, X. Ju, and Y. Li, "Performance Evaluation of Chinese Regional Innovation Systems Based on Data Envelopment Analysis," in 2006 International Conference on Management Science and Engineering, 2006, pp. 1800-1804.         [ Links ]

[109] Y. Yu, H. Sun, Y. Yan, and S. Huimin, "Research on the efficiency evaluation of industry innovation system based on DEA analysis method," in 2009 16th International Conference on Industrial Engineering and Engineering Management, 2009, pp. 908-911.         [ Links ]

[110] Z. Chen and R. Xiao-hong, "Research on the Innovative Capabily Evaluation of Innovative Enterprises: Taking National Innovative (Pilot) Enterprises in Anhui Province as Examples," in 2011 International Conference on Information Management, Innovation Management and Industrial Engineering, 2011, vol. 3, pp. 173-176.         [ Links ]

[111] Y. Nan and Y. Tian, "Performance Evaluation on Regional Innovation System Based on AHP-TOPSIS Methodology," in Proceedings of 2011 International Conference on Computer Science and Network Technology, 2011, vol. 2, pp. 1140-1143.         [ Links ]

[112] Z. Zhang and J. Mu, "The Evaluation of Business Services Innovation Capability within the Logistics Industry Cluster," in 2010 International Conference on Management and Service Science, 2010, pp. 1-4.         [ Links ]

[113] L. Jian-Hua, J. Zhao-Hua, and L. Jia, "Evaluation on the network of the national innovation systems in China from the perspective of knowledge network," Biotechnol. An Indian J., vol. 8, no. 1, pp. 22-31, 2013.         [ Links ]

[114] A. J. Wieczorek and M. P. Hekkert, "Systemic instruments for systemic innovation problems: A framework for policy makers and innovation scholars," Sci. Public Policy, vol. 39, no. 1, pp. 74-87, 2012.         [ Links ]

[115] J. Chataway, R. Hanlin, and R. Kaplinsky, "Inclusive Innovation: An Architectue for Policy Development," Innov. Dev. , vol. 4, no. 1, pp. 33-54, 2014.         [ Links ]

[116] M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, 2006.         [ Links ]

[117] N. C. Jackson, "Unit One: Background to Systematic Reviews," Handb. - Syst. Rev. Heal. Promot. public Heal. Interv., pp. 5-8, 2004.         [ Links ]

 

 

Available online 11 Nov 2016

 

 

* ssgrobbelaar@sun.ac.za