GENERAL ARTICLES

 

Research and development project selection: a comprehensive analysis of the trends and methods

 

 

G. Filiz Turkmen^*; Y.I. Topcu

Industrial Engineering Department, Istanbul Technical University, Turkey. G. Filiz Turkmen: https://orcid.org/0000-0003-2732-5801; Y.I. Topcu: https://orcid.org/0000-0001-9717-7854

 

 

---

ABSTRACT

The purpose of this paper is to present a comprehensive literature review of the research and development (R&D) project selection problem since 1977, using the PRISMA method. This study aims (i) to support researchers and practitioners to understand selection methodologies, decision-maker types, and perspectives; (ii) to guide them in choosing the most suitable project selection method; and (iii) to clarify a common confusion by proposing a more precise definition of 'R&D project selection' and 'R&D portfolio selection'. The study also highlights several areas for improvement and the future of project selection techniques for further studies.

---

OPSOMMING

Die doel is om 'n omvattende literatuurstudie van navorsing en ontwikkeling projek seleksie probleem sedert 1977 aan te bied deur middel van die PRISMA metode. Daar word beoog om hiermee (i) navorsers en praktisyns te ondersteun om seleksie metodologieë, besluitneming tipes, en perspektiewe te verstaan; (ii) om hulle te lei om die mees gepaste projeksie seleksie metode te kies; en (iii) om 'n algemene verwarring op te klaar deur 'n duideliker definisie van navorsing en ontwikkeling projek seleksie en navorsing en ontwikkeling portefeulje seleksie te verskaf. Die studie lig verskeie areas vir verbetering uit en bespreek die toekom van projek seleksie tegnieke vir toekomstige studies.

---

 

 

1 INTRODUCTION

Research and development (R&D) projects are uncertain by their very nature, and the R&D project selection problem typically aims to select the best option to sort alternatives or to obtain the best subset of them. Given its goal and solution approaches, the R&D project selection problem is similar to two specific problems: (i) new product development, and (ii) the general project selection problem. The similarity with new product development comes from the level of uncertainty while optimising insufficient and vital resources [1]. However, not all R&D studies are intended to produce a product. Instead, R&D studies involve a more comprehensive approach that can serve many purposes, such as improving processes or efficiencies. The primary elements that distinguish an R&D project selection problem from a general project selection problem (non-R&D) are a high level of uncertainty and the inadequacy of data. On the other hand, 'R&D project selection' and 'R&D (project) portfolio selection' have been used interchangeably in the literature, which causes confusion. In this study, 'R&D project selection' is used for both, and more explicit definitions are proposed in the sections that follow.

Many public and private actors focus on R&D project selection with different motivations and approaches. From a public policy view, R&D may gain importance from the perspectives of sustainable competitive advantage or social welfare. Public funding studies conducted by different organisations and in countries such as China, the European Union, Russia, Mexico, South Korea, Taiwan, and Turkey are often found in the literature. The problem of public funds distribution [2] [3] [4] [5] [6] [7] [8] [9] is one of the most frequently studied R&D project selection problems because of a fund's size and its contribution to a country's economy. Some of the notable examples cited in the literature are the R&D support programmes of the Korean government [10], public funding in Turkey [11], and the PROFIT programme (The National Plan of Scientific, Research, Development and Technological Innovation) of the Spanish government [12].

Beyond its financing capacity, the public sector also plays an important role in the execution of projects. Moreover, empirical studies [13] have shown a positive relationship between R&D expenditure and countries' economic growth. The country-level selection of emerging technologies and the allocation of public funds and subsidies has been discussed in many studies [2] [14], while other studies [3] [15] have focused on the selection of governmental R&D projects. For instance, Mehrez's study [16] handles the R&D project selection subject for a laboratory that executes R&D projects in a small university, while other studies focus on the R&D project selection problem of government-business enterprises [17] [18] [19].

Like their public sector counterparts, private sector actors need to deal with the R&D project selection problem. They conduct R&D studies to secure their position in the global market, gain competitive advantage, and increase their profits. In this connection, the related literature shows that companies focus on balancing their R&D project portfolio [20] [21] [22], optimising their R&D project portfolio [23] [24], trying to rank possible R&D project alternatives [25] [26], or valuing R&D investments in an uncertain environment [27] [28]. The literature also includes studies of balancing and valuation [29] [30] [31] [32] [33] that consider the scheduling of R&D projects. On the other hand, major private investors in R&D project selection, such as investment banks, venture capital funds, private equity funds, endowment funds, and angel investors, have different priorities. An investor's point of view is fundamentally different from that of other R&D decision-makers. They especially focus on maximising returns and controlling risk. One of the most interesting studies on the investment subject leverages the scenario tree approach to develop a portfolio of R&D projects, stocks, and bonds [34]. Wan, Xu, Wang and Dong [35] approach the R&D project selection problem from a venture capital investment decision perspective by using five factors that are not commonly found in the literature: (i) organising ability, (ii) credit quality, (iii) level of research and development, (iv) profitability, and (v) debt servicing ability.

No extensive studies have been published recently that review the literature on R&D project selection. Existing studies mostly focus on contextualising the R&D project selection problem, but do not provide a detailed classification of the field of study, point of view, decision-maker type, or object of project selection [36] [37] [38] [39] [40] [41] [42] [43] [44]. On the other hand, some of the literature [45] [46] (Talias, M. A. 2007) [48] [49] focuses on a specific field, such as the pharmaceutical sector's R&D project selection. Therefore, this study aims to update the existing literature, to fill the gaps in the research field, and to propose a distinctive source for researchers through a detailed literature review. It intends to support researchers and practitioners in understanding selection methodologies, decision-maker types, and perspectives. It also aims to support them in finding the most suitable project selection method for their work by providing a complete list of the selection methods. Finally, it proposes a clearer definition of 'R&D project selection' and 'R&D portfolio selection' to clarify the common confusion that is caused by using the same meaning for two different problems.

This study approaches the papers about R&D project selection that were published between 1977 and 2019 and that are listed in the Science Direct and Wiley Online Library Journals databases. It presents an extensive systematic literature review based on a research model named PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), a method proposed by Moher, Liberati, Tetzlaff and Altman [50]. Five research questions (RQs) are addressed to enlarge upon the literature: '(RQ1) What is the distribution of papers by publication year?'; '(RQ2) What is the distribution of papers by authors' country of origin?'; '(RQ3) What are the types of decision-makers and their motivations in the R&D project selection problem?'; '(RQ4) What theories and methods are used to solve the R&D project selection problem? What is their frequency of occurrence?'; and lastly, '(RQ5) What methods are most frequently combined with others to solve the problem?'.

Section 2 focuses on explaining the research methodology, and gives brief information about it. Section 2.1 presents the planning phase of the literature search. Section 2.2 explains the conducting of the literature review. Next, Section 3 discusses the findings and results. Finally, in Section 4, the conclusions are summarised and recommendations for further studies are proposed.

 

2 RESEARCH METHODOLOGY

The framework proposed by Xiao and Watson [51], which describes the phases and steps of a literature review, is followed in this study. The PRISMA method is used in conducting the review in Phase 2. It is a systematic method that includes a comprehensive checklist consisting of 27 items and a four-phased flow diagram [50]. It has been widely used in various research areas such as health, technology, and the environment [52].

Xiao and Watson [51] classify literature reviews into four categories, based on their purpose: (i) describe, (ii) test, (iii) extend, and (iv) critique. According to their study, this research fits into the 'describe' category, which has five types of literature review: narrative review, textual narrative synthesis, meta-summary, meta-narrative, and scoping review. This study is a good example of a 'scoping review', since (i) it aims to be comprehensive; (ii) it extracts data such as decision-maker types, perspectives, and decision-making methods; and (iii) it summarises its findings on the basis of year and research country.

2.1 Planning the literature review

Xiao and Watson [51] describe three major phases and eight common steps of the literature review process that can be applied to various types of literature review procedures. Their concept is adopted in this research, as explained in Figure 1. Planning the review (Phase 1) is summarised under this sub-heading. Conducting the review (Phase 2) and reporting the review (Phase 3) are explained in the sections that follow.

The Science Direct and Wiley Online Library Journals databases were screened in Phase 2, within the scope of this study. The advanced search tab was used in the Science Direct database, and keywords were searched using the 'Title, abstract or keywords' box. First, the keyword group of "R&D" AND select* was searched and yielded 562 results. Then the keyword group of "R&D" AND evaluat* AND NOT select* was explored in the second step, and produced 1479 results. AND NOT select* was added here to avoid duplicating articles that contain "select*" that were produced by the previous search. The reason for using "R&D" was to cover the possible outcomes of both the "R&D project" and the "R&D portfolio" keywords. Afterwards, rather than using the abbreviation "R&D", the full text of "research and development" was used to run the third and fourth searches, which were similar to the first and second searches. Again, to avoid results that had both "R&D" and "research and development", the keyword group AND NOT "R&D" was included. The third search used "research and development" AND select* AND NOT "R&D", which yielded 472 results. The fourth and final search used the keyword group of "research and development" AND evaluat* AND NOT select* AND NOT "R&D", which yielded 1306 results.

2.2 Conducting the literature review

An advanced search of the Wiley Online Library Journals database was conducted using the same keyword groups in the title and abstract fields. This search yielded a total of 155 results. Thus the two databases yielded 3974 results in all. Of those results, only those with full-text English access were included. Elimination was done by screening each article's title, abstract, and full text, and by testing its relevance to the research objectives. As a result, 86 out of 386 papers were considered relevant for inclusion in this study during the full-text assessment. The oldest study obtained by the search was dated 1977. The search process is summarised as a flowchart in Figure 2.

Studies that were not included in this study focused on (i) the procedural and organisational issues related to R&D project selection, (ii) experiences in specific cases such as international studies, (iii) the financial analysis of R&D project selection, or (iv) reviews of small or medium enterprises (SMEs). The findings of the review were examined thoroughly against the research questions in Section 3.1 below. The review and the synthesis of the results are presented in Section 3.2.

 

3 RESULTS AND FINDINGS

3.1 Reporting the review

In this section, which represents Phase 3, the findings and results are presented and discussed. First, five research questions (RQs) and their answers are clarified. Then a synthesis of the results of the literature review is provided.

3.1.1 RQ1: What is the distribution of papers by publication year?

The earliest online paper on R&D project selection examined in this study dated back to 1977 (Figure 3). When the annual publication numbers were examined over the decades in question, an increase after the 1990s could be easily observed. There was only one study before 1980, and five studies were found in the period from 1980 to 1989. There were 14 publications from 1990 to 1999 - an increase of 150% over the previous 10-year period. Last, 35 studies were found from 2000 to 2009. The total number of studies from 2010 to 2018 was 35 - the same as in the previous decade.

 

 

3.1.2 RQ2: What is the distribution of papers by authors' country of origin?

The authors' countries of origin of the R&D project selection studies were examined (Figure 4), and it was found that the majority were done in the USA (19), China (13), Taiwan (10), and the United Kingdom (10), while Canada, South Korea, and Turkey published four studies each. Germany, India, Iran, Israel, Italy, and Spain published three studies each; Brasil, France, and Japan published two studies each; Austria, Lithuania, Colombia, Czech Republic, Denmark, Finland, Hungary, Mexico, The Netherlands, Portugal, Russia, Slovenia, and Switzerland all published one study each. In cases where there were multiple contributing countries, each country was counted once.

3.1.3 RQ3: What are the types of decision-makers and their motivations in the R&D project selection problem?

R&D project selection is a complex and interdisciplinary process that is performed by various institutions and organisations with different purposes. Consequently, the selection criteria and methods show significant differences that depend on the type of stakeholder. For instance, government-linked enterprises may prefer to place social welfare before profitability, while public funds may aim to distribute budgets equally between R&D projects without optimising the benefits. On the other hand, investors may target investing in specific sectors, projects, or companies according to their strategies. Private sector companies that conduct R&D projects may focus on the project's impact on company profitability and project return rather than on its social impact. As a result, all of these stakeholders with their different priorities apply different approaches to solve the R&D project selection problem.

' Decision-maker type' refers to the groups that make an R&D project selection. Identifying the decision-maker type is an essential part of sorting the problem into sub-classes when approaching an R&D project selection problem. The sub-groups around decision-makers are similar in their project selection criteria, evaluation methods, and project selection purpose. Thus it is possible to categorise decision-makers into four groups: public funds, public sector, private sector, and investors. 'Public funds', which is 19% of decision-makers, refers to a group that includes various institutions and organisations that provide grants and other financial support to R&D projects in line with national policies. 'Public sector', which forms 8% of decision-makers, refers to institutions and organisations that execute R&D studies as a primary goal, such as universities or other government-backed commercial businesses. 'Private sector', which constitutes 67% of decision-makers, refers to companies undertaking R&D regardless of their size. Finally, 'investors' - 6% of decision-makers - refers to non-public initiatives that fund R&D projects.

Public funds are designed to distribute state resources to support sectoral or scientific development. Thus they set annual capital deployment targets in line with national strategies. Typically a pool of R&D funds is split across industries, and R&D projects are assessed to the extent of capital that is available to be distributed in a given year. For instance, if cloud technologies are prioritised over the agricultural sector, high-potential projects from the agricultural sector will go without government support. In other cases, public funds may invest in R&D projects in line with those project budgets or the funding percentage. Therefore, public funds consider R&D selection as part of the problem of government allocation of resources to sectors in line with current strategy rather than assessing them individually, on the basis on the projects' merits.

The public sector, including for-profit or non-profit enterprises, also runs many R&D projects as part of its business models. For example, universities that conduct academic research, state-owned enterprises such as public transport, or iron and steel plants may have R&D projects to improve their product/service quality. The overall motives and objectives of such public sector players are similar to those of public funds. Both actors may prioritise social welfare and sector development. Yet the significant difference between the public sector and public funds is that the public sector is the executive body behind these R&D efforts, whereas public funds are simply enablers as financing partners.

Private sector players who choose R&D projects differ from the other decision-maker groups in many ways. The first of these differences is that the decision-making occurs according to resource limitations. Risk factors (e.g., technical, financial, and environmental risks) and corporate strategies that may divert management attention away from the general business outlook play a part in the valuation. The fact that the selection criteria are weighted differently by decision-maker groups is another crucial difference that is theoretically expected and is cited many times in the literature.

The last group, investors, mainly focus on return on investment from R&D projects. Thus they prefer to approach the problem from a portfolio perspective since (i) they have more than one investment, and (ii) they consider the portfolio's cumulative risk and return. On the other hand, they may prioritise selection criteria such as reliable company management, self-dedicated and competent project teams, and transparent accounting records and/or regulations.

The distribution of methods by decision-maker type is presented in Figure 5. Correspondingly, investors mostly prefer fuzzy/grey methods, MCDM methods, and mathematical programming (MP). On the other hand, the private sector mainly uses mathematical programming, financial methods, and MCDM methods. MCDM was also the most preferred method for public funds, followed by mathematical programming and fuzzy/grey methods. Similarly, MCDM methods were the most common public sector approaches, followed by mathematical programming methods.

As a result, the four different decision-makers - public sector, private sector, public funds, and investors - have different perspectives and R&D selection methodologies, depending on their objectives and expectations from an R&D effort. Thus this studyreveals a broad range of decision-making methods to guide decision-makers about their options. However, decision-maker groups eventually make their method selections on the basis of criteria such as data availability, goals, corporate culture, and kind of institution.

3.1.4 RQ4: What theories and methods are used to solve the RäD project selection problem? What is their frequency of occurrence?

The literature offers many different theories and methods to solve the R&D project selection problem. In the current study, the authors defined nine groups: mathematical programming methods, econometrical methods, financial methods, statistical methods, multicriteria decision-making (MCDM) methods, total quality management methods, heuristic methods, fuzzy/grey methods, and others (Figure 6).

 

 

Some of these methods, such as mathematical programming, financial/econometrical methods, or heuristic methods, require a specific educational background and experience to them to be used, which may not be possible for each decision-maker. Therefore, advanced calculations of such methods may not be practical without a support tool such as the right software. If decision-makers prefer a basic method, other methods could be selected, such as ranking and scoring.

Financial and econometrical methods have a good range of applications in the private sector and in investment evaluations. However, this may not be a suitable selection method for governments, which are not concerned with financial returns but with the public's welfare. On the other hand, TQM methods are strongly related to a customer perspective and to company values during R&D project selection; but their ability to do a financial evaluation is not as strong as that of other perspectives. Thus the motivation for making a selection might emerge as a key element in choosing a methodology.

Methods such as fuzzy logic and grey system theory are good at coping with the uncertainty that comes from the nature of R&D project selection. On the other hand, some methods require accurate data, and do not work with missing/intuitive information. Thus many researchers combine them with other methods to solve the selection problem.

One of the most important results of this literature review is summarised in Figure 6, which shows the frequency with which theories and methods are used to solve the problem. Many studies suggest appropriate combinations of various theories and methods. For this reason, each method used in a study is given the value of 1 in the analysis (Figure 6). For example, if the fuzzy analytical hierarchy process (AHP) is used in a study, the value of a fuzzy logic theory is increased by 1, as is that of AHP. According to the research results, mathematical programming and multicriteria decision-making methods are the most frequently used method groups in the R&D project selection literature, followed by financial methods. It is seen that the most frequently used methods are fuzzy logic and linear integer programming followed by scoring, DEA, DSS, AHP, ranking, and real options theory.

3.1.5 RQ5: What methods are most frequently combined with others to solve the problem?

The majority of the research in the R&D project selection literature proposes hybrid models that combine different approaches and methods. Depending on the problem they tackle, researchers need to find the model that best fits their specific problem. Therefore, such a model could be a combination of different approaches and methods. The frequency of combinations in existing studies gives researchers a critical clue to, and sound guidance for, building hybrid models. Thus frequently used combinations could offer a better solution or a starting point for researchers.

Mathematical programming methods are those that are most frequently combined with others - 34 times, as can be seen in Figure 7. MCDM methods follow them, which were combined 27 times with others. Fuzzy and grey approaches are the third most frequently combined methods with others in 23 studies.

3.2 Synthesis of results

The R&D project selection problem has also been frequently referred to in the literature as ' R&D project portfolio selection'. It has rarely been called 'R&D investment portfolio selection' or 'technology portfolio selection'. However, these problems may have the same meaning. In many studies, 'project selection' and ' project portfolio selection' are used interchangeably. The authors of this study suggest that methods that assess projects independently (in which the evaluation and selection of a project does not affect others) should be called ' R&D project selection'. On the other hand, the studies that evaluate projects in a dependent portfolio should be called ' R&D project portfolio selection' . A typical example of R&D project selection is an evaluation in which projects are ranked by individual scores. In this case, project funding starts with the highest-graded project and continues until the fund runs out. Here, if a project is risky, it corresponds to the score that is totally independent of other projects. Thus the selection of a project is not affected by the risk scores of other competing projects. On the other hand, a typical example of the R&D project portfolio selection problem assumes that there is a criterion that ensures that the total risk of the fund does not exceed a certain value. Thus the risk amount of any project is not independent, and the selection of a project will be relatively affected by the risks of the other projects. It is suggested that the method in the first example be called 'project selection', and that the method in the second example be called ' project portfolio selection' . The distinction between these two concepts is also found in their decision-making behaviours. Aloysius [66] claimed that risk allocation by funding more than one project or handling a risky project with more than one company decreases the risk avoidance behaviour. Graves and Rinquest [56] also discovered that some risk scores that are accepted in the collective evaluation of the projects are not accepted if the projects are evaluated individually. Therefore, it is seen that the presence or absence of a portfolio perspective in project selection affects the behaviour of decision-makers.

A major challenge in many R&D project selection methods is the necessity that decision-makers have an advanced theoretical and analytical background. However, only 9% of studies reviewed in this paper provided decision support systems as a tool that would help decision-makers in a complex selection process. Therefore, it has been suggested that the methods be made available in software or in user-friendly applications such as Excel to overcome this challenge [26].

Many studies in the R&D literature have defined challenges such as the availability, consistency, or reliability of data. In their research, which focused on real options in the valuation of R&D projects, Newton, Paxson and Widdicks [73] pointed out the difficulty of obtaining data from public sources. Hsu et al. [3] indicated a lack of data for government-sponsored new frontier technology R&D projects. Abbassi, Ashrafi, Sharifi Tashnizi [20] and Coldrick, Longhurst, Ivey, Hannis [75] discussed the suitability of project portfolio selection models to implement the R&D project portfolio owing to there being insufficient input data. Wang and Hwang [80] referred to unavailable and unreliable data as one of the challenges in R&D portfolio decisions, and they suggested a fuzzy model to overcome that problem. Huang and Zhao [30] contextualised the problem without the historical data of project parameters; instead, they used the data provided by experts and treated it as uncertain variables. Shin, Yoo and Kwak [110] preferred to collect required but unavailable data from experts through interviews. Chu, Hsu and Fehling [58] collected data from decision-makers, and then checked the data's consistency and resource conflicts via the decision support system. Therefore, the need to implement new data collection, estimation, sharing, and processing methods in R&D project selection problems is found to be a gap in the literature.

 

4 CONCLUSION AND FUTURE RESEARCH

This study analysed the project selection literature in detail via the PRISMA method by reading and rereading selected papers. Thus a descriptive analysis was conceptualised and presented to researchers and practitioners working in this field. The findings and conclusions obtained through the interpretation have already been presented; the implications of the study and suggestions for further studies are discussed in this section.

Six main findings are defined, based on the research questions of this study:

1. In each successive decade there has been an increasing interest in the R&D selection problem. The number of studies increased 150% after the 2000s.

2. Most of the research on R&D project selection has been conducted in the US, China, Taiwan, and the UK.

3. Different decision-makers - e.g., those in the public sector, the private sector, public funds, and investors - have different approaches and criteria depending on their objectives. However, a distribution analysis of methodologies by decision-maker type (Figure 5) showed that the most common method group for all decision-makers is MCDM methods, followed by mathematical and fuzzy/grey methods.

4. There are various groups of approaches to solutions, such as mathematical programming, multicriteria decision making, and financial methods.

5. The most frequently used methods are fuzzy logic and linear integer programming followed, by scoring, DEA, DSS, AHP, ranking, and real options theory.

6. Most of the studies suggested hybrid models that combine different approaches and methods. Mathematical programming, MCDM, and fuzzy/grey methods are most frequently combined with others.

Three main suggestions are proposed from the synthesis of the results:

1. First, the study identified a common confusion caused by using the same meaning for two distinct problems: R&D project selection and R&D portfolio selection. Therefore, clearer definitions are suggested for each problem.

2. Second, the need for digitalised tools such as decision support systems or software for complex R&D project selection models was proposed to increase the practicality of the selection models.

3. Finally, the lack of modern data collection, estimation, sharing, and processing methods in the R&D project selection literature was highlighted as a significant gap.

The implications of this study can be summarised as follows:

1. It is natural to expect that the findings of the research questions presented in this study would benefit researchers and practitioners who would like to work in this field.

2. It is to be expected that the comprehensive selection methods presented in this study would reduce the effort of undertaking a similar literature review for many future researchers working in the same field.

3. The further research ideas presented in this paper can be expected to influence future researchers who work in this field and to contribute to the enrichment of the field.

4. The definitions of 'R&D project selection' and 'R&D portfolio selection' in this study can close the gap by removing the confusion that is commonly encountered in the field.

The R&D project selection literature shows that scientific decision-making processes need data, estimations based on data, and proven methods. A common reason for their limited coverage in the literature highlights two major limitations: the availability of data, and the processing capacity of such comprehensive information [15] [20] [73] [88] [82]. However, this phenomenon is no longer a major challenge because of improvements in data collection and developments in processing methods. Therefore, calculations that could not have been made in past years can be made rapidly with contemporary data processing technology. For instance, some groundbreaking approaches in the business world, such as big data, have drastically changed ways of making decisions. It has been shown that data-driven decision-making methods increase a company's productivity and lead to higher returns, better asset utilisation, and increased market value [111]. Some researchers [112] argue that big-data-driven studies supersede traditional objective-driven research in science, technology, and innovation. Although it has not happened yet, the authors foresee that the literature will expand in the future as researchers apply big data analysis to R&D project selection. Thus, in future R&D selection processes, decision-makers will be able to use big data analyses with a large amount of unstructured data that has been collected from different sources. In addition, supercomputers can reduce calculation times from days to minutes, and they are more accessible than in the past. It is common sense that improvements in calculation methods and data science would allow researchers to build more complex models and to use more data in decision-making.

More importantly, the implications of ongoing R&D projects for the selection and scheduling of future projects have been ignored in many studies. Evolving requirements in current projects may impose limitations, such as a lack of available resources, and may cause differences in the total risk calculations as well as other dependencies. In this case, assuming the foreseeable requirements of current projects seems to be a preferable option in making calcjulations easier. However, its future effects on the selection process are still uncertain. The only study found in the literature that focused on both current and potential future R&D projects was that of Hassanzadeh et al. [29]. Therefore, important priority areas for future studies might be (i) the effect of ongoing projects on R&D project selection, and (ii) the procedure to incorporate this effect into the R&D selection process.

Today, decision-making approaches have exciting areas of application, such as artificial intelligence software, which can analyse the investment decision processes of individuals by learning and imitating their behaviour, and can actively manage an investment portfolio across stock, gold, and forex products [113]. Similarly, it would be possible to develop an artificial intelligence model to analyse previous R&D project selections and to decide which R&D projects should be selected. Thus the use of decision-making and artificial intelligence approaches in project selection is promising, and could be one of the interesting focuses of further studies.

 

REFERENCES

[1] Cooper, R.G., Edgett, S.J. and Kleinschmidt, E.J., "New product portfolio management: Practices and performance," Journal of Product Innovation Management, 16(4), 333-351, 1999.         [ Links ]

[2] Giebe, T., Grebe, T. and Wolfstetter, E., "How to allocate R&D (and other) subsidies: An experimentally tested policy recommendation," Research Policy, 35(9), 1261-1272, 2006.         [ Links ]

[3] Hsu, Y.-G., Tzeng, G.-H. and Shyu, J. Z., "Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects," R&D Management, 33(5), 539-551, 2003.         [ Links ]

[4] Arratia M., N.M., Lopez, I.F., Schaeffer, S.E. and Cruz-Reyes, L., "Static R&D project portfolio selection in public organizations," Decision Support Systems, 84, 53-63, 2016.         [ Links ]

[5] Ballesteros, J.A. and Rico, A.M., "Public financing of cooperative R & D projects in Spain: The concerted projects under the national R & D plan," Research Policy, 30(2001), 625-641, 2001.         [ Links ]

[6] Jeng, D.J.F. and Huang, K.H., "Strategic project portfolio selection for national research institutes," Journal of Business Research, 68(11), 2305-2311, 2015.         [ Links ]

[7] Atzei, A. and Jensen, N. "European initiative & strategy for prioritising and harmonising space technology R&D," Acta Astronautica, 48(5-12), 257-263, 2001.         [ Links ]

[8] Feng, B., Ma, J. and Fan, Z., "An integrated method for collaborative R & D project selection: Supporting innovative research teams," Expert Systems with Applications, 38(5), 5532-5543, 2011.         [ Links ]

[9] Oral, M., Kettani, O. and Lang, P., "A methodology for collective evaluation and selection of industrial R&D projects," Management Science, 37(7), 871-885, 1991.         [ Links ]

[10] Jung, U. and Seo, D.W., "An ANP approach for R & D project evaluation based on interdependencies between research objectives and evaluation criteria," Decision Support Systems, 49(3), 335-342, 2010.         [ Links ]

[11] Karasakal, E. and Aker, P., "A multicriteria sorting approach based on data envelopment analysis for R&D project selection problem," Omega, 73, 79-92, 2017.         [ Links ]

[12] Santamaría, L., Barge-Gil, A. and Modrego, A., "Public selection and financing of R&D cooperative projects: Credit versus subsidy funding," Research Policy, 39(4), 549-563, 2010.         [ Links ]

[13] Akcali, B.Y. and Sismanoglu, E., "Innovation and the effect of research and development (R&D) expenditure on growth in some developing and developed countries," Procedia - Social and Behavioral Sciences, 195, 768-775, 2015.         [ Links ]

[14] Kim, W., Han, S.K., Oh, K.J., Kim, T.Y., Ahn, H. and Song, C., "The dual analytic hierarchy process to prioritize emerging technologies," Technological Forecasting and Social Change, 77(4), 566-577, 2010.         [ Links ]

[15] Huang, C.C., Chu, P.Y. and Chiang, Y.H., "A fuzzy AHP application in government-sponsored R&D project selection," Omega, 36(6), 1038-1052, 2008.         [ Links ]

[16] Mehrez, A. " Selecting R&D projects: A case study of the expected utility approach," Technovation, 8(4), 299311, 1988.         [ Links ]

[17] Yetis, N., "A methodology for identification, evaluation and comparison of R&D projects in R&D planning," IFAC Proceedings, 10(14), 59-64, 1977.         [ Links ]

[18] Costello, D., "A practical approach to R&D project selection," Technological Forecasting and Social Change, 23(4), 353-368, 1983.         [ Links ]

[19] Eilat, H., Golany, B. and Shtub, A., "R & D project evaluation: An integrated DEA and balanced scorecard approach," Omega, 36, 895-912, 2008.         [ Links ]

[20] Abbassi, M., Ashrafi, M. and Sharifi Tashnizi, E. "Selecting balanced portfolios of R&D projects with interdependences: A cross-entropy based methodology," Technovation, 34(1), 54-63, 2014.         [ Links ]

[21] Cluzel, F., Yannou, B., Millet, D. and Leroy, Y., "Eco-ideation and eco-selection of R&D projects portfolio in complex systems industries," Journal of Cleaner Production, 112, 4329-4343, 2016.         [ Links ]

[22] Eilat, H., Golany, B. and Shtub, A., "Constructing and evaluating balanced portfolios of R & D projects with interactions: A DEA based methodology," European Journal of Operational Research, 172, 1018-1039, 2006.         [ Links ]

[23] Beaujon, G.J., Marin, S.P. and Mcdonald, G.C., "Balancing and optimizing a portfolio of R & D projects," Naval Research Logistics, 48(2001), 18-40, 2001.         [ Links ]

[24] Bhattacharyya, R., Kumar, P. and Kar, S. "Fuzzy R&D portfolio selection of interdependent projects," Computers and Mathematics with Applications, 62(10), 3857-3870, 2011.         [ Links ]

[25] Cho, K.T. and Kwon, C.S., "Hierarchies with dependence of technological alternatives: A cross-impact hierarchy process," European Journal of Operational Research, 156(2), 420-432, 2004.         [ Links ]

[26] Hwang, H. and Yu, J., "R & D project evaluation model based on fuzzy set priority," Computers and Industrial Engineering, 35(98), 567-570, 1998.         [ Links ]

[27] Biancardi, M. and Villani, G., "A fuzzy approach for R & D compound option valuation," Fuzzy Sets and Systems, 310, 108-121, 2017.         [ Links ]

[28] Carlsson, C., Fullér, R., Heikkilä, M. and Majlender, P., "A fuzzy approach to R&D project portfolio selection," International Journal of Approximate Reasoning, 44(2), 93-105, 2007.         [ Links ]

[29] Hassanzadeh, F., Modarres, M., Nemati, H.R. and Amoako-Gyampah, K., "A robust R&D project portfolio optimization model for pharmaceutical contract research organizations," International Journal of Production Economics, 158, 18-27, 2014.         [ Links ]

[30] Huang, X. and Zhao, T., "Project selection and scheduling with uncertain net income and investment cost," Applied Mathematics and Computation, 247, 61 -71, 2014.         [ Links ]

[31] Imoto, S., Yabuuchi, Y. and Watada, J., "Fuzzy regression model of R&D project evaluation," Applied Soft Computing, 8(3), 1266-1273, 2008.         [ Links ]

[32] Latipova, A.T., "On optimization of R&D project selection and scheduling," IFAC-PapersOnLine, 48(25), 6-10, 2015.         [ Links ]

[33] Sun, H. and Ma, T., "A packing-multiple-boxes model for R&D project selection and scheduling," Technovation, 25(11), 1355-1361, 2005.         [ Links ]

[34] Fang, Y., Chen, L. and Fukushima, M., "A mixed R&D projects and securities portfolio selection model," European Journal of Operational Research, 185(2), 700-715, 2008.         [ Links ]

[35] Wan, S.-P., Xu, G., Wang, F. and Dong, J., "A new method for Atanassov's interval-valued intuitionistic fuzzy MAGDM with incomplete attribute weight information," Information Sciences, 316(168), 329-347, 2015.         [ Links ]

[36] Heidenberger, K. and Stummer, C., "Research and development project selection and resource allocation: A review of quantitative modelling approaches," International Journal of Management Reviews, 1(2), 197-224, 1999.         [ Links ]

[37] Gunasekaran, A., "Essentials of international and joint R&D projects," Technovation, 17(1l), 637-647, 1997.         [ Links ]

[38] Schrödkrf, H.H., "R&D project evaluation and selection models for development: A survey of the state of the art," Socio-Economic Planning Sciences, 5(1), 25-39, 1971.         [ Links ]

[39] Fahrni, P. and Spatig, M., "An application-oriented guide to R&D project selection and evaluation methods," R&D Management, 20(2), 155-171, 1990.         [ Links ]

[40] Baker, N.R., "R&D project selection models: An assessment," IEEE Transactions on Engineering Management, 21(4), 165-171, 1974.         [ Links ]

[41] Baker, N.R. and Freeland, J., "Recent advances in R&D benefit measurement and project selection methods," Management Science, 21(10), 1164-1175, 1975.         [ Links ]

[42] Baker, N.R. and Pound, W.H., "R and D project selection: Where we stand," IEEE Transactions on Engineering Management, 11(4), 124-134, 1964.         [ Links ]

[43] Cetron, M.J., Martino, J. and Roepcke, L., "The selection of R&D program content: Survey of quantitative methods," IEEE Transactions on Engineering Management, 14(1), 4-13, 1967.         [ Links ]

[44] Danila, N., "Strategic evaluation and selection of R&D projects," R&D Management, 19(1), 47-62, 1989.         [ Links ]

[45] Jacob, W.F. and Kwak, Y.H., "In search of innovative techniques to evaluate pharmaceutical R&D projects," Technovation, 23(4), 291-296, 2003.         [ Links ]

[46] Pandey, M., "Investment decisions in pharmaceutical R&D projects," Drug Discovery Today, 8(21), 968-971, 2003.         [ Links ]

[47] Talias, M.A., "Optimal decision indices for R & D project evaluation in the pharmaceutical industry: Pearson index versus Gittins index," European Journal of Operational Research, 177, 1105-1112, 2007.         [ Links ]

[48] Tran, G. and Pang, F., "The application of portfolio management techniques to pharmaceutical R&D project selection and cost-effectiveness analysis," Value in Health, 2(5), 408, 1999.         [ Links ]

[49] Cioffe, C., "Portfolio selection and management in pharmaceutical research and development: Issues and challenges," Clinical Pharmacology & Therapeutics, 89(2), 300-303, 2011.         [ Links ]

[50] Moher, D., Liberati, A., Tetzlaff, J. and Altman, D.G., "Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement," Journal of Clinical Epidemiology, 62, 1006-1012, 2009.         [ Links ]

[51] Xiao, Y. and Watson, M.,"Guidance on conducting a systematic literature review," Journal of Planning Education and Research, 39(1), 93-112, 2019.         [ Links ]

[52] Zare, M., Pahl, C., Rahnama, H., Nilashi, M., Mardani, A., Ibrahim, O. and Ahmadi, H., "Multi-criteria decision making approach in E-learning: A systematic review and classification," Applied Soft Computing Journal, 45, 108128, 2016.         [ Links ]

[53] Liberatore, M.J., "R&D project selection," Telematics and Informatics, 3(4), 289-300, 1986.         [ Links ]

[54] Dias, O.P., "The R & D project selection problem with fuzzy coefficients," Fuzzy Sets & Systems, 26(3), 299-316, 1988.         [ Links ]

[55] Liberatore, M.J., "An expert support system for R&D project selection," Mathematical and Computer Modelling, 11(C), 260-265, 1988.         [ Links ]

[56] Graves, S.B. and Rinquest, J.L., "Evaluating a portfolio of R&D investments," The Journal of High Technology Management Research, 7(1), 53-60, 1996.         [ Links ]

[57] Bohanec, M., Rajkovi, V., Semoli, B. and Poga, A., "Knowledge-based portfolio analysis for project evaluation," Information & Management, 28, 293-302, 1995.         [ Links ]

[58] Chu, P.-Y. V., Hsu, Y.-L. and Fehling, M., "A decision support system for project portfolio selection," Computers in Industry, 32(2), 141-149, 1996.         [ Links ]

[59] Coffin, M.A. and Taylor III, B.W., "Multiple criteria R&D project selection and scheduling using fuzzy logic," Computers & Operations Research, 23(3), 207-220, 1996.         [ Links ]

[60] Graves, S.B. and Ringuest, J.L., "Choosing the best solution in an R&D project selection problem with multiple objectives," Journal of High Technology Management Research, 3(2), 213-224, 1992.         [ Links ]

[61] Green, R.H., Doyle, J.R. and Cook, W.D., "Preference voting and project ranking using DEA and cross-evaluation," European Journal of Operational Research, 90(3), 461-472, 1996.         [ Links ]

[62] Henig, M.I. and Katz, H., "R & D project selection: A decision process approach," Journal of Multi-Criteria Decision Analysis, 5(October 1994), 169-177, 1996.         [ Links ]

[63] Kim, K., Park, K. and Seo, S., "A matrix approach for telecommunications technology selection," Computers & Industrial Engineering, 33(3-4), 833-836, 1997.         [ Links ]

[64] Rengarajan, S. and Jagannathan, P., "Project selection by scoring for a large R&D organisation in a developing country," R&D Management, 27(2), 155-164, 1997.         [ Links ]

[65] Vonortas, N.S. and Hertzfeld, H.R., "Research and development project selection in the public sector," Journal of Policy Analysis and Management, 17(4), 621 -638, 1998.         [ Links ]

[66] Aloysius, J.A., "Risk aggregation and the efficient selection of joint projects by a consortium," Omega, 27(3), 389-396, 1999.         [ Links ]

[67] Cook, W.D. and Green, R.H., "Project prioritization: A resource-constrained data envelopment analysis approach," Socio-Economic Planning Sciences, 34(2), 85-99, 2000.         [ Links ]

[68] Farrukh, C., Phaal, R., Probert, D., Gregory, M., & Wright, J., "Developing a process for the relative valuation of R & D programmes," R&D Management, 30(1), 43-53, 2000.         [ Links ]

[69] Loch, C.H. & Bode-Greuel, K., "Evaluating growth options as sources of value for pharmaceutical research projects," R&D Management, 31, 231-248, 2001.         [ Links ]

[70] Mikkola, J.H., "Portfolio management of R & D projects: Implications for innovation management," Technovation, 21, 423-435, 2001.         [ Links ]

[71] Klapka, J. and Pinos, P., "Decision support system for multicriterial R & D and information systems projects selection," European Journal of Operational Research, 140, 434-446, 2002.         [ Links ]

[72] Linton, J.D., Walsh, S.T. and Morabito, J., "Analysis, ranking and selection of R&D projects in a portfolio," R&D Management, 32(2), 139-148, 2002.         [ Links ]

[73] Newton, D.P., Paxson, D.A., Widdicks, M., "Real R & D options," International Journal of Management Reviews, 5/6(2), 113-130, 2004.         [ Links ]

[74] Ringuest, J.L., Graves, S.B. and Case, R.H., "Mean-Gini analysis in R&D portfolio selection," European Journal of Operational Research, 154(1), 157-169, 2004.         [ Links ]

[75] Coldrick, S., Longhurst, P., Ivey, P. and Hannis, J., "An R&D options selection model for investment decisions," Technovation, 25(3), 185-193, 2005.         [ Links ]

[76] Tian, Q., Ma, J., Liang, J., Kwok, R.C.W. and Liu, O., "An organizational decision support system for effective R & D project selection," Decision Support Systems, 39(2005), 403-413, 2005.         [ Links ]

[77] Lawson, C.P., Longhurst, P.J. and Ivey, P.C., "The application of a new research and development project selection model in SMEs," Technovation, 26(2), 242-250, 2006.         [ Links ]

[78] Linton, J.D., Morabito, J. and Yeomans, J.S., "An extension to a DEA support system used for assessing R & D projects," R&D Management, (37), 29-36, 2007.         [ Links ]

[79] Medaglia, A.L., Graves, S.B. and Ringuest, J.L., "A multiobjective evolutionary approach for linearly constrained project selection under uncertainty," European Journal of Operational Research, 179(3), 869-894, 2007.         [ Links ]

[80] Wang, J. and Hwang, W.L., "A fuzzy set approach for R&D portfolio selection using a real options valuation model," Omega, 35(3), 247-257, 2007.         [ Links ]

[81] Wu, M. and Yen, S.H., "Pricing real growth options when the underlying assets have jump diffusion processes: The case of R & D investments," R&D Management, 37(3), 269-276, 2007.         [ Links ]

[82] Glickman, T.S., "Program portfolio selection for reducing prioritized security risks," European Journal of Operational Research, 190, 268-276, 2008.         [ Links ]

[83] Lin, C.J. and Wu, W.W., "A causal analytical method for group decision-making under fuzzy environment," Expert Systems with Applications, 34(1), 205-213, 2008.         [ Links ]

[84] Schneider, M., Tejeda, M., Dondi, G., Herzog, F., Keel, S. and Geering, H., "Making real options work for practitioners: A generic model for valuing R & D projects," R&D Management, 38(1), 85-106, 2008.         [ Links ]

[85] Tolga, A.C. and Kahraman, C., "Fuzzy multiattribute evaluation of R & D projects using a real options valuation model," International Journal of Intelligent Systems, 23, 1153-1176, 2008.         [ Links ]

[86] Van Bekkum, S., Pennings, E. and Smit, H., "A real options perspective on R&D portfolio diversification," Research Policy, 38(7), 1150-1158, 2009.         [ Links ]

[87] Wu, J., Liang, L., Yang, F. and Yan, H., "Bargaining game model in the evaluation of decision making units," Expert Systems with Applications, 36(3 part 1), 4357-4362, 2009.         [ Links ]

[88] Yuen, K.K.F. and Lau, H.C.W.A., "Linguistic possibility-probability aggregation model for decision analysis with imperfect knowledge," Applied Soft Computing Journal, 9(2), 575-589, 2009.         [ Links ]

[89] Solak, S., Clarke, J.P.B., Johnson, E.L. and Barnes, E.R., "Optimization of R&D project portfolios under endogenous uncertainty," European Journal of Operational Research, 207(1), 420-433, 2010.         [ Links ]

[90] Liu, S.S. and Wang, C.J., "Optimizing project selection and scheduling problems with time-dependent resource constraints," Automation in Construction, 20(8), 1110-1119, 2011.         [ Links ]

[91] Davoudpour, H., Rezaee, S. and Ashrafi, M., "Developing a framework for renewable technology portfolio selection: A case study at a R&D center," Renewable and Sustainable Energy Reviews, 16(6), 4291 -4297, 2012.         [ Links ]

[92] Hassanzadeh, F., Nemati, H. and Sun, M., "Robust optimization for interactive multiobjective programming with imprecise information applied to R&D project portfolio selection," European Journal of Operational Research, 238(1), 41-53, 2014.         [ Links ]

[93] Lo Nigro, G., Morreale, A. and Enea, G., "Open innovation: A real option to restore value to the biopharmaceutical R&D," International Journal of Production Economics, 149, 183-193, 2014.         [ Links ]

[94] Rocha, A., Tereso, A., Cunha, J. and Ferreira, P., "Investments analysis and decision making: Valuing R&D project portfolios using the PROV exponential decision method," Tékhne, 12(1-2), 48-59, 2014.         [ Links ]

[95] Bhattacharyya, R., "A grey theory based multiple attribute approach for R&D project portfolio selection," Fuzzy Information and Engineering, 7(2), 211-225, 2015.         [ Links ]

[96] Bin, A., Azevedo, A., Duarte, L., Salles-Filho, S. and Massaguer, P., "R&D and innovation project selection: Can optimization methods be adequate?" Procedia Computer Science, 55, 613-621, 2015.         [ Links ]

[97] Wang, J., Wang, C.Y. and Wu, C.Y., "A real options framework for R&D planning in technology-based firms," Journal of Engineering and Technology Management, 35, 93-114, 2015.         [ Links ]

[98] Lo Nigro, G., Morreale, A. and Abbate, L., "An open innovation decision support system to select a biopharmaceutical R & D portfolio," Managerial and Decision Economics, 37, 392-406, 2016.         [ Links ]

[99] Peng, X. and Yang, Y., "Pythagorean fuzzy Choquet integral based MABAC method for multiple attribute group decision making," International Journal of Intelligent Systems, 31, 989-1020, 2016.         [ Links ]

[100] Wu, J., Chu, J., Sun, J., Zhu, Q. and Liang, L., "Extended secondary goal models for weights selection in DEA cross-efficiency evaluation," Computers and Industrial Engineering, 93, 143-151, 2016.         [ Links ]

[101] Jinfa, L. and Biting, L., "Evaluation method of R&D investment value of intelligent manufacturing enterprise based on growth option," Procedia Engineering, 174, 301-307, 2017.         [ Links ]

[102] Oztaysi, B., Cevik, S., Kerim, O. and Cengiz, G., "Evaluation of research proposals for grant funding using interval-valued intuitionistic fuzzy sets," Soft Computing, 21(5), 1203-1218, 2017.         [ Links ]

[103] Chen, T., "Remoteness index-based Pythagorean fuzzy VIKOR methods with a generalized distance measure for multiple criteria decision analysis," Information Fusion, 41, 129-150, 2018.         [ Links ]

[104] Liang, D., Xu, Z., Liu, D. and Wu, Y., "Method for three-way decisions using ideal TOPSIS solutions at Pythagorean fuzzy information," Information Sciences, 435, 282-295, 2018.         [ Links ]

[105] Zeng, S., Mu, Z. and Balezentis, T., "A novel aggregation method for Pythagorean fuzzy," International Journal of Intelligent Systems, 33(3), 573-585, 2018.         [ Links ]

[106] Musa Çaglar, S.G., "Impact assessment based sectoral balancing in public R&D project portfolio selection," SocioEconomic Planning Sciences, 66, pp. 68-81, 2019.         [ Links ]

[107] Liu, F., Chen, Y. W., Yang, J. B., Xu, D. L. and Liu, W."Solving multiple-criteria R&D project selection problems with a data-driven evidential reasoning rule," International Journal of Project Management, 37, p. 87-97, 2019.         [ Links ]

[108] Forouli, A., Doukas, H., Nikas, A., Sampedro, J., & Van de Ven, D. J, "Identifying optimal technological portfolios for European power generation towards climate change mitigation: A robust portfolio analysis approach," Utilities Policy, 57, 33-42, 2019.         [ Links ]

[109] Lopes, K. M. and Zancul, E. , "Application of set-based concurrent engineering principles in R&D project prioritization," Procedia CIRP, 84, 49-54, 2019.         [ Links ]

[110] Shin, C.O., Yoo, S.H. and Kwak, S.J., "Applying the analytic hierarchy process to evaluation of the national nuclear R&D projects: The case of Korea," Progress in Nuclear Energy, 49, 375-384, 2007.         [ Links ]

[111] Zhang, Y., Zhang, G., Chen, H., Porter, A.L., Zhu, D. and Lu, J., "Topic analysis and forecasting for science, technology and innovation: Methodology with a case study focusing on big data research,"Technological Forecasting and Social Change, 105, 179-191, 2016.         [ Links ]

[112] Provost, F. and Fawcett, T., "Data science and its relationship to big data and data-driven decision making," Big Data, 1, 51 -59, 2013.         [ Links ]

[113] Maknickienè, N., "Selection of orthogonal investment portfolio using Evolino RNN trading model," Procedia - Social and Behavioral Sciences110, 1158-1165, 2014.         [ Links ]

 

 

Submitted by authors 23 Oct 2020
Accepted for publication 16 Nov 2021
Available online 14 Dec 2021

 

 

* Corresponding author: gizemfiliz@itu.edu.tr
1 The references for each decade are as follows. Before 1980: [17]; 1980-1989: [16] [18] [53] [54] [55]; 1990-1999: [9] [56] [57] [58] [59] [60] [61] [36] [62] [63] [64] [26] [65] [66]; 2000-2009: [67] [68] [5] [23] [69] [70] [71] [72] [3] [25] [73] [74] [75] [33] [76] [22] [2] [77] [28] [78] [79] [47] [80] [81] [19] [34] [82] [15] [31] [83] [84] [85] [86] [87] [88]; 2010-2019: [10] [12] [89] [24] [8] [90] [91] [20] [29] [92] [30] [93] [94] [95] [96] [6] [32] [35] [97] [4] [21] [11] [98] [99] [100] [27] [101] [102] [103] [104] [105] [106] [107] [108] [109].
2 References by country. USA: [29] [18] [53] [55] [16] [56] [59] [60] [65] [66] [23] [72] [74] [82] [89] [92] [58] [79] [78]; China: [34] [105] [33] [76] [87] [88] [8] [30] [35] [99] [100] [101] [104]; Taiwan: [58] [3] [80] [81] [15] [83] [90] [6] [87] [103] UK: [68] [75] [77] [61] [73] [67]; Canada: [61] [78] [67] [9]; S. Korea: [63] [26] [25] [10]; Turkey: [17] [85] [11] [102]; Germany: [2] [47] [69]; India: [64] [95] [24]; Iran: [29] [91] [20]; Israel: [62] [22] [19]; Italy: [93] [98] [27]; Spain: [5] [12] [4]; Brasil: [54] [96]; France: [69] [21]; Japan: [34] [31]; Austria: [36]; Lithuania: [105]; Colombia: [79]; Czechia: [71]; Denmark: [70]; Finland: [28]; Hungary: [28]; Mexico: [4]; Netherlands: [86]; Portugal: [94]; Russia: [32]; Slovenia: [57]; Switzerland: [84]