Review of machine learning-based Mineral Resource estimation

Mineral Resources estimation plays a crucial role in the profitability of the future of mining operations. The conventional geostatistical methods used for grade estimation require expertise, understanding and knowledge of the spatial statistics, resource modelling, geology, mining engineering as well as clean validated data to build accurate block models. However, the geostatistical models are sensitive to changes in data and would have to be rebuilt on newly acquired data with different characteristics, which has proved to be a time-consuming process. Machine learning methods have in recent years been proposed as an alternative to the geostatistical methods to alleviate the problems these might suffer from in Mineral Resource estimation. In this paper, a systematic literature review of machine learning methods used in Mineral Resource estimation is presented. This has been conducted on such studies published during the period 1990 to 2019. The types, performances, and capabilities, of several machine learning methods have been evaluated and compared against each other, and against the conventional geostatistical methods. The results, based on 31 research studies, show that the machine learning-based methods have outperformed the conventional grade estimation modelling methods. The review also shows there is active research on applying machine learning to grade estimation from exploration through to exploitation. Further improvements can be expected if advanced machine learning techniques are to be used.


Introduction
Mineral Resources estimation (MRE) is one of the most important and critical stages in the mining value chain. The whole mining project depends on the reliable estimation of the grade of the mineralization. The spatial distribution of Mineral Resources (MR) depends on several known and unknown factors that cannot be incorporated in the traditional/conventional geostatistical models (Rossi and Deutsch, 2013;Hosseini, Asghari, and Emery, 2017). The basic assumption made by most of the mineral grade estimation models is that a spatial relationship exists between the grades at any two locations and that this relationship is a function of the distance between the two locations. Since the 1950s, many Mineral Resource Estimation (MRE) models have been proposed based on statistical methods, and later by incorporating the spatial dimensions in the estimations, which improved the results significantly. However, even the spatial estimation techniques are based on several assumptions and predict the MR values with some uncertainty levels. Many MRE models based on spatial statistics only incorporate the three-dimensional location of the measured value (X, Y, Z) along with the grade and thickness information. However, several other parameters like topographical variations, directions of geological structures, type of geology etc. are also very important for reliable MRE. These parameters are often neglected or not incorporated in the conventional statistical and spatial statistical methods. In the last 10 years, machine learning (ML)-based methods have become more popular in Resource estimation research. Several researchers (Samanta, Banopadhyay, and Ganduli, 2006;, Tahmasebi and Hezarkhani 2010, Zhang, 2017 have reported that ML-based methods have emerged as prediction models and as major alternatives to geostatistics for MRE. Despite the huge number of research studies that used ML-based methods, inconsistent results have been reported regarding the accuracy of the methods, the comparison between ML and non-ML methods, and comparisons among several different ML-based methods. For example, in a comparison of ML and non-ML methods for MRE, Dutta, et al., (2010) concluded that the ML methods produced more accurate results; however, Samanta, Ganguli, and Banopadhyay, (2005) showed that non-ML methods outperform the ML-based methods in producing reliable MREs. In a comparison among the ML-based methods, Tahmasebi and Hezarkhani, (2010), showed that artificial neural networks perform better than regression models. Chatterjee and Bandopadhyay (2011) however, reported opposing results. Due to the discrepancies in the research studies which applied ML methods for MRE, practitioners in the fields of mining and geosciences may be hesitant to use ML models more practically and confidently. As opposed to other fields of study where the ML methods have been well tested and applied successfully, the applications of these methods in MRE face several challenges, like limited training data-sets, the uncertainty in existing data-sets and geological conditions, as well as the human factor. Although there is an increasing trend in academic research towards the applications of ML methods, most of the experts choose non-ML methods for MRE.
To facilitate the applications of ML methods in the mining industry, it is very important to systematically summarize the empirical evidence on ML methods in current research and practice. The applications of ML methods are novel in the field of MRE, hence very little literature is available on the subject. There is no existing systematic literature review on the applications of ML methods in the field of mining related to MRE. In this paper, the literature review was performed on articles published from January 1 1990 to June 30 2019, related to ML methods applications in the field of MRE. The main purpose is to summarize the published work regarding the types of ML methods used in MRE, the comparisons between ML and non-ML methods, the performance evaluation of ML and non-ML methods, and the factors mainly considered in the application of ML methods.

Methodology
The systematic literature review methodology proposed by Kitchenham and Charters (2007) was used to conduct and report the review. The main steps include the definition of research questions, design of search strategy, selection criteria for studies, quality assessment, extraction of relevant data, and analysis, as given in Figure 1.

Research questions
Four research questions were defined based on the objective of summarizing the published work regarding the types of ML models used in MRE.

Which ML methods/models have been used for MRE?
The aim was to identify the ML methods/models that have been used in MRE to provide MRE researchers and practitioners with a range of possible methodologies to consider.

Does any publication on the comparison of ML against non-ML methods exist?
This question is concerned with the comparison of ML with non-ML methods in terms of accuracy, if performed in the studies.

Do ML methods outperform non-ML methods?
The aim is to compare the accuracy of the ML methods against non-ML methods. 4. Are there any ML methods that distinctly outperform other ML methods?
The comparison of different ML methods in order to identify these ML methods which perform better than others.

Search approach
The search strategy was based on the search terms, sources of publications, and process of search as explained below.

Search terms
The following steps were applied in order to search the terms for MRE, Wen, 2012: (a) Selection of major keywords based on the research questions (b) Identify possible different spellings and synonyms for major keywords (c) Check the major keywords given in the relevant books and papers (d) Usage of Boolean operator or to combine the different spellings and synonyms (e) Usage of and operator to link the major keywords.
The following are the main keywords identified from the published work on machine learning models and techniques used for MRE.
Mineral AND (grade OR ore OR reserve OR resources) AND (estimation OR prediction) AND (machine learning OR artificial intelligence OR mining OR data mining) AND (geochemical OR exploration OR boreholes) AND (neural networks OR support vector machine OR support vector machine OR regression tree OR random forest OR Kriging OR nearest neighbour) AND (modelling OR spatial OR analysis). Although significant research has been done on the application of ML techniques in the oil and gas or petroleum industry, that was not considered in this literature review, being outside the scope of the paper.

Publication sources
Five reliable and widely searched electronic databases (IEEE Xplore, ScienceDirect, Springer Nature, Web of Science, and Google Scholar) were used to search the most relevant literature. All the other databases are largely covered by these five primary databases, and hence have also been used by many literature reviews studies in several fields. The main keywords developed previously were used to search for journal and conference papers in the five databases. Except for Google Scholar, the search was conducted on the title, abstract, and keywords. Google Scholar returned several millions of irrelevant records when terms were searched on full text, hence the search was limited to the titles of the publications. The searches were restricted to the time period from January 1 1990 to June 30 2019, as publications on the application of ML methods to MRE begin to appear in the early 1990s; for example, Wu and Zhou (1993) used neural networks for reserve estimation in 1993.

Search process
A comprehensive search of the relevant keywords among all the databases is very important. Therefore, the search process was designed and divided into two phases to identify relevant published papers.
Phase A: Search the five databases individually and list all the papers that resulted from the searches. Phase B: The references in each resulting paper were scanned to identify additional relevant papers, which were added to the original. The inclusion and exclusion criteria set for the papers were as follows:

Inclusion criteria
1. ML methods used to pre-process the data 3. Applied more than one ML method and/or combined with non-ML methods 4. Comparative studies that compare different ML methods and/ or with non-ML methods 5. Studies which contain both conference and journal publications; only journal papers were selected for inclusion 6. Studies which have multiple versions or duplications; only the most recent and complete study was included in the list.

Exclusion criteria
1. Qualitative studies without proper ML methodsdetails and used with other than borehole or geochemical data. 2. Review papers.

Quality control and assessment
The quality control process assists with the selection of the most relevant research papers. A series of research questions was formulated in order to assess the rigorousness, reliability, and relevance of the papers. The questions are given in Table I, based on the methodology developed by Wen et al. (2012). The three options were assigned to each question, i.e. yes, partially, and no. These options were scored as 1, 0.5, and 0 respectively as in by Wen et al. (2012). The quality was assessed by summing the scores for answers against each question. The studies with a minimum score of 3.0 (50% of a perfect score) were selected for data extraction and analysis to ensure the quality of this literature research.
By applying the selection criteria, 50 papers were identified. After scanning the references in these papers, eight additional relevant papers were found. Hence, a total of 58 relevant papers were initially considered. However, after applying the quality control criteria, only 31 papers were selected for data extraction. The quality assessment is discussed in detail in the following sections. The complete list of the 31 selected papers can be found in Appendix A.

Data extraction and analysis
The selected research studies were exploited in order to collect the data that can answer the specific research questions of this exercise. All the reliable and relevant research papers were divided into specific sections, as given in Table II.
These sections were analysed combinedly to provide more meaningful information and to enhance the understandings. The extracted data was both quantitative (number. of boreholes or data-sets) and qualitative (data type, publisher, publication type). Different visualization techniques such as bar charts, pie charts, etc. were also used to enhance the data extracted from the research studies.
The vote counting method was used in order to compare the accuracy and application of different ML models (Malhotra, 2015). The vote counting method counts the number of times a model i.e., Model A outperformed Model B, or vice versa. With this method, a general idea of whether an estimation ML model outperforms another model in the estimation of mineral grades emerged.

QA1
Are the aims of the research clearly defined?

QA2
Are the estimation methods well defined and deliberate?

QA3
Is the estimation accuracy measured and reported?

QA4
Is the proposed estimation method compared with other methods?

QA5
Are the findings of the study clearly stated and supported by the results?

QA6
Are the limitations of the study analysed thoroughly?

Results and discussion
A total of 31 research studies were identified that dealt with the application of ML methods in the field of MRE. These studies were published between 1990 and 2019 in journals and conference proceedings. Of these, 30 (approx. 97%) papers were published in journals, only one (approx. 3%) was published in conference proceedings, and none were found as book chapters. The places of publication are given in Appendix B. The publishers were mainly Elsevier, Springer, the IEEE, and Taylor & Francis. All of the research studies were experimental and none of them were survey research. In terms of quality control, only those studies were selected with a minimum quality score of 50%, hence all the studies were of a high-quality level. The publication history is summarized in Figure 2 and shows that the oldest paper found was in the year 1993, followed by eight years wherein the selected publishers did not publish any paper until 2002. A significant increase in ML-based mineral grade estimation papers was found in 2010, with five papers published in that year. In both 2013 and 2017, four papers were published. However, only one paper was found to be published in 2018 and 2019. These statistics show that limited research has been conducted on the application, testing and validation ofML techniques in the field of mineral grade estimation.
Through this exercise, thirteen types of ML-based methods and techniques were identified that have been used for mineral resource exploration and estimation as listed below: ➤ Support vector machine (SVM) ➤ Support vector regression (SVR) -This is the order of discussion below • Artificial neural networks (ANN) • Adaptive neuro-fuzzy inference system (ANFIS) • Support victor regression (SVR) ➤ Local linear radial basis function (LLRBF) neural network ➤ Simultaneous perturbation artificial bee colony algorithm (SPABC) ➤ Back propagation (BP) ➤ Covariance matrix adaptation evolution strategy (CMAES) ➤ Particle swarm optimization (PSO) ➤ Naïve Bayes classifier (NBC) ➤ Radial basis function (RBF) ➤ Wavelet neural network (WNN) ➤ Random forest (RF) ➤ Regression kriging (RK) Among all these models and techniques, SVM, SVR, and ANN are the most commonly used and applied for MRE. Together they were found in 60% of the selected studies, as shown in Figure 3. Detailed information about which techniques were used in which study are given in Appendix C.
On the other hand, the non-ML methods mostly used in MRE are kriging, ordinary kriging, and inverse distance weight (IDW). Only 41% of studies (13 papers) compared the results of ML with non-ML methods in terms of mineral grade estimation. All of these studies concluded that ML models outperformed the non-ML methods except S-04, which concluded that ordinary kriging performed better than artificial neural networks. The ML methods which most frequently outperformed the other ML methods were SVM and SVR. A brief description of most the common ML techniques, including SVM, SVR, and ANN, are given in following sections.
The support vector machine is a supervised empirical machine-learning algorithm, based on statistical learning theory (Vapnik 1999). (SVM has recently been introduced in the field of mining and mineralization. The SVM is usually used for data classification and prediction; however, multi-class SVM can also be generated by combining multiple binary classifiers. It showed several unique advantages in a small data sample with nonlinear and multi-dimensional patterns recognition. The main objective of SVM is to locate a hyperplane that can separates data-points of one type from another. The best hyperplane is the one with the largest margin between two classes, and hence can separate the classes distinctively as shown in Figure 4. The maximum distance between two parallel hyperplanes results in the minimum classification error. SVM has been extensively used in several field of engineering, science, and natural languages. In the mining industry, it has recently been used for mineral classification (Patel, Chatterjee, and Gorgi, 2017), mineral prospectivity (Abedi, Norouzi, and Bahroudi, 2012), and automatic lithological classifications (Yu et al. 2012). Support vector regression differs greatly from other regression models. Whereas the other linear regression models try to minimize the difference between the estimated and the true value, SVR tries to fit the best line within a threshold value. In MRE, SVR tries to categorize all the estimation lines in two forms, those that pass through the threshold boundary and those that don't. The lines that do not pass the threshold boundary are not considered as the difference between the estimated grade value and the true grade value has exceeded the error threshold defined by ε (epsilon) as shown in Figure 5. On the other hand, the lines that pass are considered for a possible support vector to estimate the grade value at an unknown location.
Artificial neural networks is another strong machine-learning approach. This biologically inspired computational technique has wide applications in several science and engineering fields. A unique quality of ANNs is that they are able to create empirical relations between independent and dependent variables and extract the hidden variability and complex knowledge from training data-sets. The relations lies between independent and dependent variables can be built without assumptions about any mathematical depiction of the phenomena. ANN models have several benefits over regression-based ML methods, including their ability to deal with noisy data. An ANN model has thousands of interconnected artificial neurons made up of inputs and outputs. The input nodes receive the actual mineral grade values at known locations based on the internal weighting system and the neural network tries to learn the hidden patterns and produces the output, as shown in Figure 6.
An ANN model compares the actual output with what it was meant to produce, i.e., the desired output. The difference between both is corrected using back-propagation so the ANN works regressive, going from the final output node to the input nodes in order to fine-tune the weight of its connections until the variation between the actual grade value and estimated grade value produces the lowest possible error.
The types of data used for MRE estimation, regardless of technique, can be divided into three general categories: ➤ Exploratory boreholes ➤ Images ➤ Stream sediments. Twenty-three studies used exploratory borehole data, six used images photographs, and only one study utilized stream sediment data for the mineral grade estimation. The study S-13 used 3,500 exploratory boreholes and concluded that ML-based SVR is he most accurate technique for mineral grade estimation compared to the non-ML based ordinary kriging. Similarly, research study S-08, in which the authors used stream sediments, also concluded that ANN, which is another ML method, outperformed kriging, a non-ML method, when used for mineral grade estimation. The studies in which images were used did not compare the results of ML models with non-ML models.
In terms of publication domains, the studies were divided into the following three main categories: ➤ Minerals/mining ➤ Computer science ➤ Geoscience A total of seven studies were published in the field of minerals/mining, 12 in computer sciences, and 12 in the field of geosciences. This reveals that most of the publications fall into multidisciplinary applied domains. Another important aspect of this SLR is to identify the most applicable statistical/geostatistical technique used for the error assessment of the predicted mineral grades. More than 73% of the research studies used mean square error (MSE) and root mean square error (RMSE) as error assessment techniques, followed by the standard error (SE), and mean absolute error (MSE) with a total of 16%. The remaining 11% of the error assessment techniques include mainly generalization error, percentage of accuracy, co-efficient of determination, and out-of-bag error.
This review has found that most of the recent and advance ML methods and techniques such as deep convolution neural network, hierarchical convolutional deep maxout network, and hidden trajectory (generative) models, are seldom used within the field of mineral grade estimation. Therefore, researchers/ practitioners are encouraged to apply these techniques and test their applicability for mineral grade estimation.
In addition, researchers/practitioners are also encouraged to explore the other, non-tested, ML methods for mineral grade estimation. In order to become acquainted with the unexplored ML methods and to apply them in a more efficient way, researchers/practitioners should keep a close watch on the related disciplines like machine learning, deep learning, data science, and artificial intelligence, as these disciplines may provide ideas for new ML methods and techniques (Wen et al., 2012). Even though this investigation has found that the ML models are more accurate and perform better than the non-ML methods, the results of error assessment between ML and non-ML methods, and between different ML methods, are still inconclusive. Hence, it is strongly recommended that the scientific community to develop a common framework for evaluating the performance of different ML techniques, as well as against non-ML methods. The results of the studies may vary because of different data-sets and/ or different experimental designs.

Conclusion
This systematic literature review examined machine learning (ML)-based MRE models in terms of the type of ML methods or techniques, the error estimation of applied ML methods, comparison between different ML and non-ML methods, as well as ML methods with other ML methods. The extensive systematic literature review was based on research studies published in the period 1990-2019, with a total of 31 studies meeting the requirements of five research questions.
The key findings of the litrature review are that the significant ML methods applied for the MRE are, in order of application, SVM, ANN and SVR. Few studies actually compared the results of ML methods with non-ML methods for MRE. Those studies concluded that ML methods outperformed the non-ML methods in general. SVM and SVR are the most applied and tested ML methods, which yield much better results than other ML methods. Very few papers have been published in the fields of mineral and mining, whereas most of them were published in the computer and geosciences fields.
This review provides recommendations for researchers for future work as well as guidelines for practitioners. More research should be conducted detailing studies on the application of ML methods and drawing of conclusions in terms of their applicability, validation, and accuracy. Researchers should also develop a framework in terms of data usage and accuracy assessment against non-ML methods used for mineral grade estimation.
From this review, it is very clear that the application of ML methods in the industry for MRE are limited, and hence more studies should be conducted and analysis done in order to find the possible barriers to ML method applications. It is strongly recommended that ML models be used in parallel with non-ML (conventional statistical) models in the early stages of mineral grade estimations. After the error assessment and quality check, the validated ML methods can then be used for estimation of Mineral Resources. Moreover, it is advisable to consult researchers/experts from the fields of machine learning and/or data science in order to check the strengths and weaknesses of the potential ML methods and interpretation of results before and after application accordingly. Zhang, Y., Song, S., You, K., Zhang, X., and Wu, C. 2017. Relevance vector machines using weighted expected squared distance for ore grade estimation with incomplete data. International Journal of Machine Learning and Cybernetics, vol. 8, no. 5. pp. 1655-1666. Zhang, Z., Yang, J., Wang, Y., Dou, D., and Xia, W. 2014. Ash content prediction of coarse coal by image analysis and  List of research publications used in the systematic literature review after quality control and assessment

S-01
Reserve estimation using neural network techniques. Wu and Zhou, 1993 S-02 Data segmentation and genetic algorithms for sparse data division in nome placer gold grade estimation using neural network and geostatistics. Samanta, Bandopadhyay, and Ganguli, 2002 S-03 A comparative study of the performance of single neural network vs. adaboost algorithm based combination of multiple neural networks for mineral resource estimation. Samanta, Bandopadhyay, Ganguli, and Dutta, 2005 S-04 Comparing the predictive performance of neural networks with ordinary kriging in a bauxite deposit. Samanta, Ganguli, and Bandopadhyay, 2005 S-05 Comparative evaluation of neural network learning algorithms for ore grade estimation. Samanta, Bandopadhyay, and Ganguli, 2006 S-06 A machine vision approach to on-line estimation of run-of-mine ore composition on conveyor belts. Tessier, Duchesne, and Bartolocci, 2007 S-07 Genetic algorithm-based neural network learning parameter selection for ore grade evaluation of a limestone deposit. Chatterjee, Bandopadhyay, and Rai, 2008 APPENDIX A

Study ID Title Reference
S-08 Ore grade prediction using a genetic algorithm and clustering based ensemble neural network model. Chatterjee, Bandopadhyay, and Machuca, 2010 S-09 construction of a radial basis function network using an evolutionary algorithm for grade estimation in a placer gold deposit. Samanta and Bandopadhyay, 2009 S-10 Application of adaptive neuro-fuzzy inference system for grade estimation; case study, sarcheshmeh porphyry copper deposit, Kerman, Iran. Tahmasebi and Hezarkhani, 2010 S-11 Image-based quality monitoring system of limestone ore grades. Chatterjee, Bhattacherjee, Samata, and Pal, 2010 S-12 Radial basis function network for ore grade estimation. Samanta, 2010 S-13 Machine learning algorithms and their application to ore reserve estimation of sparse and imprecise data. Dutta, Bandopadhyay, Ganguli, and Misra, 2010 Publication domains of the selected research studies used for SLR