SAMPLING AND ANALYSIS PAPERS

 

Between-laboratory biases - same sample, different answers. Some guidelines

 

 

M. McWha

African Mineral Standards, South Africa

 

 

---

SYNOPSIS

Laboratory bias is a universal problem in all branches of analytical science. It results from differences in methods, techniques, equipment, and calibrations between laboratories. Notwithstanding that it is a property that can be measured by the use of standards or by inter-laboratory testing, laboratory bias remains a material issue for the mining industry.
There is extensive literature and clear guidelines on measurement from the International Organization for Standardization (ISO) and its worldwide federation of national standards bodies (ISO member bodies) and from the science of metrology. ISO guides cover the manufacture and use of standards. An accessible source of literature covering laboratory quality control comes from the public health sector, particularly pathology.
There is little doubt in financial reporting rules about the meaning of materiality. Public reporting guidelines are clear that material matters need to be disclosed for investors. Unfortunately there is currently no clear guidance what to do about biased assay results in mineral industry public reporting guidelines. This paper reviews the requirements for the manufacture and use of reference materials, public reporting requirements in the mining industry, and presents proposals to improve management and reporting of laboratory biases.

Keywords: laboratory bias, reference materials, ISO Guides, financial reporting, materiality.

---

 

 

Introduction

Different accredited analytical laboratories using the same analytical method, theoretically and ideally, should give the same results for homogeneous sample pulp splits. Unfortunately in practice, no two laboratories are the same. Differences are caused by many different versions of standard procedures, methods, temperatures, flux compositions, acid strengths, hot plate temperatures, extraction times etc. So it is common to find that results from different laboratories for the same sample are different. Which one is closest to the truth? Even a given laboratory is never exactly accurate (bias = 0), and its level of accuracy varies (hopefully a little only) from day to day and after each equipment calibration.

A traditional test for laboratory bias has been for a laboratory to submit splits of a sample pulp to a number of other laboratories for a consensus test ('round robin'). Laboratories may then also make up batches of homogenized pulp reject material to monitor their own 'batch' performance between round robins. Customers may also send a percentage of sample splits to secondary laboratories for confirmatory tests. Obvious problems with this simple practice have largely been overcome in recent times with an increased use of certified reference material's (CRMs) as control samples. The low cost and availability of modern grade- and matrix-matched CRMs makes it possible to use them routinely as control samples, allowing fast response times to problems once identified. Specific CRMs can be made for every type of sample taken in the different stages of mining and processing, covering all of the different key analytical and economic hurdles. The CRM manufacturing method ensures they can be used with confidence to compare results from a single laboratory against the results of many others.

Provided that a laboratory and the customer are both using correct CRMs for all the materials being analysed and they are both seeing correct CRM results (known values), then the laboratory will most likely also be reporting correct values for ordinary samples (unknown values). This best practice is widespread but not yet universal. Guidelines are available for the best and most cost-effective way to use CRMs in the wider science of analytical practice; but these are not yet universally adapted for the minerals industry. Public reporting guidelines need to be developed specifically for the minerals industry.

 

The International Organization for Standardization

Existing guidance on the manufacture, certification, and use of reference materials is in the form of written standards and guidelines that are produced by the International Organization for Standardization (ISO) and its worldwide federation of national standards bodies (ISO member bodies). ISO standards and guidelines are documents that provide requirements, specifications, guidelines, or characteristics that can be used consistently to ensure that materials, products, processes, and services are fit for their purpose. The ISO has published over 19 500 standards on 286 topics covering almost all aspects of technology and business (e.g. quantities and units, screw threads, food products, biotechnology, quality management, environmental management, risk management, currency codes, language codes etc.).

 

ISO guides on reference materials

The ISO working group for reference materials is the ISO Committee on Reference Materials (REMCO). Their primary term of reference is to establish definitions, categories, levels, and classification for all reference materials. The 34 countries contributing to REMCO's work include from the African continent South Africa, Kenya, and Libya.

Principal categories of reference materials that the guides are written for include those for chemical composition, biological and clinical properties, physical properties, engineering properties, and miscellaneous. Reference materials include split calibrants, which are pure standards used to calibrate instruments, and matrix CRMs, which contain an analyte in a sample. Fields for which reference materials are produced cover the full spectrum of agriculture, industry, and science and include aerospace, chemicals, construction materials, energy, foods, fuels, environmental, medicine, metallurgy, minerals, pharmaceuticals, and just about anything else that can be measured or tested.

The ISO Guides have to cover all of these, so they do not enter into any detail, but tend to be brief, ranging from 7 to 64 pages. They are copyrighted and must be purchased from national standards institutes or other licenced resellers. They are quite expensive (for what you get) and the full set is required.

The outline of their content below is the author's personal interpretation of the Guides and is purely meant to cover their scope and to help illustrate the topic of this paper. It is not intended to breach any copyright and it has not been validated by the ISO or any ISO-affiliated body.

1. ISO Guide 30. Terms and Definitions used in Connection with Reference Materials. This guide provides the basic definitions used in connection with reference materials, terms related to measurement and testing, and terms related to certification and issuance of reference materials

2. ISO Guide 31. Contents of Certificates and Labels. This guide shows all the items to be written in the table of contents of the certificate of the reference material, containing 26 items from introduction to annex. It advises that where several methods have been used to characterize the reference materials, these should be stated, also that 'A CRM and its certificate should never be parted' (3.1). The certified values and their uncertainties must be stated according to the procedures in the Guide on the Expression of Uncertainty in Measurement (GUM) and Eurachem. Note that Eurachem 2000 does not cover quantifying uncertainty in reference materials whose value assignment is determined by using consensus methods (1.2 on p. 3). This challenge is picked up by Ellison et al., 2001

3. ISO Guide 32. Calibration of Chemical Analysis and the Use of Certified Reference Materials. Covers the selection of calibration procedures in chemical analysis, calibration procedures, selection of CRMs, and the use of internal reference materials (RMs)

4. ISO Guide 33. Uses of Certified Reference Materials. CRMs should be used properly (effectively, efficiently, and economically), on a regular basis, preferably instead of in-house standards and preferably with a matrix matching the real samples. Covers statistical considerations, the role of CRMs in measurement science, the International System of Units (SI), the assessment of a measurement process, and the choice of CRM (appropriate concentration, matrix, form, and quantity)

5. ISO Guide 34. General Requirements for the Competence of Reference Material Producers. A reference material producer should be a technically competent body that is fully responsible for project planning and management, assignment of and decision on property values, authorization of property values, and issue of the certificate or other statements for the reference materials it produces (3.1). The second revision of Guide 34 set out all the general requirements within which a reference material producer has to demonstrate that it operates. The 2009 revision, in view of its use for the assessment of the competence of reference material producers applying for accreditation, made these requirements mandatory and in line with ISO/IEC 17025:2005/Cor.1:2006(E). It includes a cross-reference table to ISO 17025

6. ISO Guide 35. Certification of Reference Material -General and Statistical Principles. This guide is an application of GUM to cover peculiarities in the production of CRMs. Its purpose is to assist in understanding and developing valid methods to evaluate the properties of a reference material, including associated uncertainty and metrological traceability. RMs that comply with all steps in this guide are usually accompanied by a certificate and called CRMs. Guide 35 covers the treatment of property values based on a measurement campaign, particularly the required number of results, the treatment of data, uncertainty, and traceability. Establishing traceability is particularly complex (9.2.3) - for example, steps such as transformation are generally not traceable, therefore traceability can be assumed (10.1). In these cases comparison between different laboratories using the same method is possible and certification on the basis of agreement among independent measurement results may be used to support traceability, but not as a direct demonstration. The 1989 guide gave the minimum number of laboratories needed for a campaign to characterize a RM as 15(1989, 8.2.2). In 2003 this was revised to state that if there is a chance of obtaining statistically or technically poor results, the minimum number of laboratories is 10-15.(35:10.2.2). This guide also covers the number of units per laboratory (35:10.2.3), the treatment of stragglers and outliers, and characterization techniques using analysis of variance (A.6). It notes that the concept of determining property values of a RM based on agreement among methods and/or laboratories is based on at least two assumptions: that there exists a population of methods/laboratories that are equally capable of determining the characteristics of the RM to provide results with acceptable accuracy, and that the differences between individual results, both within and between methods/laboratories, are statistical in nature regardless of the causes (35:10). In practice a large number of laboratories are needed to overcome the effects of individual laboratory biases. The more complex the procedure; the larger the between-laboratory variances and the larger the number of participating laboratories needed to achieve a property value having a 'satisfying uncertainty' (35:10.2.2). For materials with a simple matrix and with property values comfortably above the detection limit, the minimum number of laboratories needed could be as low as 6 to 8 (35:10.2.2). Complicated materials, however, need 10 to15 laboratories to participate

7. ISO/CD Guide 80. Guidance for In-house Production of Reference Materials for Metrological Quality Control (QCMs). This guidance is intended for standards prepared for reasons of economy for in-house use. These could be prepared by the end user or by a commercial producer and are characterized during use.

 

Key definitions

1. 'Reference Material (RM). Material, sufficiently homogeneous and stable with respect to one or more specified properties, which has been established to be fit for its intended use in a measurement process'. 'Uses may include the calibration of a measurement system, assessment of a measurement procedure, assigning values to other materials and quality control' (ISO Guide 30). This definition implies that values have been assigned to the RM, but the RM need not necessarily have a certificate

2. 'Certified reference material (CRM). An RM, characterised by a metrologically valid procedure for one or more specified properties, accompanied by a certificate that provides the value of the specified property, its associated uncertainty, and a statement of metrological traceability'.

3. 'Quality Control Material (QCM). A material or substance made for 'in-house' use, one or more of whose property values are sufficiently homogeneous, stable and well established to be used for maintaining or monitoring measurement processes'. A QCM does not have formally assigned property values or uncertainties (ISO/CD Guide 80). QCMs are made to provide analytical laboratories with an economical means of checking their routine test procedures and results for precision. CRMs are still required to test accuracy.

 

Summary of the guidelines

The ISO REMCO Guides are written to cover all types of reference materials, not just the matrix-matched mineral reference materials widely used in the mining industry. Characterization of these is typically based on a measurement campaign, allocation of property values taking place on the basis of agreement among the independent measurement results. Because of this and also because of the transformation issue in many mineral analyses, these CRM values will not necessarily have direct traceability to SI units. In fact the certified value will not necessarily be the absolute value but it will be the best estimate, by that method, by the participating laboratories.

Two of the assumptions that allow characterization based on a measurement campaign are that there will be enough capable laboratories and that the results from each laboratory will be statistically compatible. Sufficient laboratories (up to 15 for difficult analytes) must be involved to overcome the effects of the 'inter-laboratory bias issue', and to achieve 'a property value having satisfying uncertainty'.

The data collected should be scrutinized with the aid of outlier treatment techniques to eliminate statistically and scientifically invalid results. This should lead to a very accurate measure for a given method, notwithstanding the underlying assumption that the inter-laboratory labs are reporting accurate results. However, both the number of laboratories involved and an amount of poor data may result in limits too broad for effective monitoring of a single laboratory or production process. Users are therefore advised to set their own quality objectives, (within the RM producers limits) based on their own control measurements.

 

Materiality

The first question is: does laboratory bias matter in the bigger scheme of mining? In financial reporting terms, is laboratory bias material?

Materiality is a concept in accounting. Essentially, an item is material if it is important enough to be mentioned in the financial statements, if the auditors need to be notified about it, or if the investors need to be notified. But how big is that? The definition of materiality leaves a lot open for interpretation. 'The omission or misstatement of an item in a financial report is material if, in the light of surrounding circumstances, the magnitude of the item is such that it is probable that the judgement of a reasonable person would have been changed or influenced by the inclusion or correction of the item' (FASB, 1980).

A decision on whether an item is or is not material could be a rule of thumb along the lines of whether it will impact more than five to ten per cent of net income (FASB, 1980). However, there are numerous circumstances in which misstatements below 5 per cent could be material (SEC, 1999). Materiality ultimately depends on the size of the item or error judged in the particular circumstances of its omission or misstatement.

 

Public reporting rules and guidelines

Maintenance of standards for reporting of mineral deposit estimates fall under the mandate of the Committee for Mineral Reserves International Reporting Standards (CRIRSCO). This is recognized as the key international organization representing the mining industry on reporting by the International Accounting Standards Board (IASB), the International Council on Mining and Metals (ICMM), and the United Nations Economic Commission for Europe (UNECE). Current CRIRSCO members include Australia, Canada, Chile, Russia, South Africa, the UK and Western Europe, and the USA. These countries maintain national reporting codes closely aligned with the CRIRSCO template.

The CRIRSCO's International Reporting Template for the Public Reporting of Exploration Results, Mineral Resources and Mineral Reserves (CRIRSCO Code) sets out the minimum standards being adopted in national codes worldwide. It does not set out to be a best practise guide; however, a checklist of assessment and reporting criteria is set out in Table I. The 'Quality of assay data and laboratory tests' is covered with a guideline that the '[n]ature of quality control procedures adopted (e.g. standards, blanks, duplicates, external laboratory checks) and whether acceptable levels of accuracy (i.e. lack of bias) and precision have been established' (CRIRSCO).

There is no further guidance on what are 'acceptable levels of accuracy'. This is left up to the Competent Person (or in Canadian terminology, the Qualified Person) responsible for the report. By default the financial concept of materiality should be used for guidance.

 

Laboratory quality control

Laboratory QC is designed to detect biased and imprecise assay results. Laboratories should be able to reproduce the same results over time and under different operating conditions. QC protocols should be designed to improve the quality of results coming from laboratories, maximize the detection of true errors, and reduce the occurrence of false errors.

Public health concerns have given us one branch of analytical science with an accessible literature, pathology. Multi-rule QC protocols as originally described by Westgard et al. (1981), now known as Westgard Rules, are widely used in clinical pathology laboratories. The idea was to combine individual control rules to minimize false rejections and maximize error detection based on the analysis of control samples (RMs or CRMs); using at least two control rules: one to detect random analytical error (precision), the other to detect systematic error (bias). This evolved into a programme that allows the user to specify an analyte, its imprecision, and the medically allowable error (Westgard, 1992) with the aim of reducing falsely rejected runs, lowering quality control expenses, and increasing laboratory efficiency. The critical parameters are the quality required for the test, the bias observed for the method, and the coefficient of variation (CV) observed for the method. In other words, the power of the protocol is that is can be adjusted according to the quality required for the test and the imprecision and inaccuracy observed for the method. Criteria can be set regarding what laboratory/method analytical bias is acceptable. Westgard recommends that the bias and the CV of control sample results should be reported, as should the detection rates for critical systematic errors and false rejections that the rules are set to detect.

 

Quantification of bias in mineral laboratories

Matrix reference materials are characterized using consensus methods, by sending samples out to a variety of government, commercial, and mining company laboratories. Production, testing and characterization are done according to ISO guidelines. Some manufacturers make the raw data received from the laboratories available. This allows customers to carry out their own statistical analysis of the results. It also allows quantification of mineral laboratory bias.

In the data-set shown in Figure 1, only 5 per cent of the laboratories fail completely. That is, they report mean values outside of the limits which are set at two standard deviations (2SD), which is what one would expect. This particular dataset comprises economic element analyses from recent AMIS standards. It contains data for 1553 sets of laboratory results, 8 results per analyte per laboratory, and 12 424 individual analyses. The data-set includes a variety of elements and methods:

1. Au, Pt, Pd by fire assay with Pb collection and inductively coupled plasma (ICP) finish

2. Au, Pt, Pd, Ru, Rh, and Ir by fire assay with NiS collection

3. Cu, Ni, Co, Fe, Ti, V, U, and rare earth elements (REE) by total digestion with ICP finish

4. Cu, Co, and Ni by X-ray fluorescence (XRF).

 

 

In the average gold reference material example below, although 95 per cent of the laboratories' z-scores are within 2SDs, there is space within the 2SDs for a 22 per cent bias between laboratories (from -2SD 1.59 g/t to +2SD 1.74 g/t).

Also, 78 per cent of the laboratories report means within single SD, within which space there is room for a 9 per cent bias (from -1SD to +1SD).

Either of these outcomes would rank as material in financial reporting terms.

 

Discussion

ISO guidelines are comprehensive and cover a vast field, but they are not all that accessible and are certainly not written to be used by mining professionals as best-practice guides. Financial reporting guidelines deal with materiality as a concept in a way that is open to a lot of interpretation. Public reporting codes and guidelines for exploration results, mineral resources, and mineral reserves only touch on laboratory bias and give no guidance on how to deal with it. The public health sector is more tightly regulated and the pathology laboratories have an extensive and accessible literature on laboratory quality control, including widely used guidance on the use of Westgard Rules, which can be adjusted to control QC expense and to increase laboratory efficiency. Examination of mineral laboratory bias shows it to be a serious issue with material consequences. Guidance is required on how to deal with its effects and the pathology laboratory protocols could serve as useful guides.

In mining it will make economic sense to spend more on QC for assays of materials where assay mistakes could have a major economic impact (resource drilling, concentrates, shipping products, samples close to cut-off). Sampling with low economic impact (geochemical exploration, tailings, grade control sampling) could have wider limits and lower QC costs.

The recommendations are:

1. The reporting of assay laboratory bias and precision should be mandatory in public reports

2. Samples with high economic impact should have tighter limits; this will attract more expensive QC and the opposite will be true for sampling with a lower economic impact

3. Reporting codes should contain guidelines on the limits of control sample biases for different types of sample (by their economic impact).

 

References

International Organization for Standardization. ISO Guide 30:1992. Terms and definitions used in connection with reference materials. Geneva.         [ Links ]

International Organization for Standardization. ISO Guide 31: 2000. Contents of certificates and labels. Geneva.         [ Links ]

International Organization for Standardization. ISO Guide 32. Calibration of chemical analysis and the use of certified reference materials. Geneva.         [ Links ]

International Organization for Standardization. ISO Guide 33: 2000. Uses of certified reference materials. Geneva.         [ Links ]

International Organization for Standardization. ISO Guide 34: 2000. General requirements for the competence of reference material producers as amended by Technical Corrigendum 1 of 15/11/04. Geneva.         [ Links ]

International Organization for Standardization. ISO Guide 35 2003. Certification of reference material - General and statistical principles. 3rd edn. Geneva.         [ Links ]

International Organization for Standardization. ISO/CD. Guide 80: 2010. Guidance for in-house Production of Reference Materials for Metrological Quality Control (QCMs). 1st edn. Geneva.         [ Links ]

ISO/IEC Guide 98-3: 2008. Uncertainty of measurement - Part 3: Guide to the expression of uncertainty in measurement (GUM1995). Geneva.         [ Links ]

JCGM. 2008. International vocabulary of metrology - Basic and general concepts and associated terms (VIM). Joint Committee for Guides in Metrology, Sèvres Cedex, France.         [ Links ]

Financial Accounting Standards Board (FASB). 1980, Statement of Financial Accounting Concepts No. 2, Qualitative Characteristics of Accounting Information, 132. Norwalk, Connecticut.         [ Links ]

Securities and Exchange Commission (SEC). 1999. Materiality. Staff Accounting Bulletin no 99. Securities and Exchange Commission 17. CFR Part 211.         [ Links ]

Committee for Mineral Reserves International Reporting Standards (CRIRSCO). 2006. International reporting template for the public reporting of exploration results, mineral resources and mineral reserves.         [ Links ]

EURACHEM/CITAC. 2000. Quantifying uncertainty in analytical measurement. Guide CG 4:2000. Eurachem Measurement Uncertainty and Traceability Working Group, Uppsala, Sweden.         [ Links ]

EURACHEM/CITAC. 2003, Traceability in chemical measurement. Eurachem Measurement Uncertainty and Traceability Working Group, Uppsala, Sweden.         [ Links ]

Ellison, S.L.R., Burke, S., Walker, R.F., Heydorn, K., MAnsson, M., Pauwels, J., Wegscheider, W., and Nijenhuis, B. te. 2001. Uncertainty for reference materials certified by interlaboratory study: Recommendations of an international study group. Accreditation and Quality Assurance, vol. 6, no. 6. pp. 274-277.

Westgard, J.O., Barry, P.L., Hunt, M.R., and Groth, T. 1981. A multi-rule Shewhart chart for quality control in clinical chemistry. Journal of Clinical Chemistry, vol. 27. pp. 493-501.         [ Links ]

Westgard, J.O. 1992. Charts of operational process specifications for assessing the precision, accuracy and quality control procedures for individual tests on a multitest analytical system. Journal of Clinical Chemistry, vol. 38. pp. 1226-1233.         [ Links ]

Westgard QC Inc. http://www.westgard.com [Accessed March 2013] ♦