Regression Analysis in Analytical Chemistry . Determination and Validation of Linear and Quadratic Regression Dependencies

The theory and practice of the extended statistical evaluation for linear and quadratic regression models used for calibration were presented. Two complete examples, solved step by step were presented as a short guide. The validation of regression dependences was based on classic F-Snedecor, Lack of Fit, FIUPAC and Mandel tests.


Introduction
In analytical chemistry practice, signal-concentration plots are often used where a concentration of analyzed samples is proportional to a respective analytical device's signal, e.g.absorbance, current, potential, peak height and peak surface.
To obtain reliable results, a calibration curve, i.e. a dependence of signal strength vs. concentration of substance in measured sample, must be drawn up first.The calibration curve is not a function dependence in mathematical sense, where one independent variable x corresponds to one and only one value of dependent variable y.It is a regression dependence, a 'slightly worse' than a functional dependence where x values may correspond to several values of y.In case of function dependence all points lie on the model curve and are indistinguishable from it.In practice, we have a set of data in the form of a matrix consisting of one x column and one or more y columns.The question, whether there is a relationship (correlation) between y and x variables has to be answered.Quantitative determination of such correlation's force is called the regression analysis.
The aim of this study is to give the reader a clear tool for an extended statistical evaluation of linear and quadratic regression models used for calibration.Therefore, in addition to the necessary minimum of theory, a number of specific examples are given.The correct choice between linear and quadratic regression models used for calibration is crucial in many biochemical tests such as Bradford or Smith protein determination methods.
Editorial offices of many analytical journals no longer accept the simple statistical analysis conducted during a calibration curve determination (straight-line equation ax + b, s a , s b or aa, ab and r 2 ).Using regression coefficient to verify the quality of correlation is not enough and can even lead to wrong conclusions as its value close to 1 can also be obtained for clearly curved dependences. 1,2For this reason, publications must present a full statistical evaluation including classic Fisher-Snedecor test, Mandel's F-test and Lack of Fit test for possible curvilinearity presence.This work pays special attention to these tests, espe-cially the second and third one mainly because they are poorly represented in literature.The existing literature on Mandel and Lack of Fit tests refers to a number of procedures, without giving clear examples solved step by step, with a clear final conclusion.Instead, reference works often contain references to statistical software, unavailable or unclear to the reader. 3,4In our approach of the problem, standard programs like Origin, Excel or Libre Office will suffice.

Calibration Curves as an Example of Regression Dependence. Function Dependencies vs. Regression Dependencies
Regression analysis aims to create a model describing a set of experimental x and y data and to predict unknown x values using created model.This generates some estimation errors.For a given set of data there may be a large number of regression models, but in order to obtain reliable results, the model showing the smallest deviation from experimental data should be chosen.This can be done intuitively by leading a line or curve between points or strictly mathematically.
For a good understanding of the statistical data processing idea and the terms such as modelling, estimation, calibration and calibration curves, the concept of functional dependence and regression dependence must be firstly distinguished. 5,6In case of function dependence y = f(x) there is a situation when every variable x i has assigned exactly one value of y i .In the case of regression dependence points do not lie exactly on the line and one independent variable x i can have assigned several values of y i .A comparison of function and regression dependences is shown in Fig. 1.
For the slope (a) and intercept (b) of the straight line equation y = ax + b, the confidence intervals aa and ab can be determined with the use of parameters generated by the Origin software.Unfortunately, the program outputs symbols which are inverse to those adopted in mathematics, namely

Confidence Interval and Determination Coefficient
Determination of the regression equation must be followed by the confidence intervals determination for a and b coefficients.This can be done with the use of respective standard deviations and t Student coefficient, t(~,df). 5Standard deviations are described by Equations ( 1) and (2) and can be calculated or obtained from Origin.
The t Student coefficient is given for the assumed level of significance ~and the number of degrees of freedom n -2, where n is the number of x values.
The confidence intervals of a and b coefficients are determined by Equation (3)   To evaluate the quality (strength) of the regression, correlation coefficient r or determination coefficient r 2 is used.
The r 2 coefficient ranges between 0 and 1.The closer the value is to 1, the better applied model describes a given set of experimental points.However, it is not a decisive criterion.
Standard deviation s a and s b alone, obtained using Origin are also not enough.Many people mistake s a and s b for confidence intervals, thus considering the former as a measure of a and b uncertainty.To obtain the actual confidence intervals, s a The confidence interval, as a uncertainty measure of determined equation, depends significantly on the number of measurement points via t Student parameter.It pays off to have 8 or even 9 of them, because then the confidence interval is significantly narrower.However, starting from n = 10, t Student begins to decrease only slightly with and therefore there is no real need to further increase the number of measurement points.
Based on the basic regression equation y 1 , and changing its a and b values by respective aa and ab according to the rule: (aÀ)(bÀ), (aÀ)(bÕ), (aÕ)(bÀ), (aÕ)(bÕ) four boundary regression equations (5) were determined: The resulting y 2 and y 5 boundary equations determine the widest range of error, i.e. confidence interval in graphic form for the whole model dependence.Thus, the value of x acquired from the model dependence also has its confidence interval ax, which should also be taken into account.The measured value of y and the basic regression equation with its confidence intervals (Fig. 2.) makes it possible to determine an unknown x value (e.g.concentration).

Summary of
bol indicates an estimated value read from the regression equation).If aa = 0 and ab = 0, we recognize a function dependence y = ax + b, which confirms that the function dependence is a special case of regression dependence, when all confidence intervals are zero.
The value read from the regression equation for given x, is always an estimated value labelled as $ y.The y i -$ y i difference, or precisely its square, is essential for the least squares method.

Least Squares Method
The above analysis was based on the Origin program without specifying the fact that it uses the least squares method to determine the regression curves (Figs. 1 and 2).This method is an optimization of a model dependence, where the criterion for optimization is to minimize the sum of model's deviations from experimental points, which can be visualized as a sum of square areas 6 marked by dotted lines in Fig. 3, a solid line denotes modelexperiment difference.Please note that the least squares method applied in analytical chemistry assumes no error in the concentration of the standards (x) and that the only variable is the signal (y).
If a function y = f(x) is drawn through a set of measurement points x i , y i , then each x value will be matched with two values: measured (y) and estimated $ y.The model, i.e. the regression equation, may be linear or curvilinear and expressed by an appropriate function.
For the linear regression y = ax + b, optimization criterion is to minimize the sum S(a,b): where $ yi represents the estimated value based on the regression equation, and n represents the number of measured x points (in the case of multiple yi measurements for each point x, average values y should be used).
Equation ( 7) is a function of two variables, a and b.Determination of such function's minimum is possible by finding a point at which the partial derivatives for all its variables are equal to zero.Hence, the system of equations ( 8) and its rearranged form (9): From the system (9) stem formulas for the a and b coefficients of a linear equation: The method used is sensitive to presence of significantly deviating points in the dataset.Each of these will deviate outlined function toward its value, weakening the fit.Therefore, before using this method, any gross errors must be eliminated from the dataset, with the use of appropriate mathematical criterion. 6

Least Squares Method -Example
Spectrophotometric method was used to measure samples containing known concentration of albumin (Bradford method) and glycine (ninhydrin method).The experiment was performed three times and the obtained experimental data are given in Table 2.
From formulas (1), ( 2) and ( 10  For Equation ( 12), r 2 coefficient is equal to 0.9998.Its value being so close to 1 suggests a nearly perfect regression equation.However, in light of the current requirements of analytical journals, even almost perfect r 2 value does not constitute enough evidence of the suitability of the adopted model.It is still required to provide the results of additional statistical tests. 7,8s was stated before, the r 2 coefficient cannot reliably indicate whether the adopted model is appropriate or not.To decide, statistical analysis should be performed on the adopted model.One of the statistical tests used for this purpose is Lack of Fit test, which can detect mismatches between the data and the adopted regression model.Now, the regression model y ij = ax i + b + É ij ; {(x i , y ij ): i = 1,...,n; j = 1,...,c} is tested for the case where each of x i matches with at least c i = 3 y values (Table 2) for the data, where n is the number of measurements as previously used and c is the number of repetitions of the measurement for a given x.To consider that linear model fits the data, a null hypothesis H 0 : y ij = ax i + b must be maintained.To test it one has to calculate the sum of squares due to errors for the full (13) and reduced (14) models and then subtract them from each other (15).

Lack of Fit Test
Results obtained from Equations ( 13) and (15) should then be divided by their respective degrees of freedom ( 16), (17) which leads to Equations ( 18) and (19).The final result of this test is the F LoF value (20).
To decide whether to accept or reject the null hypothesis posed at the beginning of the test, the calculated F LoF value (20) should be compared with the critical value F LoF # (a; df SSLF ; df SSEfull ) taken from F distribution tables.If for the assumed level of significance a, the inequality F LoF > F LoF # is true, the linearity hypothesis H 0 : y ij = a xi + b must be rejected.

Lack of Fit -Example
Lack of Fit test requires a minimum of three y values for each x.Therefore, in the example the original experimental values were used instead of the average ones (Table 2).The data were introduced consecutively into the formulas ( 13), ( 14 As one can see, the inequality F LoF > F LoF # is true for albumin data.Thus, the null hypothesis of good linear model fit has to be rejected, and the alternative hypothesis stating that the linear model for the albumin data exhibits a mismatch should be accepted. In the case of glycine data an opposite situation can be observed.F LoF < F LoF # , thus there is no basis to reject the null hypothesis of linear model fitting the experimental data. 9,10his test involves separate determination and comparison of the linear and quadratic effect's significance.The linear effect E lin , which means that part of total variability of y, that can be described by the accepted model $ y = ax + b, results from the expression (21) where y means the average value of the whole population of y.However, if the linear model is replaced with the quadratic one, i.e. $ y = ax 2 + bx + c, then the part of the total variability of the dependent variable, described by the parabola, results from expression (22), which is almost identical to Equation ( 21), with only one element different.

Classic Fisher-Snedecor Test
The quadratic effect E q of the dependent variable's total variation is the part that can be further explained by replacing the linear model with the quadratic model.In order to obtain a formula for this effect, expression (21) has to be subtracted from expression (22) resulting in Equation ( 23): This test defines the significance of the both effects as a ratio of said effect to its residual variance.The value of the residual variance describes this part of the dependent variable's variability that remains after deducting the variability described by the independent variable.It can be calculated by dividing the sum of deviations squares by the number of degrees of freedom.The number of degrees of freedom is equal to the number of independent variables minus the polynomial degree plus 1: df = n -(k + 1).
To ease the calculation, Equation (24) can be simplified by inserting an average value y.
Lastly, the obtained linear and quadratic effect significances have to be compared with critical values F # (~; 1; n -k -1) taken from F-distribution tables.If the significance of the effect is less than corresponding critical value, given effect should be considered as not significant.The more the F-value is greater than the F # , the more significant the model is.Preferably F should exceed F # by one or more orders of magnitude.
This test allows the possibility of having positive results for all tested models.Unlike other tests that give 'zero-one' like answers, the classic F-test results shows quantitative measure of each model's contribution in the data description.Therefore, despite its limited precision, it is a useful test in the evaluation of regression models.

Fisher-Snedecor Test -Example
The classic F-Snedecor test does not require multiple measurements for each independent variable.Moreover, in the case of multiple y data, y ij values should be averaged beforehand.Therefore, both albumin's linear and quadratic regression models and their determination coefficients were obtained using the averaged data from Table 2.
At first, linear and quadratic effects and residual variance for both models were calculated.Afterward, the significance of both effects was determined.

By comparing values of F lin and F lin
# it can be noticed that F lin exceeds the critical value by two orders of magnitude.This indicates high importance of this effect and could be a basis for accepting the linear model.However, the value of quadratic model's significance exceeds its critical value by two degrees of magnitude as well.Hence the conclusion that the quadratic effect is an important contribution and cannot be ignored.This situation is a case where positive results are obtained for both effects.
For comparison, the same test was performed for glycine data (Fig. 5).
All the values were calculated same way as for albumin data.
Significance of linear model F lin exceeds the critical value F lin # by four degrees of magnitude, which states greatly in favour of linear model.Moreover, the significance of quadratic model F q is lower than its critical value F q # .Such results provide a strong basis for completely rejecting the significance of the quadratic model and using the linear regression model instead.

F-Snedecor Test Modification by IUPAC and Mandel
To prevent the possibility of an inconclusive the classical F-Snedecor test results, the International Union of Pure and Applied Chemistry (IUPAC) adopted its modified version based only on residual variances of both linear and quadratic model. 10odified test checks whether the variance explained by the quadratic model is greater than the variance explained by the linear model.The null hypothesis H 0 : 'the variance explained by the additional term is not different from the residual variance' is formulated.Thus, it is assumed that the quadratic model is not significant.Test equation has the form of Equation (28): The modified test involves comparing the obtained value F IUPAC with the critical value F IUPAC # (a; 1; n -3).If the obtained value is greater than the critical value (exactly opposite to classic F-test), the null hypothesis has to be rejected in favour of the alternative hypothesis: the variance explained by the quadratic term is larger than the residual variance.That result would imply the need for a quadratic model.
The F IUPAC test is simple and easy to understand.It does not consider directly the effect of degrees of freedom, which may call into question its usefulness in most cases where the number of calibration points is relatively low.For this reason, in 1964 a chemist and statistician, J. Mandel, suggested an improved version of the test.It was summarized as a comparison of the residual standard deviation of the linear model with that of the nonlinear model. 11Unlike IUPAC, Mandel defined the F-test not as a subtraction of the variances of the two models, but as a subtraction of the sum of squares of the linear and quadratic fits, divided by the difference of their degrees of freedom.
Equation (29), can be simplified to a form similar to one specified by IUPAC (30): As can be seen from ( 28) and (30), the only difference between the IUPAC and Mandel approach is that the latter one includes respective degrees of freedom.This results in a more detailed outcome, particularly in the smaller amount of data available, wherein the differences between both tests are the most visible.
Mandel test, as an IUPAC modification, compares the test to the same critical value F IUPAC # = F M # (a; 1; n -3) and has the same condition for accepting or rejecting the null hypothesis.
Due to the use of degrees of freedom in Mandel's equation, it so happens that the result of the test differs sufficiently from IUPAC version.As a consequence, when both results are aligned with the same critical value, the obtained conclusions can be opposite.In such a situation one should accept the Mandel's F-test result as a more accurate one.

5
Figure 4 Linear and quadratic regression models for albumin calibration data points (Table 2), determined by Bradford method.2), determined by Smith method.

IUPAC and Mandel Tests -Example
To concretize the inconclusive results of the F-Snedecor test, both datasets (Table 2) were analyzed using F IUPAC and F Mandel tests.The results of all F IUPAC calculation steps are shown below.
As it can be seen for albumin data, the F IUPAC value is three times bigger than its critical value F IUPAC # .This result indicates the need of quadratic model for albumin data.
In turn, test for glycine data showed no basis for quadratic model usage since F IUPAC < < F IUPAC # .At this point one would say that the situation looks straightforward.However, F IUPAC test, by not including the respective degrees of freedom in its equation, can give misleading results for some datasets 12 This is why even though the situation may look clear, a wise move is to conduct the more accurate form of this test, namely Mandel test.For the same set of data, Mandel test results are shown below: Since for albumin data F M > F M # and for glycine data F M < F M # , the conclusions of Mandel and F IUPAC tests are alike.This confirms that albumin data requires quadratic model to be used while there is no need to do that for glycine data.

A Summary of Respective Statistical Tests
For clarity, the results of the four tests application were gathered in Table 3.
As shown in Table 3, for glycine data the linear model was consistently confirmed by all four test results and there is no justification for using the quadratic model.
Albumin data do not give such consistent results.Only three of four conducted tests, namely Lack of Fit, F IUPAC and Mandel tests, stated in favour of quadratic model while classic F-Snedecor test showed a draw.Even though this is enough to indicate the need for quadratic model, as the linear one is not sufficient in case of this dataset.
As it can be seen above, the F M value for albumin is considerably higher than F IUPAC for the same data.This shows how big of a difference makes including degrees of freedom, mainly for the cases with fewer data points.In such cases, where values given by both tests are close to the test's critical values, one has to evaluate the numerical values obtained from the tests, taking their force into account.Also, one could consider redoing the calibration procedure and adding more experimental points.Now that the statistical tests are done and the quadratic model was chosen, one could question how greatly the choice of the model impacts the outcome of an experiment that utilizes it to get results.To show that we will once more use albumin data (Table 2, Fig. 4), but this time we will be using the model rather than creating it.Albumin data points were obtained experimentally, hence their (x, y) values are certain.Therefore, one can compare those values with the x values obtained from both models to see how much a calculated value can deviate from an actual, measured value.The results are shown in Table 4.
One can easily see that the values attained from linear model deviate in general from the actual values by almost a full unit of concentration.Therefore using this model could lead to a significant error in the calculated results.On the other hand, the values obtained from the quadratic model deviate in general just by a quarter of a full unit of concentration, which means that this model is about four times less likely to cause result errors.
As much as the quadratic model is more precise in this situation it also comes with some downsides which cannot be forgotten.To properly use this regression model, one has to remember that its sensitivity is not constant along the curve as it is for linear model.Its value is equal to regression model's first derivative dAbs dC = -14.27• 10 -4 x + 0.03632.This can be easily shown by testing the model for small absorbance changes.For small absorbance range, e.g. from 0 to 0.1, a little change, DAbs = 0.05, corresponds to an equally small change, DC = 1.408 in calculated concentration, whilst the same change DAbs = 0.05 for final absorbance range from 0.400 to 0.500 corresponds to a much bigger change, DC = 4.806.It is visible that at the end of the calibration curve the model is four times less sensitive on measured factor (Abs) than at its beginning.As an effect, expected C result, obtained from final range of the model, is loaded with much greater error.If possible, one should work in the part of quadratic model where the sensitivity is still fairly high what can be achieved, e.g. by respective dilution.It can be assumed that said region is maintained up to the point where the respective derivative is greater than the linear dependence model's slope, i.e. 0.02205.The critical point obtained by simple calculation is at C = 9.948 »10 mg mL -1 , which remains in good agreement with an intuitive observation (Fig. 4).It means, that in considered case one should use the region up to C = 10 mg mL -1 to obtain reliable results.In conclusion it can be said that the presented and applied four statistical tests are useful and necessary.They also, through positive feedback, lead to an improved experimental procedures as they oblige researchers to pay more attention to the layout of their experiments.The validation of the adopted regression model adequacy should be based on several rather than one of the provided statistical tests which give a complete picture of reality.All of the discussed tests can be easily done using only basic spreadsheet programs or even just by using a calculator.Therefore, the presented statistical analysis of linear and quadratic regression models is commonly available and should be performed whenever it is possible.

5 Figure 2
Figure 2 Confidence interval in graphic form for the linear model curve.The basic regression equation y 1 is overlapped with boundary regression equations y 2 and y 5 .The latter equations are the result of determined aa and ab confidence intervals.Numerical data were taken from Fig. 1 and regression equations (5).

Figure 3
Figure 3 The essence of a linear model optimization for experimental data in Least Squares Method.The model-experiment differences are indicated by solid lines.

Figure 5
Figure 5 Linear and quadratic regression models for glycine calibration data points (Table2), determined by Smith method.
).Moreover, the nature of the parameter 'Error ' for A and B has to be clarified: they are just standard deviations s a and s b .Origin program outputs also the standard devia-RESEARCH ARTICLE R

Table 1
Explanation of Origin symbols concerning regression analysis.

Table 4
Output comparison of linear and quadratic models.The reference material was taken from Table2for albumin calibration points.The x values for quadratic model were calculated with the use of Wolfram|Alpha software; link https://wolframalpha.com/. ^'