World Journal of Analytical Chemistry
ISSN (Print): 2333-1178 ISSN (Online): 2333-1283 Website: Editor-in-chief: Raluca-Ioana Stefan-van Staden
Open Access
Journal Browser
World Journal of Analytical Chemistry. 2017, 5(1), 1-8
DOI: 10.12691/wjac-5-1-1
Open AccessArticle

Ordinary Least Squares with Laboratory Calibrations: A Practical Way to Show Students that This Fitting Model may Easily Yield Biased Results When Used Indiscriminately

Juan M. Sanchez1,

1Department of Chemistry, University of Girona, Girona, Spain

Pub. Date: December 23, 2017

Cite this paper:
Juan M. Sanchez. Ordinary Least Squares with Laboratory Calibrations: A Practical Way to Show Students that This Fitting Model may Easily Yield Biased Results When Used Indiscriminately. World Journal of Analytical Chemistry. 2017; 5(1):1-8. doi: 10.12691/wjac-5-1-1


Analytical calibration using ordinary least squares (OLS) is the most widely applied response function for calibration in all type of laboratories. However, this calibration function is not always the most adequate and its indiscriminant use can lead to obtain biased estimates of unknowns. Students need to be taught about the practical requirements needed to obtain good results with OLS and when this fitting method is not accurate. Different experimental calibration curves were obtained in laboratory sessions using two common instrumental techniques: chromatography and atomic absorption spectrometry. After discussion seminars evaluating the data obtained by students, they were able to understand that linear fitting was not the most accurate model using atomic absorption spectrometry and a quadratic fitting provided most accurate estimates. Linearity was confirmed in chromatographic calibrations, but data presented heteroscedasticity, which is very common in calibrations done in chemical and biological analyses. A simple experiment was applied to show students how the use of the regression coefficients obtained by OLS with heteroscedastic data lead to highly biased estimates near the quantification limits of the calibration curve. The results obtained allowed to show students that, despite being widely used, OLS is not the most adequate fitting model to obtain accurate and precise results with many calibration methods routinely used in chemical and biological laboratories.

calibration curves linearity homoscedasticity biased results outliers

Creative CommonsThis work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit


Figure of 4


[1]  Zabell, A.P.R., Lytle, F.E. and Julian, R.K., “A proposal to improve calibration and outlier detection in high-throughput mass spectrometry”, Clinical Mass Spectrometry, 2, 25-33, 2016.
[2]  de Souza, S.V.C. and Junqueira, R.G., “A procedure to assess linearity by ordinary least squares method”, Analytica Chimica Acta, 552, 25-35, 2005.
[3]  Rozet, E., Ceccato, A., Hubert, C., Ziemons, E., Oprean, R., Rudaz, S., Boulanger, B. and Hubert, P., “Analysis of recent pharmaceutical regulatory documents on analytical method validation”, Journal of Chromatography A, 1158, 111-125, 2007.
[4]  Welz, B., Sperling, M., Atomic Absorption Spectrometry, 3rd ed., Wiley-VCH, Weinheim, 2008.
[5]  Wild, D., The immunoassay handbook: Theory and applications of ligand binding, ELISA and related techniques, 4th ed, Elsevier, Oxford, 2013.
[6]  Van der Berg, R.A., Hoefsloot, H.C.J., Westerhuis, J.A., Smilde, A.K. and van der Werf, M.J., “Centering, scaling, and transformations: improving the biological information content of metabolomics data”, BMC Genomics, 7, 142, 2006.
[7]  Raposo, F., “Evaluation of analytical calibration based on least-square linear regression for instrumental techniques: A tutorial review”, TRAC Trends in Analytical Chemistry, 77, 67-185, 2016.
[8]  Kiser, M.M. and Dolan, J.W., “Selecting the best curve fit”, LC·GC North America, 22, 112-117, 2004.
[9]  Johnson, E.L., Reynolds, D.L., Wright, D.S. and Pachla, L.A., “Biological sample preparation and data reduction concepts in pharmaceutical analysis”, Journal of Chromatographic Sciences, 26, 72-379. 1988.
[10]  Almeida, A.M., Castel-Branco, M.M. and Falcao, A.C., “Linear regression for calibration lines revisited: weighting schemes for bioanalytical methods”, Journal of Chromatography B, 774, 215-222, 2002.
[11]  Tellinghuisen, J., “Weighted least-square in calibration: What difference does it make?”, Analyst, 132, 536-543, 2007.
[12]  Zeng, Q.C., Zhang, E., Dong, H. and Tellinghuisen, J., “Weigthed least squares in calibration: Estimating data variance functions in high-performance liquid chromatography”, Journal of Chromatography A, 1206, 147-152, 2008.
[13]  Tellinghuisen, J., “Least squares in calibration: weights, nonlinearity, and other nuisances”, in: M.L. Johnson, L. Brand, eds., Methods in enzymology, Vol 454. Academic Press, San Diego, 259-285, 2009.
[14]  Gu, H., Liu, G., Wang, J., Aubry, A.F. and Arnold, M.E., “Selecting the correct weighting factors for linear and quadratic calibration curves with least-square regression algorithm in bioanalytical LC-MS/MS assays and impacts of using incorrect weighting factors on curve stability, data quality, and assay performance”, Analytical Chemistry, 86, 8959-8966, 2014.
[15]  Marques-Marinho, F.D., Reis, I.A. and Vianna-Soares, C.D., “Construction of analytical curve fit models for Simvastin using ordinary and weighted least squares methods”, Journal of the Brazilian Chemical Society, 24, 1469-1477, 2013.
[16]  Mulholland, M. and Hibbert. D.B., “Linearity and the limitations of least squares calibration”, Journal of Chromatography A, 762, pp 73-82, 1997.
[17]  Thompson, M. and Lowthian, P.J., Notes on statistics and data quality for analytical chemists, Imperial College Press, London, 2011.
[18]  Analytical Methods Committee, “Is my calibration linear?”, Analyst, 119, 2363-2366, 1994.
[19]  Jurado J.M., Alcázar, A., Muñiz-Valencia, R.; Ceballos-Magaña, S.G. and Raposo, F., “Some practical aspects for linearity assessment of calibration curves as function of concentration levels according to the fitness-for-purpose approach”, Talanta, 172, 221-229, 2017.
[20]  De Beer, J.O., De Beer, T.R. and Goeyens, L., “Assessment of quality performance for straight line calibration curves related to the spread of the abscissa values around their mean”, Analytica Chimica Acta, 584, 57-65, 2007.
[21]  Van Loco, J., Elskens, M., Croux, C. and Beernaert, H., “Linearity of calibration curves: use and misuses of the correlation coefficient”, Accreditation and Quality Assurance, 7, 281-285, 2002.
[22]  Reichenbächer, M. and Einax, J.W., Challenges in Analytical Chemistry Assurance, Springer, Heidelberg, 2011.
[23]  Massart, D.L., Vandeginste, B.G.M., Buydens, L.M.C., De Jong, S., Lewi, P.J. and Smeyers-Verbeke, J., Handbook of chemometrics and qualimetrics: Part A, Elsevier, Amsterdam, 1997.
[24]  Van Arendonk, M.D. and Skogerboe, R.K., “Correlation coefficients for evaluation of analytical calibration curves”, Analytical Chemistry, 53, 2349-2350, 1981.
[25]  Analytical Methods Committee, “Uses (proper and improper) of correlation coefficients”, Analyst, 113, 1469-1471, 1988.
[26]  Ellison, S.L.R., Barwick, V.J. and Duguid Farrant, T.J., Practical statistics for the Analytical Scientist: A bench guide. Royal Society of Chemistry, Cambridge, 2009.
[27]  Bysouth, S.R. and Tyson, J.F. “A comparison of curve fitting algorithms for flame atomic absorption spectrometry”, Journal of Analytical Atomic Spectrometry, 1, 85-87, 1986.
[28]  Ettre, L.S., “Nomenclature for chromatography”, Pure and Applied Chemistry, 65, 819-872, 1993.
[29]  US-EPA, SW-846 Test Method 8000C: Determinative chromatographic separations, Revision3, Section 11.5.1, 2003.
[30]  Araujo, P., “Key aspects of analytical method validation and linearity evaluation”, Journal of Chromatography B, 877, 2224-2234, 2009.
[31]  LCG/VAM, “Preparation of calibration curves: A guide to best practice”, 2003. Available: [Accessed Nov 16, 2017].
[32]  Dolan, J.W., “Calibration curves, Part I: To b or not to b?”, LC·GC North America, 27, 224-230, 2009.