On Optimal Weighting Scheme in Model Averaging

Georges Nguefack-Tsague^1,

¹Department of Public Health, University of Yaounde I, Biostatistics Unit, Yaoundé, Cameroon

Cite this paper:
Georges Nguefack-Tsague. On Optimal Weighting Scheme in Model Averaging. American Journal of Applied Mathematics and Statistics. 2014; 2(3):150-156. doi: 10.12691/ajams-2-3-9

Abstract

Model averaging is an alternative to model selection and involves assigning weights to different models. A natural question that arises is whether there is an optimal weighting scheme. Various authors have shown their existence in others methodological frameworks. This paper investigates the derivation of optimal weights for model averaging using square error loss. It is shown that though these weights may exist in theory and depend on model parameters; once estimated they are no longer optimal. It is demonstrated using an example of linear regression that model averaging estimators with these estimated weights are unlikely to outperform post-model selection and others model averaging estimators. We provide a theoretical justification for this phenomenon.

Keywords:
model averaging model selection optimal weight square error loss model uncertainty

This work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

References:

[1]	Linhart, H. and Zucchini, W. Model selection, John Wiley and Sons, New York, 1986.
[2]	McQuarrie, A. D. R. and Tsai, C. L. Regression and time series model selection, World Scientific, Singapore, 1998.
[3]	Burnham, P. K. and Anderson, D. R. Model selection and multimodel inference, a practical information-theoretic approach, Second Edition, Springer-Verlag, New York, 2002.
[4]	Claeskens, G. and Hjort, N. L. Model selection and model averaging, Cambridge University Press, Cambridge, 2008.
[5]	Bancroft, T. A. On bias in estimation due to the use of preliminary tests of significance, Annals of Mathematical Statistics 15 1944, 190-204.
[6]	Breiman, L. The little bootstrap and other methods for dimensionality selection in regression: X-Fixed predictor error, Journal of the American Statistical Association 87 1992, 738-754.
[7]	Hjorth, J. Computer intensive statistical methods:Validation, model selection, and bootstrap, Chapman and Hall, London, 1994.
[8]	Chatfied, C. Model Uncertainty, data mining and statistical inference (with discussion), Journal of the Royal Statistical Society, series B 158 1995, 419-466.
[9]	Draper, D. Assessment and propagation of model uncertainty (with discussion), Journal of the Royal Statistical Society, series B 57 1995 45-97.
[10]	Buckland, S. T., Burnham, K. P. and Augustin, N. H. Model selection: An integral part of inference, Biometrics 53 1997, 603–618.
[11]	Zucchini, W. An introduction to model selection, Journal of Mathematical Psychology 44 2000, 41–61.
[12]	Candolo, C., Davison, A. C. and Demétrio, C. G. B. A note on model uncertainty in linear regression, The Statistician 158 2003, 165-177.
[13]	Hjort, N. L. and Claeskens, G. Frequentist model average estimators, Journal of the American Statistical Association 98 2003, 879-899.
[14]	Efron, B. The estimation of prediction error: covariance penalties and cross-validation, Journal of the American Statistical Association 99 2004, 619-642.
[15]	Leeb, H. and Pötscher, B. M. Model selection and inference: Fact and fiction, Econometric Theory 21 2005, 21-59.
[16]	Berk, R., Brown, L. D., Buja, A., Zhang, K. and Zhao, L. Valid post-selection inference, Submitted to Annals of Statistics, 2012.
[17]	Longford, N. T. Editorial: Model selection and efficiency-is ’which model ...?’ the right question?, Journal of Royal Statistical Society-A 168, Part 3 2005, 469-472.
[18]	Schomaker, M., Wan, A. T. K. and Heumann, C. Frequentist model averaging with missing observations, Computational Statistics and Data Analysis 54 (12) 2010, 3336-3347.
[19]	Zucchini, W., Claeskens, G. and Nguefack-Tsague, G. Model selection, In International Encyclopedia of Statistical Sciences, Editor: M. Lovric, Springer. Part 13, 830-833, 2011.
[20]	Liu, W. and Yang, Y. Parametric or nonparametric? a parametricness index for model selection, Annals of Statistics 39 (4) 2011, 2074-2102.
[21]	Nguefack-Tsague, G. and Zucchini, W. Post-model selection inference and model averaging, Pakistan Journal of Statistics and Operation Research 7(2-Sp) 2011, 347-361.
[22]	Nguefack-Tsague, G. An alternative derivation of some commons distributions functions: A post-model selection approach, International Journal of Applied Mathematics and Statistics 42(12) 2013, 138-147.
[23]	Nguefack-Tsague, G. On bootstrap and post-model selection inference, International Journal of Mathematics and Computation 21(4) 2013, 51-64.
[24]	Nguefack-Tsague, G. Bayesian estimation of a multivariate mean under model uncertainty, International Journal of Mathematics and Statistics 13(1) 2013, 83-92.
[25]	Nguefack-Tsague, G. Estimation of a multivariate mean under model selection uncertainty, Pakistan Journal of Statistics and Operation Research, 2014, forthcoming.
[26]	Nguefack-Tsague, G., Zucchini, W. and Fotso, S. On correcting the effects of model selection on inference in linear regression, Syllabus Review 2(3) 2011, 122-140.
[27]	Hoeting J., Madigan D., Raftery A. and Volinsky C. Bayesian model averaging: A tutorial, Statistical Science 4 1999, 382-417.
[28]	Wasserman, L. Bayesian model selection and model averaging, Journal of Mathematical Psychology 44 2000, 92-107.
[29]	Wan, A. T. K., Zhang, X. and Zou, G. Frequentist model averaging estimation: a review, Journal of Systems Science and Complexity 22 (4) 2009, 732-748.
[30]	Hansen, B. E. Least squares model averaging, Econometrica 75 2007, 1175-1189.
[31]	Hansen, B. E. Least squares forecast averaging, Journal of Econometrics 146 2008, 342-350.
[32]	Hansen, B. E. Averaging estimators for regressions with a possible structural break, Econometric Theory 35 2009, 1498-1514.
[33]	Hansen, B. E. Averaging estimators for autoregressions with a near unit root, Journal of Econometrics 158 2010, 142-155.
[34]	Wan, A. T. K., Zhang, X. and Zou, G. Least squares model averaging by Mallows criterion, Journal of Econometrics 156(2) 2010, 277-283.
[35]	Li, K. C. Asymptotic optimality for Cp, CL, cross-validation and generalized cross-validation: discrete index set, Annals of Statistics 15 1987, 958-975.
[36]	Liang, H., Zou, G., Wan, A. T. K. and Zhang, X. Optimal weight choice for frequentist model average estimators, Journal of the American Statistical Association 106 2011, 1053-1066.
[37]	Hansen, B. E. and Racine, J. S. Jackknife model averaging, Journal of Econometrics 167 (1) 2012, 38-46.
[38]	Zhang X., Zou G. and Liang H. Model averaging and weight choice in linear mixed-effects models, Biometrika 101 (1) 2014, 205-218.
[39]	Akaike, H. Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory, eds. B. Petrov and F. Csáki, Budapest: Akadémiai Kiadó 1973, 267-281.
[40]	Schwarz, G. Estimating the dimension of a model, The Annals of Statistics 6 1978, 461-464.
[41]	Leblanc, M. and Tibshirani, R. Combining estimates in regression and classification. Journal of the American Statistical Association 91 1996, 1641-1650.
[42]	Claeskens, G. and Carroll, R. J. An asymptotic theory for model selection inference in general semiparametric problems, Biometrika 94 2007, 249-265.
[43]	Yan, Y. Combining different procedures for adaptive regression, Journal of Multivariate Analysis 74 2000, 135-161.
[44]	Yan, Y. Regression with multiple candidate models: selecting or mixing?, Statistica Sinica 13 2003, 783-809.
[45]	Yan, Y. Combining forecasting procedures: some theoretical results, Econometric Theory 20 2004, 176-222.
[46]	Yan, Y. Aggregating regression procedures to improve performance, Bernoulli 10 2004, 25-47.
[47]	Zou, H. and Yang, Y. Combining time series models for forecasting, International Journal of Forecasting 20 2004, 69-84.
[48]	Yuan, Z. and Yang, Y. Combining linear regression models: when and how?, Journal of the American Statistical Association 100 2005, 1202-1204.
[49]	Shan, K. and Yang, Y. Combining regression quantile estimators, Statistica Sinica 19 2009, 1171-1191.
[50]	R Development Core Team. R: A language and environment for statistical computing, R Foundation for Statistical Computing, Austria, 2011.
[51]	Mallows, C. L. Some comments on Cp, Technometrics 15 1973, 661-675.