American Journal of Educational Research
ISSN (Print): 2327-6126 ISSN (Online): 2327-6150 Website: Editor-in-chief: Ratko Pavlović
Open Access
Journal Browser
American Journal of Educational Research. 2018, 6(1), 38-42
DOI: 10.12691/education-6-1-6
Open AccessArticle

Psychometric Validation of a Medical and Health Professions Course Evaluation Questionnaire

Kenneth D. Royal1, , Leigh Jay Temple2, Jennifer A. Neel3 and Laura L. Nelson1

1Department of Clinical Sciences, North Carolina State University, Raleigh, NC, USA

2Education Support Services, North Carolina State University, Raleigh, NC, USA

3Department of Population Health & Pathobiology, North Carolina State University, Raleigh, NC, USA

Pub. Date: January 11, 2018

Cite this paper:
Kenneth D. Royal, Leigh Jay Temple, Jennifer A. Neel and Laura L. Nelson. Psychometric Validation of a Medical and Health Professions Course Evaluation Questionnaire. American Journal of Educational Research. 2018; 6(1):38-42. doi: 10.12691/education-6-1-6


Education in the medical and health professions is considerably different than education in most higher education contexts (e.g., multiple instructors, clinical focus, student cohorts, etc.). Thus, despite a mature research literature relating to course and instructor evaluations in higher education, there largely is an absence of such literature in medical and health professions education. This study sought to contribute to the medical and health professions education literature by: 1) introducing a new instrument for evaluating course effectiveness; 2) demonstrating how to conduct a state-of-the-art psychometric validation study of an instrument’s psychometric properties; and 3) providing a framework for interpreting construct validity evidence. Results of the validation study indicated a considerable amount of construct validity evidence is discernible to conclude the instrument is capable of producing valid and reliable measures of course quality and effectiveness. Other medical and health professions educators are encouraged to adopt the instrument for use at one’s own campus, and/or replicate the validation procedures on other survey instruments.

medical education health professions education veterinary education evaluation measurement survey validation psychometrics course evaluation

Creative CommonsThis work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit


[1]  Smart, D.T., C.A. Kelley, and J.S. Conant. “Mastering the art of teaching: Pursuing excellence in the new millennium.” Journal of Marketing Education, 25(1), 71-78, 2003.
[2]  Kogan, J. R., & Shea, J. A. “Course evaluation in medical education.” Teaching and Teacher Education, 23, 251-64, 2007.
[3]  Billings-Gagliardi, S., Barrett, S. V., & Mazor, K. M. “Interpreting course evaluation results: insights from thinkaloud interviews with medical students.” Medical Education, 38, 1061-70, 2004.
[4]  Haidet, P., & Stein, H. F. “The role of the student-teacher relationship in the formation of physicians. The hidden curriculum as process.” Journal of General Internal Medicine, 21 Suppl 1, S16-20, 2006.
[5]  Royal, K. D., & Stockdale, M. R. “Are Teacher Course Evaluations Biased Against Faculty That Teach Quantitative Methods Courses?” International Journal of Higher Education, 4(1), 217-224, 2015.
[6]  American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association, 2014.
[7]  Royal, K. D. “Four tenets of modern validity theory for medical education assessment and evaluation.” Advances in Medical Education and Practice, 8, 567-570, 2017.
[8]  Royal, K. D. “Making meaningful measurement in survey research: A demonstration of the utility of the Rasch model.” IR Applications, 28: 1-16, 2010.
[9]  Rasch, G. Probabilistic models for some intelligence and achievement tests. Copenhagen: Danish Institute for Educational Research, 1960.
[10]  Bond, T.G., & Fox, C. M. Applying the Rasch Model. Fundamental measurement in the human sciences (3rd ed.). Routledge, New York, 2015.
[11]  Andrich, D. “A rating formulation for ordered response categories.” Psychometrika, 43, 561-573, 1978.
[12]  Linacre, J. M. WINSTEPS® (Version 3.92.0). Computer Software. Beaverton, OR:, 2017.
[13]  Wright, B. D., & Masters, G. N. Rating scale analysis: Rasch measurement. Chicago, IL: MESA Press, 1982.
[14]  Royal K. D., & Guskey, T. R. “On the appropriateness of norm- and criterion-referenced assessments in medical education.” Ear, Nose & Throat Journal, 94(7):252-4, 2015.
[15]  Royal, K. D., & Guskey, T. R. “The perils of prescribed grade distributions: What every medical educator should know.” Journal of Contemporary Medical Education, 2(4):240-1, 2014.
[16]  Linacre, J. M. “Dimensionality: contrasts & variances.” Available at:, Accessed on July 6, 2017.
[17]  Wright, B. D., & Masters, G. N. “Number of person or item strata: (4*Separation + 1)/3.” Rasch Measurement Transactions, 16(3):888, 2002.
[18]  Linacre, J. M. “Optimizing rating scale category effectiveness,” Journal of Applied Measurement, 3(1):85-106, 2002.
[19]  Wright, B. D., & Linacre, J. M. “Reasonable mean-square fit values.” Rasch Measurement Transactions, 8:370, 1994.
[20]  Royal, K. D. “The impact of item sequence order on local item dependence: An item response theory perspective.” Survey Practice, 2016. Available at: Accessed on July 6, 2017.
[21]  Marais, I., & Andrich, D. “Effects of varying magnitude and patterns of response dependence in the unidimensional Rasch model.” Journal of Applied Measurement, 9(2): 105-124, 2008.
[22]  Smith, R. M. “Fit analysis in latent trait measurement models.” Journal of Applied Measurement, 1(2): 199-218, 2000.
[23]  Messick, S. “Validity,” In R. L. Linn (Ed.) Educational Measurement (3rd ed., pp. 13-103). New York: Macmillan, 1989.
[24]  Royal, K. D., & Hecker, K. G. “Understanding reliability: A review for veterinary educators.” Journal of Veterinary Medical Education, 43: 1-4, 2016.
[25]  Royal, K. D., & Puffer, J. C. “The consequential validity of ABFM examinations.” Journal of the American Board of Family Medicine, 27(3): 430-431, 2014.
[26]  Linacre, J. M. “When to stop removing items and persons in Rasch misfit analysis?” Rasch Measurement Transactions, 23(4):1241, 2010.
[27]  Linacre, J. M. “Removing Rasch Misfit and Cleaning Windows.” Rasch Measurement Transactions, 23(4):1241, 2010.