Candidate Surveys on Program Evaluation: Examining Instrument Reliability, Validity and Program Effectiveness

Ruchi Bhatnagar^1, , Jihye Kim² and Joyce E. Many¹

¹Dean’s office, Georgia State University, Atlanta, Ga, USA

²University System of Georgia, Atlanta, Ga, USA

Cite this paper:
Ruchi Bhatnagar, Jihye Kim and Joyce E. Many. Candidate Surveys on Program Evaluation: Examining Instrument Reliability, Validity and Program Effectiveness. American Journal of Educational Research. 2014; 2(8):683-690. doi: 10.12691/education-2-8-18

Abstract

This self-study conducted by an urban college of education examined the effectiveness of its teacher education programs in emphasizing its social justice conceptual framework, as perceived by its teacher candidates. The study included development of an electronic survey based on the ten learning outcomes of the unit’s conceptual framework, and testing the internal consistency reliability and the construct validity of the survey items. The survey was administered to teacher education candidates who rated their program’s effectiveness in fulfilling the unit’s vision of preparing educators that are informed and empowered, committed, and engaged with students and their communities. We analyzed data from teacher candidates at mid or end points in their programs over two academic years. Our results indicated that the survey was a valid and reliable instrument for collecting candidates’ feedback on program improvement. Candidates’ feedback showed differences among the three programs in their emphasis on the conceptual framework. Additionally, there were some learning outcomes that were rated low in both years by the candidates, implying that some outcomes are being more successfully emphasized in the programs than others. These results were important feedback for the teacher education unit to move along the path of self-reflection and continuous improvement.

Keywords:
teacher education effectiveness candidate surveys validity reliability

This work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

References:

[1]	Agarwal, R., Epstein, S., Oppenheim, R., Oyler, C., & Sonu, D. (2010). From ideal to practice and back again: Beginning teachers teaching for social justice. Journal of Teacher Education, 61(3), 237-247.
[2]	Berry, A. (2004). A self-study in teaching about teaching. In. J. Loughran, M.L. Hamilton, V.K. LaBoskey, & T. Russell (Eds.), International handbook of self-study of teaching and teacher education practice (pp. 1295-1332). Dordrecht, The Netherlands: Kluwer.
[3]	Bhatnagar, R. (2011). Use of standards in assessment of student teachers: Purposes fulfilled, untapped potentials. Paper presented at the annual meeting of the American Education Research Association, New Orleans, LA.
[4]	Boyles, D. (2009). Considering Lorraine Code's ecological thinking and standpoint epistemology: A theory of knowledge for agentic knowing in schools? Philosophical Studies in Education, 40, 125-137.
[5]	Bronfenbrenner, U. (1986). Ecology of the family as a context for human development: Research perspectives. Developmental Psychology, 22, 723-742.
[6]	Cibulka, J. G. (2009). Improving relevance, evidence, and performance in teacher preparation. Education Digest: Essential Readings Condensed for Quick Review, 75(2), 44-49.
[7]	Cochran-Smith, M. (2004). Walking the road: Race, diversity and social justice in teacher education. New York: Teachers College Press.
[8]	Cochran-Smith, M., & Boston College Evidence Team (2009). “Re-culturing teacher education: Inquiry, evidence, and action. Journal of Teacher Education, 60, 458-468.
[9]	Coggshall, J. G. (2007). Communication framework for measuring teacher quality and effectiveness: Bringing coherence to the conversation. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from www.http://www.tqsource.org/communicationFramework.phpwww.
[10]	Council of Chief State School Officers. (2013, April). Interstate teacher assessment and support consortium InTASC Model core teaching standards and learning progressions for teachers 1.0: A resource for ongoing teacher development. Washington, DC: Author.
[11]	Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson,W. E. (2003). Advance mixed methods research designs. In A.Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209-240). Thousand Oaks, CA: Sage.
[12]	Darling-Hammond, L., & Bransford, J. (2005). Preparing teachers for a changing world: What teachers should learn and be able to do. San Francisco: Jossey-Bass.
[13]	Delandshere, G., & Arens, S. A. (2003). Examining the quality of the evidence in preservice portfolios. Journal of Teacher Education, 54, 57-73.
[14]	Delandshere, G. & Petrosky, A. (2004). Political rationales and ideological stances of the standards-based reform of teacher education in the U.S. Teaching and Teacher Education, 20, 1-15.
[15]	Delpit, L. (1995). Other people’s children: Cultural conflict in the classroom. New York: New Press.
[16]	Dewey, J. (1938). How we think: A restatement of the relation of reflective thinking to the educative process. Boston: D. C. Heath & Co.
[17]	Diez, M. E. (2010). It is complicated: Unpacking the flow of teacher education’s impact on student learning. Journal of Teacher Education, 61, 441-450.
[18]	Dooley, C. M., & Assaf, L. C. (2009). Contexts matter: Two teachers’ language arts instruction in this high-stakes era. Journal of Literacy Research, 41, 354-391.
[19]	Fairbanks, C. M., Duffy, G. G., Faircloth, B. S., He, Y., Levin, B., Rohr, J., & Stein, C. (2010). Beyond knowledge:
[20]	Exploring why some teachers are more thoughtfully adaptive than others. Journal of Teacher Education, 61, 161-171.
[21]	Feimen-Nemser, S. (2001). Helping novices learn to teach: lessons from an exemplary support teacher. Journal of Teacher Education, 52, 17-30.
[22]	Feiman-Nemser, S. (2012). Teachers as learners. Harvard Educational Press: Cambridge, MA.
[23]	Gay, G. (2010). Acting on beliefs in teacher education for cultural diversity. Journal of Teacher Education, 61,143-152.
[24]	Goe, L., Bell, C., & Little, O. (2008). Approaches to evaluating teacher effectiveness: A research synthesis. Washington D.C.: National Comprehensive Center for Teacher Quality.
[25]	Goodlad, J. (2008). Advancing the public purpose of schooling in teacher education. In M.Cochran-Smith, S. Feiman-Nemser, D. J. McIntyre, and K. E. Demers (Eds.) Handbook of research on teacher education: Enduring questions in changing contexts (3rd ed.), pp. 111- 116. New York: Routledge.
[26]	George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference 11.0 update (4th ed.). Boston: Allyn & Bacon
[27]	Grossman, P., Hammerness, K. M., McDonald, M., & Ronfeldt, M. (2008). Constructing coherence: Structural predictors of perceptions of coherence in NYC teacher education programs. Journal of Teacher Education, 59, 273-287.
[28]	Hammerness, K. (2006). From coherence in theory to coherence in practice. Teachers College Record, 108, 1241-1265.
[29]	Hollins, E., & Guzman, M. T., (2005). Research on preparing teachers for diverse populations. In M. Cochran-Smith and K. Zeichner (Eds.), Studying teacher education: The report of the AERA panel on research and teacher education (pp. 477-548). Mahwah, NJ: Lawrence Erlbaum Publishers.
[30]	King, E. J. (2000). White teachers at the crossroads: A moral choice for White teachers. Teaching Tolerance Magazine, 2000, 14-15.
[31]	King, E.J. (2006). On dysconciousness: An interview with Joyce E. King. Educational Studies, 40(2), 197-208.
[32]	King, P. M., & Kitchener, K.S. (2004). Reflective judgment: Theory and research on the development of epistemic assumptions through adulthood. Educational Psychologist, 39, 5-18.
[33]	Kline, p. (2000). The handbook of psychological testing (2nd ed.). pp.13. London:Routledge
[34]	Kozol, J. (2005). The shame of the nation: The restoration of apartheid schooling in America. New York: Crown Publishing.
[35]	Knapp, M.Copland, M. & Swinnerton, J. (2007). Understanding the promise and dynamics of data informed leadership. In P. Moss (Ed.) Evidence and decision making, 106th yearbook of the National Society for the Study of Education. (Pt. 1, pp. 74-104). Chicago: National Society for the Study of Education.
[36]	Koerner, M., Rust, F. O., & Baumgartner, F. (2002). Exploring roles in student teaching placements. Teacher Education Quarterly, 29, 35-58.
[37]	Korthagen, J., Loughran, J., & Russell, T. (2006). Developing fundamental principles for teacher education programs and practices, Teaching and Teacher Education, 22, 1020-1041.
[38]	Kornfeld, J., Grady, K., Marker, P., & Ruddell, M. (2007). Caught in the current: A self-study of state-mandated compliance in a teacher education program. Teachers College Record, 109 (8), 1902-1930.
[39]	LaBoskey, V.K., & Richert, A. E. (2002). Identifying good student teaching placements: A programmatic perspective. Teacher Education Quarterly, 29, 7-34.
[40]	Landow, G.P. (2006). Hypertext 3.0. Baltimore: The Johns Hopkins University Press.
[41]	Lieberman, A., & Pointer Mace, D. (2011). Making practice public: Teacher learning in the 21st century. Journal of Teacher Education, 61, 77-88.
[42]	Loughran, J. (2006) Developing a pedagogy of teacher education: Understanding teaching and learning about teaching. Routledge: London.
[43]	Louis, K. S. (2008). Creating and sustaining professional communities. In A. Blankstein, P. Houston, & R. Cole (Eds.), Sustaining professional learning communities (pp. 41-47). Thousand Oaks, CA:Corwin.
[44]	Millet, C.M., Stickler, L.M., Payne, D.G., & Dwyer, C.A. (2007). A culture of evidence: Critical features of assessments for post-secondary learning. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from https://www.ets.org/Media/Resources_For/Higher_Education/pdf/4418_COEII.pdf.
[45]	Milner, H. R. (2010). What does teacher education have to do with teaching? Implications for diversity studies. Journal of Teacher Education, 61, 118-131.
[46]	Noddings, N. (2002). The caring teacher. In V. Richardson (Ed.), Handbook of research on teaching (pp. 99-105). Washington, DC: American Educational Research Association.
[47]	Peck, C. A., Galluchi, C., & Sloan, T. (2010). Negotiating implementation of high-stakes performance assessment policies in teacher education: From compliance to inquiry. Journal of Teacher Education, 61, 451-463.
[48]	Plecki, M. L., Elfers A. M., & Nakamura, Y. (2012). Using evidence for teacher education program improvement and accountability: An illustrative case of the role of value-added measures. Journal of Teacher Education, 63, 318-334.
[49]	Pianta, R. C. (1998). Enhancing relationships between teachers and children. Washington, DC: American Psychological Association.
[50]	National Council for Accreditation of Teacher Education. (2008). Professional standards for the accreditation of teacher preparation institutions. Washington, DC: NCATE.
[51]	National Council for Accreditation of Teacher Education. (2010).Transforming teacher education through clinical practice: A national strategy to prepare effective teachers. Washington, DC: NCATE.
[52]	United States Department of Education. (2002). No Child Left Behind Act of 2001, pub. L. No. 107-110, 115 stat. 1425. Washington, DC: U.S. Department of Education.
[53]	Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge: Harvard University Press. (Original works published 1930, 1933, and 1935).
[54]	Wilson, S. M., Floden, R., &Ferrini-Mundy, J. (2001). Teacher preparation research: Content knowledge, gaps, and recommendations: A research report prepared for the U.S. Department of Education. Seattle, WA: Center for the Study of Teaching and Policy.
[55]	Wysocki, A.F. (2004). Opening new media to writing: Openings & justifications. In A.F. Wysocki, J. Johnson-Eilola, C.L. Selfe, & G. Sirc (Eds.), Writing new media: Theory and applications for expanding the teaching of composition (1-33). Logan, UT: Utah StateUniversity Press.
[56]	Zeichner, K. M. (2005). A research agenda for teacher education. In M. Cochran-Smith and K. M. Zeichner’s (Eds.), Studying teacher education: The report of the AERA plan on research and teacher education (pp. 761-766). Lawrence Erlbaum Associates: Mahwah, NJ.