Elicitation of L2 learners reading comprehension skills and strategies through cognitive diagnostic assessment

Karim Shabani

Abstract

Cognitive diagnostic assessment (CDA) as an innovative criterion-based approach to assessment has been the center of increasing attention in recent decades due to its inherent merits over its previous psychometric counterpart. Due to the lack of research on the post hoc analyses of classroom assessments to diagnose L2 learners’ strengths and weaknesses in the multidivisible skill of reading comprehension, and also its significance in high-stakes tests, this article offers the implementation of samples of high-stakes tests in the classroom environment due to its beneficial consequences and washback for the learners to increase their educational chances. Therefore, the present article is intended to shed light on the mental processes which the examinees went through in responding to the multiple choice questions of four reading comprehension passages, elicited from the recent Iranian University Entrance Exam 2015. Time-series design was conducted to forty female students aged between 16 and 17 years old who made the control and experimental groups in an EFL context at a high school to help them enhance their reading comprehension skills and strategies by giving proper diagnostic feedback and intervention. The analysis of data collected through triangulation of methods, namely interview, think-aloud, and self-assessment confirmed the usefulness of consistent cognitive diagnostic assessment (CDA) for both teacher and students. Moreover, the CDA analysis led to the development of a number of online reading comprehension strategies which are normally belittled in non-interactive assessments. Finally, the article suggests the use of consistent CDA as a dependable and trustworthy procedure to diagnose and enhance L2 learners’ reading comprehension processes.

References

1. Alavi, S. M., Kaivanpanah, S. & Shabani, K. (2012). Group dynamic assessment: An inventory of mediational strategies for teaching listening. Journal of Teaching Language Skills, 3(4), 27-58. 2. Alderson J. C. (1990). Testing reading comprehension skills. Reading in a Foreign Language, 6(2). 425-438. 3. Alderson, J. C. (2000). Assessing reading.UK: Cambridge University Press. 4. Alderson, J.C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. London/New York Press. 5. Alderson, J. C., Brunfaut, T., & Harding, L. (2014). Towards a theory of diagnosis in second and foreign language assessment: Insights from professional practice across diverse fields. Applied Linguistics 1-25. doi: 10.1093/applin/amt046. 6. Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources.Language Testing,8(1), 41-66. 7. Bachman, L. F. (2000). Modern language testing at the turn of the century: Assuring that what we count counts. Language testing, 17(1), 1-42. 8. Badrasawi, K. J. I., Kassim, N. L. A. & Daud, N. M. (2017). The effects of test characteristics on the hierarchical order of reading skills. Malaysian Journal of Learning & Instruction, 14(1), 63-82. 9. Baird, J., Hopfenbeck, T. N., Newton, P., Stobart, G. & Steen-Utheim, A.T. (2014). Assessment and learning: State of the field review. Oslo: Knowledge Centre for Education. 10. Baily, K.M. (1999). Washback in language testing. Educational Testing Service, Princeton, New Jersey. 11. Berg, B.L. (2004). Qualitative research methods. Boston: Pearson Press. 12. Brown, J.D. (2005). Testing in language programs. Prentice Hall. 13. Cauley, K. M., & McMillan, J. H. (2010). Formative assessment techniques to support student motivation and achievement.The Clearing House: A Journal of Educational Strategies, Issues and Ideas,83(1), 1-6. 14. Doe, C. D. (2013). Validating the Canadian Academic English Language Assessment for diagnostic purposes from three perspectives: Scoring, teaching, and learning (Unpublished Doctoral dissertation). Queen’s University:‏Kingston, Ontario, Canada. 15. Doe, C. (2015). Student interpretations of diagnostic feedback. Language Assessment Quarterly, 12(1), 110-135. 16. Farhady, H., Jafarpour, A., & Birjandi, P. (2009). Testing language skills. Tehran: SAMT Publishers. 17. Gallagher, N. (2000). DELTA’s key to the TOEFL test. McHenry, IL: Delta. 18. Gibbs, G. & D. Simpson, C. (2004-05). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1, 3-31. 19. Gipps, C. V. (1994).Beyond testing: Towards a theory of educational assessment. Psychology Press. 20. Gu, Z. (2011). Maximizing the potential of multiple-choice items for cognitive diagnostic assessment (Unpublished doctoral dissertation). University of Toronto. 21. Gu, L. (2014). At the interface between language testing and second language acquisition: Language ability and context of learning. Language Testing, 31(1), 111-133. 22. Hancock, C. R. (1994). Alternative assessment and second language study: What and why. Eric Documents: ED376695. 23. Harding, L., Alderson, J. C., & Brunfaut, T. (2015). Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles. Language Testing, 32(3) 317-336. 24. Herna´ndez, R. (2008). The challenges of engaging students with feedback. Paper presented at the 4th Biennial EARLI/Northumbria Assessment Conference, 27–29. August, postdam/Berlin. 25. Hernández, R. (2012). Does continuous assessment in higher education support student learning?Higher Education,64(4), 489-502. 26. Huff, K., & Goodman, D. P. (2007).The demand for cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education. 19-60. Cambridge: Cambridge University Press. 27. Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press. 28. Hughes, G. (2011). Towards a personal best: A case for introducing ipsative assessment in higher education.Studies in Higher Education,36(3), 353-367. 29. Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG-TOEFL (Unpublished doctoral dissertation). University of Illinois at Urbana–Champaign, Urbana, IL. 30. Jang, E. E. (2008). A framework for cognitive diagnostic assessment. In C. A. Chappelle, Y.-R. Chung, & J. Xu (Eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 117-131). Ames, IA: Iowa State University. 31. Jang, E. E. (2009). Demystifying a Q-matrix for making diagnostic inferences about L2 reading skills. Language Assessment Quarterly, 6(3), 210-238.‏ 32. Jang, E. E., & Wagner, M. (2014). Diagnostic feedback in language classroom. In A. Kunnan (Ed.), Companion to language assessment (pp. 693-711). Wiley-Blackwell. 33. Kiani, G. R., Alibakhshi, G., Akbari, R. (2009). On the consequential validity of ESP tests: A qualitative study in Iran. The Journal of Applied Linguistics, 2(1), 103-126 34. Kim, H. S. J. (2011). Diagnosing examinees' attributes-mastery using the Bayesian inference for binomial proportion: A new method for cognitive diagnostic assessment (Unpublished doctoral dissertation). Georgia Tech University. 35. Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263.‏ 36. Lee, Y. W. (2015). Diagnosing diagnostic language assessment. Language Testing, 32(3), 299-316. 37. Liu, H. H. T. (2014). The conceptualization and operationalization of diagnostic testing in second and foreign language assessment. Teachers College. Columbia University Working Papers in TESOL & Applied Linguistics, 14(1), 1-12. 38. Lougheed, L. (2003). How to prepare for the TOEFL test. Hauppauge, NY: Barron. 39. McNamara, T. (2006). Validity in language testing: The challenge of Sam Messick's legacy. Language Assessment Quarterly: An International Journal, 3(1), 31-51. 40. McCann, T., & Clark, E. (2003). Grounded theory in nursing research: Part 2—Critique. Nurse Researcher, 11(2), 19-28. 41. McDowell, L., K. Sambell, V. Bazin, R. Penlington, D. Wakelin, H. Wickes, & J. Smailes (2005). Assessment for learning: Current practice exemplars from the Centre for Excellence in Learning and Teaching. Newcastle: Centre for Excellence in Teaching and Learning. University of Northumbria. 42. Messick, S. (1996). Validity and washback in language testing. Language Testing, 13, 241-256. ‏ 43. Moss, C. M. (2013). Research on classroom summative assessment. In J. H. McMillan (Ed.), Handbook of research on classroom assessment(pp. 235–255). Los Angeles: Sage. 44. Nation, P. (2009). Teaching ESL/EFL reading and writing. New York, NY: Routledge. 45. Ozuru, Y., Rowe, M., O’Reilly, T., & McNamara, D. S. (2008). Where’s the difficulty in standardized reading tests: The passage or the question? Behavior Research Methods, 40(4), 1001-1015. 46. Ravand, H., Barati, H., &Widhiarso, W. (2013). Exploring diagnostic capacity of a high Stakes reading comprehension test: A pedagogical demonstration. Iranian Journal of Language Testing, 3(1), 12-37.‏ 47. Rost, D. (1993). Assessing the different components of reading comprehension: Fact or fiction? Language Testing, 10(1), 79-82. 48. Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective.Language testing,23(4), 441-474. 49. Rymniak, M. & Shanks, J. (2002). TOEFL CBT exam. New York: Kaplan. 50. Spolsky, B. (1990). Social Aspects of Individual Assessment. In de Jong, John H.A.L., & Stevenson, D.K. (Eds.), Individualizing the Assessment of Language Abilities, 3-15. Clevedon, Philadelphia: Multilingual Matters Ltd. 51. Sullivan, P.N., Brenner, G.A. & Zhong, G.L.Q. (2004). Master the TOEFL 2005. Lawrenceville, NJ: Thomson/Peterson. 52. Trotter, E. (2006). Student perceptions of continuous summative assessment. Assessment and Evaluation in Higher Education, 31(5), 505-521.