by Karen R. Tellez-Trujillo, New Mexico State University
In Genre-based automated writing evaluation for L2 research writing, Elena Cotos provides a broad overview of theoretical and operational frameworks that reinforce research writing pedagogy for second language (L2) writers. Her audience is vast: teachers of research writing, researchers, developers of intelligent writing technologies, and scholars. The book relies on empirical evidence and theoretical discussions as it advocates the development of an Automated Writing Evaluation (AWE) program, defined by Cotos as a technology used to “complement instruction with computerized affordances that are otherwise unavailable or extremely time and labor-intensive” (p. 5). Through formative assessment of graduate student writing and discipline-specific feedback produced by a scaffolded computer-assisted learning environment (Harasim, 2012), Cotos presents genre-based approaches to academic writing for research writers and L2 research writers. The AWE presented by Cotos is an L2 research writing program that includes a model for designing and evaluating a corpus and genre-based Automated Writing Evaluation (AWE) technology prototype. She closely considers the research writing needs of novice scholars, uses a mixed methodological approach for empirical evaluation of the Computer-Assisted Language Learning (CALL) materials, and presents a sound resource for educators interested in exploring learning technologies that address writing challenges faced by L2 graduate student writers.
Genre-based automated writing evaluation for L2 research writing is well organized and comprehensive in its design. Seven chapters presented in two parts include sections on learning and teaching challenges in linguistics and rhetoric, automated writing evaluation, and conceptualization and prototyping of genre-based AWE. The second half of the book assesses the implementation and evaluation of genre-based AWE for L2 research writing. Cotos explores and evaluates the Intelligent Academic Discourse Evaluator (IADE) prototype she developed, and later discusses the cognitive, socio-disciplinary dimension and learning experiences for students who use the IADE. The analysis engine within Cotos’ IADE can be trained through John Swales’ move schema – establishing a territory, establishing a niche, and occupying a niche –which can lead to the identification of textual rhetorical structures. As a result, the IADE can provide feedback to students on their rhetorical structure, as well as information regarding the distribution of moves in their text in comparison to other moves presented in the student’s discipline. Cotos concludes by introducing the Research Writing Tutor (RWT), an extended “full-fledged corpus-based AWE program for L2 research writing pedagogy” (p. 214) capable of providing discipline-specific feedback attentive to the conventions of the genre for which the student is writing.
This book is ambitious: It provides an extensive list of figures and tables and an overview of technologies designed to assist students with writing challenges as it discusses and evaluates the design of the IADE system. A strength of this book is Cotos’ acknowledgement that research writers ought to write “like an authoritative member of the discourse community” (Boote & Beile, 2005, p. 18). Doing so is dependent upon the writer’s understanding of the genre for which they are writing and the expectations of the discourse communities to which they belong. Cotos emphasizes in the introduction, “For graduate students as aspiring scholars, research writing is also the foundation of their academic career and of the credibility of the scholarly dossier” (p. 2), thus strengthening the necessity for a graduate student to gain credibility and develop skills for effective written communication within their disciplinary community.
A point of consideration in this book is the controversial AWE technology. Enthusiasts have looked to it as “a silver bullet” or simple solution perceived to be immediately successful when applied to language and literacy development (Warschauer & Ware, 2006, p. 175). Cotos explains the possibility of AWE’s use in the writing classroom and in L2 research writing if “conceptualized at the earliest design stages” (p. 40) and not as a quick cure for a fundamental problem. Numerous examples of Automated Essay Scoring (AES) technologies are discussed in support of the author’s argument that AWE technology can serve the student well when discipline specific genre and validity concerns are addressed. Various instructional tools such as timely formative and summative feedback, data-analysis and reporting features, and teacher accessibility for setting of parameters are provided by AWE and, Cotos argues, can be utilized with less time and effort than when done free of technology.
Automated evaluation of writing remains controversial based on the lack of the “human factor.” Cotos notes the drawbacks and apprehensions about AWE and speaks to issues that have come to surface through research, recognizing that without attention to these issues, technology will suffer when put into action. And while the author’s attention to the drawbacks of AWE is admirable, leading scholar-teachers in Rhetoric and Composition/Writing Studies remain unconvinced that technology – no matter now sophisticated – can replace a human.
Paul Deane et al. (2013) suggested in their article, “Automated essay scoring in innovative assessments of writing from sources,” that human raters are needed to evaluate more complex factors such as critical reasoning, strength of evidence, or accuracy of information. Further, Brent Bridgman et al. (2012) presented scores on two high-stakes timed essay tests that use ETS’s e-rater® software: the Test of English as a Foreign Language (TOEFL) iBT and the Graduate Record Exam (GRE) in their article, “Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country.” The study revealed that e-rater scored writing by Chinese and Korean speakers more highly than did human raters, but gave lower scores to writing by Arabic, Hindi, and Spanish speakers. The authors hypothesized that these scoring differentials are attributable to stylistic differences human readers often accommodate but e-rater does not, and that some of these differences may be cultural rather than linguistic (Elliot, et al. 2013). Elliot et al. in “Uses and limitations of automated writing evaluation software” reminded readers that computational methods of assessing writing rely on cognitive and psychological models of language processing that can be at odds with theoretical understandings of writing as a rhetorically complex and socially embedded process that varies based on context and audience (2013).
Implications
In addition to the application of Swales’ moves schema, one of the themes of this book is reflection. Regardless of the specific method used to evaluate student writing, Cotos emphasizes the need for writers to continually reflect upon their writing moves and practices. She not only addresses the practice of graduate student research writing, but also works with three recursive processes to include planning, translating and reviewing, in addition to knowledge sub-stages (Flower et al., 1986; Hayes et al., 1987). While student writing is analyzed for moves by the IADE, the student is engaging in recursive practices and is improving metacognitive awareness of their writing. It is possible that L2 students utilizing the IADE will produce quality writing as a result of detailed feedback given by the IADE, however, outside the hands of the composition teacher style and cultural factors cannot be considered, thus providing the students incomplete feedback.
Whether used as an introduction to the various types of intelligent writing technologies or for the sake of research, Genre-based automated writing evaluation for L2 research writing is an ideal resource for teachers and researchers in search of an instrument to aid students in learning to write into their academic discourse communities. All too often in English departments and composition programs, L2 writing issues are left to TESOL specialists. This book helps all teachers of composition recognize that it is within their means to aid all students – including L2 students – with scholarly writing.
References
Boote, D.N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34(6).
Bridgeman, B., Trapani, C., and Attali, Y. (2012). Comparison of human and machine scoring of essays: Differences by gender, ethnicity, and country. Applied Measurement in Education, 25(1), 27-40.
Deane, P., Fowles, M., Baldwin, D., & Persky, H. (2011). The CBAL summative writing assessment: A draft eighth-grade design (Research Memorandum 11-01). Princeton, NJ: Educational Testing Service.
Elliot, N. Gere, A.R., Gibson, G., Toth, C., Whithaus, C., Presswood, A. (2013). Uses and Limitations of Automated Writing Evaluation Software, WPA-CompPile Research Bibiliographies. WPA-CompPile Research Bibliographies, 23.
Flower, L., Hayes, J.R., Carey, L., Schriver, K., & Stratman, J. (1986). Detection, diagnosis and the strategies of revision. College Composition and Communication, 37, 16-55.
Harasim, L. (2012). Learning theory and online technologies. New York: Routledge.
Hayes, J. R., Flower, L., Schriver, K.A., Stratman, J.F., & Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg (ed.), Advances in applied psycholinguistics (Vol. 2, pp. 176 241). New York, NY: Cambridge University Press.
Warschauer, M. & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 1-4.
