Communities, stakeholders, and quality in assessment: A review of Sally O’Hagan’s Variability in Assessor Responses to Undergraduate Essays: An Issue for Assessment Quality in Higher Education
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG.
By Joe Cirio, Florida State University
In light of the increased growth of linguistic diversity in Australian universities, Sally O’Hagan’s book, Variability in Assessor Responses to Undergraduate Essays, seeks to explore how a more linguistically diverse student population affects assessment practices for disciplinary writing. The increased growth of diversity, as O’Hagan points out, poses a challenge to the quality and fairness of assessment—particularly, it poses a challenge to “consistency of assessment criteria and standards” (p. 11). But these challenges are only exacerbated by the lack of consensus on the meaning of quality within a discipline, and, moreover, the fact that assessors often rely upon “tacit knowledge of close communities of academics” (p. 17). This context is what ground her empirical study of assessor behaviors when responding to essays from native-speakers (NS) and non-native-speakers (NNS) of English.
Both Chapter One and Two contextualize her study: the former chapter lays out the immediate context of the Australian university; the latter details the scholastic conversations with which her study intersects. She starts by describing the ways in which assessors vary in both marks and the evaluative judgments in support of those marks. As noted by O’Hagan, a multitude of factors can contribute to variability in scores. To rectify this variability, institutions have begun to implement policies to encourage transparency and clear communication about assessments. However, O’Hagan writes, “it has been argued that the explicit specifications of criteria and standards does little to improve the quality of assessment in the absence of supportive measures” (p. 37). She points to the fostering of genuine communities of practice as a way to ground disciplines in sharing an understanding of quality, criteria, and standards for assessment. The concluding section of Chapter Two reviews relevant research into verbal reporting methodologies, the central method used by O’Hagan to understand assessor behaviors. While verbal reporting methodologies do not “directly represent cognitive processes” (p. 76), they contribute an approach to assessor behavior that may provide a different data set to uncover implications not yet realized. Other research into assessor behaviors on second language writers—most recently in an edited collection by Cox and Zawacki (2014)—have used methods such as interviews (Zawacki & Habib, 2014; Ives et al., 2014; Dan, 2014), surveys (Ives et al., 2014), or analysis of student examples (Lancaster, 2014).
Collecting data from an academic department at a major university in Australia, the study hinged on ten assessors—nine tutors and one instructor—who would offer verbal reports as they assessed ten student essays (half NS and half NNS—this information was not made available to assessors). Chapter Four walks through the statistical analyses of three sets of data that were collected: the marks assigned to the essays, the features of the essay that prompted an evaluation, and the evaluative judgments made by assessors. The conclusions of the study can be distilled into four points: NNS essays (1) received lower marks, (2) with more negative comments, and (3) with comments more often directed toward mechanics. But (4) there was also an increased range of comments for NNS essays indicating a lack of consensus about how assessors, within the same discipline, respond to writing of second-language writers (p. 207).
The remaining chapters of O’Hagan’s book point to improvements in assessment quality that can account for the wide range of disagreement in marks and the wide variability in evaluative judgments, regardless of student language background. As she describes her solutions, it is clear that she does not see utility in assessors simply exchanging criteria for the sake of quality assurance, but instead, she directs us toward the fostering of community to discuss how assessment values should be socialized across assessors. However, fostering community does not necessarily guarantee shared understanding of quality, but it is “through the interactions of participants” [emphasis hers] (p. 230) that is important for O’Hagan to have the constituents of quality grow and evolve dialogically.
The final chapter is a brief description of the limitations of the presented study and future research endeavors with which her findings might align. As she recognizes, because the scope of the study was limited to assessor behaviors, no attention was given to students themselves. As she writes, the study might have stretched its scope to include how students might benefit from transparency in assessment criteria or the degree to which students are familiarized with disciplinary knowledge. However, the suggestions offered by O’Hagan position students in a particular role: namely, on the outside. While she recognizes that students “are the primary stakeholders of assessment” (p. 248), there doesn’t seem much room to involve students in the negotiation of quality, much less involve students in the communities of practice that O’Hagan advocates. She does, however, point to some research that explores student involvement in assessment (Sadler, 2005; Woolf, 2004; Starfield, 2001), but even within the research she cites, students still do not have much of an active role in their assessment: Citing Sadler (2005) and Woolf (2004), students become empowered by transparency as the veil of the “mystique” is lifted from assessment, but their empowerment grows in only having the criteria available to them instead of their active negotiation. And citing Starfield (2001), students only negotiated aftera mark was given to receive higher marks as opposed to negotiating quality with instructors in the process. If we see assessment as constituent in the formation of self (see: Faigley, 1989; Yancey, 1999), positioning students on the outside when it comes to defining assessment quality poses a problem to how we see students, how students see themselves, and how that affects their agency within writing.
O’Hagan offers a pragmatic contribution to questions of quality for assessment. Her study is notable for its detailed description of methods and rigorous attention to critical literature in writing assessment scholarship. As such, her book can be a useful resource for those who are designing research projects of their own or considering improvements to disciplinary writing assessment. However, we must not lose sight of the students’ stake in disciplinary considerations of quality.
References
Cox, M. & Zawacki, T. M. (Eds.) (2014). WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices. Anderson, SC: Parlor Press LLC.
Dan, W. (2014). Let’s see where your Chinese students come from: A qualitative descriptive study of writing in the disciplines in China. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 233-255). Anderson, SC: Parlor Press LLC.
Faigley, L. (1989) Judging Writing, Judging Selves. College Composition and Communication, 40(4), 395-412.
Goen-Salter, S., Porter, P., & van Dommelen, D. (2009). Working with generation 1.5 pedagogical principles and practices. In M. Roberge, M. Siegal, & L. Harklau (Eds.), Generation 1.5 in college composition: Teaching academic writing to US-educated learners of ESL (pp. 235-259). New York, NY: Routledge.
Ives, L., Leahy, E., Leming, A., Pierce, T., and Schwartz, M. (2014). ‘I don’t know if that was the right thing to do’: Cross-disciplinary/cross-institutional faculty responses to L2 writing. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 211-232). Anderson, SC: Parlor Press LLC.
Lancaster, Z. (2014). Making stance explicit for second language writers in the disciplines: What faculty need to know about the language of stance taking. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 269-298). Anderson, SC: Parlor Press LLC.
Nielsen, K. (2014). On class, race, and dynamics of privilege: Supporting generation 1.5 writers across the curriculum. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 129-150). Anderson, SC: Parlor Press LLC.
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG.
Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175-194.
Starfield, S. (2001). ‘I’ll go with the group’: Rethinking ‘discourse community’ in EAP. In M. Peacock and J. Flowerdew (Eds.),Research perspectives on English for academic purposes (pp. 132-147). Cambridge, England: Cambridge University Press.
Woolf, H. (2004). Assessment criteria: Reflections on current practices. Assessment and Evaluation in Higher Education, 29(4), 479-493.
Yancey, K.B. (1999). Looking back as we look forward: Historicizing writing assessment. College Composition and Communication, 50(3), 483-503.
Zawacki, T. M. & Habib, A. S. (2014). Negotiating ‘errors’ in L2 writing: Faculty dispositions and language differences. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 183-210). Anderson, SC: Parlor Press LLC.
Source: jwa