Category: Uncategorized

A Review of White, Elliot, and Peckham’s Very Like a Whale: The Assessment of Writing Programs

White, E.M., Elliot N., & Peckham, I. (2015). Very like a whale: The assessment of writing programs. Logan: Utah State University Press.

By Peggy O’Neill, Loyola University Maryland

This volume offers readers a model for writing program assessment grounded in an overview of relevant theory and practice as well as case studies of two writing programs—Louisiana State University’s, where Peckham was the WPA, and New Jersey Institute of Technology’s, where Elliot served for many years. The text is organized into five main chapters—Trends, Lessons, Foundations, Measurement, and Design. It opens with an introduction and ends with a glossary of terms, references and the index. The text also includes 17 tables and 13 figures, one of which is the model for a genre of writing program assessment that the authors are putting forth (see Fig 1.1 and 5.1).

The introduction, which is available on the publisher’s website, summarizes each of the chapters and explains the authors’ approach and the framework of the text. While it provides standard features such as a summary of each chapter, it also explains the title this way: “With AARP cards embedded firmly in their wallets, the three seniors, formally educated in literary studies, selected a passage from Hamletfor the title” (p. 2). This opening threw me off, as a reader, (although I was wondering why this title) because of the way it positioned the authors and left me wondering why they situated themselves this way. A few paragraphs later, when articulating the audience for the book, they ask readers to “Imagine running into the three authors . . . at the annual meeting of the Conference on College Composition and Communication” (p. 4). They then present a dialog—“Let’s imagine just such a conversation” (p. 4) to illustrate “the tone for [their] book” (p. 4), which they describe as “chatting with colleagues and students” (p. 4). At this point, I was not sure where this book was going or what it was doing and felt a bit exasperated at the tone of the opening. However, the introduction then proceeds into a more straightforward overview of their approach and the chapter summaries.

The chatty tone that opened the book pops up now and again throughout the text. As a reader I found myself rushing through passages that address the reader directly (e.g., “Because the LSU case study is the first of four complex studies, you may want to review it briefly now and then review it again after completing the book” [p. 39]) or give background information that seems unnecessary (e.g., the brief tangent about the philosopher who distinguished between nomothetic and idiographic knowledge and the reference to Henry Fielding’s comment about The History of Tom Jones to make a point about history [p. 73]). For the most part, however, the book is more focused, which I think is the authors’ goal.

No doubt, readers charged with conducting program review, which the authors define  “as the process of documenting and reflecting on the impact of the program’s coordinated efforts” (p. 3), will benefit from the explanation of theory, methods, and practice that the authors offer. They seek, in their words, “to make clear and available recent and important concepts associated with assessment to those in the profession of rhetoric and composition/writing studies” (p. 3).  

In keeping with this goal, the authors provide a range of strategies, examples, and best practices for conducting a program assessment, grounded in the scholarship from writing studies as well as educational measurement. The strategies and approaches aren’t necessarily presented step-by-step so readers will need to read through the text and pull what they want if they are looking for a guide.

Although the case studies can help readers understand different questions and documentation methods, the level of detail sometimes seemed too much. While I realize case studies require detail, I felt some details were not important or were distracting, such as a brief history of WAC (p.50).  Or, in another example, referencing tagmemics  (p. 103) and Toulmin (p. 104) in discussing the way eportfolios would be evaluated seemed beyond the needs of most readers. Yet, I found myself wanting more explanation at other times. In discussing the assessment of eportfolios for a Writing about Science, Technology and Society, for instance, the explanation of the interreader reliability rates (p. 56-7) and the conclusions drawn from that information seemed to need more explanation, especially for readers less experienced with assessment. It also wasn’t clear how the data presented on interrater reliability demonstrated that students are improving over time (p 57). Although the authors explain the reasoning about student improvement, there seems to be a missing piece here. Yes, scores improved over the five years, but does that mean student writing improved? I am assuming different students were tested and other variables were in play (although admission test scores were consistent, they note). In other words, if the authors are assuming that many readers need basic information on WAC and WID, then I would expect that readers  would need a more complete and nuanced explanation of the technical data and analyses.

Lists of questions, such as that found in Chapter 3, Lessons, (p. 67) or the scoring sheet for a technical communication eportfolio in the same chapter (p. 56), can be of interest to readers looking for help in designing their own program assessments. Sharing examples of how eportfolios have been used is valuable for those of us trying to convince administrators to invest in the technology and faculty development time needed to implement them, yet I think this is somewhat limited view of the potential of eportfolios.

In addition to some practical examples, readers will get a sense of educational and writing theories that inform the authors’ approach to writing program assessment. However, the authors want to focus on more than practice—that is, how to conduct a program assessment. They want to contribute to the theoretical concept of writing program assessment: the “main purpose of this book,” they explain, is “to advance the concept of writing program assessment as a unique genre in which constructs are modeled for students within unique institutional ecologies” (p. 7).  

The book seems to achieve its first goal—providing readers with practical approaches and strategies—which is, I imagine, what most readers will be interested in. The second goal, to propose a genre of writing program assessment, is a bit more ambitious. While the model is unveiled in the first chapter, it is explained more fully in the last one, where it is presented with each of its nine components explained in more detail. Before delving into each of the components, the authors review fourteen of the key concepts that they have used throughout the book. These concepts are more general about the field of rhetoric and composition/writing studies (e.g., “Epistemologically, advancement of our field is best made by both disciplinary and multidisciplinary inquiry” [p. 151]); measurement (e.g., “In matters of measurement, analyses are most useful if they adhere to important reporting standards, including construct definitions” [p. 152]); and writing program assessment (e.g., “Imagining a predictable future for the assessment of writing programs reveals a need for attending to . . . . “ [p. 152]).

From here, the authors expound on their model, reminding readers that “acceptance of the model” (p. 153) is predicated on validity as Messick defined it in 1989: that validity is at the core of assessment and that it involves making a theoretical and empirical argument about the “adequacy and appropriateness of the inferencesand actions based on test scores or other modes of assessment” (Messick, qtd. in White, Elliot, and Peckham, p. 154).

Their proposed model of assessment of writing programs, which is presented as a flow chart that loops around with the results feeding back into the writing program, is then explained. Although some of the terminology and/or concepts in the framework are unfamiliar in writing program assessment literature (e.g., standpoint) most of it will seem very familiar to those involved in assessment theory and practice (e.g., construct or documentation) or in writing program administration (e.g., communication). All in all, I didn’t think the actual processes, strategies, and approaches for program assessment presented in this monograph are all that new or different; instead, I think the book provides an overview of work in writing and writing program assessment that has been going on for the last several decades, pulling it together and presenting it in an attempt to link it to the broader fields of both writing studies and educational measurement.

Source: jwa

A Review of White, Elliot, and Peckham’s Very Like a Whale: The Assessment of Writing Programs

White, E.M., Elliot N., & Peckham, I. (2015). Very like a whale: The assessment of writing programs. Logan: Utah State University Press.

By Peggy O’Neill, Loyola University Maryland

This volume offers readers a model for writing program assessment grounded in an overview of relevant theory and practice as well as case studies of two writing programs—Louisiana State University’s, where Peckham was the WPA, and New Jersey Institute of Technology’s, where Elliot served for many years. The text is organized into five main chapters—Trends, Lessons, Foundations, Measurement, and Design. It opens with an introduction and ends with a glossary of terms, references and the index. The text also includes 17 tables and 13 figures, one of which is the model for a genre of writing program assessment that the authors are putting forth (see Fig 1.1 and 5.1).

The introduction, which is available on the publisher’s website, summarizes each of the chapters and explains the authors’ approach and the framework of the text. While it provides standard features such as a summary of each chapter, it also explains the title this way: “With AARP cards embedded firmly in their wallets, the three seniors, formally educated in literary studies, selected a passage from Hamletfor the title” (p. 2). This opening threw me off, as a reader, (although I was wondering why this title) because of the way it positioned the authors and left me wondering why they situated themselves this way. A few paragraphs later, when articulating the audience for the book, they ask readers to “Imagine running into the three authors . . . at the annual meeting of the Conference on College Composition and Communication” (p. 4). They then present a dialog—“Let’s imagine just such a conversation” (p. 4) to illustrate “the tone for [their] book” (p. 4), which they describe as “chatting with colleagues and students” (p. 4). At this point, I was not sure where this book was going or what it was doing and felt a bit exasperated at the tone of the opening. However, the introduction then proceeds into a more straightforward overview of their approach and the chapter summaries.

The chatty tone that opened the book pops up now and again throughout the text. As a reader I found myself rushing through passages that address the reader directly (e.g., “Because the LSU case study is the first of four complex studies, you may want to review it briefly now and then review it again after completing the book” [p. 39]) or give background information that seems unnecessary (e.g., the brief tangent about the philosopher who distinguished between nomothetic and idiographic knowledge and the reference to Henry Fielding’s comment about The History of Tom Jones to make a point about history [p. 73]). For the most part, however, the book is more focused, which I think is the authors’ goal.

No doubt, readers charged with conducting program review, which the authors define  “as the process of documenting and reflecting on the impact of the program’s coordinated efforts” (p. 3), will benefit from the explanation of theory, methods, and practice that the authors offer. They seek, in their words, “to make clear and available recent and important concepts associated with assessment to those in the profession of rhetoric and composition/writing studies” (p. 3).  

In keeping with this goal, the authors provide a range of strategies, examples, and best practices for conducting a program assessment, grounded in the scholarship from writing studies as well as educational measurement. The strategies and approaches aren’t necessarily presented step-by-step so readers will need to read through the text and pull what they want if they are looking for a guide.

Although the case studies can help readers understand different questions and documentation methods, the level of detail sometimes seemed too much. While I realize case studies require detail, I felt some details were not important or were distracting, such as a brief history of WAC (p.50).  Or, in another example, referencing tagmemics  (p. 103) and Toulmin (p. 104) in discussing the way eportfolios would be evaluated seemed beyond the needs of most readers. Yet, I found myself wanting more explanation at other times. In discussing the assessment of eportfolios for a Writing about Science, Technology and Society, for instance, the explanation of the interreader reliability rates (p. 56-7) and the conclusions drawn from that information seemed to need more explanation, especially for readers less experienced with assessment. It also wasn’t clear how the data presented on interrater reliability demonstrated that students are improving over time (p 57). Although the authors explain the reasoning about student improvement, there seems to be a missing piece here. Yes, scores improved over the five years, but does that mean student writing improved? I am assuming different students were tested and other variables were in play (although admission test scores were consistent, they note). In other words, if the authors are assuming that many readers need basic information on WAC and WID, then I would expect that readers  would need a more complete and nuanced explanation of the technical data and analyses.

Lists of questions, such as that found in Chapter 3, Lessons, (p. 67) or the scoring sheet for a technical communication eportfolio in the same chapter (p. 56), can be of interest to readers looking for help in designing their own program assessments. Sharing examples of how eportfolios have been used is valuable for those of us trying to convince administrators to invest in the technology and faculty development time needed to implement them, yet I think this is somewhat limited view of the potential of eportfolios.

In addition to some practical examples, readers will get a sense of educational and writing theories that inform the authors’ approach to writing program assessment. However, the authors want to focus on more than practice—that is, how to conduct a program assessment. They want to contribute to the theoretical concept of writing program assessment: the “main purpose of this book,” they explain, is “to advance the concept of writing program assessment as a unique genre in which constructs are modeled for students within unique institutional ecologies” (p. 7).  

The book seems to achieve its first goal—providing readers with practical approaches and strategies—which is, I imagine, what most readers will be interested in. The second goal, to propose a genre of writing program assessment, is a bit more ambitious. While the model is unveiled in the first chapter, it is explained more fully in the last one, where it is presented with each of its nine components explained in more detail. Before delving into each of the components, the authors review fourteen of the key concepts that they have used throughout the book. These concepts are more general about the field of rhetoric and composition/writing studies (e.g., “Epistemologically, advancement of our field is best made by both disciplinary and multidisciplinary inquiry” [p. 151]); measurement (e.g., “In matters of measurement, analyses are most useful if they adhere to important reporting standards, including construct definitions” [p. 152]); and writing program assessment (e.g., “Imagining a predictable future for the assessment of writing programs reveals a need for attending to . . . . “ [p. 152]).

From here, the authors expound on their model, reminding readers that “acceptance of the model” (p. 153) is predicated on validity as Messick defined it in 1989: that validity is at the core of assessment and that it involves making a theoretical and empirical argument about the “adequacy and appropriateness of the inferencesand actions based on test scores or other modes of assessment” (Messick, qtd. in White, Elliot, and Peckham, p. 154).

Their proposed model of assessment of writing programs, which is presented as a flow chart that loops around with the results feeding back into the writing program, is then explained. Although some of the terminology and/or concepts in the framework are unfamiliar in writing program assessment literature (e.g., standpoint) most of it will seem very familiar to those involved in assessment theory and practice (e.g., construct or documentation) or in writing program administration (e.g., communication). All in all, I didn’t think the actual processes, strategies, and approaches for program assessment presented in this monograph are all that new or different; instead, I think the book provides an overview of work in writing and writing program assessment that has been going on for the last several decades, pulling it together and presenting it in an attempt to link it to the broader fields of both writing studies and educational measurement.

Source: jwa

Link to NCTE’s webinar: "No Test is Neutral: Writing Assessments, Equity, Ethics, and Social Justice"

If you missed NCTE’s webinar, “No Test is Neutral:  Writing Assessments, Equity, Ethics, and Social Justice,” on November 2, 2015, you can watch the video linked to NCTE’s page above.  We greatly appreciate NCTE’s generosity to host this important conversation. Supplemental information from the webinar can be accessed by clicking on the “be part of the conversation” banner.

We were pleased to share the news that Journal of Writing Assessment will publish two special issues within the next six months.  On November 16, 2015, we will post the special issue on the Common Core State Standards Assessments, and in mid-March 2016, we will publish a special issue on a theory of ethics for writing assessment.

Stay tuned!
Source: jwa

Link to NCTE’s webinar: "No Test is Neutral: Writing Assessments, Equity, Ethics, and Social Justice"

If you missed NCTE’s webinar, “No Test is Neutral:  Writing Assessments, Equity, Ethics, and Social Justice,” on November 2, 2015, you can watch the video linked to NCTE’s page above.  We greatly appreciate NCTE’s generosity to host this important conversation. Supplemental information from the webinar can be accessed by clicking on the “be part of the conversation” banner.

We were pleased to share the news that Journal of Writing Assessment will publish two special issues within the next six months.  On November 16, 2015, we will post the special issue on the Common Core State Standards Assessments, and in mid-March 2016, we will publish a special issue on a theory of ethics for writing assessment.

Stay tuned!
Source: jwa

NCTE Webinar on Writing Assessment, Equity, Ethics, and Social Justice Featuring JWA Editors Diane Kelly-Riley and Carl Whithaus

Journal of Writing Assessment editors, Diane Kelly-Riley and Carl Whithaus, will be participating in NCTE’s Webinar, “No Test is Neutral: Writing Assessments, Equity, Ethics, and Social Justice” on Monday, November 2, 2015 at 6 PM Pacific/9 PM Eastern.

Here is NCTE’s description of the event:  Join us next Monday November 2 at 9 PM ET for ‪#‎NCTEonAir‬. Samuel Messick, one of the leading assessment researchers of the 20th Century, once argued that implementing an assessment program without first assessing its effects on students, teachers, and systems of education is like releasing an untested new drug on the market.

This final event of Connected Educator Month explores what a theory of ethics might mean for our field, and what implications it might have for the design and implementation of both large-scale and classroom writing assessments.
The one-hour web seminar will feature Diane Kelly-Riley and Carl Whithaus, editors of the Journal of Writing Assessment, posing questions on the ethics of writing assessment to a panel of experts—Bob Broad, Ellen Cushman, Norbert Elliot, Mya Poe, and David Slomp.

• What are the key issues teachers, students, and stakeholders need to know about problems with current large-scale writing assessment models?

• What key ideas and practices should guide the field of writing assessment? How could explicitly addressing ethical issues (such as fairness) shift major assumptions within the field of writing assessment? How might those considerations affect the daily lives of teachers?

• How might a consideration of ethics change how we design classroom and/or large scale writing assessments?

• How would consideration of ethics enable us to better attend to the needs of the diversity of students in our classrooms?

• How might a theory of ethics enable us to better advocate for and inform changes in the design of large-scale writing assessments?

Across all of these discussions, panelists will focus on the implications for classroom teachers. 

You are invited to tweet your questions to #whatwehonor. Our panel moderators will select some of these questions to pose to our panelists.  


To join in the conversation, please go to NCTE’s free Google Air event link:  https://plus.google.com/events/cofhn9uvknm16huk36m9jho1rq0 .

Source: jwa

NCTE Webinar on Writing Assessment, Equity, Ethics, and Social Justice Featuring JWA Editors Diane Kelly-Riley and Carl Whithaus

Journal of Writing Assessment editors, Diane Kelly-Riley and Carl Whithaus, will be participating in NCTE’s Webinar, “No Test is Neutral: Writing Assessments, Equity, Ethics, and Social Justice” on Monday, November 2, 2015 at 6 PM Pacific/9 PM Eastern.

Here is NCTE’s description of the event:  Join us next Monday November 2 at 9 PM ET for ‪#‎NCTEonAir‬. Samuel Messick, one of the leading assessment researchers of the 20th Century, once argued that implementing an assessment program without first assessing its effects on students, teachers, and systems of education is like releasing an untested new drug on the market.

This final event of Connected Educator Month explores what a theory of ethics might mean for our field, and what implications it might have for the design and implementation of both large-scale and classroom writing assessments.
The one-hour web seminar will feature Diane Kelly-Riley and Carl Whithaus, editors of the Journal of Writing Assessment, posing questions on the ethics of writing assessment to a panel of experts—Bob Broad, Ellen Cushman, Norbert Elliot, Mya Poe, and David Slomp.

• What are the key issues teachers, students, and stakeholders need to know about problems with current large-scale writing assessment models?

• What key ideas and practices should guide the field of writing assessment? How could explicitly addressing ethical issues (such as fairness) shift major assumptions within the field of writing assessment? How might those considerations affect the daily lives of teachers?

• How might a consideration of ethics change how we design classroom and/or large scale writing assessments?

• How would consideration of ethics enable us to better attend to the needs of the diversity of students in our classrooms?

• How might a theory of ethics enable us to better advocate for and inform changes in the design of large-scale writing assessments?

Across all of these discussions, panelists will focus on the implications for classroom teachers. 

You are invited to tweet your questions to #whatwehonor. Our panel moderators will select some of these questions to pose to our panelists.  


To join in the conversation, please go to NCTE’s free Google Air event link:  https://plus.google.com/events/cofhn9uvknm16huk36m9jho1rq0 .

Source: jwa

Communities, stakeholders, and quality in assessment: A review of Sally O’Hagan’s Variability in Assessor Responses to Undergraduate Essays: An Issue for Assessment Quality in Higher Education

Communities, stakeholders, and quality in assessment: A review of Sally O’Hagan’s Variability in Assessor Responses to Undergraduate Essays: An Issue for Assessment Quality in Higher Education

O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG.

By Joe Cirio, Florida State University

In light of the increased growth of linguistic diversity in Australian universities, Sally O’Hagan’s book, Variability in Assessor Responses to Undergraduate Essays, seeks to explore how a more linguistically diverse student population affects assessment practices for disciplinary writing. The increased growth of diversity, as O’Hagan points out, poses a challenge to the quality and fairness of assessment—particularly, it poses a challenge to “consistency of assessment criteria and standards” (p. 11). But these challenges are only exacerbated by the lack of consensus on the meaning of quality within a discipline, and, moreover, the fact that assessors often rely upon “tacit knowledge of close communities of academics” (p. 17). This context is what ground her empirical study of assessor behaviors when responding to essays from native-speakers (NS) and non-native-speakers (NNS) of English. 
Both Chapter One and Two contextualize her study: the former chapter lays out the immediate context of the Australian university; the latter details the scholastic conversations with which her study intersects. She starts by describing the ways in which assessors vary in both marks and the evaluative judgments in support of those marks. As noted by O’Hagan, a multitude of factors can contribute to variability in scores. To rectify this variability, institutions have begun to implement policies to encourage transparency and clear communication about assessments. However, O’Hagan writes, “it has been argued that the explicit specifications of criteria and standards does little to improve the quality of assessment in the absence of supportive measures” (p. 37). She points to the fostering of genuine communities of practice as a way to ground disciplines in sharing an understanding of quality, criteria, and standards for assessment. The concluding section of Chapter Two reviews relevant research into verbal reporting methodologies, the central method used by O’Hagan to understand assessor behaviors. While verbal reporting methodologies do not “directly represent cognitive processes” (p. 76), they contribute an approach to assessor behavior that may provide a different data set to uncover implications not yet realized. Other research into assessor behaviors on second language writers—most recently in an edited collection by Cox and Zawacki (2014)—have used methods such as interviews (Zawacki & Habib, 2014; Ives et al., 2014; Dan, 2014), surveys (Ives et al., 2014), or analysis of student examples (Lancaster, 2014).
Collecting data from an academic department at a major university in Australia, the study hinged on ten assessors—nine tutors and one instructor—who would offer verbal reports as they assessed ten student essays (half NS and half NNS—this information was not made available to assessors). Chapter Four walks through the statistical analyses of three sets of data that were collected: the marks assigned to the essays, the features of the essay that prompted an evaluation, and the evaluative judgments made by assessors. The conclusions of the study can be distilled into four points: NNS essays (1) received lower marks, (2) with more negative comments, and (3) with comments more often directed toward mechanics. But (4) there was also an increased range of comments for NNS essays indicating a lack of consensus about how assessors, within the same discipline, respond to writing of second-language writers (p. 207). 
The remaining chapters of O’Hagan’s book point to improvements in assessment quality that can account for the wide range of disagreement in marks and the wide variability in evaluative judgments, regardless of student language background. As she describes her solutions, it is clear that she does not see utility in assessors simply exchanging criteria for the sake of quality assurance, but instead, she directs us toward the fostering of community to discuss how assessment values should be socialized across assessors. However, fostering community does not necessarily guarantee shared understanding of quality, but it is “through the interactions of participants” [emphasis hers] (p. 230) that is important for O’Hagan to have the constituents of quality grow and evolve dialogically.
             
The final chapter is a brief description of the limitations of the presented study and future research endeavors with which her findings might align. As she recognizes, because the scope of the study was limited to assessor behaviors, no attention was given to students themselves. As she writes, the study might have stretched its scope to include how students might benefit from transparency in assessment criteria or the degree to which students are familiarized with disciplinary knowledge. However, the suggestions offered by O’Hagan position students in a particular role: namely, on the outside. While she recognizes that students “are the primary stakeholders of assessment” (p. 248), there doesn’t seem much room to involve students in the negotiation of quality, much less involve students in the communities of practice that O’Hagan advocates. She does, however, point to some research that explores student involvement in assessment (Sadler, 2005; Woolf, 2004; Starfield, 2001), but even within the research she cites, students still do not have much of an active role in their assessment: Citing Sadler (2005) and Woolf (2004), students become empowered by transparency as the veil of the “mystique” is lifted from assessment, but their empowerment grows in only having the criteria available to them instead of their active negotiation. And citing Starfield (2001), students only negotiated aftera mark was given to receive higher marks as opposed to negotiating quality with instructors in the process. If we see assessment as constituent in the formation of self (see: Faigley, 1989; Yancey, 1999), positioning students on the outside when it comes to defining assessment quality poses a problem to how we see students, how students see themselves, and how that affects their agency within writing.
             
O’Hagan offers a pragmatic contribution to questions of quality for assessment. Her study is notable for its detailed description of methods and rigorous attention to critical literature in writing assessment scholarship. As such, her book can be a useful resource for those who are designing research projects of their own or considering improvements to disciplinary writing assessment. However, we must not lose sight of the students’ stake in disciplinary considerations of quality.
References
Cox, M. & Zawacki, T. M. (Eds.) (2014). WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices. Anderson, SC: Parlor Press LLC. 
Dan, W. (2014). Let’s see where your Chinese students come from: A qualitative descriptive study of writing in the disciplines in China. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 233-255). Anderson, SC: Parlor Press LLC.
Faigley, L. (1989) Judging Writing, Judging Selves. College Composition and Communication,  40(4), 395-412.
Goen-Salter, S., Porter, P., & van Dommelen, D. (2009). Working with generation 1.5 pedagogical principles and practices. In M. Roberge, M. Siegal, & L. Harklau (Eds.), Generation 1.5 in college composition: Teaching academic writing to US-educated learners of ESL (pp. 235-259). New York, NY: Routledge. 
Ives, L., Leahy, E., Leming, A., Pierce, T., and Schwartz, M. (2014). ‘I don’t know if that was the right thing to do’: Cross-disciplinary/cross-institutional faculty responses to L2 writing. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 211-232). Anderson, SC: Parlor Press LLC. 
Lancaster, Z. (2014). Making stance explicit for second language writers in the disciplines: What faculty need to know about the language of stance taking. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 269-298). Anderson, SC: Parlor Press LLC.
Nielsen, K. (2014). On class, race, and dynamics of privilege: Supporting generation 1.5 writers across the curriculum. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 129-150). Anderson, SC: Parlor Press LLC. 
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG. 
Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175-194. 
Starfield, S. (2001). ‘I’ll go with the group’: Rethinking ‘discourse community’ in EAP. In M. Peacock and J. Flowerdew (Eds.),Research perspectives on English for academic purposes (pp. 132-147). Cambridge, England: Cambridge University Press.
Woolf, H. (2004). Assessment criteria: Reflections on current practices. Assessment and Evaluation in Higher Education, 29(4), 479-493. 
Yancey, K.B. (1999). Looking back as we look forward: Historicizing writing assessment. College Composition and Communication, 50(3), 483-503.
Zawacki, T. M. & Habib, A. S. (2014). Negotiating ‘errors’ in L2 writing: Faculty dispositions and language differences. In M. Cox & T. M. Zawacki (Eds.),  WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 183-210). Anderson, SC: Parlor Press LLC.

 
Source: jwa

Communities, stakeholders, and quality in assessment: A review of Sally O’Hagan’s Variability in Assessor Responses to Undergraduate Essays: An Issue for Assessment Quality in Higher Education

Communities, stakeholders, and quality in assessment: A review of Sally O’Hagan’s Variability in Assessor Responses to Undergraduate Essays: An Issue for Assessment Quality in Higher Education

O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG.

By Joe Cirio, Florida State University

In light of the increased growth of linguistic diversity in Australian universities, Sally O’Hagan’s book, Variability in Assessor Responses to Undergraduate Essays, seeks to explore how a more linguistically diverse student population affects assessment practices for disciplinary writing. The increased growth of diversity, as O’Hagan points out, poses a challenge to the quality and fairness of assessment—particularly, it poses a challenge to “consistency of assessment criteria and standards” (p. 11). But these challenges are only exacerbated by the lack of consensus on the meaning of quality within a discipline, and, moreover, the fact that assessors often rely upon “tacit knowledge of close communities of academics” (p. 17). This context is what ground her empirical study of assessor behaviors when responding to essays from native-speakers (NS) and non-native-speakers (NNS) of English. 
Both Chapter One and Two contextualize her study: the former chapter lays out the immediate context of the Australian university; the latter details the scholastic conversations with which her study intersects. She starts by describing the ways in which assessors vary in both marks and the evaluative judgments in support of those marks. As noted by O’Hagan, a multitude of factors can contribute to variability in scores. To rectify this variability, institutions have begun to implement policies to encourage transparency and clear communication about assessments. However, O’Hagan writes, “it has been argued that the explicit specifications of criteria and standards does little to improve the quality of assessment in the absence of supportive measures” (p. 37). She points to the fostering of genuine communities of practice as a way to ground disciplines in sharing an understanding of quality, criteria, and standards for assessment. The concluding section of Chapter Two reviews relevant research into verbal reporting methodologies, the central method used by O’Hagan to understand assessor behaviors. While verbal reporting methodologies do not “directly represent cognitive processes” (p. 76), they contribute an approach to assessor behavior that may provide a different data set to uncover implications not yet realized. Other research into assessor behaviors on second language writers—most recently in an edited collection by Cox and Zawacki (2014)—have used methods such as interviews (Zawacki & Habib, 2014; Ives et al., 2014; Dan, 2014), surveys (Ives et al., 2014), or analysis of student examples (Lancaster, 2014).
Collecting data from an academic department at a major university in Australia, the study hinged on ten assessors—nine tutors and one instructor—who would offer verbal reports as they assessed ten student essays (half NS and half NNS—this information was not made available to assessors). Chapter Four walks through the statistical analyses of three sets of data that were collected: the marks assigned to the essays, the features of the essay that prompted an evaluation, and the evaluative judgments made by assessors. The conclusions of the study can be distilled into four points: NNS essays (1) received lower marks, (2) with more negative comments, and (3) with comments more often directed toward mechanics. But (4) there was also an increased range of comments for NNS essays indicating a lack of consensus about how assessors, within the same discipline, respond to writing of second-language writers (p. 207). 
The remaining chapters of O’Hagan’s book point to improvements in assessment quality that can account for the wide range of disagreement in marks and the wide variability in evaluative judgments, regardless of student language background. As she describes her solutions, it is clear that she does not see utility in assessors simply exchanging criteria for the sake of quality assurance, but instead, she directs us toward the fostering of community to discuss how assessment values should be socialized across assessors. However, fostering community does not necessarily guarantee shared understanding of quality, but it is “through the interactions of participants” [emphasis hers] (p. 230) that is important for O’Hagan to have the constituents of quality grow and evolve dialogically.
             
The final chapter is a brief description of the limitations of the presented study and future research endeavors with which her findings might align. As she recognizes, because the scope of the study was limited to assessor behaviors, no attention was given to students themselves. As she writes, the study might have stretched its scope to include how students might benefit from transparency in assessment criteria or the degree to which students are familiarized with disciplinary knowledge. However, the suggestions offered by O’Hagan position students in a particular role: namely, on the outside. While she recognizes that students “are the primary stakeholders of assessment” (p. 248), there doesn’t seem much room to involve students in the negotiation of quality, much less involve students in the communities of practice that O’Hagan advocates. She does, however, point to some research that explores student involvement in assessment (Sadler, 2005; Woolf, 2004; Starfield, 2001), but even within the research she cites, students still do not have much of an active role in their assessment: Citing Sadler (2005) and Woolf (2004), students become empowered by transparency as the veil of the “mystique” is lifted from assessment, but their empowerment grows in only having the criteria available to them instead of their active negotiation. And citing Starfield (2001), students only negotiated aftera mark was given to receive higher marks as opposed to negotiating quality with instructors in the process. If we see assessment as constituent in the formation of self (see: Faigley, 1989; Yancey, 1999), positioning students on the outside when it comes to defining assessment quality poses a problem to how we see students, how students see themselves, and how that affects their agency within writing.
             
O’Hagan offers a pragmatic contribution to questions of quality for assessment. Her study is notable for its detailed description of methods and rigorous attention to critical literature in writing assessment scholarship. As such, her book can be a useful resource for those who are designing research projects of their own or considering improvements to disciplinary writing assessment. However, we must not lose sight of the students’ stake in disciplinary considerations of quality.
References
Cox, M. & Zawacki, T. M. (Eds.) (2014). WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices. Anderson, SC: Parlor Press LLC. 
Dan, W. (2014). Let’s see where your Chinese students come from: A qualitative descriptive study of writing in the disciplines in China. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 233-255). Anderson, SC: Parlor Press LLC.
Faigley, L. (1989) Judging Writing, Judging Selves. College Composition and Communication,  40(4), 395-412.
Goen-Salter, S., Porter, P., & van Dommelen, D. (2009). Working with generation 1.5 pedagogical principles and practices. In M. Roberge, M. Siegal, & L. Harklau (Eds.), Generation 1.5 in college composition: Teaching academic writing to US-educated learners of ESL (pp. 235-259). New York, NY: Routledge. 
Ives, L., Leahy, E., Leming, A., Pierce, T., and Schwartz, M. (2014). ‘I don’t know if that was the right thing to do’: Cross-disciplinary/cross-institutional faculty responses to L2 writing. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 211-232). Anderson, SC: Parlor Press LLC. 
Lancaster, Z. (2014). Making stance explicit for second language writers in the disciplines: What faculty need to know about the language of stance taking. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 269-298). Anderson, SC: Parlor Press LLC.
Nielsen, K. (2014). On class, race, and dynamics of privilege: Supporting generation 1.5 writers across the curriculum. In M. Cox & T. M. Zawacki (Eds.), WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 129-150). Anderson, SC: Parlor Press LLC. 
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang AG. 
Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education, 30(2), 175-194. 
Starfield, S. (2001). ‘I’ll go with the group’: Rethinking ‘discourse community’ in EAP. In M. Peacock and J. Flowerdew (Eds.),Research perspectives on English for academic purposes (pp. 132-147). Cambridge, England: Cambridge University Press.
Woolf, H. (2004). Assessment criteria: Reflections on current practices. Assessment and Evaluation in Higher Education, 29(4), 479-493. 
Yancey, K.B. (1999). Looking back as we look forward: Historicizing writing assessment. College Composition and Communication, 50(3), 483-503.
Zawacki, T. M. & Habib, A. S. (2014). Negotiating ‘errors’ in L2 writing: Faculty dispositions and language differences. In M. Cox & T. M. Zawacki (Eds.),  WAC and second language writers: Research towards linguistically and culturally inclusive programs and practices (pp. 183-210). Anderson, SC: Parlor Press LLC.

 
Source: jwa

Who Assesses the Assessors? A Review of Assessing the Teaching of Writing: Twenty-First Century Trends and Technologies

Who Assesses the Assessors? A Review of Assessing the Teaching of Writing:
Twenty-First Century Trends and Technologies
Dayton, A. E. (Ed.). (2015). Assessing the teaching of writing:Twenty-first century trends and technologies. Boulder, CO: University Press of Colorado.
by Way Jeng, Washington State University
Overview
           
Assessing the Teaching of Writing:Twenty-First Century Trends and Technologies examines the performance of teachers. That is to say, it offers methods to investigate whether curricula are taught so students learn the material and are well-positioned to perform. The scholarship of assessment in writing studies generally looks at the performance of students, the achievement of learning outcomes for entire programs, and helps to define the curricular values of educational institutions. This volume addresses the missing middle link in the assessment chain: the teaching performance of individual educators.
Over the course of the book’s 12 chapters, the authors (Amy E. Dayton, Meredith DeCosta, Duane Roen, Brian Jackson, Gerald Nelms, Kara Mae Brown, Kim Freeman, Chris Gallagher Chris M. Anson, Nichole Bennett, Cindy Moore, Amy C. Kimmie Hea, Charles Paine, Robert M. Gonyea, Paul Anderson, Deborah Minter, and Amy Goodburn) discuss topics as varied as student course evaluations and teacher portfolios to administrative priorities and issues of how to assess the efficacy of writing center consultations for students.
           
The first half of the book looks at frameworks and methods for teacher assessment. These chapters offer clear methods from those who have already implemented the discussed programs, along with a discussion of each method’s validity construct. The second half of the book steps outside of the classroom to examine the larger institutional and administrative context of teacher assessment. The authors in this section show awareness that concerned teachers and administrators may perceive the assessment of teachers may as unfair, punitive, or otherwise designed to hurt teachers. Here, the emphasis on formative feedback of teaching, rather than a summative or purely evaluative frame, is very welcome.
Highlights
           
The volume’s third chapter, Amy E. Dayton’s “Making Sense (and Making Use) of Student Evaluations,” presents the use of student course evaluations. This is the most logistically-straightforward method of teacher evaluation for many institutions simply due to the ubiquity of student course evaluations. Administrators likely already have years of data already collected. Dayton provides a detailed discussion of validity concerns surrounding course evaluations (i.e., Can our students be relied upon when they are not experts on the subject matter themselves?), as well as a framework for interpreting student comments in course evaluations in the context of programmatic learning outcomes statements.
Cindy Moore’s chapter, “Administrative Priorities and the Case for Multiple Methods,” encapsulates one of the main argumentative threads that permeates each chapter: No one assessment can adequately describe an activity as rich and varied as teaching. Using multiple methods allows stronger conclusions to be drawn by triangulating data to reinforce observations. Moore further expands this core argument by discussing ways administrators can overcome the obvious issue of time practicality since more assessments and more assessment data can often mean more time. Moore suggests identifying work that can be spread between stakeholders in the assessment and strategically choosing assessment materials to ensure a complete yet concise assessment procedure.
Last Thoughts
           
As the volume is only around 200 pages, readers can expect that some aspects of teacher assessment will be left unexamined. For example, none of the authors specifically discuss the effects of race and class in assessing teachers, though it seems likely that race and class bear effects for teaching just as they do for learning. Because Assessing the Teaching of Writingis concise, it is a practical guide for readers who want to revise or develop their methods for assessing teachers. Rather than providing an exhaustive treatment, this text provides a foundation for implementing teacher assessment and offers clear methods to do so. Readers are therefore well-positioned to expand that base by looking at other texts that explore specific aspects of assessment to incorporate those ideas into the existent framework of teacher assessment.
           
As such, existing works that discuss assessment complement the book very well. Newcomers to assessment will probably want to read some of the texts referenced by numerous authors, notably Brian Huot’s (2002) (Re)articulating Writing Assessment for Teaching and Learning, Linda Adler-Kasser and Peggy O’Neill’s (2010) Reframing Writing Assessment to Improve Teaching and Learning, Chris Anson’s (1994) Portfolios for Teachers, and Peter Seldin’s (1991) The Teaching Portfolio. Assessing the Teaching of Writing is a solid addition to any scholar’s collection of assessment texts. It’s not so much a foray into a qualitatively different model of assessment as it is the transfer of established theory into a new context. This book helps to imagine and implement assessment of teachers in nuanced ways, and as such helps to inform an important part of the assessment of writing.

References
Adler-Kassner, L., and O’Neill, P. (2010). Reframing Writing Assessment to Improve Teaching and Learning. Logan, Utah: Utah State University Press.
Anson, C. (1994). Portfolios for teachers: Writing our way to reflective practice. New Directions in Portfolio Assessment. Eds. Black, L., Daiker, D., Sommers, J., and Stygall, G. Portsmouth, NH: Neinemann.
Huot, B. (2002). (Re)articulating Writing Assessment for Teaching and Learning. Logan: Utah State University Press.
Selden, P. (1991). The Teaching Portfolio: A Practical Guide to Improved Performance and Promotional/Tenure Decisions. Bolton, MA: Anker.

Source: jwa

Who Assesses the Assessors? A Review of Assessing the Teaching of Writing: Twenty-First Century Trends and Technologies

Who Assesses the Assessors? A Review of Assessing the Teaching of Writing:
Twenty-First Century Trends and Technologies
Dayton, A. E. (Ed.). (2015). Assessing the teaching of writing:Twenty-first century trends and technologies. Boulder, CO: University Press of Colorado.
by Way Jeng, Washington State University
Overview
           
Assessing the Teaching of Writing:Twenty-First Century Trends and Technologies examines the performance of teachers. That is to say, it offers methods to investigate whether curricula are taught so students learn the material and are well-positioned to perform. The scholarship of assessment in writing studies generally looks at the performance of students, the achievement of learning outcomes for entire programs, and helps to define the curricular values of educational institutions. This volume addresses the missing middle link in the assessment chain: the teaching performance of individual educators.
Over the course of the book’s 12 chapters, the authors (Amy E. Dayton, Meredith DeCosta, Duane Roen, Brian Jackson, Gerald Nelms, Kara Mae Brown, Kim Freeman, Chris Gallagher Chris M. Anson, Nichole Bennett, Cindy Moore, Amy C. Kimmie Hea, Charles Paine, Robert M. Gonyea, Paul Anderson, Deborah Minter, and Amy Goodburn) discuss topics as varied as student course evaluations and teacher portfolios to administrative priorities and issues of how to assess the efficacy of writing center consultations for students.
           
The first half of the book looks at frameworks and methods for teacher assessment. These chapters offer clear methods from those who have already implemented the discussed programs, along with a discussion of each method’s validity construct. The second half of the book steps outside of the classroom to examine the larger institutional and administrative context of teacher assessment. The authors in this section show awareness that concerned teachers and administrators may perceive the assessment of teachers may as unfair, punitive, or otherwise designed to hurt teachers. Here, the emphasis on formative feedback of teaching, rather than a summative or purely evaluative frame, is very welcome.
Highlights
           
The volume’s third chapter, Amy E. Dayton’s “Making Sense (and Making Use) of Student Evaluations,” presents the use of student course evaluations. This is the most logistically-straightforward method of teacher evaluation for many institutions simply due to the ubiquity of student course evaluations. Administrators likely already have years of data already collected. Dayton provides a detailed discussion of validity concerns surrounding course evaluations (i.e., Can our students be relied upon when they are not experts on the subject matter themselves?), as well as a framework for interpreting student comments in course evaluations in the context of programmatic learning outcomes statements.
Cindy Moore’s chapter, “Administrative Priorities and the Case for Multiple Methods,” encapsulates one of the main argumentative threads that permeates each chapter: No one assessment can adequately describe an activity as rich and varied as teaching. Using multiple methods allows stronger conclusions to be drawn by triangulating data to reinforce observations. Moore further expands this core argument by discussing ways administrators can overcome the obvious issue of time practicality since more assessments and more assessment data can often mean more time. Moore suggests identifying work that can be spread between stakeholders in the assessment and strategically choosing assessment materials to ensure a complete yet concise assessment procedure.
Last Thoughts
           
As the volume is only around 200 pages, readers can expect that some aspects of teacher assessment will be left unexamined. For example, none of the authors specifically discuss the effects of race and class in assessing teachers, though it seems likely that race and class bear effects for teaching just as they do for learning. Because Assessing the Teaching of Writingis concise, it is a practical guide for readers who want to revise or develop their methods for assessing teachers. Rather than providing an exhaustive treatment, this text provides a foundation for implementing teacher assessment and offers clear methods to do so. Readers are therefore well-positioned to expand that base by looking at other texts that explore specific aspects of assessment to incorporate those ideas into the existent framework of teacher assessment.
           
As such, existing works that discuss assessment complement the book very well. Newcomers to assessment will probably want to read some of the texts referenced by numerous authors, notably Brian Huot’s (2002) (Re)articulating Writing Assessment for Teaching and Learning, Linda Adler-Kasser and Peggy O’Neill’s (2010) Reframing Writing Assessment to Improve Teaching and Learning, Chris Anson’s (1994) Portfolios for Teachers, and Peter Seldin’s (1991) The Teaching Portfolio. Assessing the Teaching of Writing is a solid addition to any scholar’s collection of assessment texts. It’s not so much a foray into a qualitatively different model of assessment as it is the transfer of established theory into a new context. This book helps to imagine and implement assessment of teachers in nuanced ways, and as such helps to inform an important part of the assessment of writing.

References
Adler-Kassner, L., and O’Neill, P. (2010). Reframing Writing Assessment to Improve Teaching and Learning. Logan, Utah: Utah State University Press.
Anson, C. (1994). Portfolios for teachers: Writing our way to reflective practice. New Directions in Portfolio Assessment. Eds. Black, L., Daiker, D., Sommers, J., and Stygall, G. Portsmouth, NH: Neinemann.
Huot, B. (2002). (Re)articulating Writing Assessment for Teaching and Learning. Logan: Utah State University Press.
Selden, P. (1991). The Teaching Portfolio: A Practical Guide to Improved Performance and Promotional/Tenure Decisions. Bolton, MA: Anker.

Source: jwa