SOM Implements Writing Assessment Program

by Frank Allen Philpot, School of Management

Most teachers have a store of examples of student papers that consist of one paragraph that runs on for three or four pages, contains seemingly random capitalization or endlessly confuses “their” and “there.” These are the problems we share around the lunch table or in teaching seminars when we complain, “Students today just can’t write!” But are these anecdotal examples representative of a systematic problem or are they just the items that are seared onto our brains?

Toward a Systematic Assessment

In the spring of 2007 the School of Management (SOM) set out to address the question of student writing quality in a systematic manner. Alison O’Brien, the school’s Associate Dean for Undergraduate Education, invited a group of faculty to serve as a task force to review student writing. As a marketing professor passionate about good written communication, I served as chair of that task force.

The School of Management does not profess to graduate students who are necessarily gifted writers. Elegant analogies and rhetorical flourishes are, in fact, out of place in business writing. As faculty, we want to send students into the workplace whose writing is not noticeable, since writing that calls attention to itself gets in the way of communicating a clear and direct message. Our goal is workmanlike prose that fades into the background because the ideas expressed are important and the message is clear.

Justification and Procedures

Our task force undertook this project because we think writing is important and because our accreditation body (The Association for the Advancement of Colleges and Schools of Business) asks us to assess learning goals continuously, while the Mason Writing Assessment Group (WAG) expects a formal review every five years. Our goal was to know with greater certainty the level of competency in student writing and, if necessary, to provide a foundation for recommendations to improve our curriculum – especially our writing intensive course, SOM 301.

As a first step, Dean O’Brien asked faculty to save student papers from the spring semester ’07. We received a total of 260 papers from six courses representing assignments in SOM 301 and other major and core courses. We asked that the papers come without names or faculty comments or grades, and a sample of 111 papers received was prepared for use in the scoring process.

For this project we focused on the distinction between competent and not-competent writing although we also discussed the category of highly competent writing for future assessments.

Creating a Scoring Rubric

Our first task was to determine if a group of faculty could agree on the elements of competent writing. With the help of Dr. Terry Zawacki, Director of the University’s Writing Across the Curriculum program, our task force met to create a rubric, or assessment matrix, that would allow us to identify the elements of good writing and provide a tool to determine if a specific paper met the standard of competency. Five faculty of the School of Management participated in this process. For this calibration process, we used papers from our introductory business communications course, SOM 301.

As you might imagine, the five faculty had differing opinions on the level of competence each writing sample represented. After about three hours of review and debate, we agreed upon the elements of good writing for business students and the common types of errors that would render an assignment not competent.

Over the next week, we exchanged electronic versions of the rubric and collectively produced the matrix we would use for the next phase of the writing assessment process. Individual faculty differences and preferences were considered, and sometimes compromises were made, to produce a workable assessment tool. Next, a team of six SOM faculty members and three faculty members from outside the school met to review a sample of papers collected from spring semester. We agreed that each paper would be read by two faculty members and designated as either competent or not-competent; a third reading was used in the event of disagreements.

Results of the Scoring Process

During the six-hour rating session, the group read 51 papers – all from major courses. The raters agreed on 82% of the papers – a surprisingly high level of inter-rater reliability for what initially seemed a subjective task. An examination of the nine papers that had required a third reading to resolve a difference of opinion among raters showed that the initial readers had usually agreed in identifying problems but had made slightly different decisions as to whether the papers fell just over or under the competent line.

An analysis of the papers showed 73% to be considered competent and 27% not-competent. The number of competent papers was slightly higher than some faculty had estimated, an outcome that pleased the task force. However, the 27% of not-competent papers represents a problem and a challenge to the School of Management – particularly since these came from SOM students who had successfully passed English 302, as well as the SOM writing-intensive course.

Next Steps and Goals

Our next step to address this problem is to find a way to reduce the number of students not competent in written communication from graduating. We are also developing proposals to set a writing standard for admission to the SOM to insure that students who lack minimal writing skills do not begin our major coursework until they demonstrate competent writing skills. We may also recommend that students who are already highly competent writers be allowed to pass out of SOM 301 and spend their time on additional major electives further developing their writing skills and content knowledge. The combination of these two factors would allow us to concentrate the resources we have for SOM 301 on the students who can best benefit from the course.

We would like to reach the situation in which every SOM graduate is a competent writer; however, we know there will always be a few who slip through the cracks. In the long run the task force believes that we can reduce the number of non-competent writers we graduate and increase the written communication skills for those who successfully pass through our program. By emphasizing a higher level of writing competence, we will better prepare our students for the world of work, satisfy the needs of the employers who hire our graduates, and improve the reputation of George Mason University graduates.

To see the rubric that the School of Management generated, click here. A collection of rubrics, broken down by the department, is available here.