How the New ACT Writing Test Went Wrong

[On June 28th, 2016 ACT threw in the towel on the 1-36 score and decided to start reporting scores on a 2-12 range starting September 2016. To find our more about the impact, check out our extensive explanation of the changes.]

[Just after this post was finalized, ACT announced on October 30th that scoring of September tests had been completed internally and that reports would be posted over the weekend in time for early admission applications. This development is a big relief for impacted test takers, but does not change the conclusions of this piece.]

The September 12 administration of the ACT was the unveiling of the new and improved ACT Writing, the optional essay component that more than a million students choose to take each year. There were new prompts and instructions, of course. The test length was extended and a new scoring guide (rubric) for essay graders put in place. New domain scores, a scaled score, and a combined ELA score were a part of the revised package. Thrown into the mix were mostly seniors, many of whom had grappled with a different essay assignment and different scoring only months before. All this was taking place approximately 5 weeks before November 1 early application deadlines. What could go wrong? As it turns out, plenty.

Quality Assurance
Scores have been delayed by weeks for tens of thousands of students, upending some early application plans and causing confusion and anxiety on the part of students, parents, counselors, and admission officers. More troubling for the long-term health of ACT Writing is that the scores students have received are causing almost as much confusion and anxiety as the delay. The questions around ACT Writing come at a time when colleges are re-evaluating the importance of the essay. Initially, ACT told students that the delay was “a routine part of our quality assurance and scoring process.” More recent statements have acknowledged that “scoring and reporting of results is taking longer than typical due to the introduction of the enhanced design of the writing test, which uses a new scoring rubric.”

Despite the fact that Writing is an optional component of the ACT, the testing organization has said that it is unable to send an ACT with Writing student’s official score report — even of just the core multiple-choice sections — until the essay score has been finalized. ACT does not expect to be fully out from under the backlog until November 6 [we now know that ACT was able to push this target forward to 10/31.]. In an unprecedented statement for a major testing organization, ACT has recommended that students send colleges screenshots of their multiple-choice scores as a stopgap measure. Admission offices have had a mixed reaction to this recommendation, and it has done little to tamp down anxiety. In an effort to downplay the delays, ACT has pointed to the fine print of their testing policy that notes that scoring may take up to 8 weeks. Typically this only comes into play for exceptional cases where sheets have been misplaced or mangled, or additional investigation needs to be done. A college admission testing delay impacting this many students for this long has not occurred since the College Board’s “humidity problem” in 2005.

Thousands of New Scores Every Day
College Board and ACT both have a history of secrecy when it comes to registration numbers and scoring issues. While the organizations disclose the annual number of test takers, they do not release the number of students testing on a particular date. ACT has followed that playbook by not specifying how many students took the September 12 test, how many students received their results “on time,” or how many students are still awaiting scores. Given ACT statements that they are “releasing thousands of new scores every day” and given the length of the delay, we can project that tens of thousands of scores were impacted.

Scoring several hundred thousand exams requires a massive scaling up of part-time readers and a sudden scaling down when the readers’ work is done (or estimated to be done). ACT takes advantage of the web to distribute grading responsibility, but any change in deployment plans for its reader army would not have come without hiccups. ACT’s scoring capacity may have dropped when it needed it most because it had, in a sense, already planned for much of its army to go home.  An irony is that ACT became what it is today because of a revolution in test scoring speed — ACT founder E.F. Lindquist patented a machine for pencil mark scoring in 1962.

A Preventable Error
The choice of September for a rollout of the new essay was an error — a preventable error. A February 2016 launch made more sense in several important ways and would have mimicked the launch of the original ACT Writing in February 2005. Why was the September introduction a foreseeable and preventable error?

1. Fall scores are more time sensitive than spring scores. A delay in February scores would not have jeopardized college applications in the way that the September slowdown has. Field tests of several hundred or several thousand students cannot replicate the processes needed to score hundreds of thousands of essays. A scoring overhaul of this magnitude — readers need to assess and score an essay in four different areas instead of giving a single “holistic” score — should have come with better contingency planning and a longer runway.
2. Testers who were repeating the ACT this fall had to prepare for a very different essay. Taking the ACT in the spring of junior year and repeating the exam in the fall of senior year is a testing pattern followed by an estimated half-million students each year. It is unfortunate that some students were taken by surprise by the revised essay format. A February rollout would have more neatly separated the class of 2016 (original ACT essay) and the class of 2017 (new ACT essay). The delay also would have provided more time to educate students about the change. [Compass had been tracking the new essay since its announcement and made sure that its tutors, students, materials, and practice testing facilities were ready.]
3. The September introduction has left students and admission offices to face the perplexing issue of how to compare the raw, holistic 2-12 score on the original ACT essay to the four 2-12 domain scores and scaled 1-36 score of the new Writing test. The contributions and limitations of test results are more easily understood when all applicants have taken the same exam. A February start would have meant that essentially all 2015-2016 freshman applicants would have 2-12 scores and almost all 2016-2017 freshman applicants would have 1-36 scores. Instead, a concordance is used to provide comparable scores. Concordances suffer from notable liabilities, and the limited score range of the original essay exacerbates those liabilities. For example, a score of 8 on the old essay concords with a scaled score of 23 on the new essay, while a 9 leaps to a concordant score of 30. As for domain scores, nothing on the old essay compares to the analytic rubric used on the new essay, so admission offices use the scores at their own risk. “Is ‘Development and Support’ more important than ‘Ideas and Analysis’?”

It’s not clear what benefits ACT saw in a September rollout. ACT publishes its Preparing for the ACT on a school-year calendar, but the convenience of keeping a single booklet does not seem to justify splitting an admission cycle down the middle. Perhaps ACT wanted the stage to itself (now a mixed blessing). A February 2016 launch could have buried the essay revamp in the noise surrounding the redesigned SAT’s debut in March. College Board and ACT regularly compete to “get there first.”

The Future of ACT Writing and the SAT Essay
The problems with ACT Writing come at an awkward time. College Board is making the essay optional on the redesigned SAT. As long as the essay was an integral part of the SAT, many colleges felt that the logical response was to require ACT with Writing (or ACT Plus Writing as it used to be called). Now that both College Board and ACT have made the essay optional, colleges are re-evaluating writing tests that increase fees by approximately 40% and lengthen testing by 20-30%. Do the SAT and ACT essays add sufficient value to the admission process to justify their requirement? A growing number of schools are saying “No,” and few are coming out strongly in favor of preserving the timed writing sample. Harvard, Yale, and Princeton will continue to require the SAT or ACT essay for the class of 2017, but Penn, Cornell, and Columbia have dropped, or are dropping, the requirement. Boston College, Boston University, and Northwestern are also doing away with the essay. University of Chicago, Georgetown, and Washington University in St. Louis never adopted the writing tests in the first place. The University of California system is the largest recipient of score reports in the country and will continue to require the essay portions of the exams. University of Wisconsin is a system on the other side of the divide and is eliminating the writing requirement for next year’s applicants. Most colleges and universities are still in the “To be determined” column.

ACT cannot undo the mistakes of the essay rollout, although there are changes we hope are made. Scoring inadequacies need to be permanently addressed. If the September administration of the ACT risks regularly becoming “too late for early admission,” large swaths of students will have their options limited. The organization should make a new commitment to transparency when problems develop. ACT needs to devote resources to making students’ essays available online so that students can use the test and the rubric as a true learning tool. The essays are available to high schools and colleges, but not to the authors. More research needs to be presented to students and educators about the reliability and predictive validity of the new prompt and new scores — this is true for the ACT and SAT.

Meanwhile, Compass continues to recommend that students take the ACT with Writing. Application plans change and a skipped essay could negate great scores on English, Mathematics, Reading, and Science, as schools that require Writing will not necessarily “superscore” Writing and non-Writing test dates. The long-term prognosis for ACT Writing and the SAT Essay depends not on ACT and College Board stating that writing is a fundamental skill for college success — it undoubtedly is — but on their ability to make the case that an impromptu essay is useful in cultivating or assessing that skill. The “To be determined” column is being rapidly determined.