The Trembling State of College Admission Tests: Part II — Are They Getting ‘Harder’?

Part I left off pondering the idea that college admission tests can be quietly made harder for various reasons.  ACT’s rising popularity has made that test in particular a target of such accusation.  Here I address these myths and explain where these unfounded suspicions originate.

Has the ACT gotten harder?

This has been a bold claim in some test prep circles, and the big miss. It’s understandable when students depend on what feel like finely tuned testing palates to extrapolate their own experiences to 1.85 million ACT takers. It’s troubling when experts do so and then wrongfully report it. This mistakes evolutionary changes in the test for changes in difficulty when in fact the former are required to avoid the latter. Test content has to shift — on a very gradual schedule — with the times and based on external pressures. It’s hard to separate the difficulty one senses on certain problems from the overall test difficulty encountered by all high schoolers. That’s where psychometrics comes in.

Difficult is a loaded word. Just about anything can be described, emotionally and subjectively, as difficult. We know what students mean when they use the word but that doesn’t cut it for test-makers. When psychometricians use the word, they are not describing an item, they are measuring it. If too few or too many students miss a question, it’s useless. If this were a footrace, we wouldn’t be measuring speed (whether with a tailwind or into a headwind) but rather how many racers you beat. Golf is a better analogy: lots of individuals competing on a common course (the test) whose slope (difficulty rating) has been set by previous golfers (reference group) and then rank-ordered on the leader board. But golfers aren’t then scaled, so that’s where that comparison ends. With standardized tests, there are scales and every scale is a bit different from exam to exam. We all know this, and then we all sort of forget it.

Over the last 15 years, there has not been much variation in the raw-to-scaled stability — especially when we look where we should be looking — where the kids are. The meaty 16-28 range is a far better indicator of how the majority of college bound students are doing. Even in non-ideal situations where the scaled score dropped to 34 based on 1 error (which indicates an “easy” raw test for the top 1%), the scales stabilize quickly. You don’t see multiple gaps happening (the SAT is always going to display more gapping because of the 61-point scale, but the result is the same).

Test prep experts tend to score at the top themselves and often limit their analysis (and recollection of the test) to that narrow band. I regularly mention the weakness of these tests at the extreme ends, and how a test loses optics at its tails. The makeup of the test from a student’s experience of difficulty has not changed in at least the last several years. The national average has been stable. The p-value for the average student has been about where it should be on a normed test — hitting the sweet spot. It is easy to remember the really tricky problems, the ones strong test takers belabor and remain haunted by. They are terribly skewed in how much time they require for one added point. What we are all bad at doing, though, is remembering the ones we pick up between questions 1-30 — let alone taking inventory of any slight changes there. A high scoring student quickly conquers those, which is exactly why she has 2-3 minutes to burn on that polynomial division problem that pops up late.

Could the items underneath be shifting? Is there a creeping Common Core?

There is some paranoia in just about any testing discussion these days. Fears feel validated everywhere we look even when we know that we shouldn’t be thinking about them. Yes, Common Core is playing an important role in the redesign of the SAT, although note that President Coleman has handpicked from among a range of standards in Common Core, and Texas, and Virginia. That’s a different path than ACT who touts its test aligns with 100% of the Common Core standards. The College Board neither can nor wants to say that. They have intentionally given a cold shoulder, for example, to geometry. ACT, meanwhile, was already maintaining full compliance with Common Core by 2010. But the last major test overhaul was 1989. Huh? Well, ACT had a hand in writing the Common Core standards and there’s been some squinting-until-you-see-it happening both at ACT and on the ACT ever since. Everything they do now is seen through the lens of the New SAT and Common Core.

The difficulty claim is problematic. One article recently proclaimed that an old 36 is like a new 31 (which would be like saying a 690 on the SAT is the new 800). I was then further bewildered when that claim was linked to a spike in smart and well-prepped kids on the coasts discovering the ACT. That is absurdly inaccurate. This ignores that ACT scales were set based on the performance of a sample of seniors in October 1988. Californians and Iowans are all equated to that same gang (ACT did do some additional equating work in 2005 when the calculator option was added, but their testing showed that it really didn’t make a difference.). If more strong test-takers take the test, we get more 34’s, 35’s, and 36’s (and indeed we’ve seen that; it’s just that a fairly stable mean disguises the outliers). What we don’t get is countervailing pressure to push the scale back down again. Why would ACT even want to do that? To preserve some sense of cohort difficulty while completely abandoning their core mission of a commonly accepted baseline? And if such a radical change were happening on the ACT, wouldn’t that totally scramble the relationship between the SAT and ACT?

That said, the ACT has always cycled in some oddballs. They want to claim to test law of sines or double angle formula or matrix addition, but there just isn’t enough room – literally or psychometrically — on any one test. Most of any ACT is same-old-same-old. But outside of those retreads, you will see the occasional oddity, and maybe that’s happening more given the trembling state of testing these days. The former head of R & D at the College Board is now at ACT (and vice versa). These are interesting times.

There is the supposition that we are seeing more problems in context. I would contend that this is mostly paranoia. The ACT has always prided itself in putting things in context — even to the point of hilarity. Swimming pool, oil drills, hot air balloon ropes, blimp tethers, lighthouses, and car rides are just a few of the situations ACT has created for trigonometry problems. Basically, if you can make a right triangle with it, ACT will try to work it into an item at some point. But that’s all these questions are: here is a right triangle that we’ve spent too much time describing; solve for one of the missing pieces of information using SOHCAHTOA. This is not really what the standards writers were hoping for when they talked about students proving and modeling math using real world examples. This is just an item writer’s need for variety. No matter how convoluted the background material, the first thing every good student does is just redraw a simple right triangle. Actually, the diagram usually provides a nice right triangle. “The context” is gravy.

We’ve heard other recent conspiracy claims to which there is always a reasonable explanation. Maybe it’s a geometry problem with more steps, none of which is that hard. Bread-and-butter for the ACT, because they simply do not jump into difficult math. So they put rhombuses inside of squares inside of circles inside of isosceles triangles. These are times tables with tricky triangles. Annoying, sometimes time-consuming, and not without their slip-ups. But not dramatically more difficult. Higher order polynomials appear every few ACT’s. So does inverse trigonometry. Functions requiring being careful about the order of operations is classic ACT. They love the confusing functions (they also love easy functions, which is where all of the sub-32 students are getting their points). Difficult probability problems? If anything, ACT has made probability way too easy, not too hard. Common Core actually calls for much more here.

So maybe ACT has been just mixing things up, or maybe they’ve self-identified a weakness in light of Common Core. But strictly speaking, no, the ACT has not suddenly gotten harder (and psst — the redesigned SAT won’t either, so don’t believe the hype!).