The New York Times reports today that scoring errors in thousands of SATs have led some to question the validity of the tests.

It's about freaking time!

The SAT measures little or nothing. It's a test of reasoning skills that presents itself as something more. The stated purpose of the SAT is to accurately predict the first-year college grades of the test-taker, but studies have shown that the SAT's predictions are unreliable. For example, as a generality, men score better than women on the SAT, but women receive higher grades their first year in college than men do. This fact in itself should cause anyone and everyone to question why we still use the SAT -- or any other standardized test -- in college admissions. It boils down to two things:

1. Money. The SAT is administered by The College Board, an arm of the powerful Electronic Testing Service (ETS), which runs pretty much every standardized test in the country. The SAT's biggest rival is the ACT, administered by a rival company. Some major colleges and universities switched to the ACT a few years back because it doesn't pretend to be an objective test of objective reasoning. The ACT knows that it's a test of what you have learned in school, and that's what it's supposed to measure. There are Algebra II problems on the ACT, and if you don't know Algebra II, then you won't do well on the questions that require Algebra II knowledge. In 2005, ETS introduced the "new" SAT in its March test administration after its largest client, the University of California system, threatened to switch to the ACT for college admissions of the SAT wasn't made more relevant. Analogies were out; essays were in. ETS spends a lot of money on phony statistical studies to make it appear as though its tests are actually measuring something. And why not? If ETS admitted that its tests were biased (no more racial bias; in studying the SAT -- which was my job this summer and fall -- I found more of a geographical bias than anything else) and unreliable, the company would fall apart. Imagine if thousands of students weren't paying $50 every year to take the SAT (and some students take the test more than once in a calendar year). That's a lot of lost revenue.
2. Sheer Size. Huge state schools like University of Michigan (the largest public university in the country, I believe) and Ohio State University (the second-largest public university, I think) need some way of reducing its applicant pool every year. Admissions employees can't possibly look through the thousands and thousands of applications individually, so they need something to reduce the size. In comes the SAT, which provides a handy number that stands in for "aptitude" or something like that. If OSU or U of M specifies that it will cut off applicants below a certain score, that makes its life easier. The theory is that those schools don't want students who will have poor first-year grades, but the SAT doesn't reliably measure first-year grades, so the point is moot. In any security system, a false positive means that the system isn't working. The SAT is a security system because it's trying to control access to something, and its criterion for access is the composite SAT score. What if ETS released a study showing that terrorism was linked to SAT scores? The higher the score, the lower the probability that a person would blow something up. So, we'd have people taking SATs before they boarded planes. This is a more obvious example of a security system in action, and it's ridiculous. The SAT controlling access to colleges is just as ridiculous and meaningless, but we don't think it's so ridiculous because we've accepted it as "normal." The SAT is a way of restricting access to a scarce resource, and that's fine. But the problem is that the SAT doesn't restrict that access to the people it says it's going to restrict access to. Plenty of smart people are shut out of the system because they don't take tests well. Plenty of dumb people learn to circumvent the system because of test preparation companies <cough>. Like Bruce Schneier says, any person can invent a system so secure that he can't figure out how to get past it. For years, ETS claimed the SAT couldn't be cracked, but in 1981, The Princeton Review figured it out. Now many other companies have followed. Test preparation is an industry. We should be concerned about this.

And now we found out that kids are losing, in some cases, 400 points due to errors caused by someone else. 400 points on the new test (for those of you who aren't familiar with the new test) is about 266 points on the old test. This can be the difference between getting into a college and being rejected from a college. We're talking life-altering stuff, here.

And the appeals process for the SAT isn't easy. And since you're not permitted to remove any exam materials from the room, it's not like you can prove that something was scored badly. You'd have to wait until you received your score report three weeks later and noticed that a correct answer was marked incorrect, in which case, you'd have to go through a lengthy appeals process. During the time this is happening, you're also applying to colleges, and they're not going to sit and wait while you appeal your SAT score.

With the rise of standardized testing standing in for intelligence, aptitude, or learning, we must question machine-scored results. Machines are not infalliable, and in some cases, the supposed infalliabiltiy of those machines means that people don't go to the colleges they should, people don't graduate from high school, and presidents are dubiously elected.