"Fantasy hockey pool guidebooks use more rigorous assessment and criteria than the Fraser Institute," Dix declared in a statement.
He's referring to school rankings supplied by the libertarian, right-leaning Fraser Institute, a think tank favouring private solutions and free markets. But it made me curious what analysis the Institute actually does to reach its widely cited rankings. So I took a look.
The rankings are most often criticized because they are based (partly) on the Foundational Skills Assessment (FSA), a standardized assessment widely derided by educators. Here's Jane Friesen of SFU:
"It’s simply telling you how is a particular cohort of students in a school doing in a particular year. I think we have to be careful to not interpret those results as a measure of the effectiveness of the school and I think that’s where the real issue comes in,” she said.
Friesen said there are a number of factors that the rankings don't take into account, such as students' backgrounds.
In other words, the test was never intended to be used to directly compare schools, even though that's precisely how the Fraser Institute uses them. But then I dug into the actual Fraser Institute 2010 report card for B.C. and the Yukon and found that the problems go well beyond the use of the FSA.
The Fraser Institute rankings actually depend on 7 factors, only 3 of which involve the FSA. The other four are Math 10 gender gap, English 10 gender gap, graduation rate and delayed advancement rate. Combining these factors with various weights (graduation rate and delayed advancement rate accounting for fully 25% of the final score), the Fraser Institute scores schools on a 0-10 scale and ranks them "best" to "worst."
You may have spotted the problem already. Given the factors I just mentioned, which schools would you expect to do worst and best? Unsurprisingly, the rankings favor private schools that cater to the wealthy and well-to-do. Social context is completely ignored; schools are directly compared with one another despite some schools facing much more of a challenge regarding students coming from poverty and neglect and facing myriad obstacles not often seen by private school attendees. Should we conclude that West Point Grey Academy in Vancouver is really a better school than a public school in Prince George because it has higher graduation rates and better test scores? That these outcomes reflect superior teachers and better curriculum? Of course not; the challenges faced are worlds apart. We shouldn't be surprised that these 7 factors give a higher score to wealthy private schools. We should be surprised that the Fraser Institute uses those scores to rank schools as being better or worse.
To put it another way, you shouldn't look at these rankings and think "wow, I should put my kids into West Point Grey Academy." You should look at them and think "wow, I should have a wealthy family with few social problems."
I would hope that people's skepticism would be aroused even before inspecting the Fraser's Institute's dodgy use of statistics. After all, what are the chances that a right-leaning libertarian think-tank in B.C. and a right-leaning, libertarian think-tank in Washington State would use the exact same ranking scheme and that the scheme just happens to favor schools advocated by right-leaning libertarians.
Sadly, it gets worse. Many people would look at these stats and say that we should pay teachers according to how well their students do on such rankings. I think most people are sympathetic to the idea of rewarding good teachers, but punishing teachers who are faced with troubled students and real-world problems just makes the problem infinitely worse.
Finally, we should probably be skeptical of any ranking that gives a perfect score to the Bountiful Elementary-Secondary school.