Yeah yeah, they claim that.
I’m just not sure that I buy it. I’ve talked to exam committee folk too, and Bruce Schobel wrote about it extensively on AO with a considerably more cynical take.
As the exam committee members are increasingly steeped in the material it all starts to look easy to them. Once they’ve asked all of the obvious questions on a topic if they want to ask new and different questions each sitting they have to go to increasingly esoteric material. And the study guides get better at the same time as the questions get harder so the candidates are better prepared but they answer similar numbers of questions correctly.
Look at the 2000 and 2004 sittings of Course 2, or Course 3, or Course 4 side by side… there’s no comparison in terms of difficulty. (2004 was way harder.) Yet the pass marks were largely unchanged. There was one particularly easy sitting of Course 4 (Fall 2002 IIRC) that had a much higher pass mark, but in general they didn’t move much despite the exams, with that glaring exception, getting much harder.
But the study guides were also way better by 2004 than they’d been in 2000. I mean, not a study guide but when the Macroeconomics study note first came out it literally had the words “supply” and “demand” backwards on something like 50 pages (presumably an example of Replace All gone horribly horribly wrong). I can’t even begin to comprehend the torture that students must have gone through trying to make sense of that before the errata list came out.
So sure, it was a lot easier for candidates to absorb the material after they got that fixed. But … are the exams measuring effort or knowledge? It always seemed to me like it was a lot closer to the former… while claiming to be about the latter. If it was about the latter then the pass rates should have been markedly increasing over time and they weren’t.