The pass rate does not seem to be predetermined in this case since the CAS said that the delay in results was from setting the pass mark. Also, if there is a clear advantage to having two attempts then why not have two pass marks? It could be they did and the reddit sample is biased since only smart people are on reddit, but I hope that the long awaited CAS report addresses this point.
2 Likes
Think about what you’re asking for though. You want the CAS to have a separate pass mark to fail more candidates because they were better prepared through no fault of their own. That goes against the idea of the MQC.
1 Like
No, I am asking for a different pass mark since they saw the material already.
The idea of a MQC is very different between taking a fresh exam or re-taking an exam one has already seen.
4 Likes
There is a world of difference between being minimally qualified on a broad syllabus, and being minimally qualified when you already know how and what specific areas are being tested.
6 Likes
You seriously think the pass mark should be exactly the same between the two groups? You even outlined what a massive advantage it was for one group over the other, so why do you think they should get the huge advantage that others didn’t?
Everyone should have the same chance to pass. The only thing that should vary is your own preparation, not which day you take the exam.
My exam-taking days were done before Blooms and the MQC concepts were formally introduced, and I have never served on an exam committee. So, what follows is from the viewpoint of merely someone who was brutalized by the exams in the olden days.
Back then, the conventional wisdom among us victims was that the pass mark for a particular exam×sitting was selected by examining the distribution of raw scores, and making a judgment call. If there wasn’t a clear break in the distribution / if there was a concentration of results near the notional ideal, there would be a bias preferring to pass “too few” rather than “too many”.
(I reiterate that this was not necessarily reality; it was just the belief that exam-takers at my then-employer held over 20 years ago.)
With that in mind…if that were accurate today, I would expect that the process for the situation this sitting would have been:
- Look at the distribution of raw scores of all exams except those on May 1 and the retakes, and set a pass mark.
- Look at the distribution of raw scores from May 1. Candidates who exceed the mark from step 1 pass. This distribution would inform how much of a penalty the May 1st situation created.
- Look at the distribution of raw scores from the retakes, and note the difference from 1 and 2, to get a biased indication of the advantaged gained with extra time and knowledge of the exam.
- Identify candidates who exceeded the mark from step 1 on May 1, who also retook the exam, and look at the distribution of raw scores from this subset of retakes
- Set a pass mark for the retakes, informed by steps 1-4. Presumably this pass mark will be slightly higher than that from step 1.
To re-reiterate…I don’t know that that’s what was done, and I don’t have recent exam experience on either side of the exam paper (or Pearson terminal). However, if my old understanding of the old way things were done is close to accurate…I’d bet that the five steps I described would be pretty close to how they would have handled this debacle.
EDIT: I could see someone arguing that a lower pass mark for the May 1 non-retake exams would be appropriate. While I would agree that that makes sense…that would be inconsistent with my “old days” experience. I don’t have an informed opinion about whether it would be consistent with current thought processes.
2 Likes
The CAS have said that they only graded the second attempt if you failed the first, so I don’t think they have a great distribution of second attempts, let alone separate pass rates (in theory they could have set a higher bar based on some judged relativity, but I doubt it)
There’s a reason I was heavily disclaiming my post as having the viewpoint of only someone who took exams a couple of decades ago, and not someone familiar with current thinking or inside knowledge of the sausage-making of deciding who passed. 
That being said, I can see an interpretation of “only graded second attempt” that doesn’t exclude the possibility that raw scores were generated on the second attempt papers, and there are a few of us who get paid the big bucks to make judgment calls based on having very limited data points describing a distribution.
EDIT: And thinking about it a little more, I can also see where the process I described would work if they did only mark the papers of those deemed to have not passed on May 1. It’d just require a bit more of a judgment call.
If the exams were exactly the same then adjusting the pass mark for the second sitting doesn’t seem unreasonable. I’m pointing out that the pass mark is a blunt tool, and raising the pass mark will fail more people, full stop. Maybe people are okay with this, but in my experience candidates aren’t too keen when the pass mark is set really high.
From my understanding, there were candidates who were able to get through the entire first sitting, some who couldn’t even start the exam, and everything in between. Ideally, you would want to suss out the candidates who saw most or part of the exam, weren’t going to pass the first time, and went home and memorized how to answer those specific questions. There isn’t a straightforward way of detecting this, and I imagine that’s a major reason why those consultants were hired.
Historically, when an exam debacle occurs, the pass rates tend to go up, as the CAS errs on the side of being lenient. There’s some undercurrent that they may have been too lenient this time?
Not that they are being more lenient, rather they are being more lenient to those who potentially saw the exam twice. If the pass rate was high for both groups then people would not be antsy, but since the May 1 cohort has a much higher pass rate (according to reddit surveys) this gives the impression that those candidates had some advantage.
For more reference see the following reddit polls:
https://www.reddit.com/r/actuary/comments/1ea7z7t/exams_59_poll_discussion/
https://www.reddit.com/r/actuary/comments/1eaai81/exams_59_may_1st_poll/
https://www.reddit.com/r/actuary/comments/1ea99h0/exam_7_poll_exam_7_only_scroll_down_for_exam_59/
https://www.reddit.com/r/actuary/comments/1ea9yi0/exam_9_poll/
All of these show that May 1 candidates, and especially those that saw the exam twice, had an advantage.
Three questions you should be asking yourself as you look at the Reddit survey results:
- What proportion of exam-takers attempted the exams on May 1, and how does that compare to the distribution claimed by survey participants?
- Is there a potential for certain exam-takers to be more-or-less likely to participate in a Reddit poll?
- Is it possible that there is a bias in the qualifications / likelihood of passing of May 1 candidates who signed up to re-test?
I thought about that but in my opinion it would require a lot of non-May 1 sitters to be non-reddit people and the results seem intuitive so I gave some credibility to the results.
If 3 of the 4 polls were representative, more people sat for exams on May 1 than didn’t.
I don’t think that’s the case.
Consider the possibility that exam-takers that are particularly riled up waiting for results because they were involved in the May 1 debacle, because they knew they were going to be close to the pass mark, or because they thought they passed but actually didn’t might be more motivated to express their frustrations and possible relief on Reddit than other candidates.
(The fourth poll was for May 1st folks only.)
I’m sure some of the May 1st takers did have some advantage. But I don’t know how you’d even attempt to translate that into an adjustment to the passing score. Every candidate’s exposure to questions was different, and I believe the only information the CAS has is whether the candidate typed anything on the question.
Also, every candidate’s recollection of the exam is going to be different, which is a big factor in how much gain they get from seeing it twice. Some people can practically regurgitate an exam afterwards. Others come away with a few vague memories of categories where they struggled. If you try to offset the advantage gained by the first, you end up disadvantaging the second, which clearly isn’t fair.
I think I passed on May 1. On the retake, I went from not having time to attempt the last question, to finishing with 5 minutes to spare. And I had one question where my answers made a lot more sense the second time. So I probably did do better on the second try, perhaps by 3-4 points. But it wasn’t an overwhelming difference. But like I said, even if they wanted to, I don’t see how they could make any adjustment that wouldn’t end up disadvantaging someone. And I think that had to be the overriding concern.
2 Likes
This actually seems reasonable to me because the number of exam takers on May 1 is likely what caused the Pearson issue in the first place. It was also the last day to take the upper level exams, and I would expect more candidates to register for the last day for various reasons.
1 Like
I thought Pearson had some software issue that affected non-CAS tests, too. An i misremembering?
1 Like
That’s what I heard as well, but honestly I have not followed it very closely.
The software issue did affect other exams “dependent on memory, including CAS exams”. This doesn’t really explain why the issue occurred on the last day instead of the first day, however.