In fact, the most common thought I have heard coming out of this exam 9, was that I could have studied another six months it would have made no difference. This is precisely because the bloom’s level is too high and people can’t prepare for it any more.
Ah yeah I think we fundamentally agree on this. I agree there are probably more clever ways you can differentiate qualified actuaries from non qualified actuaries that doesn’t seek to put much time pressure on people, but I think it would be hard to do this without effectively lowering the bar. I do think the CAS has a bar. I don’t think they say “ok only x through this time”, but I don’t think the pass rates are a cruel accident.
I think the more recent low pass rates on exam 8, reflects the change in the admissions and exam committee being more strict.
Or effectively raising the bar for the exam.
Yeah I agree, there certainly has been a downward trend on 8’s pass rates.
To me this is almost we make the test harder, but don’t offset the required pass mark as much.
I believe this actually dates back to when the new head of admissions took over…
"For example, exam 8 a few years ago threw out a bunch of triangles to do one of the smoothing out methods mentioned in Shapland. Most people got this wrong because they didn’t realize the CAS gave the wrong triangles on purpose and that you need to do weird manipulation to get to paid triangles from incurred.
Instead of testing actual knowledge they focused on exam 5 manipulation embedded in the question. They could have asked the question in a bunch of different ways that would have focused on knowledge of pearson residuals and what not."
Yea, so I agree that’s bullshit, but only in the context of how the current exam is structured where you need to be writing full speed from minute one. Obviously in the real world it’s a very relevant concern to make sure the data you are looking at is paid and not reported, and I have had to translate paid triangles from reported triangles. It’s important enough that a qualified actuary knows how to do that, so I don’t mind it being a random curveball that can show up anywhere on the upper levels. But only on a test where candidates have ample time to review all the relevant details, which the current test does not.
But if that exam 8 was half as long as it was, that would be a totally fair thing to test.
One thing I will note, is that I do feel like the graders have been more lenient grading to offset the harder tests.
It’s fair in the context of being a good actuary. And realizing what data you have and how to use it with manipulations.
It’s not testing the material though. And that’s my problem.
I think we’ve got a lot of concerns that conflict in areas. To test just the things that determine whether someone is a qualified actuary and just the things that are exactly on the syllabus and they can’t be too algorithmic but not too puzzley either. This is why we’d be paying so much per hour for people to write questions.
I don’t know what the SOA is like. But, do they complain as much about the SOA exams. I have this feeling that people don’t complain as much. Those tests are like six hours and made up of 8 questions. I think they have higher pass rates too.
I get the sense they do a better job creating a fair test. I could be completely wrong though.
I can say as having extensive training on Bloom’s Taxonomy (my background is actually mathematics education–research level, not mere teaching it) that the CAS’s implementation of evaluating candidates at the higher levels of Bloom’s Taxonomy complete ignores all of the literature of how to do so. Namely:
- You don’t use timed test (in fact, I’ve read articles that demonstrated that timed test actually give false readings on the knowledge base of the student)
- For a (true) open ended question, you give adequate time for a candidate to consider multiple angles on that question. That is, they should be done as a project over a long(er) time frame.
For this last item, I think the CSPA has it right with the use of a capstone project that a candidate gets ~60 days to complete. And even in that case, if something is lacking, they get a chance to address that lack. THIS is a proper means of evaluating a candidate’s knowledge and ability at the higher levels of Bloom’s Taxonomy.
3 Likes
The CAS spends too much time trying to craft challenging problems or things candidates never would have dreamed of. The questions are often too long and complex as well.
The best part of it is the CAS asks open ended questions and then often only accepts 1 or 2 of their intended responses. I have seen this on quite a few exam reports.
Yeah that’s all fair, I don’t think the CAS really cares to properly follow the relevant literature on that, they merely seek to make their exams more based on critical thinking than memorization of prior problems.
“Ah yeah I think we fundamentally agree on this. I agree there are probably more clever ways you can differentiate qualified actuaries from non qualified actuaries that doesn’t seek to put much time pressure on people, but I think it would be hard to do this without effectively lowering the bar. I do think the CAS has a bar. I don’t think they say “ok only x through this time”, but I don’t think the pass rates are a cruel accident.”
My whole point is the current system is warping your perception. The current bar isn’t “qualified actuary” it’s “qualified high stakes test taker”. So yes, we would be lowering the bar of how difficult our famously difficult high stakes tests are, but we would NOT be lowering the bar of who is a qualified actuary and who isn’t. Take any exam you passed. Now imagine two people who failed that exam. Let’s say hypothetically if these two people were given an extra hour, one would have passed and one would have failed. Is the person who passed with an extra hour really less qualified to be an actuary? I don’t think so, for the reasons I outlined in my initial post. And if I’m correct, that means the current test is failing at differentiating. We’re given 3 people, 2 are qualified, 1 is not. But the result is 1 qualified and 2 not, because we’re testing speed and not knowledge.
1 Like
I think the key constraint people have to recon with with any exam changes is a constant pass rate. Whether the pass rate should go up or down is a separate concern. If you want to make changes to the exam how would you implement those while keeping the pass rate constant. Otherwise it’s just like arguing for what things we should spend money on without contemplating what should be cut.
Why should the pass rate have to be constant? If say there’s 40% of total test takers who are qualified, but 25% of the 40% fail for reasons outlined above by other posters then shouldn’t the ideal pass rate be 40% instead of the 30% we’d currently see?
On the margin, sure we might say that other person is qualified. But I do think we need some sort of time limitation. If you give me 8 hours to take an exam I’ll probably sit there long enough and just figure stuff out eventually which isn’t a great differentiator either. You get more time on the job but you’re also expected to know ur stuff.