“Ah yeah I think we fundamentally agree on this. I agree there are probably more clever ways you can differentiate qualified actuaries from non qualified actuaries that doesn’t seek to put much time pressure on people, but I think it would be hard to do this without effectively lowering the bar. I do think the CAS has a bar. I don’t think they say “ok only x through this time”, but I don’t think the pass rates are a cruel accident.”
My whole point is the current system is warping your perception. The current bar isn’t “qualified actuary” it’s “qualified high stakes test taker”. So yes, we would be lowering the bar of how difficult our famously difficult high stakes tests are, but we would NOT be lowering the bar of who is a qualified actuary and who isn’t. Take any exam you passed. Now imagine two people who failed that exam. Let’s say hypothetically if these two people were given an extra hour, one would have passed and one would have failed. Is the person who passed with an extra hour really less qualified to be an actuary? I don’t think so, for the reasons I outlined in my initial post. And if I’m correct, that means the current test is failing at differentiating. We’re given 3 people, 2 are qualified, 1 is not. But the result is 1 qualified and 2 not, because we’re testing speed and not knowledge.
I think the key constraint people have to recon with with any exam changes is a constant pass rate. Whether the pass rate should go up or down is a separate concern. If you want to make changes to the exam how would you implement those while keeping the pass rate constant. Otherwise it’s just like arguing for what things we should spend money on without contemplating what should be cut.
Why should the pass rate have to be constant? If say there’s 40% of total test takers who are qualified, but 25% of the 40% fail for reasons outlined above by other posters then shouldn’t the ideal pass rate be 40% instead of the 30% we’d currently see?
On the margin, sure we might say that other person is qualified. But I do think we need some sort of time limitation. If you give me 8 hours to take an exam I’ll probably sit there long enough and just figure stuff out eventually which isn’t a great differentiator either. You get more time on the job but you’re also expected to know ur stuff.
Because we’re arguing for two separate things then. One is how the test is given the other is how difficult it should be (measured in % pass). If your test is more effective at differentiating you should still be able to draw the line at the same %, you just do a better job sorting.
Yes if the population of unqualified who pass = the population of qualified who fail, but if they don’t then the pass rate wouldn’t be the same which is an okay result. I don’t think % who pass should even be a thought in the exam makers mind, if people are qualified they’re qualified.
“I think the key constraint people have to recon with with any exam changes is a constant pass rate. Whether the pass rate should go up or down is a separate concern. If you want to make changes to the exam how would you implement those while keeping the pass rate constant. Otherwise it’s just like arguing for what things we should spend money on without contemplating what should be cut.”
Why should the exam rate remain constant? It’s already fluctuates sitting to sitting, from 30%-50% sometimes. My hypothesis is that the current system is artificially keeping the pass rate down. Why should fixing it involve offsetting it with some other artificial constraint to keep pass rates constant?
The exams are supposed to be “Here are a list of things a qualified actuary should know how to do. We are going to ask you a series of questions to determine if you know how to do them.” If by removing the time constraint a significantly higher percentage of people demonstrate that they know how to do them, then that makes the exams a BETTER differentiator of candidates while as a consequence raises the pass rate. That’s an acceptable outcome.
Note that the Examiner’s Report has limited space and the graders/Exam Chair are likely only drawing on a few examples that illustrate what they felt were good answers (and received most, if not all, of the points available).
I don’t think that the Examiner’s Report was ever intended to be “comprehensive” in possible solutions for those open ended questions.
Please note that I am giving the CAS the benefit of the doubt in that statement and not because I have any sort of inside knowledge.
The unfortunate thing about the CAS is a failing candidate’s ability to see what they did write vs. a sample answer in order to learn where they’re deficient. Some years ago, the CAS did release actual responses for a couple of candidates (one who passed and one who scored a 5), and the AO community was able to point out how the one who scored a 5 fell short in most cases.
I don’t think so, and I think people make the issue too easy for themselves. If you increase the time then you’ll also pass people who aren’t qualified just because they were able to figure things out in the moment.
Assume the CAS wants an exam of difficulty where 40% pass. They accept some % of the group is sufficiently qualified but don’t know who. All the complaints posit that we’re passing lots who we shouldn’t and missing a bunch who we should. How can we change the exams to keep the % passing constant while more effectively sorting.
Just opening the floodgates doesn’t make your exam more effective, it just increases your type 1 error.
I think the issue with 7-9 is that the CAS is starting to see the FCAS credential as something prestigious that should only be awarded to people who are great critical thinkers with a really solid work ethic. It seems to be moving more and more away from whether you are a qualified actuary. People who are very good at critical thinking on the fly probably make better actuaries on average (not in every case obviously), so I see why the CAS is going with this approach. At the end of the day if the point of the exam 7 is to show familiarity with different reserving methods and that you can apply them in straightforward ways, then I agree the exam is too difficult. If the goal is to hold up the FCAS credential to be a gold star that shows you’re good at critical thinking and have a great work ethic, then I think the exams aren’t doing a poor job.
I do absolutely agree in any case though that the exams have been too long historically. The move to the spreadsheet format is definitely a step in the right direction. When I sat for both 7 and 9 I finished each with around an hour remaining because of the format, I definitely would’ve been pressed for time if they were written. I think a good solution would be to leave the exam length/difficulty as is and increase the allotted time to 5 or 6 hours.
Yes, I used TIA for 9 as well. I haven’t touched the source for any exams other than P and FM.
If the time limit was raised the questions would have to be written to a higher standard to avoid this I agree. A good exam question in a 4 hour exam or a 6 hour exam or an exam with no time limit shouldn’t be able to just be figured out by staring at it long enough. I do agree this is an issue on a few problems on current exams.
If after the questions were adjusted to avoid this the pass rates ended up higher I don’t think it would be an issue.
How does passing rate measure an exam’s difficulty?
Put another way, why should a qualitative characteristic be measured by a single arbitrary quantitative measure?
At present, the pass rate is purely subjective. MQC, as implemented, has a lot of qualifications around it as to render it meaningless as a quantitative measure for difficulty. Why you might ask? Because the points assigned to a problem are arbitrary. And if you look between two exams, that point assignment only grows in intensity in its arbitariness.
Sure, it’s not a perfect metric. But if we’re suggesting real improvements we need to keep within some bounds of reality otherwise how do you know which exam changes actually improve the exam and which just make it easier?
And dependent on the given graders and the exam answers they see.
The MQC is one of the examples where the CAS screams transparency, but in reality they just look at statistics and make a decision how many people will pass a given test based on judgment.
I think what they are trying to see if you understand the reasons behind the formulas. The brosciuos exam 7 question from I believe 2018, I believe did an okay job at this.
If you didn’t understand the fundamentals then you wouldn’t be able to figure out the formula to setup the algebra. But then it becomes an annoying algebra problem.
The cape cod one from that year they had two unknown variables and one cancelled out, and it left you with one of those gnarly formulas that x^2 + 1/x^2 in the equation. I believe I solved this by trial and error since x was the ELR. So you knew where it had to be around.