14 posts were merged into an existing topic: GOOD LUCK ON YOUR EXAM 8
Yes if the population of unqualified who pass = the population of qualified who fail, but if they don’t then the pass rate wouldn’t be the same which is an okay result. I don’t think % who pass should even be a thought in the exam makers mind, if people are qualified they’re qualified.
“I think the key constraint people have to recon with with any exam changes is a constant pass rate. Whether the pass rate should go up or down is a separate concern. If you want to make changes to the exam how would you implement those while keeping the pass rate constant. Otherwise it’s just like arguing for what things we should spend money on without contemplating what should be cut.”
Why should the exam rate remain constant? It’s already fluctuates sitting to sitting, from 30%-50% sometimes. My hypothesis is that the current system is artificially keeping the pass rate down. Why should fixing it involve offsetting it with some other artificial constraint to keep pass rates constant?
The exams are supposed to be “Here are a list of things a qualified actuary should know how to do. We are going to ask you a series of questions to determine if you know how to do them.” If by removing the time constraint a significantly higher percentage of people demonstrate that they know how to do them, then that makes the exams a BETTER differentiator of candidates while as a consequence raises the pass rate. That’s an acceptable outcome.
3 Likes
Note that the Examiner’s Report has limited space and the graders/Exam Chair are likely only drawing on a few examples that illustrate what they felt were good answers (and received most, if not all, of the points available).
I don’t think that the Examiner’s Report was ever intended to be “comprehensive” in possible solutions for those open ended questions.
Please note that I am giving the CAS the benefit of the doubt in that statement and not because I have any sort of inside knowledge.
The unfortunate thing about the CAS is a failing candidate’s ability to see what they did write vs. a sample answer in order to learn where they’re deficient. Some years ago, the CAS did release actual responses for a couple of candidates (one who passed and one who scored a 5), and the AO community was able to point out how the one who scored a 5 fell short in most cases.
But there is no venue for such training.
I don’t think so, and I think people make the issue too easy for themselves. If you increase the time then you’ll also pass people who aren’t qualified just because they were able to figure things out in the moment.
Assume the CAS wants an exam of difficulty where 40% pass. They accept some % of the group is sufficiently qualified but don’t know who. All the complaints posit that we’re passing lots who we shouldn’t and missing a bunch who we should. How can we change the exams to keep the % passing constant while more effectively sorting.
Just opening the floodgates doesn’t make your exam more effective, it just increases your type 1 error.
I think the issue with 7-9 is that the CAS is starting to see the FCAS credential as something prestigious that should only be awarded to people who are great critical thinkers with a really solid work ethic. It seems to be moving more and more away from whether you are a qualified actuary. People who are very good at critical thinking on the fly probably make better actuaries on average (not in every case obviously), so I see why the CAS is going with this approach. At the end of the day if the point of the exam 7 is to show familiarity with different reserving methods and that you can apply them in straightforward ways, then I agree the exam is too difficult. If the goal is to hold up the FCAS credential to be a gold star that shows you’re good at critical thinking and have a great work ethic, then I think the exams aren’t doing a poor job.
I do absolutely agree in any case though that the exams have been too long historically. The move to the spreadsheet format is definitely a step in the right direction. When I sat for both 7 and 9 I finished each with around an hour remaining because of the format, I definitely would’ve been pressed for time if they were written. I think a good solution would be to leave the exam length/difficulty as is and increase the allotted time to 5 or 6 hours.
Yes, I used TIA for 9 as well. I haven’t touched the source for any exams other than P and FM.
Right now, they aren’t letting us to discuss the exams. Making it even harder to figure out how you improve your knowledge.
If the time limit was raised the questions would have to be written to a higher standard to avoid this I agree. A good exam question in a 4 hour exam or a 6 hour exam or an exam with no time limit shouldn’t be able to just be figured out by staring at it long enough. I do agree this is an issue on a few problems on current exams.
If after the questions were adjusted to avoid this the pass rates ended up higher I don’t think it would be an issue.
2 Likes
How does passing rate measure an exam’s difficulty?
Put another way, why should a qualitative characteristic be measured by a single arbitrary quantitative measure?
At present, the pass rate is purely subjective. MQC, as implemented, has a lot of qualifications around it as to render it meaningless as a quantitative measure for difficulty. Why you might ask? Because the points assigned to a problem are arbitrary. And if you look between two exams, that point assignment only grows in intensity in its arbitariness.
1 Like
Because we have to quantify it some how.
Sure, it’s not a perfect metric. But if we’re suggesting real improvements we need to keep within some bounds of reality otherwise how do you know which exam changes actually improve the exam and which just make it easier?
And dependent on the given graders and the exam answers they see.
The MQC is one of the examples where the CAS screams transparency, but in reality they just look at statistics and make a decision how many people will pass a given test based on judgment.
Can we talk about the reasoning behind the CAS testing algebra on fellowship exams? I mean, honestly, where do I even begin on that? No words.
I think what they are trying to see if you understand the reasons behind the formulas. The brosciuos exam 7 question from I believe 2018, I believe did an okay job at this.
If you didn’t understand the fundamentals then you wouldn’t be able to figure out the formula to setup the algebra. But then it becomes an annoying algebra problem.
The cape cod one from that year they had two unknown variables and one cancelled out, and it left you with one of those gnarly formulas that x^2 + 1/x^2 in the equation. I believe I solved this by trial and error since x was the ELR. So you knew where it had to be around.
But that was a pretty terrible question.
1 Like
“I don’t think so, and I think people make the issue too easy for themselves. If you increase the time then you’ll also pass people who aren’t qualified just because they were able to figure things out in the moment.”
Ah, so here we’re coming to a fundamental disagreement. These exams aren’t multiple choice. You can’t just guess and check the choices until you get a right answer. If you’re someone who didn’t memorize the formula for the credibility of a claims free year from Bailey and Simon, but instead spent 10 minutes deriving it from first principles, I don’t consider you any less of a qualified actuary than the person who memorized the flash card and took 20 practice exams and knows how to answer it just by glancing at the table (I’m the latter one btw). Knowing how to answer that question faster and quicker isn’t the same thing as being the more qualified actuary.
"Assume the CAS wants an exam of difficulty where 40% pass. They accept some % of the group is sufficiently qualified but don’t know who. All the complaints posit that we’re passing lots who we shouldn’t and missing a bunch who we should. How can we change the exams to keep the % passing constant while more effectively sorting.
Just opening the floodgates doesn’t make your exam more effective, it just increases your type 1 error."
So a couple of points here. I think your premise if flawed. The CAS doesn’t even claim this to be the case. Why should there be an arbitrary 40% pass rate? Why are you assuming a coherent rank-ordering strategy of “qualified actuary” even exists? In the extreme case where everyone who is sitting for exam 9, having passed 9 other very difficult actuarial exams already, is actually qualified (not that extreme when you think about it), why would you want to rank-order them and only pass the top 40%? Isn’t it pretty obvious that any such rank-ordering will be fairly arbitrary? i.e. if they are all qualified actuaries, we can rank order them based on their ability to solve algebra problems very quickly, and result in a given 40% cohort. But we can also rank order them on their language and grammar usage (which is even more relevant for a professional actuary) and come up with another rank ordering. Or we can rank order them on their ability to solve complex logical puzzles, and have yet a 3rd ordering. Neither of these 3 methods are any better at differentiating what the test is allegedly a measure of, is the person a qualified actuary.
By constraining yourself to a set pass mark you are necessarily making an arbitrary choice that results in a worse ordering. Making your test actually be about what it’s supposed to be doesn’t increase your risk of type 1 errors. Sticking with an arbitrary ~40% pass rate and testing algebra speed does increase your type 1 errors though.
As an example: Last years test had a really dumb problem where we had to solve a system of 3 equations to get the answer. Some people did this quickly. Some people did this slowly. Then we get to the last question of the exam. Imagine the person who did this quickly gets to the last question of the exam and says “Wow, I have no idea how to answer that question, I don’t even understand it.” Whereas the person who does it slowly runs out of time before they even read it, but had they had time they could have answered it for full points. The CAS then looks at their exam statistics and says “wow, question 20 sure was difficult, so many people didn’t even attempt it, I guess we’ll set the MQC relatively low for that one.” The fast algebra guy might score higher, despite not being fully qualified. Any time you’re testing on some other variable you’re going to get these kind of errors.
In exam 9 they were much more puzzle focused, no algebra. They are shying away from this.
I believe exam 9 two/three years ago had an algebra question with three system of equations. And you had to do it twice in part a and part b both worth 1.5 points. From what I heard is that those questions took forever.
Being able to do algebra is part of being an actuary? Explain.
Algebra falls under puzzle imo.
I can confirm. I was pretty certain that I had failed the exam because of this question. Thankfully, I ended up scraping together enough points to still get the pass.