Yup, I think this is the main difference from exams like the SATs
How does this make it harder to change the syllabus?
If you need a large body of questions on any topic, it’s expensive to add a topic.
I don’t think it’s so much a matter of the (new) system makes changing the syllabus difficult so much as the logistics behind a change in the syllabus would require both a review of current problems (to ensure conformity to new material) as well as development of new questions to cover the added material.
To be sure, additional consideration of whether or not to accept an answer that would be true under prior syllabus material but may be contradictory with new material.
The CAS just simply doesn’t have the robust resources off of volunteers to make any huge syllabus changes. Back when the SAT got rid of analogy and added new types of questions, no one questioned anything (and do we really think no one wanted to challenge some of the analogy questions), because they’re a trusted to handle such a task. People don’t trust the CAS to make changes like this (past blunders, unpaid writers and graders, etc.)
I expect there to be far fewer defective questions since they’re now building a reusable problem bank.
What’s going to happen in this context is that two, maybe three, questions will be “new” problems to be assessed for quality. So, if there are 100 points available on the Exam and 6 of those points are from “new” problems, then the passing mark would be determined based on the other 94 points available.
You have to get 94 points worth of questions without defects to start though. Year 1, presumably, won’t all questions be new? The CAS has a long history of defective and ambiguous questions on upper level exams.
Also, can’t really figure out how to quote correctly on this forum.
Yeah I get that. I guess my hope is that if they can reuse some questions for parts of the syllabus that aren’t changing that should free up some resources to focus on writing really good questions for the new stuff.
1 Like
https://www.casact.org/press/index.cfm?fa=viewArticle&articleID=4836
If you’d like, I could read this to you, but it would probably be better and faster if you read it yourself, out loud if you want to hear it. 
ETA:
Basically, it’s due to the switch to computer-based testing (CBT) and having testing windows rather than a single time when all candidates take a particular exam. There are some other side benefits (and of course some drawbacks) but my understanding is that the primary short-term reason is CBT testing windows.
There are about a decade’s worth of examiners reports with this info, and the announcement said sample questions will be provided when any syllabus material is added in the future.
As far as I can tell, question types and the grading approach aren’t directly affected by this change.
2 Likes
Well, it may solve the problem of some exams being too long, and it should decrease the number of defective questions over time. So that’s good. The first couple of years will be rough than the subsequent ones, after a question bank has been built up.
I have to disagree with:
- A higher level of exam security, to address concerns with the potential for cheating associated with the move to computer-based testing and exam windows.
however. This is a small field. A large fraction of the people sitting for these exams are close friends with other people sitting for them, including people sitting next year. Some socially connected candidates WILL learn about some exam questions. But perhaps that won’t be a large problem.
1 Like
One important key for some of the “core” calculation problem is to “change the numbers” along with what part of the formula is missing from the information given.
So even some of the Fellowship Exams can have a good bank that would be easily managed from that stand-point. But I do agree that the Fellowship Exams need to be more flexible to address current issues; which would be difficult to account for in a “well-established” bank. But if there is a reduced need for Exam Writers across the board, I’ll bet that you’ll be able to find the right volunteers to focus their energies on writing far better questions for this situation.
1 Like
Well, that’s up to CAS, not really the SOA, at this point.
TIL: The CAS Candidate Code has been updated to prohibit discussing any exam questions ever, not just during the exam window. That really feels like a mistake.
3 Likes
Hi Marcie thanks for all your input. I attended the CAS CBT webinar today and I was a bit disappointed with their response.
Someone asked if you can discuss exam questions after the window with another person who took the exam. They said no. This seems shortsighted.
1 Like
They’ll have a very difficult time enforcing that requirement.
Perfectly legit to expect folks to not talk during active administration. Something entirely different to view it as a NDA. After all, how can they prove definitively that the discussion was on the problem itself and not about the topic the item is intended to evaluate?
2 Likes
Yeah, that’s where I heard it. You described it better than I did. Also, “shortsighted” is a great way of putting what I thought of it, too.
This is a little bit off topic. The pearson environment will be a huge improvement. But there are a number of bugs in the environment. They didn’t mention these bugs. I am concerned test takers will be surprised when they run into these issues on the exam.
They also didn’t address what happens if someone gets sick or is in quarantine or what not. It seemed like they were avoiding certain topics. Maybe there were just too many questions.
Appearance of threat.
If the environment from now on is that anyone can rat anyone out and invalidate their candidacy completely, nobody will dare even bring up the topic. Similar to how most actuaries don’t talk about their recreational drug use during non office times, because it might come back to haunt them, even if there’s no way to really prove it either way.
I don’t think that the CAS can withstand a lawsuit to enforce their claim. Plus the need to have someone monitoring outlets of discussion to see if something is being discussed. granted that you can have AI and crawlers, but if one is having a discussion on the topic, you’re going to have a really tough time filtering out messages that actually are attempting to discuss actual exam questions.
Bottom line, I don’t think the CAS leadership is really interested in penalizing people who are legitimately trying to learn material, but wanting to insure security of their exams. One problem that I see that is present is the unnecessary expectation that all problems on an Exam should never have been seen before in order to be valid items to assess knowledge.
1 Like