That is subjective and will highly depend on the candidate’s experience. There are many situations where I’d find (fully) discussing what I read in a footnote to be easier than setting up a lengthy yet basic series of calculations.
Further, should the goal of a timed exam be to assess whether or not a candidate read a footnote? I’m not arguing that the discussion of the footnote doesn’t demonstrate a candidate’s knowledge, but should MQC be set in such a manner that requires this discussion to pass?
Disclaimer: I am not an officer of the CAS or the examination committee. I have served as a volunteer, but that’s it. As such, I speak for myself not for the CAS; my comments are my own and do not necessarily reflect the views of the CAS exam committee.
These 2 statements are absolutely true, ime.
This is mostly true. Exam committees go into grading with an a priori MQC score for each question, based purely on a standard related to the LOs & the perceived difficulty of the item. Graders get some input into the final MQC score for their question, which is based on the same standard, but they can arrive at a different number than the a priori score due to different interpretations of the standard/difficulty or due to actual candidate results, especially if there is an accepted alternate answer that wasn’t anticipated in the a priori MQC.
There can also be overall length/difficulty adjustments to the pass mark. From what I have seen in the past, these (almost?) always result in the pass mark being adjusted downward because the exam was deemed overall too long or too difficult, based on stats such as how many questions were left blank. I have never seen an exam deemed too short or too easy in total.
True.
True, but one thing to remember about the prior CBT exam 5 is that many candidates had essentially 2 attempts, with a pass on either of them giving a pass for exam 5. If every candidate had a random 40% chance of passing any given exam…
(Yes, I realize that would be a ridiculous assumption.)
Yes, this is possible. It’s also possible that, like the exam 5 CBT experiment, the new format could go terribly wrong for a large number of people. I hope that doesn’t happen.
Can you explain why it is inappropriate to apply the concept in this way? I get that it’s slightly different than how the theory was developed, but that in itself doesn’t make it inappropriate. The multiple choice test where the result of each item is binary (fully right or wrong) could just be a special case of the way the CAS is applying the idea.
How is it not just using the MQC idea at a higher Blooms level?
If the MQC is expected to know topic X only somewhat (say, 50% knowledge, whatever that means), then s/he/it will likely be able to earn more (partial) credit on an easier question on topic X than a more difficult topic X question.
This is likely more valid when higher difficulty is based on higher-level understanding of a topic’s nuances, and not simply additional calculations or complicated algebra.
By “misapplication,” I’m referring to the situation that MQC theory is validated on MC-exams. I’m not aware of any studies/research of using this sort of theory in other contexts (like short answer and essay forms of evaluations).
At its core, MQC theory is a binary theory (the candidate either knows it or doesn’t) and the available points for a response item is all-or-nothing. If anyone is aware of scholarly studies using MQC theory in a setting other than MC-exams, I’d love to see them.
In response to the “application of MQC theory at a higher Bloom’s level” . . . I’d love to see the research that validates such application. In addition, if one looks at the literature on the use of Bloom’s Taxonomy for evaluation, they’ll find that timed exams are the worst vehicle to assess those higher levels.
If MQC theory was applied (based on the established theory), then “difficult questions” would still be applied at the entire question level. For purposes of this, problems that are broken down to parts a, b, c, etc., each identified “sub-part” would be evaluated individually. So, if it’s believed that the MQC would not be able to answer the part in its entirety, then the MCQ threshold would be zero (not some some percentage).
FTR, using the idea of a minimally-qualified-candidate as a way to establish a pass mark is a good one, IMO. My beef is the CAS making claims that they’re using a theory with well established background but using it out of context of that well established background.
NOTE: I’ll look to extract some of the discussion from this thread and move them to a new discussion in the General section as I fear that we’ve drifted from the main topic of this thread. But I’ll have to do that this afternoon.
In addition, if one looks at the literature on the use of Bloom’s Taxonomy for evaluation, they’ll find that timed exams are the worst vehicle to assess those higher levels.