via email (I will see if they post it on their website later):
Setting Expectations: The New Normal for SOA Exams
The Society of Actuaries (SOA) announced in June of 2020 that it was moving to a computer-based testing (CBT) approach for all examinations beginning with the fall 2020 written-answer exam administration. Employers, exam committees and candidates have long recommended the change because it allows for the use of questions and responses that are more reflective of the real tools an actuary uses daily. The need to meet the challenges of the COVID-19 pandemic precipitated quick action, and our longstanding relationship with Prometric test centers facilitated the transition to CBT.
As the pandemic continued into 2021, our philosophy was that any candidate who wished to sit for exams should have the opportunity to do so. Education volunteers, staff, and exam vendors collaborated on thoughtful solutions for the myriad situations that were affecting candidates. Providing reliable technology to our global candidate pool is mission-critical to all involved in the administration and delivery of our CBT exams. As we look back now on the spring 2021 administration, our third since the pandemic began, we believe it is important to share information on the issues brought about by the pandemic as well as our move to CBT, how we have adapted and what we anticipate for the future of the SOA’s CBT exams.
Moving written-answer exams to CBT provides a better assessment experience
- CBT exam administration allows for the integration of programs such as Word, Excel and RStudio which better aligns with the current working tools of actuaries.
- Electronic collection of exam papers facilitates secure storage and distribution to graders.
- Many candidates surveyed commented that typing in Word, rather than writing on paper, was a welcome change.
- Candidates enjoyed the functionality of Excel and not needing to use a calculator.
- Some exam committees commented that the CBT format allows them to construct more robust questions with the additional computing power of Excel.
- Graders reported that handwriting legibility was no longer an issue, enabling them to focus exclusively on the content and quality of candidates’ responses.
- The CBT format allows for the delivery of the exams in emergency situations through alternative vendors with additional seating capacity.
The use of professional testing centers supports consistency, security and access
- The use of Prometric test centers supports the SOA’s goal of the equitable administration of examinations across the globe. Prometric operates worldwide and keeps current with the regulations and requirements in all jurisdictions. During the pandemic, the SOA was able to retain essential service status as a result of using Prometric as our testing vendor. This facilitated the offering of examinations that would have been otherwise cancelled in a traditional volunteer-proctored environment.
- Prometric provides a consistent experience for candidates throughout their exam pathway. All centers are outfitted with the same level of computer technology ensuring that candidates receive an equitable testing environment, unencumbered by variations in bandwidth, computing power and screen size.
- The use of Prometric testing centers eliminates the need to rely on volunteer proctors and employer-supplied exam sites while at the same time provides a higher level of security and proctoring.
Why the SOA won’t return to paper and pencil exams
- In addition to the advantages outlined above, the pandemic demonstrated the effectiveness of CBT at professional testing centers during emergency situations. Many SOA organized centers would have had to close during the last few administrations due to local restrictions, company policies and the health and well-being of the individual proctors.
- Relying on volunteer proctors and employer-supplied exam sites has become increasingly problematic due to inconsistencies at different centers and the possibility of a center suddenly being unable to operate or a proctor being unable to participate. Some examples include:
- The COVID-19 mitigation measures in the Canadian province of Ontario that began on April 8 would have shuttered our traditional pencil and paper centers. Prometric centers continued operations, although with reduced capacities, providing testing opportunities for the vast majority of candidates in that region.
- Malaysia implemented COVID-19 control measures just before the start of our spring exam session; our CBT exam format allowed the SOA to swiftly pivot and provide a testing opportunity for candidates displaced due to capacity restrictions.
- The risks associated with the shipping of exam booklets to centers and the collection of exam papers is eliminated. Although extremely rare, we have had exam scripts go missing in transit in the past – CBT delivery eliminates that risk.
Benefits outweigh the new risks of CBT exam delivery
- Despite the many benefits, CBT delivery also introduces some new risks, most of which center on potential technology issues. Reliance on technology will mean a degree of risk is always possible.
- Computer, network, server or power failures will happen from time to time.
- In addition, Prometric has had some issues with exams loading properly at some workstations at some centers due to computer upgrades. Regretfully, this has caused delays and rescheduling for some candidates. Prometric has a solution for this problem, and its prevalence will diminish over time. At this point, fewer than 2% of candidates have been affected by this issue. We continue to work with Prometric to eliminate it.
Why remote proctoring is not the answer: cheating, exam security, inequity
- Remote proctoring systems that operate without proctors physically on site cannot currently support the integrity of the SOA examination process and provide the level of assurance that our standards, and our stakeholders, demand.
- Remote proctoring comes with far greater cheating risks and perceptions of such risks, which in turn affect exam fairness for all candidates. This will undermine the ability of a credentialing organization to provide assurance that passing candidates have the requisite knowledge and skills.
- Remote proctoring does not offer a fair and equal opportunity for all candidates to progress toward a designation. Hardware and internet capabilities will have a direct effect on a candidate’s ability to complete a test, leading to an unfair advantage for those with better access to higher-quality tools. Any association between access to technology and socio-economic factors could support the view that remote proctoring is inherently discriminatory.
- Remote proctoring creates reputational risks to the value and integrity of the SOA’s credentials.
- Remote proctoring raises privacy and security concerns. Third-party software is often installed on the candidate’s computer that allows the remote proctoring company access to content. Multiple cameras are often implemented which pose a privacy concern for candidates who are unable to secure a separate and distinct testing space.
Management of Spring 2021 CBT-related issues
- 72 candidates experienced a launch issue. All 72 candidates either had their exam started after a delay or were rescheduled. This is an unnecessary disruption for candidates, and we are working with Prometric to minimize these issues. The technical issue that causes the launch problem occurs at the workstation level and is difficult for Prometric to predict, but each workstation only has the issue once, so we expect this concern to be diminished significantly in the Fall 2021 exam session.
- A further 29 candidates experienced a technical issue such as a power outage or loss of internet connection at a testing site, highlighting the risks of delivering exams in computer-based format. All 29 candidates were promptly rescheduled and were able to complete their exams.
- We received several reports of candidates having file upload issues, and we are continuing to work on process improvements to minimize these. We have learned that, in all but for a very few cases, we can successfully recover candidates’ files after an upload issue, the only exception being a total server crash during the exam.
- We expect that the Fall FSA exam administration will not be as affected by the pandemic and that many more seats will be available, particularly in Canada where test center capacities were severely restricted in Spring.
Looking to the future, we are confident that the change to CBT at professional testing centers was the right decision for the SOA exams. This modern solution provides a level playing field for all and provides the security needed for high-stakes examinations. Some risks are part of the new normal, but with experience and diligence we will continue to reduce those risks. While we may never completely eliminate technical issues in a CBT format, we will work industriously toward a low and manageable number of concerns. We look forward to continuing to enhance our assessment tools and building examinations that evaluate candidates for the industry’s current and future needs.
Ken Guthrie
Managing Director, Education
