Given the delays over Qualification Exam results, I thought I would provide some information regarding the Angoff Method that IMCA is using to determine the pass rate. (1)

Rationale Exams may have different levels of difficulty, due to the fact that questions are randomly pulled from a test bank. Fixed passing rates, of say 70%, may result in an exam not being “reliable”. Reliability deals with the ability of a test to measure a test-taker’s ability consistently.(3)(4)  As an example: for a scale to be reliable, it needs to consistently estimate a one-pound object as weighing one pound.  For an exam to be reliable, the scoring method needs to take into account the difficulty of each exams to be able to consistently evaluate a qualified candidate as a qualified candidate.  Major contributions in this field by William H. Angoff (1919-1993) have led to national tests and educational reform to determine cutscores for tests.(2)(3)  The Angoff Method (or its updated version, the modified-Angoff) has been adopted by many standardized tests, including the SATs and CFA.(5)

Method The Angoff method essentially uses field experts to determine the difficulty of each question.  It defines the cutoff score as the lowest score a minimally acceptable candidate is likely to achieve.  For the CIMA(R), a minimally acceptable candidate would be defined as someone who has an adequate knowledge of the learning objectives provided in the Candidate Handbook.  Essentially, it works this way:

  • A panel of experts independently rate each question, based on what they believe the % of minimally qualified candidates would answer the question correctly.  As an example, a question that is rated as 50% means that the panel believes that only 50% of the minimally qualified candidates would answer the question correctly.  This would be a much harder question, than one that is rated as 80%.
  • The panel then reviews each test question as a group and comes to a consensus for the rating of each question.  This process is performed periodically to include comments made by test-takers.  Questions are then edited, removed, or re-rated.(6)
  • At the exam level, the cut score is determined as a function of the ratings for each question in that test.  Often, it is just the average of the ratings.  As an example: an exam that consists of 50 questions with average ratings of 65%, would require at least a score of 65% to be passed.  An easier exam, with a rating of 80% (say), would require a score of at least 80%.

Conclusion There is a substantial empirical evidence showing the benefit of the modified-Angoff method.  However, it has come under some scrutiny with regards to the balance between Type I and Type II errors (false positives and false negatives: passing people who should fail and failing people who should pass).(7)  However, this would be an issue for fixed pass rates as well.  The use of item-judgment methods, specifically the modified-Angoff method, seems to be a significant improvement over fixed pass rates.  In my humble opinion, this creates a sweeter and fairer testing process.

References
1. Source at IMCA.
2. Angoff, William H: Technical Problems of Obtaining Equivalent Scores on Tests, Journal of Educational Measurement, Vol. 1, No. 1 (Jun., 1964)
3. Thorndike, Robert L. and William H. Angoff: Educational Measurement, American Council on Education, 1971
4. http://en.wikipedia.org/wiki/Standard-setting_study
5. http://www.cfainstitute.org/cfaprog/overview/pdf/IntoOur5thDecade.pdf
6. http://www.cwea.org/cert_howcert_testresults_howpassdeter.shtml
7. Tiratira, Niclie: CutOff Scores: The Basic Angoff Method And the Item Response Theory Method, The International Journal of Educational and Psychological Assessment, April 2009, Vol. 1, Issue 1, pp. 39-47


Category: CAIA, CFA, CIMA

About the Author


6 Responses to Angoff Method to determining passing grades

  1. Jeff Turnage says:

    Determining the exam cut score is only one step in a complex process. First an practice analysis is performed. This consists of a huge questionnaire filled out by a wide cross section of people working in the field. Their feedback determines what material is relevant to the field for which the exam is being developed. That was the exam is objectively relevant. From that analysis questions are written and reviewed and re-revieded and re-re-reviewed by subject matter experts to assure that the question is clear, the one correct response is truly correct and the distractors are truly incorrect. The consensus of a group of experts makes each question as objective as possible. Questions are written in such a way as to discriminate between low-performers and high-performers. That is, some are easier than others. But you wouldn’t want to be unlucky enough to get an exam packed with all the really hard questions if the passing score was set at a random point, say 70% That would be very subjective. That’s why a panel of experts in the field have to rate each question on the exam and use that objective data to establish a passing score for that particular exam. After each exam the performance of the questions and examined. If a particular question is too easy or too hard, it’s reevaluated. Too hard questions may be rewritten for clarity, for example. Every step in the process is designed to be as objective as possible. It’s not complexity for complexity sake.

  2. Chronos says:

    @ pierre
    I have to agree with “pierre’s” statement re what makes most sense. I’m not sure if it accurately describes Angoff/M-Angoff simply b/c I don’t know enough to make that judgement. I hope it’s the case, though.
    ~~~
    “pierre” stated:
    … So, in the end, a question ends up being mostly weighted by participant performance. For example, a question that is answered correctly by 80% of the exam takers is deemed easier than one that is answered correctly by 60% of exam takers.

  3. pierre says:

    Perhaps I could offer some clarification. The above was written for brand new exam questions. When a question is “in the mix”, the “opinion” about its difficulty is updated by the percentage of participants who answer it correctly. So, in the end, a question ends up being mostly weighted by participant performance. For example, a question that is answered correctly by 80% of the exam takers is deemed easier than one that is answered correctly by 60% of exam takers.

  4. I almost never write responses, however i did a few searching and wound up here Angoff Method
    to determining passing grades | Beta Education Blog.
    And I actually do have a couple of questions for you if
    it’s allright. Could it be simply me or does it seem like some of these comments come across as if they are left by brain dead folks? 😛 And, if you are posting at other social sites, I would like to follow anything new you have to post. Could you make a list of all of your public sites like your Facebook page, twitter feed, or linkedin profile?

  5. SYED says:

    The use of item-judgment methods, specifically the modified-Angoff method, seems to be a significant improvement over fixed pass rates. In my humble opinion, this creates a sweeter and fairer testing process.

    IF YOU BELIEVE SO ,YOU LOOKS ME ONE OF IDIOT WHO WANT TO MAKE OTHERS IDIOTS TOO..THIS TOTALLY MADE TO MAKE MONEY MORE BY SOME SO CALLED CERTIFYING BODIES BY DECLARING FAIL THE CANDIDATE ,WITH OUT ANY CAUSE.

  6. Ann Pitt says:

    This is far too subjective. The test should just consist of questions related to the profession and determined if correct or not. Far too much thought has created a far too complex procedure.

Leave a Reply

Your email address will not be published. Required fields are marked *

February 2010
M T W T F S S
« Dec   Mar »
1234567
891011121314
15161718192021
22232425262728

Categories




Hit Counter provided by Curio cabinets