TestMagic Forums - View Single Post - The Reply from The FPGEC. What should we do next?

knok · 09-16-2007, 07:55 PM

Please read an article written by Dean Papa John regarding the use of panelist setting standard.

Source: http://tesl-ej.org/ej37/a1.pdf

Quote (last paragraph):

Conclusions

Based on these observations it should be clear that the results from the ETS panel of ITA developers
should be looked at cautiously and interpreted carefully. "All standard setting is judgmental," as Jaeger
(1976, p. 22) succinctly puts it. So, what can we do with the results from the ETS sponsored standard
setting panel? The point of this article is not to criticize the standard setting process recommended by
ETS. They have applied a number of methods used in assessment and research in the past. However, it is
important to know why certain methods were chosen, what the strengths and weakness of different
methods are, and the issues that are raised in the steps of standard setting. Understanding the standard
setting process can lead to a better interpretation of the standards that are set.
The results from the ETS standard setting panel should be considered as a starting point rather than as
an end in itself. It is data you can use as a comparison to your own institutional context. Before
establishing a cut off score for your institution you should investigate ITA responsibilities, go through
local standard setting, and follow through with validating the cut score. The contrasting groups method
could be applied at the campus level. ESL instructors could be asked to classify their students as ITAhttp://
tesl-ej.org/ej37/a1.html (5 of 7)7/2/2006 2:24:25 AM
TESL-EJ 10.1 -- Standard setting for next generation TOEFL Academic Sp...ctions on the ETS Panel of International Teaching Assistant Developers
ready or not, then the overlap of frequency distributions of scores could be reviewed (Hambleton et al.,
2000). In addition, you can also utilize other assessments such as, listening cut scores, interview, and
performance tests. ITA supervisors and students of ITAs can be surveyed to validate the cut off score for
ITAs (Papajohn, in press). Finally, rather than adopt or accept someone else's cut score, use the standard
setting process as an opportunity to raise awareness of ITA issues at your institution.

09-16-2007, 07:55 PM	#7 (permalink)
knok TestMagic Guru Moderator Join Date: Nov 2003 Location: USA Posts: 3,100	Please read an article written by Dean Papa John regarding the use of panelist setting standard. Source: http://tesl-ej.org/ej37/a1.pdf Quote (last paragraph): Conclusions Based on these observations it should be clear that the results from the ETS panel of ITA developers should be looked at cautiously and interpreted carefully. "All standard setting is judgmental," as Jaeger (1976, p. 22) succinctly puts it. So, what can we do with the results from the ETS sponsored standard setting panel? The point of this article is not to criticize the standard setting process recommended by ETS. They have applied a number of methods used in assessment and research in the past. However, it is important to know why certain methods were chosen, what the strengths and weakness of different methods are, and the issues that are raised in the steps of standard setting. Understanding the standard setting process can lead to a better interpretation of the standards that are set. The results from the ETS standard setting panel should be considered as a starting point rather than as an end in itself. It is data you can use as a comparison to your own institutional context. Before establishing a cut off score for your institution you should investigate ITA responsibilities, go through local standard setting, and follow through with validating the cut score. The contrasting groups method could be applied at the campus level. ESL instructors could be asked to classify their students as ITAhttp:// tesl-ej.org/ej37/a1.html (5 of 7)7/2/2006 2:24:25 AM TESL-EJ 10.1 -- Standard setting for next generation TOEFL Academic Sp...ctions on the ETS Panel of International Teaching Assistant Developers ready or not, then the overlap of frequency distributions of scores could be reviewed (Hambleton et al., 2000). In addition, you can also utilize other assessments such as, listening cut scores, interview, and performance tests. ITA supervisors and students of ITAs can be surveyed to validate the cut off score for ITAs (Papajohn, in press). Finally, rather than adopt or accept someone else's cut score, use the standard setting process as an opportunity to raise awareness of ITA issues at your institution. _ _ _ _ SIG _ _ _ _ "I see so much testing in the world, and see how badly most of it is done. I also see the faith many people put in tests and test results, and the consequences for people when test results are used to make decisions about their lives. --Liz Hamp-Lyons (2002)"