Meeting with NABP: Are you ready to talk to NABP?

[knok]

Hi pharmacists,

Will you be willing and ready to talk to NABP? We have requested NABP to discuss our concerns regarding TOEFL iBT.

Please suggest what you think about how to deal with this unfair English proficiency requirement and vote. Your opinions are valuable to our group. TOEFL iBT administration and scoring process are problematic.

I hope to hear from you. Please vote again. We need an update poll. Thank you.

Becoming united and fighting until we get a good and fair outcome is our strategy.

Note: This poll is different from another poll. This thread is moderated. Any irrelevant discussion will not appear.

Best wishes,

Knok

[subramanian]

Dear Knok,

I am unable to vote this poll. How to vote, anyway my vote is for 2nd choice.
-

[knok]

Hi Subramanian,

Sign in and click the option. You are fine. I saw your name on it. Thank you for taking the time to vote.

Good luck.

Knok

[I.Q]

Hi guys,

Good effort, count me in. Thanks

i.q

-

[AJRAYES]

Medical students seeking residency in US are not asked for the Toefl test anymore. They have somthing called CS; an interview with some paid actors pretending to be patients.
Now, if we're going to talk to the NABP, then I suggest that we find a way somehow to convince them that such an alternative, or similar to it, is applicable for foreign pharmacists.
They should examine our english proficiency by themselves or by the boad of the state we apply for it's internship license, or they can make the Toefl with an overall score rather than requiring a separate minimum score for each section.
I believe that this BLACKMAIL should stop in oneway or another.
You have all my support guys.
A.J

[knok]

Hi AJ, IQ, Subramanian, and others,

We sent out a letter last year to NABP with no good result. AJ, about doctors, is it the same test that we mentioned in this petition?

Here is the petition to NABP.

June 19, 2007


The FPGEC
National Association of Boards of Pharmacy
1600 Feehanville Drive
Mount Prospect, IL 60056

Subject: English Proficiency Examination

Dear FPGEC:

We are a group of foreign pharmacists who would like to relay the test practice of Educational Testing Services (ETS) to the board. Some of us have taken the Internet based Test of English as a Foreign Language (TOEFL iBT) and continue to be frustrated by this test. In May 2005, we reported the problems of the Test of Spoken English (TSE) and the FPGEC suggested us to contact ETS directly. We have contacted ETS, but unfortunately, we did not receive satisfactory resolution from ETS. Presently, the TSE has been replaced by the TOEFL iBT. We feel that the TOEFL iBT is of poorer quality than the TSE, regarding testing format, testing environment and equipment, convenience of test takers, and the length of the test.

The TOEFL iBT complaint is enclosed with this correspondence. As you might note, the number of problems and deficiencies that this relatively new test has are really astonishing. Hence, we do not think that ETS will be able to improve or change the test dramatically soon. It may take years or never happen as we wait for ETS to address our needs as test takers. Not to mention the validity of the exam and conflict of interests may have caused ETS to continue with the same practice for years.

We strongly believe that it is in your hands to look for a proper and fair solution. All we are asking is the opportunity to serve our communities as registered pharmacists. That opportunity has been taken away from us since the implementations of the new TOEFL iBT requirements. You have the statistics of the number of foreign pharmacists that have already passed the FPGEE satisfactorily and have not been able to be certified only because of this language requirement. This is not an isolated case; there are thousands of us stranded in the same island.

We are sure that FPGEC is aware of many of the problems related to the TOEFL iBT. In appreciation and recognition of the importance of FPGEC’s responsibility for maintaining the high standards of practice of foreign pharmacy certification, we respectfully request that the board consider five possible solutions regarding the requirement of English proficiency testing for the FPGEE certificate program.

1. Allow other speaking test as an alternative

1.1 Oral Competency exams
We do not mind passing the speaking test, but we urge the board to use other exams if IELTS is not an option. In the state of MD, the U.S. Pharmacy graduates are required by the state to undergo an English Proficiency test (Maryland Board of Pharmacy, p. 3). They cannot be licensed in the state unless they pass the English test, even if they are native speakers. The tests on the list below offer a face-to-face interview. They are administered by “Oral Competency Vendors” (Maryland Board of Pharmacy, p. 6).

If NABP is not sure about the tests in above list, we ask NABP to consider this as a pilot program for a period of six months or so. Later on, the data collected from those who have taken these exams can be used to compare among speaking tests.

1.2 Create pharmacy English proficiency exams
In addition, for foreign physicians, the English requirement is measured through an English proficiency exam administered by the same certification authority. It is part of the three steps of the certification process of United States Medical Licensing Examination (UMSLE Bulletin, 2007):

Table 3: The Subcomponents of Step 2 CS
Integrated Clinical Encounter (ICE) Subcomponent
• Data gathering – patient information collected by history taking and physical examination
• Documentation – completion of a patient note summarizing the findings of the patient encounter, diagnostic impression, and initial patient work-up
Communication and Interpersonal Skills (CIS) Subcomponent
• Questioning skills
• Information-sharing skills
• Professional manner and rapport
Spoken English Proficiency (SEP) Subcomponent
• Clarity of spoken English communication within the context of the doctor-patient encounter (e.g., pronunciation, word choice, and minimizing the need to repeat questions or statements (UMSLE, 2007)
Basically, the English Test is a simulation of a patient-doctor interaction, as occurs in real life. We feel that physicians have to communicate exclusively and directly to patients more than pharmacists but they do not follow ETS and any other institution’s advice. What they care is that foreign physicians have to be able to communicate with their patients generally well in situations and context of healthcare, which is close to the real world. This might set an example for certifying foreign pharmacists.
2. IELTS as an Alternative

2.1 IELTS offers a face-to-face interview
Even though the IELTS is harder than the TOEFL iBT, the IELTS offers a real interview. A real interview is able to solve the problems of the recording, noisy environment, and provide two-way communication with human beings, not a machine. The common complaint about the TOEFL iBT speaking is a recording process which is too mechanical. The TOEFL iBT lacks interactive communication which is the most important part of conversation in the real world. The recording process and time constraint requiring candidates to answer a question in seconds degrade the performance of test takers who need to reach an almost perfect score of 26 on the speaking section.

In the meantime, the Wikipedia (Wikipedia, IELTS) documented that the number of IELTS users has risen from 100,000 in 1999 to half a million in 2003.
Renshaw, an English teacher (Renshaw) compared the TOEFL iBT and IELTS:
To my mind, the first of the writing tasks on IELTS clearly requires integration of reading and writing skills, and the fact that their speaking tasks involve interacting with a face-to-face interviewer (and one task where test-takers have to interact with another test-taker on-site) should lead to the general assumption that integration of skills is intrinsic to these tasks.

We shouldn't lose sight of the fact here that IELTS scores have always had a stronger correlation to actual academic ability in English-speaking tertiary contexts, and the iBT speaking section in particular was added to make the iBT more competitive with IELTS, which has had a tried and true speaking section for decades.

Don't forget also that ETS has been staring down the barrel of IELTS starting to infringe on its home territory (the United States), with dozens of US universities starting to accept IELTS scores due to its stronger reputation as a genuine measure of academic English ability in all skill areas (Renshaw, 2007).
2.2 IELTS provides a separate speaking section in an individual room
The testing conditions provided by ETS in TOEFL iBT are unacceptable. The IELTS provided a real interview in an isolated room. This can avoid the problem of a disturbing environment. Even though the recording is done during the interview, the test is graded by interviewers during a real test. The recording will be replayed to the test takers to make sure that the recording is done properly. The IELTS informed test takers that the recording is done as a reference, in case it is requested to prove the transparency. In contrast, in the TOEFL iBT, test takers have no chance to listen to their recording, no matter what.

2.3 ETS faces shortage of testing capacity
ETS has only limited available seats for the TOEFL iBT candidates, usually filled four months before deadline date (Wikipedia, 2007). As a result, alternative exams are likely to be considered in assessing candidates’ proficiency of the English language as part of the certificate program. Not to mention difficulties logging into the TOEFL.org web site when test takers want to register for tests or check their score results. It may take two weeks to log in because the web site is too busy.

2.4 Allow competition
If the IELTS or other speaking test is chosen to fulfill the board’s requirement, we believe that competition will force organizations to improve their products and services. Nowadays, the monopolistic power that the ETS holds keeps ETS from improving their efficiency and the efficacy of the testing procedure, but allows ETS to increase its profit. This situation creates unfair practice and frustration to test takers who pay the money but continue to receive this poor testing procedure. We hope that competition will change everything. Presently, it is really difficult to get a seat for the TOEFL iBT.

When a complaint has been made to ETS, it is ignored or the best resolution ETS offers is allow test takers to take another test without charge. This practice does not help a test taker because the real problems have never been solved. Test takers will not be able to improve their score if they have to take another test in the same condition and receive the same vague evaluation sheet, which requires test takers to read raters’ minds on where to improve.

3. Reduce the speaking score requirement to 20

NABP stated in the correspondence in June 2005 that, for public safety, NABP would not reduce the score requirement. Due to annoying testing conditions which degrade performance of test takers tremendously, we feel that test takers who receive a score of 20 or higher are able to communicate well in disturbing environment, time constraint, and unrealistic conversation. To receive a score of 26, test takers must receive almost a perfect score. This is unfair because the test content has nothing to do with the pharmacy field. When test takers are tested to speak on different unfamiliar topics in seconds, test takers are not able to receive a score of 26 on the speaking section because it is highly possible that they do not have an idea to answer the questions, not because they are not able to speak in English language.

For instance, if FPGEE test takers had to take an essay FPGEE exam in an annoying environment and the time was reduced to one hour, we believe that only few pharmacists would be able to receive a scale score of 75. This is not because they would be lacking pharmacy knowledge and experience but because of unfair testing conditions and not having enough time to think and organize the answer in allotted time. We do not think that it is fair for test takers when testing practice of ETS is an issue.

In addition, for those who receive a score of 20 or higher, ETS mentions in the score report that their pronunciation and grammatical mistakes do not interfere with the overall communication. They can communicate in that area. Please see the details in the copy of the score reports. This should be enough.

3.1 The mean score of American candidates was 22 on the Speaking section
According to TOEFL® Test and Score Data Summary for TOEFL Internet-Based Test: September 2005-December 2006 Test Data (Educational Testing Service, page 9-10), the mean score of the candidates from the United States on the speaking section of TOEFL iBT was only 22 whereas the mean score of the candidates from country that English is their primary language is 23. This result indicates that the test is invalid and unrealistic in the real world.

This test seems to measure intelligibility or other aspects rather than the language ability. For these reasons, we think that the current requirement is too high for foreigners whose native language is not English. Even though some of these Americans may not have grown up speaking English in America, it is important to recognize the relatively low score for those who have been in an English speaking country for their entire lives.

3.2 Is a score of 26 on the Speaking section reasonable?
In the speaking section, a score of 4 is the highest score according to ETS. To score 26 in TOEFL iBT speaking, test takers should get 20/24 for each of six questions. The highest possible score for each question that a test take is able to receive is a score of 4 so that a result will be (4*6 = 24), which is very high for any standards.

To get a score of 26, test takers should get high level score(4) in two questions and fair level scores(3) in remaining four questions, which added up to a total score of 20/24, 20/6 = 3.3 that equivalent to 26/30.

The following rating method is used by ETS:

1. (Q.1-2)  4+3 = 3.5 (GOOD), (Q.3-4) 4+3 = 3.5 (GOOD), (Q.5-6)4+3 = 3.5 (GOOD) Total Score 21/24 = 3.5 = 27/30

2. (Q.1-2)  4+3 = 3.5 (GOOD), (Q.3-4) 4+3 = 3.5 (GOOD), (Q.5-6) 3+3 (FAIR) Total Score 20/24 = 3.3 = 26/30

3. (Q.1-2)  4+3 = 3.5 (GOOD), (Q.3-4)3+3 = 3 (FAIR), (Q.5-6)3+3 (FAIR) Total Score 19/24 = 3.1 = 24/30

4. (Q.1-2)  3+3 = 3 (FAIR), (Q.3-4)3+3 = 3 (FAIR), (Q.5-6)3+3 (FAIR) Total Score 18/24 = 3.0 = 23/30

(Q.) = question number

A test taker was informed by English as a Second Language teacher (ESL teacher) who attended ETS workshops and is a member of ETS training that a score of 26 in TOEFL iBT is equivalent to 55 in the TSE, but ETS publishes in its official guide that a score of 50 on the TSE is equal to a score of 26 on the Speaking section of TOEFL iBT (ETS official guide, p. 350). This is not always true. These two tests provide different formats. They are not comparable. However, ETS suggests institutions that a score of 23 is a minimum score for International teaching assistants (ITAs) (Bowman).

The tricky part is that ETS has set a score of 26 in the speaking section as good communication skills by conducting a studying group to testify to its assumption. This studying method is criticized by educators whether it is practical in a real world or not. Please refer to Papajohn’s article. Papajohn states:

Decision makers may have to consider their own belief about the effect of passing less than qualified applicants and failing qualified applicants (Papajohn, p. 6, Faggen, p. 4).

For ITA, the mean score for panelists suggest the next generation TOEFL Academic Speaking Test (TAST) score of 23 for the cut off of the minimally acceptable speaker (Papajohn, p.5).

He also mentions that information that is conveyed in a way that is too compact may be difficult to understand. In other words, information that is too diffuse may make it harder to comprehend the connections. Response lengths of 60 seconds do not sometimes allow for correct evaluation of compactness and diffuseness (Papajohn, p. 2). He comments that no one communicates perfectly 100% of the time. The result from the ETS standard setting panel is likely to be considered as a starting point rather than as an end in itself. Rather than adopt or accept someone else’s cut score, use the standard setting process as an opportunity to raise awareness of ITA issues at their institution.

4. Count the Speaking section separately from the other three sections
We request the board to count a score of three sections as a total score by calculating from adding up all three sections as NABP required before. The total score of these three sections of 60 on the TOEFL iBT is equal to a score of 213 in the Computer Based TOEFL (ETS, 2004, p.10). It is a waste of time and money when a majority of pharmacists are able to receive a high score or passing score of the other three sections but not the speaking section to retake the total of four sections. Others have passed the TOEFL CBT but they have to retake it again.

ETS put the speaking section on the third part of the TOEFL iBT. After test takers spend three hours in the reading and listening test, they feel tired and lose the ability to concentrate because they use up their energy and spirit, especially in bad testing conditions. Some of them are so hungry that they cannot perform at their maximum potential. The ten minute break does not help reduce their stress and rejuvenate them to take the active parts, speaking and writing. Please remember that the test is operated in four to five hours or more, including waiting time for checking in.

5. Receive the highest score from the tests on different test dates or receive the total score
Since the TOEFL iBT testing conditions are unpredictable, scores may vary dramatically. For example, if the monitor blinks, test takers are not able to read from the screen clearly and fast enough in time constraint because the low quality of equipment. In addition, some test centers are too hot or too cold, making the test takers feel uncomfortable and irritated. Their reading score will be much lower than in the other tests. In the listening section and writing section, test takers are not able to hear well and concentrate on their tests while others are shouting loudly, especially when the test rooms do not have partitions or set them too close to each other.

Testing conditions affect our scores. NABP can check from test taker to test taker from their score report. If this TOEFL iBT is valid, their score is not likely to vary enormously. After asking many test takers, their scores can vary up to eight points or more. In the TOEFL iBT, in general, academic institutions require only a total of three sections, reading, listening, and writing or a total of four sections. The design of this test is likely to be counted in a total
score. In two different tests, A and B, the reading section in test A may be more difficult than that in test B but the listening section of test A may be more difficult than that of test B.

We think that a total score should be used instead of separating the score into 4 sections. In other words, a combination of the passing score should be considered to fulfill the board requirement or the use of the highest section from each test should be considered. If it is not the case, we request the board that we register and take only sections of the test we need and omit the sections we receive passing scores.

In addition to physicians, the Commission on Graduates of Foreign Nursing Schools® (CGFNS) requires foreign nurses to receive a total score of 83 for the TOEFL iBT, or a score of 725 on TOIEC, or an overall score band of 6.5 on IELTS (Academic Module) (CGFNS, 2007 Edition).

From our perspectives, nurses and doctors need to communicate with patients more than pharmacists, but their boards allow candidates to be more flexible to select the language proficiency tests. CGFNS requires candidates to pass a total score of language proficiency test rather than focus on separating score. One reason is probably that there is a high correlation among different sections of tests. In the TOEFL iBT, the speaking and writing sections become integrated tasks of four skills; reading, listening, writing, and speaking. If public safety is such an important issue that NABP prohibits candidates to use other tests to fulfill the board requirement, when compared with these healthcare professionals, we feel that this requirement should be reconsidered annually as NABP stated in 2005.

Follow Up
Finally, our group is seeking positive solutions to the concerns we have listed. It is our hope to be contributing members of society and practice our profession of pharmacy in the USA. If NABP is not able to understand our requests or if NABP does not feel that our complaints matter enough to incur changes, we are willing to meet with NABP in person. We would appreciate it if the board allows us to meet in order to explain details of our problems and difficulties. You can see that we have a thoughtful and professional request. We politely request a written response or a meeting. Thank you for your serious consideration of these issues.

Sincerely,

Your signature….

Mr./Ms.(Your name) B. Pharm (Pharmacy), M.S.,….
EE#

Enclosure: TOEFL iBT Complaint, TOEFL score reports, Test center complaint table
Cc: Competency Assessment Director, State Board of Pharmacy, Members of Congress, Governors, Attorney General

References
1. Bowman, M. S. (Nov 2005), Graduate program directors hand-out: the next generation TOEFL test-TOEFL iBT, Related issues for Speaking/TSE, ESL Program, [College of Graduate Studies, Cleveland State University]. Available at: http://www.csuohio.edu/gradcollege/f...%2011-2005.pdf Accessed June 3, 2007.

2. Commission on Graduates of Foreign Nursing School (CGFNS) Certification Program for Registered Nurses (2007 Edition), Program Requirement: English Language Proficiency. Available at: http://www.cgfns.org/files/pdf/req/cp-requirements.pdf Access June 1, 2007.

3. Educational Testing Service (2004), iBT/Next Generation TOEFL Score information, p.10. Available at: http://www.ets.org/Media/Tests/TOEFL...nformation.pdf
Accessed May 31, 2007.

4. Educational Testing Service (2007), Total and Section Score Means: TOEFL® Test and Score Data Summary for TOEFL Internet-Based Test: September 2005-December 2006 Test Data [report], p. 9-10. Available at:
http://www.ets.org/Media/Research/pd...M-0506-iBT.pdf Accessed May 29, 2007.

5. Educational Testing Service (2007) The official guide to the new TOEFL iBT. New York: McGraw-Hill, p. 350.

6. Faggen J. (1994), Setting standards for constructed response tests: an overview. Educational Testing Service. [Report]. Available at: Setting Standards for Constructed Response Tests: An Overview Accessed June 2, 2007.

7. Maryland Board of Pharmacy (2005), Test of Oral English Competency, Pharmacist Licensure application packet, p.3. Available at: http://www.dhmh.state.md.us/pharmacy...at/newexam.pdf Accessed June 2, 2007.

8. Maryland Board of Pharmacy, Oral English Competency Vendors, Pharmacist Licensure application packet, p. 6. Available at: TEST OF ORAL ENGLISH COMPETENCY Accessed May 30, 2007.

9. Papajohn, D.,(2006, June 1) Standard setting for next generation TOEFL Academic Speaking Test (TAST): Reflections on the ETS panel of international teaching assistant developers, Teaching English as a Second or Foreign Language (TESL-FJ 10.1), Volume 10, Number 1, p. 2, p. 4-5. Available at: http://tesl-ej.org/ej37/a1.pdf Accessed June 1, 2007.

10. Renshaw, J. (2007, April), TOEFL iBT and IELTS Tests Compared [Renshaw’s weblog]. April 2, 2007. Available at: iBT Speaking and Writing: News and General Gossip Accessed 2007, June 2.

11. United States Medical Licensing Examination, 2007 UMSLE Bulletin, Step 2 Clinical Skills (CS). Available at: http://www.usmle.org/bulletin/2007/examcontent.htm Accessed May 28, 2007.

12. Wikipedia (2006, June 6), The Free Encyclopedia. IELTS, Location and dates, 2007. Available at: IELTS - Wikipedia, the free encyclopedia Accessed June 6, 2007.

13. Wikipedia, The Free Encyclopedia, TOEFL. 2007. Available at: TOEFL - Wikipedia, the free encyclopedia Accessed June 1, 2007.

Note: “Dean Papajohn has worked with international teaching assistants for over twelve years teaching English as a second language classes, coordinating orientations for international teaching assistants, and conducting oral English assessment. The second edition of his book Toward Speaking Excellence has been released by the University of Michigan Press.” (TESL EJ)

[knok]

TOEFL iBT Complaint:

Subject: TOEFL iBT Complaint

June 19, 2008

Dear_____


We are a group of the Test of English as Foreign Language (TOEFL iBT) test takers who are concerned with the test practices of Educational Testing Service (ETS). We would like to express our frustration about the testing center availability, testing environment and equipment, and test scoring process. ETS has published many reports and official guide to give an impression that the TOEFL iBT is given fairly. However, the speaking test is too subjective and the evaluation is too vague for test takers. Hundreds of test takers continue to have bad experiences with TOEFL iBT test and services while they have paid a high testing fee of $150 for each test. We would like your office to help us with this situation.

TOEFL iBT Complaint

1. Defective and Low-Standard Equipment

Overall, Thomson Prometric testing centers are bad, others are worse.

1.1 Computer Monitor

Some Thomson Prometric testing centers are using old monitors, which are not the LCD type. Some computers used in this test center should be thrown away since they do not provide good images anymore. They are hard to read because they blink. Some computers are not set to have high contrast between images and background. This makes it hard for test takers to read the passages or, if they can read them, they will have to stare at the screen intensely and hurt their eyes. Particularly, test takers are required to read perhaps five passages on the reading section, for a total time of two hours for the reading section alone.
This problem usually occurs at Thomson Prometric testing centers. When test takers asked to change their monitors, the test proctors responded that this was the way they were set up and they had nothing to do with them and suggested test takers to write the complaint to ETS directly. Unfortunately, nothing had been resolved after several hundred test takers filed their complaints to ETS. This problem has continued for years.

We suggest that LCD monitors, which offer the latest technology be used for the current TOEFL iBT. This type of monitor can be acquired at less than $170 each.

1.2 Internet Connection Errors or Computer Problems

In addition, in some test centers, it takes too long to see another question after clicking the answer on the previous question. It is not fair for test takers because this test is giving in a time constraint. On some occasions, test takers cannot see some buttons on the test or they cannot change the answers after clicking on the answers.

It may be true that test takers can raise their hands and request a test proctor to stop the test and re-operate it but in some occasions it is impossible because test proctors do not see test takers when they need help immediately. Test takers waste their test time at least 5-10 minutes asking for help. Also, in the reading section, the information that test takers already read before calling test proctors is lost from their memory. They must re-read it again. This results in a lower score on that section. This problem usually happens at some other testing centers that ETS uses, such as high schools or colleges. These facilities have limited hardware or Internet capacity. Most importantly, they set tables too close to each other with no partition, less than one meter apart.

1.3 Insufficient Paper Notes and Pencils

Furthermore, test takers must take notes during the test. Some test centers give only a few pieces of paper to test takers and tell them to ask for paper later when candidates use it up. When candidates run out of paper during the test, it is almost impossible for them to ask for paper in the middle of the test. This is because it is not convenient and time consuming to ask for paper. Also, some test centers do not provide enough sharp pencils to candidates. Writing with a dull pencil will reduce the speed of taking notes which reduces ability of test takers to perform well, especially on the Speaking and Writing Section.

2. Disturbing Testing Environment

2.1 Noisy Environment

Most importantly, the test environment should be quiet at all time. Nobody should be allowed to speak in a test room at any time. First, when people are testing microphones at the beginning, some test takers are working on the reading section. It is difficult to focus and perform well in answering questions. Second, when some test takers are listening to the passages on the listening section, others are shouting to answer the six speaking questions. Remember that test takers are checked in at different times so the time of speaking may take longer than 40 minutes. It is hard to concentrate on listening or writing when they must be in distracting environment.
2.2 Unprofessional Test Administration

Moreover, the test proctors, who should be ensuring that we have a proper environment for taking the test, are sometimes the biggest problem. They have been known to carry on conversations while the test is in progress. Some test proctors even sign candidates in, take pictures of candidates inside the test room, surf the Internet next to test takers while the test is in session, and even listen to music without headphones.

3. Format and Time Constraint in the Speaking Section

ETS states:

The raters who listen to your response will analyze it in three general categories. These categories are Delivery, Language Use, and Topic Development. All three categories have equal importance. Use the sample Independent and Integrated Speaking Rubrics on pages 242 to 246 to see how responses are scored. This section includes important points that should be covered when answering each question. All of these points must be presented in a response in order for it to receive the highest score in the Topic Development category. These important points are guides to the kind of information raters expect to hear in a high-level response. (ETS official guide to the new TOEFL iBT, p.233)

ETS also published the strategies for raising the TOEFL Speaking score by telling test takers to ask themselves these questions:
Did I complete the tasks? Did I speak clearly?
Did I organize my ideas clearly and appropriately?
Did I provide a complete response?
Did I use the time effectively?” (ETS official guide, p. 227)

The ETS official guide recommends candidates to time themselves when practicing the speaking tasks so that they can get an idea of how much can be said in the allotted time (ETS official guide, p. 247).

Officially, in the Speaking Section, there are only six questions. The first two questions are called Independent Speaking Tasks because they require test takers to draw entirely on test takers’ ideas, opinions, and experiences when responding. Test takers have 15 seconds to think and 45 seconds to speak. It is really difficult for test takers to think and formulate their thoughts in 15 seconds and organize and speak in 45 seconds, particularly when the questions are not familiar in their daily lives.

For the rest of the questions, candidates are required to summarize the conversations, lectures, and reading provided. In some tests, the topics are too difficult or are not in the interest of test takers. This test is unrealistic for measuring speaking ability of test takers. It is the test that puts more focus on organizing speech in the time limit. ETS mentions on its ETS official guide that test takers may not finish the tasks but can still get a high score and, in the meantime, those who can speak fluently may not get a high score in the Speaking section test if they cannot answer intelligibly and sustainably.

Not surprisingly, after asking many test takers, even though they can speak fluently, most of them are very nervous during the test and cannot finish answering the questions, none of them can get a score higher than 24. ETS informs institutions that a score of 26 is a score that is equal to a score of 50 on the Test of Spoken English (ETS official guide, p. 350). However, a score of 26 on the speaking section of TOEFL iBT is currently a minimum score for professionals to fulfill their requirements for their professional licenses.

According to TOEFL® Test and Score Data Summary for TOEFL Internet-Based Test: September 2005-December 2006 Test Data (ETS, page 9-10), the mean score of the candidates from the United States on the Speaking section of TOEFL iBT was only 22 whereas the mean score of the candidates from countries in which English is their primary language is 23. Even though some of these Americans may not have grown up speaking English in America, it is important to recognize the relatively low score for those who have been in an English speaking country for their entire lives.

This result indicates that the test is invalid and unrealistic in the real world. This test seems to measure intelligibility or other aspects rather than the language ability. This seems to be unfair for test takers when it is obvious that getting a high score on the Speaking section has nothing to do with the ability to speak English fluently.

4. Grading System:

4.1 Transparency of Recording

In TOEFL iBT, in the speaking part, a narrator tells candidates to check the microphone and one warm-up question is given to check its system, but how can we know that the volume and clarity of our voice is sufficient to be understood or clear enough to produce a good impression on the raters? Test takers will never know whether their recording is done properly or not because we cannot hear our own response.

Obviously, ETS has candidates sign a document before taking the test stating that after finishing the test, all exam results or recordings will solely become a property of ETS. As a consequence, not a single organization, not even the Better Business Bureau (BBB) can hear or audit the recording when a test taker requests to prove transparency of the exam.

Accordingly, this seems highly unfair for test takers when they are suspicious about their recording. When they ask to disclose the recording, ETS responds that it cannot give parts of speech or run the recording to prove transparency of the Speaking section because of security reasons. This is ridiculous and it shows that ETS uses monopolistic power to take advantage of test takers. The causes of failure might be the low volume of their voice, or technical errors resulting from defective equipment, or incomplete voice transmission resulting from static, or the echo of a microphone or Internet transmission errors. As you may know, voice transmission through the Internet is not comparable to true voice as in a face-to-face interview.

In IELTS, after an interview, the interviewer will replay the tape to make sure that the recording is done properly. If not, another interview or exam will be done immediately until the tape is satisfactorily recorded. Moreover, IELTS clearly states that the grading is done during the interview, not from the recording. The recording is used only as a reference in case that a candidate requests to check for transparency of the exam.
We are told by test proctors that we should speak slowly and loudly in the test so that the recording will be better recorded. In real life, we do not have to worry about speaking to the microphones when we deliver a presentation or have a conversation with our peers or professors or patients and we do not have to be sure that our voice will be recorded properly. Most importantly, we can repeat or correct our speech immediately without worrying of time limit if we make mistakes.

4.2 Vague Evaluation

4.2.1 Useless Evaluation

We are very disappointed to receive the canned evaluation in which ETS copies and pastes the detail of range of score evaluation of 1.5-2.5, 2.5-3.0, and 3.0-4.0 for every two questions rather than giving us the individually detailed evaluation for each question. The evaluation sheet given to test takers is too general and too vague. For example, when the evaluation says that the speech is difficult to understand because of pronunciation or grammatical mistakes, the raters should give specific examples of part of test takers’ speech. For example, some speech patterns may be interpreted as an accent rather than mispronunciation. It depends on the ability of the raters to interpret the speech whether as to it is acceptable or not.

4.2.2 Identify a phrase with the problem

Identifying sentences or phrases with the problem, we strongly believe, is crucial and necessary. Pointing out which part of delivery is unacceptable or wrong is a key for the test takers’ success. Otherwise, they will continue to make the same mistakes without knowing that their mistakes are so severe that the raters will punish them. This causes serious frustration to test takers who spend time, money, and effort hoping to receive a desired score to improve. We do not know exactly where to improve. We all have to read raters’ minds. This is unfair for test takers and causes emotional distress to test takers and their families. If ETS is willing to improve testing practice, giving a clear evaluation is a must. Therefore, if ETS cannot give a specific evaluation to test takers, it should not include the Speaking section in the TOEFL iBT at all.

To make it clearer, please see the enclosed document that demonstrates the differences between the evaluation given in the real test, which is given the same for everyone who receives the same score range, and the copied evaluation published in its official guide sold in the market, which explains differently for each individual.

4.3 Is TOEFL iBT speaking a valid test for the real world?

According to ETS (ETS: Converting Rubric Scores to Scaled Scores, 2004, ETS: Score Data Summery, 2007), to receive a score of 26, test takers must receive at least 3.33 out of a rubric score of 4 which is very high. In addition, it is speculated that to receive a score of 26, a candidate must be able to be at about 85 percentile. It means that when ranking all candidates, the successful candidate who receives higher than a score of 26 must be at about the top 15 percentile.

As a result, this practice raises questions about a possible conflict of interest for ETS and the TOEFL iBT test design. We ask ourselves whether the questions are too difficult or there is not enough time given to organize our response, or are the questions even appropriate given the vast cultural diversity of candidates.

Keeping this in mind, some TOEFL iBT test takers have received advanced degrees from American universities, have worked for universities and companies successfully and continue to receive a score lower than 26 on Speaking section of TOEFL IBT. In addition, others have responded that even though English is their primary language, they have lived in the U.S. for several years, and communicate exclusively in English on a daily basis; they continue to score below 26 on the Speaking section of TOEFL iBT. This indicates that the TOEFL iBT Speaking may not be a valid test for the real world.

4.4 Scoring Inconsistencies and Errors

Furthermore, inconsistencies with scoring occur very often. ETS allows test takers to rescore the Speaking section of TOEFL iBT by paying $60. It is discouraging when candidates receive a score lower by several points or even up to 5 points. We feel that the discrepancies of score errors are too high and the grading system is not reliable and depends on the mercy of the raters. It seems that rescoring system becomes another source of income for ETS.

5. Registration and Customer Feedback

5.1 Availability of TOEFL iBT and Improper Testing Schedule

Another frustration test takers have is that ETS is not able to provide a test at a good test center at a convenient time and location. At many locations in the United States, all test centers are occupied a few months before the test dates. Test takers must register at least a few months ahead at a location they sometimes are not satisfied with as there are no available seats in the desired location.
Indeed, this information can be confirmed by Wikipedia, the free encyclopedia (Wikipedia, TOEFL iBT, 2007):
Candidates complain about the TOEFL iBT, mainly because of the high noise level during the speaking section: Everyone is responding orally to six questions at the same time. In addition, the number of seats available is limited (usually filled four months before deadline date)[citation needed] (Wikipedia, 2007).
Inside Higher Ed, the online source for news, opinion, and jobs for all of higher education (Jaschik) announced that access is a problem for TOEFL candidates:
Since ETS introduced the internet based TOEFL in September 2005, thousands of students all over the world have endured immense stress, inconvenience and financial loss: waiting months to get a seat, traveling far to testing centers, and then having tests postponed or cancelled at the last minute. Why? Because there are not enough test centers. Why not? Because ETS does not offer institutions enough of a financial incentive to become a test center. Moreover, the infrastructural support offered to the few existing test centers is weak.
On the other hand, I wish that ETS had paid attention to the suffering of international students and fixed the administrative problems of the TOEFL iBT. If they had done that, they would not be running into these access problems now. ETS, please invest some of your millions into infrastructure and access (Priscilla Allen, 2007, April 4).
In addition, ETS continues to offer the test at inappropriate time such as 4 a.m., 6 a.m., 6 p.m., 7 p.m. and 10 p.m., which are the times that many people are supposed to rest.
When there are no test centers left, test takers have no choice. They must register for a test and take it. These inappropriate time schedules affect test takers’ performance, which result in poor scores on their tests.

5.2 Customer Response Failure

After facing on-going problems, many members feel that ETS continually ignores their inquiries regarding score evaluations and the poor testing environments. When emails or letters are sent to ETS advising them of these problems, a “canned” response is what we get. The problems are not addressed. Often, we never receive response.

6. Number of test questions and total time differences among test takers.

It is unfair that test takers may be required to perform different amounts of tasks and a different number of questions on the reading and listening section. Candidates may receive either three or five passages during the reading section and six to thirteen passages of listening section. We think that everyone should get the same amount of questions. We were informed that ETS gave dummy questions which were not scored for some test takers. These dummy questions are probably for ETS's exam trial. This exhausts the test takers. This practice should not be allowed.

Test takers are exhausted after reading five passages for the reading section before doing the Speaking section. Test takers are not allowed to go to the rest room during any sessions. It is impossible to go to the rest room during the reading because the reading is divided into two to three subsections:

1 reading passage ---20 minutes
2 reading passages-- 40 minutes
2 reading passages ---40 minutes

Furthermore, average waiting time for registration process is about 1 hour. Occasionally, test centers encounter technical or server problems which result in longer waiting time. Test takers might have to wait two to three hours to complete registration process. After that, they have to complete the reading and listening section before they can take a break. We think that test takers should be allowed to take a break after they finish 60 minutes of the reading section or before they go on the listening section. We feel that these issues should not be taken lightly because they involve health-related problems, such as urinary tract infection.

[knok]

To be continued....(TOEFL complaint)
...

...
B. Suggestions

1. Speaking Section

1.1 Separate Speaking section

In the current format, we feel that at least test takers should have been able to listen to their voices after finishing recording. This would allow them to make sure that their speech is recorded properly (while others are shouting at the same time) but it is not likely that this practice is going to happen. Therefore, we suggest ETS entirely change the format and separate the Speaking section.

During the Speaking section, it is impossible to maintain a quiet environment. We suggest that ETS separate the Speaking test to another room and that test takers should be taken to this section at a different time. Test takers should have their own individual room.

In addition, the Speaking section was arranged to be the third part of TOEFL iBT after the Reading and Listening section. It sometimes takes up to three to six hours starting from checking in process to the end of the Listening section. For example, if the test starts at 9 a.m., test takers are required to check in at 8:30 a.m. They may finish the second part of listening section at or after 12 p.m. They will have a chance to break for 10 minutes before doing the Speaking for about 20 minutes and writing section about an hour. By this time, test takers are very tired from using their eyes to read several passages and their source of energy in their body has been used up. They will feel hungry. Therefore, they should be able to have lunch and take a rest before doing the rest of the test.

It seems that the Speaking section is a real problem of the TOEFL iBT. If ETS cannot separate the Speaking section to a different day or continue the Speaking section after at least 1 hour break, and if ETS cannot provide quiet environment for test takers, the TOEFL iBT should not include the Speaking section at all.

1.2 Face to Face Interview in the Speaking Section

According to information given above, we do not think that the current format of the Speaking section is good enough to measure the speaking ability of test takers. It is too mechanical. Test takers are required to speak at a certain time and finish at a certain time. We are not robots. The test is base on recorded questions and answers. There is no face-to face interaction. In the real world, 60% of communication is non-verbal. (Facial expressions, reading lips, and body language) There is no chance to clarify the meaning of the question and no chance to judge the effectiveness of our speech through the immediate feedback one gets in a real conversation.

2. Separate Writing Section

Since the Writing section is at the last part of the test, it is also as active an activity as the Speaking section, which requires test takers to think clearly and concentrate on writing a good essay. The first part of writing is the integrated task which requires test takers to listen to the passage. It is difficult to hear the passage when others are speaking. We suggest that ETS separate the Writing section to a separate room. The typing activity is quite annoying because of the noises of keyboards. The required space for each test taker should be far enough at least two meters so that test takers will not be disturbed by one another. Some test centers currently set test tables close to each other. The noises of keyboards are annoying, making it hard for test takers to perform well.

3. Experience the TOEFL iBT

To help your office understand our situations and problems, we would like your office to experience the TOEFL iBT. We can have our representatives help you to prepare for this test for 4-5 hours and some of your representatives can take this test. Then, we are willing to show you how to register and take the test in 4-5 hours. This will help you to profoundly understand our frustration.

Conclusion

ETS has not been responsive to our request to provide a good testing environment and meaningful scoring system. Therefore, we appreciate any assistance you can give to help ETS understand the importance of our request and justified indignation.

Thank you very much for taking the time to read about our concern. We will appreciate any effort and want to see an improvement. We are looking forward to hearing from your office. We hope that we can work together to resolve these issues. If your office has any further inquiry, we are willing to have our representative talking with you.


Sincerely,

Your signature

Ms./Ms. (Your name)

Enclosure: TOEFL iBT Score Reports, Description of TOEFL iBT test center problem
Cc: FPGEC, Competency Assessment Director (NABP)

References

1. Educational Testing Service (2004), Converting Rubric Scores to Scaled Scores, Writing and Speaking Sections of the New TOEFL iBT Test. Available at: http://www.ets.org/Media/Tests/TOEFL...ing_Rubric.pdf Accessed June 1, 2007

2. Educational Testing Service (2007), the official guide to the new TOEFL iBT, p.233, p. 227, p. 247, p. 350, p.233-241. New York: McGraw-Hill

3. Educational Testing Service (2007), TOEFL® Test and Score Data Summary for TOEFL Internet-Based Test: September 2005-December 2006 Test Data, p. 9-10. Available at:
http://www.ets.org/Media/Research/pd...M-0506-iBT.pdf Accessed June 1, 2007.

4. Inside, Higher Ed, Access also a Problem for TOEFL Candidates, April, 2007, Available at: ETS Aborts New GRE :: Inside Higher Ed :: Higher Education's Source for News, and Views and Jobs Accessed June 5, 2007.

5. Wikipedia, TOEFL Criticism, May 30, 2007. Available at: TOEFL - Wikipedia, the free encyclopedia Accessed June 1, 2007.