Columns on ELTNEWS.com View All Columns
Visit ELTBOOKS - all Western ELT Books with 20% discount (Japan only)

The Uni-Files

A candid look at EFL life and lessons from a university teacher's perspective.

November 12, 2009

Q. When is a Native Speaker not good enough to be a Native Speaker? A. On the TOEIC test

Have you ever wondered if you might fail an English test that your students take even though you are a native speaker of English? And if you did fail, what would that say about the test (Assuming that you were giving it your best shot)? It could happen. It has happened...

One of the more memorable ELT presentations I've attended recently was given by Terry Fellner, Associate Professor at Saga University. Terry is not only a native speaker of English (duh!), he's a particularly well-educated and articulate one. But earlier this year Terry, out of curiousity as well as a means of 'testing the test', decided to take on the new Speaking/Writing TOEIC test (also known as 'Walmart').

I won't keep you in suspense here regarding the results. You can probably guess what's coming:
Terry's score was not in the highest percentile but in the second rank, which meant that Terry was judged not to be a proficient speaker/writer of his native language. Among the weaknesses cited were:
- errors when using complex grammar
- imprecise use of vocabulary
- minor difficulties with pronunciation, intonation, or hesitancy
In none of these categories (or rubrics, if you will) did the highly articulate Terry Fellner deliberately or willfully fall short.

Hmmm.

Terry was also judged to have some problems regarding the relevancy of his responses (readers should note that the test was all done online in real-time but obviously in a depersonalized manner). And this is where it gets interesting.

Terry decided to test the pragmatic preconceptions of the test by giving slightly unexpected responses but responses that were nonetheless, given the questions and tasks, logical, orderly, comprehensive and, of course, expressed with fluency. Let's take a look at some of these...

1. Terry was asked to describe a photograph. What he chose to do though was not start with an explanation of the foreground image (apparently a cart and horse) but rather focused upon the surrounding qualities of the picture: the weather, the background scenery etc. In one sense his choice might seem to be facetious, deliberately obverting the evaluator's expectations, but why should examines be expected to conform to certain narrow Western notions of centrality or importance on what is purpotedly a test of INTERNATIONAL communication?

Not only is it considered to be a cultural trait by many to focus on background and surroundings before articulating the 'center' but some personalities may also have this attribution (a skirt-chasing friend long ago displayed an incredible to spot, focus upon, and remember any 'hot babe' at locales such as The Parthenonor Notre Dame cathedral while forgetting what city he was in). A better example may be to look at a classical Chinese landscape painting. While there may be a hermit/poet scrawled in a lower corner somewhere this is not what the painting is 'about'. More central is the background- the atmosphere of the mountains, the textures and shapes that surround the 'subject'. Notions of background and foreground are blurred, mixed.

This can happen with music too. In many non-Western music styles the 'melody' is not nthe foreground or center, but rather tonality, timbre, texture, polyrhythm etc. Fans of modern classical music and most modern jazz, which tend to incorporate non-Western tonalities, will also be familiar with this aesthetic.

In other words, the test assumed a Eurocentric model of both viewing and description- again for a test of INTERNATIONAL communication. But wait, there's more...

For one speaking task, which required Terry to propose a solution to the problem of high office expenses, he suggested the use of clay tablets to replace computers and paper. This he supported logically and consistently (in keeping with the demands of the task) arguing that it was 'proven technology', with cheap and easily accessible materials, that are environmentally friendly, and that clay tablets could also recoup costs by being resold as housing material.
Maybe not what the test evaluators were suggesting but still expresed with relevance, logical consistency, sufficient support, and of course, fluency.

Terry also assumed a very familiar stance with his superiors in the task, going as far as to apologize for his lateness as being the result of a hangover. Here he was testing the TOEIC's notion of appropriacy- what kind of appropriacy? Whose standard?

Similarly, in a writing task in which he was required to respond to an email from a real estate agent with two requests and a question, Terry complied by not only thanking her for the email but expressing surprise that she was now out of jail. He also requested a location that would be near, among other things, a nunnery and a German bakery. His request was to pass along a hefty 'gift' to a police sargent while asking how the buasiness license was coming.

Socially inappropriate? In some (but certainly not all) cases. Does his response display a lack of knowledge or understanding of English discourse? Not at all. Was it unconnected to the demands of the the task? No. Was it expressed in an intelligible manner? Certainly. So where did Terry go 'wrong' on the test? What was the scoring rubric? And was it geared towards certain localized 'norms' that do not reflect a flexible, or international, standard? Seems likely.

It goes on..

In an essay writing task in which he had to explain qualities that Customer Service Representatives need to be succesful he responded by expounding upon the ability to project false sincerity and the willingness to work for a low salary. Again, he met the demands of the question, and utilized his English skills but not exactly in a way that the evaluators would be looking for. Perhaps then the place that they are looking is too narrow. To pragmatically focused upon a North American model. Culturally loaded. Morally loaded too.

Apparently, a lage number of TOEIC test takers end up being placed in the same percentile where Terry was rated. So, if a native speaker ends up there, what does this say about the accuracy and relevance of the scoring? A huge variety of skill levels seem to converge onto this one evaluation slot. It becomes rather meaningless.

And what about the feedback he received? Doesn't it sound a little like an o-mikuji bought at your local shrine at New Year's, where one's allotment of good luck or bad luck is already printed on the paper, prefabricated 'fortunes' completely independent of the individual actually buying them?

Now this isn't meant to rag on the TOEIC people. I know how hard it is to make a comprehensive test that is completely valid (it tests what it claims to be testing) and reliable (the result will be unaffected by happenstance). In many ways the TOEIC is admirable and comprehensive, but it is very very far from being foolproof- especially on the new Speaking/Writing version. I'll go even further. It is still very far away from being an accurate measure of a students' ability to speak or write English for international communication.

Recently, many universities have been getting all hot 'n sweaty about the alleged 'objective' value of TOEIC scores, which is supposed to represent an evaluation that is better than that of a trained in-house English teacher. Terry Fellner has shown though that this is still an illusion. Universities who think that a TOEIC orientation should replace normal communicative English learning had better think twice.



« 1. The image of MEXT and 2. Learning from open-ended conversation tasks | Main | Two mini entries: 1. Grammar, plurality, agreement and 2. Formalized brainstorming »

Comments

I've similarly played around with TOEFL to see what answers I could get away with. It seems to not allow for as many whimsical answers.

Interesting stuff. Of course this weird talk into a computer in Japan and have an American score the test is now the only way to do evaluate your speaking ability via the TOEIC brand. The face-to-face Language Proficiency Interview has been cancelled due to lack of demand.

On the other hand, learners and cheeky native speakers will soon be able to take the speaking and writing tests separately.

Don't get too excited about TOEIC being a test of "international communication". The Japanese businessman responsible for starting the TOEIC had a role in thinking up the name using some of his favourite words. The name of his main company at the time was called International Communication Inc. and all of his subsidiary companies also included the word "international" and/or "communication".

While I don't teach the Speaking and Writing TOEIC test (focusing instead on the TOEIC Listening & Reading), when I teach TOEFL Speaking, one thing I stress is the need for students to "know their audience".

It seems that Terry Fellner did not keep that in mind, and also failed to understand that though the Test Scorers are supposed to keep their emotions out of their evaluation, when they feel the test-taker is not taking the test seriously, they will grade the test-taker more harshly. As teachers, I think we've all had that "annoying" student in class who isn't taking our lecture seriously, thereby coloring our response to what they say & do. I strongly suspect that the test-scorers reacted badly to Terry's tongue-in-cheek responses and acted as any of us would react if we felt someone was belittling our job.

Personally, I view the TOEIC & TOEFL as hoops my students must jump through, and as such they need to just give the test-scorer what they are looking for. Bringing humor into a testing situation is, in my experience, a recipe for disaster.

However, that aside, I do agree that Speaking & Writing tests are tricky things to grade. I would be curious if IELTS system works any better. Has any study been done on IELTS Speaking section?

If Chris is right then there is no objectivity in the marking of TOEIC, which makes it a somewhat problematic in the area of reliability. That may be true but unlike Terry, Chris provides no proof for his contention other than how he treats students when they don't take his classes as seriously as he thinks they should.

Hi Everyone,

I am a foreigner who lived in US for many years and just took the TOEIC (Listening and Reading - total 200 questions) and my total average is about 83% out of the whole test.

I completely agree with Mike as he mentioned that "I know how hard it is to make a comprehensive test that is completely valid (it tests what it claims to be testing) and reliable (the result will be unaffected by happenstance)", but think that the test does not prove much.

I assume that there are more the same (TOEIC Listening and Reading) "but varying differently in words" than the one I took about a month ago. I know many native English-speakers, who mentioned that they could not do well in this test. Then, I am thinking, what's the point of this test and what's trying to prove - probably not much.

How many more different scenarios and words could a test such as this one could be used and somehow change the grade of a tester differently?

How about idiomatic expressions? I did not see it at all at "my TOEIC test."

Thanks.

Marco

The parent of this examination is traditional schooling. Schooling's purpose is not to measure ability, nor to educate, but rather to manufacture labor that conforms to a preconceived social model. There is ample historical evidence that public education's 19th century origins were seeded by industrialists in order to expand a pool of a dependent, obedient and non-questioning class (notice the pun). Since this particular test stems from the schooling system and is used for academic assessment, why should it be any surprise that the people who grade it are academics themselves and are admitted into the job of scoring based on their academic prow? Therefore would giving outside-the-box unconventional answers to those who are trained and hired to enforce conformity earn higher scores?

Recent Columns

Recent Comments

Categories

Comments

Events

World Today