November 12, 2009
November 12, 2009
Have you ever wondered if you might fail an English test that your students take even though you are a native speaker of English? And if you did fail, what would that say about the test (Assuming that you were giving it your best shot)? It could happen. It has happened...
One of the more memorable ELT presentations I've attended recently was given by Terry Fellner, Associate Professor at Saga University. Terry is not only a native speaker of English (duh!), he's a particularly well-educated and articulate one. But earlier this year Terry, out of curiousity as well as a means of 'testing the test', decided to take on the new Speaking/Writing TOEIC test (also known as 'Walmart').
I won't keep you in suspense here regarding the results. You can probably guess what's coming:
Terry's score was not in the highest percentile but in the second rank, which meant that Terry was judged not to be a proficient speaker/writer of his native language. Among the weaknesses cited were:
- errors when using complex grammar
- imprecise use of vocabulary
- minor difficulties with pronunciation, intonation, or hesitancy
In none of these categories (or rubrics, if you will) did the highly articulate Terry Fellner deliberately or willfully fall short.
Terry was also judged to have some problems regarding the relevancy of his responses (readers should note that the test was all done online in real-time but obviously in a depersonalized manner). And this is where it gets interesting.
Terry decided to test the pragmatic preconceptions of the test by giving slightly unexpected responses but responses that were nonetheless, given the questions and tasks, logical, orderly, comprehensive and, of course, expressed with fluency. Let's take a look at some of these...
1. Terry was asked to describe a photograph. What he chose to do though was not start with an explanation of the foreground image (apparently a cart and horse) but rather focused upon the surrounding qualities of the picture: the weather, the background scenery etc. In one sense his choice might seem to be facetious, deliberately obverting the evaluator's expectations, but why should examines be expected to conform to certain narrow Western notions of centrality or importance on what is purpotedly a test of INTERNATIONAL communication?
Not only is it considered to be a cultural trait by many to focus on background and surroundings before articulating the 'center' but some personalities may also have this attribution (a skirt-chasing friend long ago displayed an incredible to spot, focus upon, and remember any 'hot babe' at locales such as The Parthenonor Notre Dame cathedral while forgetting what city he was in). A better example may be to look at a classical Chinese landscape painting. While there may be a hermit/poet scrawled in a lower corner somewhere this is not what the painting is 'about'. More central is the background- the atmosphere of the mountains, the textures and shapes that surround the 'subject'. Notions of background and foreground are blurred, mixed.
This can happen with music too. In many non-Western music styles the 'melody' is not nthe foreground or center, but rather tonality, timbre, texture, polyrhythm etc. Fans of modern classical music and most modern jazz, which tend to incorporate non-Western tonalities, will also be familiar with this aesthetic.
In other words, the test assumed a Eurocentric model of both viewing and description- again for a test of INTERNATIONAL communication. But wait, there's more...
For one speaking task, which required Terry to propose a solution to the problem of high office expenses, he suggested the use of clay tablets to replace computers and paper. This he supported logically and consistently (in keeping with the demands of the task) arguing that it was 'proven technology', with cheap and easily accessible materials, that are environmentally friendly, and that clay tablets could also recoup costs by being resold as housing material.
Maybe not what the test evaluators were suggesting but still expresed with relevance, logical consistency, sufficient support, and of course, fluency.
Terry also assumed a very familiar stance with his superiors in the task, going as far as to apologize for his lateness as being the result of a hangover. Here he was testing the TOEIC's notion of appropriacy- what kind of appropriacy? Whose standard?
Similarly, in a writing task in which he was required to respond to an email from a real estate agent with two requests and a question, Terry complied by not only thanking her for the email but expressing surprise that she was now out of jail. He also requested a location that would be near, among other things, a nunnery and a German bakery. His request was to pass along a hefty 'gift' to a police sargent while asking how the buasiness license was coming.
Socially inappropriate? In some (but certainly not all) cases. Does his response display a lack of knowledge or understanding of English discourse? Not at all. Was it unconnected to the demands of the the task? No. Was it expressed in an intelligible manner? Certainly. So where did Terry go 'wrong' on the test? What was the scoring rubric? And was it geared towards certain localized 'norms' that do not reflect a flexible, or international, standard? Seems likely.
It goes on..
In an essay writing task in which he had to explain qualities that Customer Service Representatives need to be succesful he responded by expounding upon the ability to project false sincerity and the willingness to work for a low salary. Again, he met the demands of the question, and utilized his English skills but not exactly in a way that the evaluators would be looking for. Perhaps then the place that they are looking is too narrow. To pragmatically focused upon a North American model. Culturally loaded. Morally loaded too.
Apparently, a lage number of TOEIC test takers end up being placed in the same percentile where Terry was rated. So, if a native speaker ends up there, what does this say about the accuracy and relevance of the scoring? A huge variety of skill levels seem to converge onto this one evaluation slot. It becomes rather meaningless.
And what about the feedback he received? Doesn't it sound a little like an o-mikuji bought at your local shrine at New Year's, where one's allotment of good luck or bad luck is already printed on the paper, prefabricated 'fortunes' completely independent of the individual actually buying them?
Now this isn't meant to rag on the TOEIC people. I know how hard it is to make a comprehensive test that is completely valid (it tests what it claims to be testing) and reliable (the result will be unaffected by happenstance). In many ways the TOEIC is admirable and comprehensive, but it is very very far from being foolproof- especially on the new Speaking/Writing version. I'll go even further. It is still very far away from being an accurate measure of a students' ability to speak or write English for international communication.
Recently, many universities have been getting all hot 'n sweaty about the alleged 'objective' value of TOEIC scores, which is supposed to represent an evaluation that is better than that of a trained in-house English teacher. Terry Fellner has shown though that this is still an illusion. Universities who think that a TOEIC orientation should replace normal communicative English learning had better think twice.