Columns on ELTNEWS.com View All Columns
Visit ELTBOOKS - all Western ELT Books with 20% discount (Japan only)

The Uni-Files

A candid look at EFL life and lessons from a university teacher's perspective.

January 21, 2010

'Misses' and 'objectivity'

The Center Shiken (National University Entrance Exam) took place a week back and I'm sure many readers were involved at some level, most likely by proctoring. And if you were proctoring, (even if you were a back-up proctor, yes, there are benchwarmers in Japan's Center Shiken proctoring world) you will know the intricate protocols, steps, conditions, and general hoop jumping that is involved in what many might mistakenly think of as an easy process.

The key notion is of course that the Center Shiken must be fair and fully objective. That's why it is held nationwide with the same subjects being tested at the same time in over a thousand locales Japan-wide with over 500,000 students taking part. In order to maintain this integrity the surrounding system has to be airtight. Details are meticulous and must be adhered to under threat of your photo appearing in newspapers regarding a breach of Center Shiken protocol. No compromises. Nothing slipshod is allowed.

Lengthy protocol explanation sessions, complete with instructional CD ROMS, are prepared for proctors. The instruction booklet is the size of a small telephone book and, as far as I can read, contains provisions regarding appropriate actions to take if an examinee freaks out, becomes physically ill, if an alien lands in the testing room, and if an examinee suddenly morphs into The Dave Clark Five.

You know, the Japanese are generally very good with this type of thing. One old school generalization about Japan that I hold on to is the fact that the couuntry is pretty risk adverse and great lengths will be taken to ensure that there are no 'misses' ('miss' being the standard abbreviation for 'mistake', and it is the default term used in Japanese). If you've ever been involved, or merely watched, a kindergarten or elementary school undo-kai (sports day) you can see the meticulous, orderly planning manifested in a seamless- but somewhat tense and regimented- performance. (Whether people actually ENJOY it is another matter).

The thing is though, the more you try to avoid 'misses' by fine-tuning, tightening the screws, or devising manuals that try to cover every contingency, the tighter the system the more likely that a 'miss' is likely to occur- precisely because you've created a huge checklist of protocols that now could go wrong. As analogies, think of pure-bred dogs and how finnicky they are. Think of the guy (it's almost always a guy) who tweaks his computer to a T but it's always malfunctioning when any new software is introduced. Think of body builders where each muscle teeters on the brink of both 'perfection' and complete physical breakdown. The fact is, the tighter you build the foundation, and the more pieces that you use, the greater the likelihood that one piece will falter and lead the whole thing to collapse.

Hence, the near fetishistic emphasis upon 'miss' avoidance can actually induce scenarios where more misses are likely to occur. At the Center Shiken we proctors were quite tense, with almost every second accounted for and formally backed up in some way, making sure that the myriad steps were taken in precise order, with military obedince to the manual. This meant that we had to act with speed and efficiency but also meant that any screw ups would lead delays or claims from examinees of some breach of norm. And the more nervous, cluttered, and time constrained you are, the more likely that a 'miss' will occur. (There was also a ubiquitous stretcher placed outside the examination area, as if to underscore the severity of it all).

Now, here's the twist.
A miss in the test administering protocol is considerede a huge black mark. Therefore, about 95% of the pre-test information sessions and meetings focus upon the avoidance of a 'miss'. But, as an English teacher, I am more concerned about 'misses' at the larger level. Let me explain.

At the orientation sessions for teachers making the second-stage university entrance exams (NOT the Center Shiken orientation sessions) the overwhelming emphasis is also placed upon not having any 'misses' in the test. There is, in my opinion, too little emphasis placed upon producing a test that is valid and reliable. In other words, the overriding rubric is negative: "Don't have any mistakes on the test. That's all we ask". The endless fix-up and follow-up sessions are designed to make sure that no misses get through.

A big, get-called-before-a-committee mistake would be something like the following:
Match the four paraphrased sentences below with the undelined sentences (1,2,3,4) in the passage.
a.
b.
d.
e.

Although the lack of a 'c' answer should not really confuse students or cause them to answer incorrectly, this would be a huge black mark for the test makers.

Anyway, administrators usually want 'objective' style tests because objectivity, it is believed, reduces the likelihood of mistakes. So, in order to meet the heavy 'no-miss' criterion you could make discrete English language test questions like the following:
1. The Montreal Canadiens last won the Stanley Cup in [ ].
a. 1998
b. 1984
c. 1993
d. 2004

2. Hitler's [ ] regime lead to the restructuring of Europe's political boundaries
a. nebulous
b. soporific
c. pernicious
d. sendentious

As you will see, there are officially NO misses in the above questions. But they are clearly absolutely crap questions for an English test. (I've exaggerated the samples- I can't imagine any exam actually making such questions although they did come close in the not-too-distant past- to make a point).

The first question does not measure English skill in any way but rather teasts localized knowledge which happens to be presented in English. And even if this was accompanied by a passage containing the answer (c) it still would not be indicative of English skill, especially in terms of measuring suitability for university entrance. Also, if the answer was contained in the passage 99.9% of the examinees would get it correct which renders the stratifying force of the question meaningless. So, while there are technically no 'misses' in the question it is nonetheless both invalid (it doesn't measure what an English entrance exam is supposed to be measuring) and unreliable (it's either too hard, based on chance specialist knowledge, or -if the answer is in the passage- it is too easy) and thus cannot have any stratifying function for placing examinees.

But it IS 'objective'. It contains no 'misses'. Also, the answers can be immediately measured numerically: 2 out of 2. Administrators love this type of thing and consider it somehow more 'objective' because the results can easily be rendered as numbers- even though these numbers basically indicate NOTHING about actual English ability. "Hey, if it's mathematical it must be objective!"

In the second example, the vocabulary choices are obviously way over the students' heads which means that if the correct answer is chosen it will almost certainly be chosen randomly (and of course a trained chimpanzee has a 25% chance of getting the correct answer on a 4-item multiple choice question).

Hey, but it is still 'objective' and contains no 'misses'--- despite the fact that it is thoroughly invalid and unreliable.

OK- I can't imagine any university entrance exam test maker making such egregious errors (in fact, in my research I have found that many second stage entrance exams and recent Center Shiken are quite valid and reliable). But the point is that an inordinate focus upon avoiding misses and maintaining this surface, shallow notion of objectivity can obscure the bigger picture- that of makng valid and reliable tests that acuurately or reasonably measure a wide range of student English skills.

Questions that demand deep thinking or skills such as making inferences, reading between the lines, predicting, summarizing and so on tend to be both more complex and nebulous than simple kigou (so-called because they can be answered by a letter mark- a,, b, c, d) questions. This complexity or lack of clarity can often led to what overseeing commitees think of as 'misses'. Overseeing commitees don't like the alleged 'subjectivity' or interpretive element that such questions demand. Hence the safety factor in making more discrete TOEIC-type questions

I find this fear of alleged subjectivity odd. After all, as trained professionals it is precisely we who should be expected to be able discern which students display the greatest ability in a subjective or essay-type question. By taking away the subjective evaluation element from a trained, experienced pro (who is supposed to be an expert in the field- that's why you've hired them to teach at a university) you've basically narrowed the scope of the test. You're no longer measuring extensive English skills but discrete item knowledge. You're no longer testing English ability but knowledge about English.

Your emphasis on 'no misses' at the expense of greater test validity and an artificial sense of objectivity that in fact often reduces test reliability means that you've messed up the bigger picture of measuring holistic student English ability.

And that's the biggest 'miss' of all.

A QUICK FUNNY- My all-time greatest classroom mistake

A long time back, when I was new to Japan, I had a small class in which I asked the students to tell me about the Japanese person who they admired most. One of the students answered 'I admire Chiyonofuji'. At that time I had no idea who Chiyonofuji was, so I asked. "He is a small restaurant," came the reply. "Non, no," I responded. "He OWNS a small restaurant or he runs a small restaurant. Not 'He IS a small restaurant'". The student looked both frustrated and amused. "But he IS a small restaurant" he insisted. A few seconds later another student spoke up. "Chiyonofuji is a sumo wrestler," he explained.

Oh (blush).
But come to think of it, some sumo wrestlers are actually like small restaurants.



« Japan- Love it, Leave it, 'Change' it, Or...? | Main | The Politics of 'Hello' »

Comments

As always good stuff, Mike. The infatuation for objectivity, I would guess, comes from a couple angles. One, which you mention, is to make the test as fair as possible. If everyone is taking the same test and there are clear right and worng answers then it is easy to measure who gets the best score. Another reason the tests are probably made this way is for practical reasons, seeing as it would be quite cumbersome to grade 500,000 essays. And I question whether the officials who make the test would trust NJs to be able to evaluate the essays as objectively as possible. They probably assume we could never understand the finer details of a japanese university entrance exam and would therefore not be able to accurately grade the essays (even though they hirte us as trained professionals as you point out).

What the test makers CLEARLY do NOT understand is that language ability (which is what they are supposedly trying to measure) can not be tested in an objective way. That's not to say language ability cannot be measured at all, it's just that it's almost impossible to assign any kind of a numerical, objective score to langauge ability. Obviously the sentence "Me go Tokyo last week" is not as good as the sentence "I went to Tokyo last week" but how could the accuracy of the utterances be assigned a score (maybe this is a bad example seeing as the second sentence is perfect and could be given a 10 out of 10 score, but you hopefully get my point).

This idea that English is a science which can be broken down and analyzed so as to be able to be measured objectively is precisely the reason why so many students graduate from high school knowing the meaning of words like "nebulous" but can barely introduce themselves in a natural way. As you say, this is the biggest miss of all.

The questionable reliability and validity of my own uni's entrance exam used to trouble me. Then I saw the minuscule numbers of students actually admitted to through the test. There are so many other ways to get into the uni that I don't give it a second thought and just make sure there are no "misses".

Who was blushing at the mistake? I don't think your classroom mistake is anything to be embarrassed about. It just shows teachers aren't teaching pronunciation properly. Students should get more moments like that. They might learn something.

Recent Columns

Recent Comments

Categories

Comments

Events

World Today