Columns on ELTNEWS.com View All Columns
How good is your English? Free test app from Oxford University Press

The Uni-Files

A candid look at EFL life and lessons from a university teacher's perspective.

May 02, 2011

A response to Tim Murphey's criticism of Japan's university entrance exams- and more on Kumiko Torikai

Tim Murphey’s plenary speech at the 2010 JALT Conference was one of the best I’ve ever seen. Tim’s outstanding address provides a strong argument for inviting local practitioners to give the plenaries, as opposed to those foreign names of note who might not be familiar with the EFL scene in Japan or, as has happened on occasion, just go through the plenary motions.

Tim usually has something of interest to say on any EFL-related topic so I was eager to read his interview regarding university entrance exams with ELTNEWS.com. . Unfortunately though (and no, I’m not being contrarian just for the sake of creating controversy), I was a little disappointed in some of Tim’s comments and I think that some readers who serve or have served on university entrance exam committees, and have tried to create more meaningful examinations, might take umbrage at some of the more sweeping comments. Of course, there is still more of this interview to come so I expect that some points may still be fleshed out. Nonetheless several items struck me immediately.


Taking the exams seriously means having serious people on board

In a similar vein to my critique of Kumiko Torikai (more on that later) let me start with a clear point of agreement. Tim mentions that many members of entrance exam committees do not want to be on the committee and, moreover may have absolutely no knowledge of testmaking or testing theory in general. So, if universities are taking these exams as seriously as they appear to, and as seriously as most examinees do, surely you want to make sure that the team entrusted with the task has some testing expertise on board, and moreover, that everyone involved wants to be involved.


Is the problem really the washback effect or…?

Ok. Readers are probably more interested in what I found disappointing (jackals that we are). Well, it seems to me that Tim is holding on to a viewpoint that I would say is a popular, but outdated, myth-- specifically that the problem with university entrance exams is their washback effect on to high school (and further) English education. In fact, there has been a lot of research which questions, and in fact appears to disprove, this notion. Among these are:

1) Michael Stout’s 2003 ETJ Journal piece, 'Not Guilty as Charged'

2) Bern Mulvey ‘s 'A Myth of Influence' from the JALT Journal in 1999
and from my own research:

3) '...But I Have To Teach Grammar,' from The Language Teacher (2000)

4) 'A Comparative Analysis of the Japanese Senta Shiken Based on a 25 Year Gap,' from the JALT Journal of May 2008

5) 'Some new proposals and responses in ascertaining the reliability and validity of Japanese university entrance exams', from the journal Shiken in 2008 and,

6) 'Teaching Progressively... For the Center Shiken!' from the JALT 2005 Proceedings (published in 2006)

Each of the papers mentioned above argue (in different ways) that the alleged washback effect is either a myth, greatly overstated, or simply cannot be blamed upon university entrance exams. Each of these pieces shows (again in different ways) that a greater number of university entrance exams are demanding productive skills from examinees than before, a wider range of cognitive testing, a broad range of analytical skills- that the old discrete-point, receptive ‘kigou’ questions, usually redolent of grammatical minutiae, have decreased significantly, even on the Center Shiken. In other words, if high schools and jukus want their student to succeed on university entrance exams they should be helping students develop these more integrative, holistic skills- which have general pedagogical value as well.


The pressure spirals up the system, not downward

The fact that many high school or jukus seem to ignore this cannot be blamed on the universities but either upon an ignorance (or gap in time-comprehension) of the tasks and skills demanded on the exams (although this is unlikely since the better jukus and schools go over these tests like 17 year old boys checking out each others’ wheels in high school parking lots). The more likely culprit is the education publishing industry who make big yen churning out guides to entrance exam preparation and don’t like to see universities change their approach- which would mean a fundamental revamping of their materials- some of which might be beyond their academic comprehension.

In support of this I also point to a recent conversation I had with a researcher at a national university who is currently conducting an in-depth survey regarding the roles of non-Japanese members on university entrance exam committees. I don’t want to steal her thunder by revealing too much about her research but she did tell me that many university teachers remarked on the pressure they received from both high schools and jukus to keep the test focus upon discrete-points and receptive items. In other words, if and when universities are still maintaining non-productive entrance exams, I would suggest (as I have before) that it is more a case of ‘washforth’ than washback.


Avoiding the stress associated with entering a prestigious university- How?

Tim sympathizes with the stress felt by examinees at the mercy of this system- and understandably so- although with current demographics being what they are it is not nearly as difficult to gain entry as in previous generations-- in fact almost anyone can get into some university now if they wish to. But what I would like to ask Tim is this: even if there was no entrance exam as we know it, there would still have to be some meritorious criteria by which one student would be able to enter and another not. In other words, there will always be stress in terms of making the grade and gaining acceptance no matter what the criteria is. Some students will not be able to enter a prestigious university and will be crushed by that fact. I’m wondering how Tim would develop an entry criteria that would not lead to stress and/or possible disappointment. In short, what would Tim Murphey do in place of the current exams to eliminate the psychological fallout from not making the cut?

I must also admit being surprised at Tim's suggestion that one possibility be entry based only on the Center Shiken score. This would seem to me to make the testing system even more of a crapshoot- with a corresponding increase in related stress- since any local colour that the individual wishes to gauge examinees by would be eliminated. In short, such an entry process would be very mechanical and detached, lacking the local elements that Tim seems to be otherwise arguing in favour of, such as a greater focus on the individual examinee's academic background when considering candidacy and the more productive, holistic focus that is actually suggested as being the intended purpose of individual university exams by Monkasho (since the Center Shiken cannot address such skills adequately). I understand that Tim's responses were made in a real-time interview and not as a research paper but I still find this oddly incongruous with the rest of his thought.


The question of validity: what’s the purpose of university entrance exams?

Then there is the matter of ‘validity’. In the preview to the published interview, ELTNEWS.com paraphrased Tim as saying that the entrance exams were not valid since they didn’t test what they said they were testing. Of course, this is a perfect definition of test validity but Tim doesn’t seem to address this in the interview (later perhaps?).

In a paper I wrote for The Language Teacher in 2008 I suggested a purpose for the entrance exams (since there seem to be none extant in Japanese or English, so it appears that either everyone is supposed to know the purpose instinctively or that they are just done this way because they have always been done this way). In the article, I proposed that the purpose of university entrance exams is to ascertain, as best one can, which students among the candidates is more likely to perform well on that subject in a research capacity on academic topics in the future- the suitability criterion. After all, that is fundamentally what university education is all about, is it not?

This should preclude any focus on grammatical or lexical minutiae as being a valid basis for testing (although obviously the examinee would still have to display competency in these areas). It also precludes daily, practical English of the conversational or ‘travel abroad’ type (although such skills might still be rewarded on the exam). It would not be an achievement test- a culmination of high school knowledge (surely, a university entrance exam is forward-looking, that is for placement, rather than acting as a summary of a completed course). Nor should the exam try to be an indicator of the examinees’ extant knowledge on the subject of the faculty they hope to enter (e.g., the entrance exam for a Nursing school should not be a test of knowledge of nursing). It should reveal an examinee’s aptitude for the academic pursuit of English in a non-English language setting. It should therefore, challenge the examinees' cognitive skills and duly reflect these. It should focus upon productive skills, and multiple competencies. It should allow for intellectual and emotional expression.

If the exam meets this criteria, and I believe that some (many?) second-stage entrance exams certainly do, then it would be valid. I wonder if Tim would find the above acceptable (he does mention that holistic testing should be encouraged- so why not do so within the existing entrance exam format?) and, if not, to be clear as to what aspect of incongruity informs his claim that the exams are (all?) invalid- because so far he has spoken mostly in generalities on the issue of validity.

Then there's the connected issue of reliability. Tim suggests that entrance exam committee members carry out a follow-up item-by-item reliability analysis. But I see three problems here. One is that this means more work for the team- and Tim has already complained about the administrative workload involved. Second is the question as to how to apply reliability measurements when items are changed every year? Sure, a certain item may have proved unreliable this year but you wouldn't be putting the same item on next year's test anyway so... I wonder if Tim has any concrete suggestions on how to achieve this. Thirdly, doesn't this approach assume and/or encourage the very discrete-point kigou type questions that make Tim want to scream?


Part 2: A partial vindication of Prof. Kumiko Torikai- Interview of April 28th in the Daily Yomiuri

A number of readers read, and supported, my critique of Prof. Torikai’s Asahi Shimbun article . Torikai was subsequently further interviewed on April 28th by the Daily Yomiuri newspaper (as well as by our own ELTNEWS.com). In my opinion, the Yomiuri interview was much more palatable and well-thought out than the Asahi piece. She seems to have responded indirectly to some of the criticisms sent her way by clarifying certain points of import utilizing a more academic than emotional argument.

Primary among these is her partial clarification as to what constitutes a ‘native speaker’ in which she seems to support Kachru’s concentric circles model (although I think she distorts the status of English in Singapore, India and the Philippines somewhat as being ‘not a native tongue but an official second tongue’ when in fact it is an official language in each nation, on equal official footing with ‘native’ tongues, serving particularly as a unifying factor in these linguistically diverse areas).

Unfortunately, she still seems to lump these countries in with EFL countries such as China, Japan etc. (the outer circle) when talking about non-native numbers vis-a-vis 'native speakers' whereas I'd say these are clearly two distinct categories. This conflation distorts the picture and is rather unfair to Singapore, India and The Philippines, not to mention Kachru's model, although it does reinforce her point that English is not owned by some guy in Iowa.

Torikai also makes reference to Jennifer Jenkins and the Vienna-Oxford International Corpus (VOICE), which aims to establish a worldwide standard that is not completely beholden to any local standard or variety (read: UK or U.S.). Fine (Here she might be seen to be supporting Quirk’s World Standard, but not his identification of U.S./U.K. English as representing that standard, which was strongly opposed by Krachu. Nerds may want to look further into the references I'm making here). Thankfully, this is very, very different from her Asahi claim that Japanese could legitimately say, “But this is how we say it in Japan!” when confronted with criticism about their English shortcomings. After all, Japanese English (which is an interlanguage stage and not a variety) will surely not be equivalent to the proposed VOICE standard.

She also seems to be aware in this interview that there is a big difference between being ‘a variety of English in the world’ and being ‘a world English’. Good. But this does beg the greater question as to how her perception of the validity of Japanese English fits in to the mix.


Communication over conversation

Finally, I also support Torikai's assertion that conversation is overemphasized in most current curricula (a point that Tim Murphey seems to take issue with). I have long questioned the notion of ‘teaching conversation’ and have always emphasized productivity and meaning-based communication (spoken or otherwise) as being a suitable basis for both secondary and tertiary English education instead. Conversation is just one minor piece of the much bigger communicative, meaning-oriented, cognition-engaging picture. And while the inherent 'discourse analysis' might be a university-level academic term, this type of consciousness-raising can still be applied to high school learners, without employing the highbrow nomenclature.

Unfortunately, in the Murphey interview, Torikai's criticism of the overinflated role of conversation seems to be treated as an ipso facto endorsement of grammar-teaching approach which is a bit of a jump because Torikai simply affirms the importance of reading and writing (correctly, imo, questioning the popular paradigm which says that Japanese students can write and read but not speak in English). Reading and writing can be taught meaningfully and communicatively without resorting to discrete-point teaching. In fact, intelligibility- as opposed to detail- is the thrust of Torikai's entire Yomiuri interview.

I'm looking forward to the remaining sections of Tim's interview as well as the future Torikai interview here on ELTNEWS.com



« The F word on trial | Main | Putting conversation in its proper place- a few ideas and practices »

Comments

Mike, another thought-provoking read here. Thanks for these fine articles.

I think your section, "Avoiding the stress associated with entering a prestigious university" could be better served by exploring the situation in other countries. What about Canada and the US, for example (to take two countries that we might/should know about)? I've always thought the better universities--and I have no experience of their application process --look a lot at extracurricular activities (i.e. volunteer work), other accomplishments (class president), high school grades and general personality (via interview). Is that your impression? Isn't that desirable to putting so many eggs in the uni entrance basket?

In my experience, uni made test questions are pretty absurd. One uni I worked at used remarkably difficult passages--some tough for native speakers. In most cases, the questions could be answered, in my view, by some sort of trick, i.e. just picking off a key word or two and tracing it to the right part of the article. Or simply making the questions blantantly easy (so the test-taker would be best served skimming or skipping the article and just trying the questions based on common sense). Then, other questions were clearly gimmes, probably to balance with the doctorate-level ones. Altho Tim's argument might not be air-tight given the research you cite (I haven't read it yet of course but would like to), I still think his main argument that university entrance exams are not made by experts, waste a lot of faculty time, and are far from the best measures of future success at university. Not pushing students through to graduation would be another step in the right direction.

I look forward to reading Tim's next interview--and your response.

John Spiri

Hi John. Thanks, as always, for your comments.

I have seen some tests like the ones you describe above. They are simply poorly thought-out, badly designed tests and I don't think they warrant a wholesale condemnation of entrance exams or 'the entrance exam system' anymore than the fact that there are some shoddy textbooks out there means that we have a moral obligation to throw the entire textbook baby out with that bathwater.

As for the North American entry model... I don't know much about the U.S. and GMATs. How similar is that to the Center Shiken? Don't U.S. students trying to enter prestigious schools suffer great anxiety too? I don't know.

Right now, many J universities (including my own) do incorporate interviews and short essays into the total score. Most also have some spots for entry based on recommendations- which includes a perusal of character, HS and community activities etc. But it still seems to me at some point that even if you widen the criteria like this there would still be great stress as to whether that candidate will make the grade or not- it's just spread out over a longer period.

I think geography is a factor in the heirarchical nature of J universities and shows similarities to the U.K. and eastern U.S. Since there are numerous universities one could physically attend with ease, students in these regions will naturally vie to enter the best one- hence a ranking system emerges organically. This is quite unlike Western Canada for example, where almost everyone who wants to continue studying, whether excellent or mediocre studenrs, will head to one of the two or three nearby universities, and thoughts of heading to the Ivy league or even McGill are left to either the rich, connected, or otherwise remain in the same likelihood category as marrying Kim Kardashian.

Hi Mike,

I don't think the stress of an interview, or the stress related to having a university look at my accomplishments (awards, leadership, musical instruments played, etc.), if there is any stress for that at all, can compare to the stress of an exam which consumes massive amounts of time spent mostly memorizing facts. Students who are rojin, for example, take a year off to do nothing but study for a certain exam. I know a young man planning to be a rojin for two years for Todai's exam. That seems outrageous to me, two years mostly wasted.

But I look forward to reading more about this soon.

John

Hi John.

I agree that accruing credits for citizenship while in HS or interview skills is not as stressful as a two day exam fest but I think that the spectre of subjectivity starts to loom larger here. Campaigning or lobbying for entry into prestigious universities would seem to weigh more than actual academic merit. I know that this is a great fear among test administrators in Japan.

By the way, if students try to just memorize facts in order to succeed on the exams of the better universities, I think they'll be in for a rude surprise. Maybe someone has done a precise analysis of other exam subjects but from what I've seen and heard, many exams do not emphasize memorization, and especially not rote memorization. Certainly not in English.

Recent Columns

Recent Comments

Categories

Comments

Events

World Today