« February 2010 | Main | April 2010 »

March 2010 Archives

March 3, 2010

Putting together a half-decent achievement test

If you work at a JHS, HS, college, senmon gakkko, or university in Japan you have probably just completed several year or semester end achievement tests. After all, you need grades for your students so some kind of evaluation is required. But this is an area in which a lot of mistakes are made, a lot of educational principles violated...

I'd like to think that testing is something I know a little about, an area that I've become at least a little sophisticated with. It was one of my specializations during my MA days as well as one of those areas in which I've kept up the research level, so I'm hoping that a few of the things I mention below might carry some weight above and beyond the 'some guy on the internet' level of credibility.

First point-
Achievement tests are not placement tests nor, usually, are they proficiency tests.
In an achievement test you are evaluating the students' course work. That means the focus of test content must be upon what students have, or were supposed to have, covered in the course. This means that any content that was not dealt with in the course should not be part of the test. It means that the skill emphasis should match the skills that you were trying to teach in your class. Test tasks should resemble those tasks which were practiced during the course. You are not gauging the students' overall English ability or general skill- which would be more representative of a placement or proficiency test- so don't try to. The test should measure a student's ability to meet the specific course goals as set out in the syllabus.

Second point-
If you are an educator the test should have an educational function.
It should have a pedagogical purpose as well as an evaluative function. Students should be learning from their tests. This means that students must know what they did right, what they did wrong and be given a chance to fix it. In other words a good achievement test has a diagnostic function. This has several administrative implications:
1. You must give the test back to the students. It belongs to them.
2. There must be some type of review or feedback for the students.
3. You shouldn't give the test in the final class or else you can't review it.
4. Students should be able to find out what the correct or model answers are.
5. Students who did poorly should be made to do a re-test, or two, until they show that they have learned the material (or skill).
6. Why not have students obtain good or correct answers on those sections where they did poorly by checking with peers? I do a 'test interview' where students ask one another those questions they didn't answer correctly and if the partner knows the proper answer, they can teach (not just 'tell') it to the other student.

Third point-
You can and should diagnose your own teaching effectiveness from the test results.
If students do poorly on the test, or on specific items on the test, it is very likely because either 1) the question, task, or entire test was invalid ( the test didn't actually test what is was supposed to) or unreliable (if a similar test was given to the similar students at a different time and place scores would be very different- meaning that happenstance affected the test results, usually as a result of poor test design).
2) you didn't teach whatever it is that you were testing well enough.
This should be telling you sometyhing. After all, tests test the teacher's effectiveness as well as the students'.

Fourth point-
You need to test more than just recognition (memory) and discrete-item knowledge.
Memory is a limited skill. Not only that but memory is not just recognition (the most passive, receptive aspect of memory) but also recall (contextual understanding), and reproduction (application). If you were teaching a class that was expected to focus on developing productive skills but give a test that measures only memory-recognition you have an invalid test.

Likewise, language is not just a collection of discrete-item knowledge. It is a dynamic system that involves numerous social and pragmatic considerations. So again, if your class was expected to develop student skills in using English within meaningful and/or practical contexts, if you focus mainly (or solely) on discrete-items you will have made an invalid test, since the skills you are supposedly trying to inculcate will have escaped the net of evaluation.

Fifth point-
The test can easily be used as a study and/or review experience
Open-book tests are great. Students can once again review material and find those things that the teacher wants them to understand. Open-book test success also relies more on a general comprehensive understanding of a subject as opposed to memorizing discrete items. Of course, given that the test is open-book we should also expect standards to be high. I have come to notice that students who are well-organized and think actively succeed at these tests while the laggards who weren't paying much attention or making much of an effort all year rarely rise above their 'stations'- at least on the first test. This doesn't always happen on discrete-point knowledge-based TOEIC-type tests.

Providing students with the test tasks or questions or old exams in advance (they'll usually get them from their seniors anyway) can help too. By letting students know what to study for, you focus their energies on those things you really want to inculcate and leave less to random chance, circumstance or wasted/misguided student effort.

Sixth point-
Ongoing evaluation, especially if you are using a variety of evaluative means and measures, is more effective than the traditional 'one final paper exam' format.
Language learning is a process and so the evaluation should be process-based and focus less on the one, final 'this-is-your-official-result' mode of testing. Using a variety of testing methods and means allows students who respond differently to different challenges to strut their stuff. Not all 'good' students are sharp at paper tests and may do much better on a role-play, report, or some type of visual/tactile task. Ideally, using all test types you can get a panoramic view of their all-round skills, and therefore a more accurate reading of their English abilities (assuming that you are trying to educate them in holistic way, that is).

Weighting tests is also important. Putting something like 80% on a final test might not be a good indicator of actual student ability over the entire course of the class. Breaking evaluation up into 20% increments allows for more types of evaluation and widens range of the criteria. It also tends to keep students alert and focused.

Seventh point-
Let students have some say in the test content
Productive, open-ended tasks are to be encouraged as these allow for some self-expression and variety, letting students use the language while actively thinking and engaging it. Most teachers will tell you that in terms of marking, these tasks and problems are easier to grade- and tend to provide a more comprehensive view of actual student abilities. Even better, allow students to make some tests themselves. This will allow for a good review of content and also show the teacher what students have learned (or not), or feel is important (or not). And what a teacher learns from this can be applied to next year's lesson plans.

I allow my students to appeal their test grades too- as long as they do so in English. If they feel that the grade on a 'subjective' test or item was unfair they have the opportunity to explain to me why their score should be higher, a process which demands that they consider both the test result and content but also how they will plead their cases in front of me.

Reader suggestions on testing are more than welcome in the comments section.

March 10, 2010

Some exceptional students; and one 'Debito moment'

I suppose the popular stereotype of medical students is that they are a bit nerdy, diligent and thorough, and come from fairly well-to-do families with a history of medicine in the background (Daddy runs his own clinic). As I've mentioned before, there is in fact a wide variety among our number.

Let me tell you about some students who stand out in particular:

Student Y: 5th year female. Exceptionally sociable, a real person's person. Comes from a family of seven (seven!) children and- get this- was raised by a single, welfare mother (her father was absent from the time she was born- I didn't ask why). Her mother worked at any number of odd jobs to help get her kids through school. When her daughter was accepted for medical school it was obviously a huge triumph for the family and for the mother in particular. Suffice to say that this student needs NO motivation and never seems to find the rigours of medical study to be too taxing. After all, it's probably a breeze compared to what she has already been through.

Student S: 6th year male. This is less 'inspiring' and more personally memorable. In their first year 'getting to know you' lessons students interview one another and one of the common questions is 'Who is your favourite singer/musician?'. The answers typically include the popular Western and J-pop divas, a few rap/reggae acts, the odd boy band (J or otherwise), indie J bands like Qururi or Spitz, and the odd folkie/MOR act like Kobukuro, but I couldn't believe my eyes when I saw that one completed form had 'King Crimson' listed as 'favourite musician'. Somebody in Miyazaki U. knows (and appreciates) those dissident tritones! After talking to the guy, he admitted a fondness for Van Der Graf Generator too. Ahh- back to my musically mind expanding post-high school days of the mid 70's...

Students A and K: Y is in her 5th year and K is starting his 2nd. Both come from tiny, remote islands. A is from one of those Okinawan outposts of about 500 people where the idea of going to university, let alone becoming a doctor, is rare and exotic. K comes from an island of about 100 people off a forgotten part of the Kyushu coast, accessible only by a once-a-day ferry. He's clearly a diligent and bright fellow- one of those kind who is always thinking and challenging himself. Somehow the dilligence required to succeed despite his locale followed him through junior high school where he was deemed academically fit to get full-funding to an elite boy's school in Kansai and then on to medical study...

Student E became pregnant during her second year, the father being a classmate. They did the 'right' thing, had the baby, and grew up very quickly, supporting each other and the child all while studying. Neither of them have failed a course despite now having two young children and a third on the way (!). Compared to this couple, students who think that the notorious physiology test represents the ultimate challenge don't know what tough is. Suffice to say that I would certainly trust a doctor with this much energy and gumption with my health.

We have numerous other interesting students, some with disabilities that they have to try to overcome, some who were raised abroad (of course some people in the J education system might consider THAT a disability), a few Todai grads who returned to Miyazaki wanting to become doctors, a few students who scored at the very top of the Center Shiken nationally but chose to stay in Miyazaki...However, I haven't asked their permission to mention them here (unlike those mentioned above) so I'll end this section by saying something about discretion and valour.

My 'Debito moment'

If you read this blog much you are probably aware that I'm not a big supporter of Japan's most well-known (notorious?) NJ human rights actvist (agitator/gadfly?). Debito bats about .100 for me, with about one out of every ten of his pieces in my opinion being accurate, balanced or worthy ('culture' as an overused and convenient excuse for dubious practices and the obsolescence of the koseki system being two that I agree with). But I'm sure that all NJs have our moments when we feel a bit put out by authorities in this, our adopted homeland.

This story concerns getting an international driver's license (I have a J license already) in Miyazaki. First, in filling out the international license application form I noticed a section asking us what our 'birthplace' was. Now this is tricky for me because, as you probably know, my citizenship is Canadian, as is my passport. But I was born in the U.K. (my family emigrated to Canada when I was 1 year old).
So I asked the clerk, "Why do you want to know my birthplace?".
"Because your citizenship must be noted on the license", she replied.
"But what if my birthplace and citizenship are different?" This took a few seconds to register with her.
"Oh. Ok. The country of your passport should be written in". I duly did so but mentioned that 'citizenship' or 'country of passport' should be the category, not 'birthplace' (you can just feel the long arm of the koseki here can't you?).

I then proceeded to the bottom part of the form where I was asked:
1. Where are you going?
2. When are you leaving and returning to Japan?
3. What is the purpose of your trip?

Now, for a driver's license this seems to me to be rather intrusive. What business is it of theirs as to why I'm going abroad, or where? This isn't the freakin' immigration office, is it? So, I told the clerk that this was private information irrelevant to issuing a license and said that I didn't want to divulge my private information in this way and so wouldn't fill that part in. I said this kindly but firmly, mentioning that I'm sure she was aware of the current importance of privacy issues in Japanese public affairs.

So she did what you could expect. She called the old Kacho guy from the adjoining office and explained it to him. I have less patience with these kind of people. You'll soon see why. He approached me and said "You have to fill this in. It's a requirement".
"Why"
"Because it's necessary"
"I'm afraid you didn't answer my question. Why is my private information, such as the reason I plan to travel abroad, necessary for a prefectural MV licensing center to know"
"Because we can't issue the license without it"
"Ummm you seem to be evading my quesition" (I then raised my voice- not in anger- but so that customers nearby could hear).
"It this because you plan to give citizen's and resident's private information to the police or immigration authorities?".
Saying this directly made him nervous, and rightly so. I didn't actually think this was the explanation but yes, I did want to rattle him.
"No. It's information like a census. If we know the applicants' travel data we can serve them better".
"Shouldn't it be voluntary then? After all this isn't North Korea, is it, where every reason for every movement has to made known to officials. Anyway, this data would already be known to immigration officials or travel agencies."
"We just collect the data, but it's not collated with the driver's personal details".
"It's not the Edo Period, where you couldn't move without permission from authorities, right? It's Heisei 22 and Japan is a democrracy, right?". (Now I was sounding like Debito. Yikes!)
"Look you don't have to write in detail. Absolutely anything you write there will do. But we can't move until you fill it in with something".
"OK".

So under "purpose" I wrote "private". Under destination I wrote "various" (this makes sesnse of course because the license is valid for a year and therefore for multiple visits. It's not like sigle permit re-entrry visa). And under 'departure and retuirn dates' I wrote that day's date (although I am not due to leave until later in March). He took the form away for processing.

I then asked the clerk, "I'm sorry about this but privacy is a current issue I'm sure you know and none of this seems relevant for a prefectural driver's license office. So as a resident and as a customer (you pay ¥2680 for the license) I'd like to make a complaint about this application form and ask that these questions be abolished in the future. Please mention this to your superiors or however you may process complaints. Oh- and one question. I'm curious. Do Japanese people sometimes complain about these questions?"
"Yes," she responded, "a few".
"Thank you", I said, "So please pass my comments on".

A few days later some beefy men in sunglasses in an official DMV car came to my home, demanded to see my passport, and tore out my Japanese visa. They also ridiculed my wife for being impure in marrying a foreigner and my children for being of mixed blood. Then, upon leaving, one added that 'Only Japan has four seasons' (I'M JOKING!!!)

But I admit that I did do myself in a bit. By being obstinate about the departure date I inadvertently caused that date to be named as the starting date of my license's validation, and not the day I leave- as a result I waste about three weeks' validity. Of course, instead of asking the intrusive "departure and return from Japan" question they should just ask, "From which date would you like validation to begin?".

March 18, 2010

Teachers' Meeting! (A 'Morality' Play)

Watanabe: First, I’d like to welcome you all to this meeting. As senior teacher I’ve been asked to create this working group on student morality by the Prefectural Board of Education, who seem to be worried about the alleged decline in student morals and want us to do something about it within the context of English education. (Aside) Hmmm I seem to remember my teachers saying the same thing when I was a student but whatever…
Anyway, other teachers will be addressing the issue within their own subject’s working groups and a report of suggestions and plans from us in the English department will be sent to the Board so feel free to offer your ideas.

Saito: I think the answer here is obvious. Morality means following rules. Therefore the more rules we create, as long as we rigidly enforce them, the greater the amount of morality.

Watanabe: Uhh, what kind of rules do you have in mind, Saito?

Saito: Any arbitrary rule will do. How about this? Whenever a student speaks English in response to a teacher’s question they have to stand and move one away from their desks to the right, starting with a lateral step of 80 to 100 centimeter’s length. This will demonstrate respect for others, particularly those who create arbitrary rules and have the ability to punish those who violate them.

Watanabe: Saito, I think you are talking about some artificially imposed idea of 'manners'. Morality means something more than that.

Hayashi: That’s right Saito, where is love of country in your proposal?

Watanabe: Love of country? What’s the connection? Can you elaborate, Hayashi?

Hayashi: Come on, Watanabe sensei! Are you really Japanese? Morality is basically patriotism. Patriotism demonstrates care for others- as long as they are our fellow countrymen that is. Love of nation leads to moral acts.

Watanabe: Such as?

Hayashi: Well in terms of English teaching it means helping our students explain Japanese culture, the Japanese way of thinking, and Japan’s positions to foreigners so that they will agree and come to appreciate the beauty of our country. (Eyes well up with tears). I can think of nothing more moral than sacrificing the fun part on their homestays for the betterment of Japan.

Watanabe: Ummm, I’m not sure there is a single Japanese way of thinking or a set ‘Japanese position’ on most issues or that students should be fodder for national propaganda.

Kobayashi: I think you are all missing the point. Morality means respect for life. Students have to learn that life is precious.

Watanabe: And how do you intend to teach that, Kobayashi?

Kobayashi: Well, we tell them in our classes that life is precious and that we must respect it in all forms. (Silence)

Watanabe: And this will be achieved by just telling them that this is so?

Kobayashi: Well, I’ll tell them to say it in English. I also think it’s important to remember that each person has his or her own morals. Who’s to say who’s right and wrong?
(Long, awkward silence)

Saito: We are. We’re the teachers.

Watanabe: Kobayashi, I know you mean well but I don’t think that really helps us in our current situation. After all, some students recognize no moral authority at all and many simply do not understand the nature of the social contract, how to interact in society.

Yamamoto: Ladies and gentlemen, you are all avoiding the inevitable. Morality is connected to grammar. Proper grammar leads to greater morality. Look at our own language. Back when everyone said “taberareru”, the correct form, we lived a peaceful co-existence in Japan based on respect for our fellow man… and syntax.

Saito: Hear, hear!

Yamamoto: But now kids, and even (shudders) some adults say (gulps) “tabereru”. And with this increase in sloppy grammar it is no coincidence that we see a rise in drug usage and threatening hairstyles. In fact I was talking about this just the other day with the girls at the Pink Thrill club. They all agreed that morals loosen when prepositions do. Or at least I think that’s what I said. I’d had a few too many that night. (Takes a long drag on his cigarette and blows the smoke across the meeting table).

Nishimura: Well I came of age in late sixties and we had some pretty radical ideas about morality and I think a lot of them are still valid. Morality is something that is imposed by the man, man. So, I call for counter-morality, morality that seeks to destroy the corporate industrial morality that oppresses the human spirit.

Watanabe: More concretely?

Nishimura: Like, I envision Ummagumma-era Pink Floyd jamming in the background while the students stage a sit-in, where they take back the streets from Big Oil, turning it into a people’s street.

Yamamoto: Nishimura, you know what that leads to don’t you? It might start with street protests but it ends with uncouth grammatical contractions.

Hayashi: Not to mention interracial marriage.

Watanabe: Nishimura, I’m not sure that’s a viable option in our current situation.

Nishimura: Fascist! Just wait until Narita airport expands into your backyard!

Hayashi: Communist! Nishimura, are you really Japanese?

Saito: Well Watanabe sensei, what do you say? As the senior teacher here and as head of this working group I will gladly submit to your authority on the topic.

Watanabe: Well, I agree that morality is not something that can be imposed from above or taught as a series of discrete facts. When we do that the students are not learning morality they are simply obeying orders to avoid punishment and not really dealing with any moral notions at all. In fact, I believe it retards their moral development. Confusing morals with arbitrarily chosen manners or rules, or conflating it with patriotism, is just a form of bullying, or in the latter case, is just chauvinism masquerading as ethics. Morality implies that the individual acts from a consistent, principle-based ethical foundation and is not purely driven by self-interest, momentary caprice, or simply by acceding to authority.
For moral development, young people have to engage human nature, understand complex relationships, decision-making and its consequences and have to actively engage these. English case or situational examples exposing them to moral dilemmas in complex characters and situations and asking for descriptions, explanations, opinions and so on might help them to reflect on the notion of right and wrong at a deeper level and thereby provide a strong foundation for moral principles. By presenting such issues in English and having our students deal with them productively, perhaps our students can not only further their English skills but become engaged at a deeper cognitive level too.

Saito: Whatever you say, Watanabe sensei.

Nishimura: Lackey!

March 25, 2010

ESS- How Tomoyuki lost his groove

(The following is a bit o’ fiction based on a series of real incidents, sewn together with a bit of -ahem- artistic license. The way in which peoples’ good intentions get misinterpreted and misdirected in a foreign language, and ultimately leads to tension and frustration, is an interesting topic for me)

There it was near the bottom of the list of clubs. ESS- English Speaking Society. Tomoyuki liked the sound of that. It had an air of sophistication and worldliness about it. Coming from a small provincial town Tomoyuki couldn’t really think of himself as a man of ‘society’, especially since until this April he had focused almost solely on the university entrance exam. But now, having entered a prestigious university in a bigger city he felt eager to shake off his provincialism and perhaps joining ESS was the way to start.

Ryota, the only other student from his high school to have entered the same university, tried to convince Tomoyuki to join him in the tennis club. “The seniors seem cool, there are lots of social events, and there are some freshman hotties who are managers”. But while Ryota was more of a sports and party guy, Tomoyuki yearned to be erudite and sophisticated. And joining ESS at the university was his first-stage ticket.

Although he knew that his high school English classes had not really been practical, despite Fukushima sensei’s attempts to give them life and relevance and the occasional visit from an ALT from Kenya (to whom Tomoyuki was one of the few to listen with rapt attention and respond to), he had scored well on the exams and felt that he had a better overall grasp of English than most of his classmates, who seemed to only be able to produce individual words or set phrases.

He arrived several minutes early for the first ESS meeting, eager to show his interest. A few students were already there, one or two faces he recognized as other freshmen from orientation, plus a sprinkling of those who were clearly seniors. He nodded at the few familiar faces but kept his head down. One older guy had a notably casual, almost arrogant, air about him. Legs stretched out forward, crossed at the ankles, a little too relaxed.

They’ll probably ask me to introduce myself in English, Tomoyuki thought, and started practicing the mantra in his head. Just as his brain was weighing up "come from" vs. "came from" he heard English chatter coming down the hall towards the ESS room.

The foreign teacher who led ESS, was Goertzen. Tomoyuki remembered the name from the class schedule distributed just the day before. He assumed Goertzen would start by introducing himself and welcoming everyone in English but instead Goertzen strode in chatting amiably in English with a female student as if they were on a private date. Somehow, that cavalier approach made Tomoyuki feel uneasy, as did the fact that the girl crossed her legs when she sat down.

But wait a second, he thought, the girl is good. I bet she’s a returnee- that’s why she’s so fluent. He heard her call the teacher "Dave". OK, Tomoyuki thought, foreigners are usually rather informal with each other, but this seemed to be overly familiar to him. It was almost as if the girl was saying, “I’m not one of you, I’m an English speaker”. OK, maybe you’re just feeling jealous because she’s fluent, he thought. After all, wouldn’t you like to be able to communicate in English with that degree of confidence and control?

Goertzen began. “Today Kanako, a fourth year student, will lead us. But feel free to speak at anytime. And relax!”

Relax, on my first day, yeah right! How long has this guy been in Japan? Then Kanako began to speak, just a little faster than Tomoyuki could follow comfortably, her chirpy banter filled with "yeahs" and "wannas". OK- tone it down already Ms. Returnee he thought, and then realized he hadn’t been paying much attention to what she was saying.

As fate would have it, she called on him first. A self-introduction is natural at this point, he thought. “My name is Sakai Tomoyuki, Tomoyuki Sakai” he blurted out, correcting the name order to suit the English style. “Sorry, what was that”? Goertzen butted in. What was what? Tomoyuki’s mind raced. “It’s my name”, he said. What did you think it was? “Tomoyuki Sakai” Kanako concluded with an air of finality, and fixed him with a look that was either of encouragement and compassion or condescension and pity. Tomoyuki assumed it was the latter.

Just as he was about to continue, Kanako asked him something else, ending in the word ‘from’. What? He wanted to check what her question had been. “My hometown?” he asked, but realized that his intonation was flat and that it had come out like a statement instead: “Where are you from?” “My hometown!” Duh!

He wanted to smack himself in the head. Kanako flashed him that look of pity again. A few other students shifted uncomfortably. Goertzen spoke up. “Well of course you come from your hometown. We all do. But where is your hometown?” There were a few chuckles, especially from Mr. Casual. Goertzen did nothing to discourage them. Tomoyuki felt his cheeks burning and answered, but in his lingering embarrassment the discussion that followed completely eluded him.

When he re-focused, the topic had changed and Goertzen was now saying something about “…six years of high school English …you can’t speak English yet.” Tomoyuki was angered by this. Why don’t we speak English?! Because this is Japan! Are we expected to suddenly change our national language after high school? Was Goertzen one of those arrogant foreigners who thought that Japanese people were somehow obligated to speak English, and who thought that people who didn’t speak English well were less than himself? Tomoyuki didn’t think of himself as being particularly nationalistic but now he felt that part of himself burning and thought he might redeem his earlier awkwardness by volunteering an answer to this question. Foreigners speak directly, he thought, so I will too.

“Because here is Japan!” he blurted out, inadvertently pointing to his nose. “I know this is Japan.” Goertzen looked a bit exasperated. “I just wanted to know how and why the English system here has failed the students!” Who said I failed English? Tomoyuki thought. I actually had one of the highest English scores in my high school! Was this arrogant gaijin already judging him?

“Ba chew wanna get better at English, yeah?” Kanako chimed in. “Yes. I want to be”, he responded. Then he realized that English verbs usually require objects. “It”, he added awkwardly several seconds later. “I want to be… it”. He saw Mr. Casual sigh and ostentatiously check his cell phone.

Tomoyuki wanted to smack himself again. Every twelve-year old in Japan can say, “I want to become good at English” and here he had messed up even this, the simplest of English sentences. He felt his cheeks burning again, kept his head down, and checked his watch.

Tomoyuki ran into Ryota in the passageway later that day. “How was that English thing you went to” “OK, I guess” “Any hotties? “I didn’t notice” “There’s still room for freshmen in the tennis club!” Tennis sounded good to Tomoyuki.

After the ESS meeting, Goertzen was chatting with Kanako in his office. “I’m not sure why that Tomoyuki guy came to ESS today. He didn’t seem interested in English and was even a little hostile. And he can’t speak it at all although I suppose that ESS can help him get a bit better”

“Well he’s a small town boy,” Kanako responded, “I tried to be nice and help him but he just seemed, well, awkward. He doesn’t know how to interact with people like us. Sometimes I pity people like that”.

About March 2010

This page contains all entries posted to The Uni-Files in March 2010. They are listed from oldest to newest.

February 2010 is the previous archive.

April 2010 is the next archive.

Many more can be found on the main index page or by looking through the archives.

Powered by
Movable Type 3.35