JB: The "global economy" relies more and more on back-and-forth spoken communication, most often in English. We realized that today's corporations, universities, schools and governments all need to know that their employees or students can function well in an English-speaking environment. At the same time, we noticed that there was no service that provided a fast and sure way to measure a person's spoken English ability. Considering this and our combined areas of expertise, the idea of a system like PhonePass was very natural.

PhonePass scores are based on the exact words used in the spoken responses, as well as the pace, fluency, and pronunciation of those words in phrases and sentences. PhonePass technology uses a speech recognition system that is specially designed to handle different rhythms and varied pronunciations commonly used by native speakers of English and people who speak English as a second language. In addition to recognizing the words uttered, the system also aligns the speech signal, i.e., it locates that part of signal that contains relevant segments, syllables, and words.


Base measures are then derived from the linguistic units (segments, syllables, words), based on statistical models of native speakers. The base measures are combined into five diagnostic subscores using advanced statistical modeling techniques. The diagnostic subscores are: Listening Vocabulary, Repeat Accuracy, Pronunciation, Reading Fluency, and Repeat Fluency. An overall summary grade is calculated as a weighted combination of the five diagnostic subscores, re-scaled and combined so that they optimally predict the human manner-of-speaking judgments.

It was extremely difficult for us to create a system that could exceed the abilities of a human rater. There were significant hurdles in every area - linguistics, speech recognition and statistical modeling.

It took us several years of research and an inventive flurry before we could overcome the difficulties we encountered. Mainly, we needed a clear scientific model of how people speak, and we needed data from thousands of native and non-native speakers to fill in and validate those models.

During the development of the PhonePass SET-10 test, human graders assigned over 26,000 scores to thousands of utterances from thousands of different examinees. Item response analysis of the human grader scores indicates that human ratings produce relatively consistent grades for fluency, pronunciation, and conversational skill, with inter-grader reliabilities between 0.82 and 0.86. When we combined several human judgements of the overall conversational skill based on examinee responses to open questions, the resulting score had a reliability of 0.93.

Well, because of the immediate and accurate results, PhonePass Spoken English Tests can be used anywhere, at anytime, to monitor progress and guide instruction. Uses are everywhere, including the evaluation of potential international teaching assistants, placement of students into appropriate English courses, benchmarking progress within English courses, or as an exit exam. The SET tests are less expensive than TOEFL, TOEIC, TSE, and SPEAK, and have demonstrated high correlation to these other English language tests.

The benefits to businesses are similar - a fast, flexible and reliable way to know a person's English skill level - while the applications are slightly different. Corporations can use the SET Spoken English Tests to efficiently screen potential job candidates, candidates for international duties, and candidates for upper level management. The benefits are clear for industries that require high English proficiency, such as the call center, hospitality, and health care industries. SET tests can help employers in these industries identify qualified candidates early in the process and ensure that they have the right employees in those positions that require real use of spoken English.

We are pleased with the results of a recent successful trial done at Waseda University, where SET tests were implemented for measuring improvements in spoken English proficiency of students in the School of Law. The University plans to expand usage of the tests to other undergraduate schools for measuring proficiency and verifying effectiveness of English courses. Another example of SET tests in use was at last year's World Cup. In one week, the Korean World Cup Committee administered over 15,000 PhonePass tests to volunteers. Because they could take the test over the phone, the volunteers didn't need to travel to a testing center. Since results were typically available within minutes after the tests were taken, the Korean Organizing Committee could quickly determine the appropriate position for each volunteer who had taken the test.

I look forward to seeing you and your readers next time I'm in Japan.