|Simulations: A Tool for Testing "Virtual Reality" in the Language Classroom|
This article appeared in similar form in the On JALT "95: Curriculum and Evaluation. (1996). Tokyo: The Japan Association for Language Teaching, pp. 313-317. Copyright @ Randall S. Davis.
One of the foremost responsibilities of language teachers should be to create and implement speaking assessment methods that actually measure oral language proficiency in action. To do this, many researchers and practitioners alike have suggested the power of simulations to transpose the normal classroom into an authentic setting where language skills can be evaluated under more realistic conditions. I will outline the rationale and unique benefits of simulations and describe one such language assessment model designed at a business college in Tokyo, Japan, to test business English. I will then report on the results of a questionnaire administered to the participants to evaluate (a) the usefulness of the simulation and (b) the assessment techniques in providing diagnostic feedback for improvement, concluding with a short discussion of results of this research.
Yet, while many skills can be assessed using pencil-and-paper tests, oral proficiency "is widely regarded as the most challenging of all language exams to prepare, administer, and score" (Madsen, 1983, p. 147). Creating standard criteria of assessment, solving problems of administration, designing test items that resemble tasks in normal language use, and testing the complex and interlocking nature of language and skills in content-based courses are only a few of the logistic hurdles teachers must surmount in creating a sound testing instrument (Hughes, 1989; Littlejohn, 1990; McClean, 1995).
In Japan, the result has been that many teachers have resigned themselves to giving written tests instead; however, the concerns of creating a more enriched communicative environment for students and then assessing their language proficiency have led some to shift their attention to the use of simulations as a means of testing the language skill in action.
The most common view of simulations is that they provide a way of creating a rich communicative environment (a representation of reality) where students actively become a part of some real-world system and function according to predetermined roles as members of that group. More important, however, is the notion that a simulation becomes reality and the "feeling of representivity fades" (Crookall & Oxford, 1990, p. 15), so much so that the world outside the simulation becomes, paradoxically, imaginary (see Black, 1995; Jones, 1982, 1985, 1987; Taylor & Walford, 1978, for a more detailed explanation of the mechanics of simulations).
The innate benefits of simulations include: (a) fulfill students' need for realism---a desire to "relate to life 'out there' beyond the classroom's box-like walls" (McArthur, 1983, p. 101); (b) increase student (and teacher) motivation, especially for those in EFL situations who might see English as a deferred need at best (Jones, 1982; Stern, 1980); (c) dismantle the normal teacher-student relationship so that students take control of their own destiny within the simulation, leading towards "declassrooming" the classroom (Sharrock & Watson, 1985); (d) help the learner confront and identify with the target culture (Oxford & Crookall, 1990); (e) reduce anxiety levels which is essential to language development (Dulay, Burt, & Krashen, 1982; Krashen, 1982); and (f) allow teachers to monitor the participants progress unobtrusively.
As part of this movement, Littlejohn (1990, p. 125) suggests that "the use of simulations as a testing device is . . . an important development since it should be possible to replicate the situations in which learners will have to use the language." He also feels that this kind of replication "allows us to view not only the language product but also the process by which that language emerged" (p. 125). Whereas standardized methods give us insight on how the student might do in a real setting, "simulations will show us how the student actually performs" (Littlejohn, p. 128; italics, the author's).
Overview. To bridge this gap between simulations and testing, I have developed a task-based model at Tokyo Foreign Language Business Academy as part of an ongoing research project to evaluate the effects of simulation techniques in ESP classes, taking in account the need and desire to measure language proficiency (in this case, business English) at the intermediate level.
Design. Students are required to participate in a business simulation called "Let's Do Business" as part of the final evaluation near the end of the second year. This simulation deals with the rise of a travel agency called Fly Company from its inception through the research and development of a new sales promotion over a six-month period (which actually takes place during four consecutive class periods of 90 minutes each). I allocate each student the role of office manager, sales representative, or office clerk, and they are required to put into full use the language, behavioral, and business skills they have acquired during the past two years. In this case, I divide students into four branch offices of the company that are supposedly located in cities throughout Japan by partitioning the room into four sections, each equipped with a computer and printer, table and chairs, white board, phone, calculator, and access to a fax machine.
I make elaborate preparations to fulfill, what Jones (1982, pp. 4-5) terms, the three essential elements of simulations: (a) Reality of function (participants are assigned roles and are told they must fully accept them both mentally and behaviorally as if they were actually those people); (b) simulated environment (a realistic setting constructed to enhance role-acceptance by utilizing a variety of realia, e.g., in this case, specially printed business cards, time cards, name tags, letterhead, technical support including computers and a fax machine, and memorandums); and (c) structure (the whole action is built around a set of problems or tasks---not invented by the participants but rather evolve as the action progresses).
The groups are asked in a memorandum from the company president, William Johnson, to devise a new marketing strategy for domestic travel tours in Japan based on the results of a comprehensive survey of Japanese consumers' tastes and preferences. After analyzing the data, participants at each branch discuss their target market, decide how they are going to promote their services (e.g., television or radio spot, newspaper advertisement, direct mail, fliers, etc.), communicate their ideas and progress with the other branches by fax, phone, or mail, and then write and submit a proposal to the president.
In the end, our main goal is to provide some measure of both the process (how they approached the task orally in English by reviewing, organizing and weighing alternatives, deliberating over the information available to them, etc.) and the product (the proposal they draft demonstrating their English writing, computer, and reading skills).
The most challenging step is to evaluate the process. Three techniques that work well in tandem include a student-generated checklist, video or audio recordings, and a debriefing session.
1. Job appraisal checklist. One useful assessment tool I use is a student-created job appraisal checklist (see Appendix A, for one example) that, in reality, serves as a prop used by employees within this simulation as a way of measuring performance. Participants fill out this checklist based on whether they feel they fulfilled the duties as outlined in their job descriptions. The advantages of utilizing such a discovery approach are: (a) it empowers the participants with the know-how to evaluate their strengths and weaknesses without the constant feedback from an external evaluator; (b) its application is not limited to the classroom, but can be used later on the job; and (c) it satisfies the students' belief that their work should be fairly judged based on a system they clearly understand rather than be graded, in one of my student's opinion, "by a subjective scale created at the whims of the teacher."
Because I feel participant-reported responses often lack impartiality, I spend time training students how to be more objective by putting them in charge of writing the checklist as part of the regular coursework and then having them view past students on video engaged in similar business tasks and identifying positive models of the skills they want to acquire. Then, they practice evaluating each other in short role plays that resemble situations found in the simulation. At the same time, I take notes, record my own evaluations, and later discuss how my ratings coincide with those the students wrote down. My feedback at this point reinforces in their minds the validity and reliability of their own marks.
At the close of the simulation, the regional manager asks each participant to complete the job appraisal checklist before a year-end performance interview. The purpose of the interview, they are told, is to review their progress for possible promotion and pay raise in the near future. At this point, the simulation ends.
2. Videotaping or tape recording. Recording simulations can serve as a powerful tool for encouraging self-correction as well as student and/or teacher-initiated feedback. First, I try to position the camera so it will blend in with the surroundings without inhibiting students from assuming their roles in a more natural setting. I make sure the camera has become a regular fixture of the classroom weeks before (or months through repeated use) I carry out the simulation. By that time, students have accepted its presence and are not aware of whether it is rolling or not. Also, because four different meetings are going on simultaneously, I rotate the camera among groups to ensure that everyone appears on the video.
Furthermore, because tape recorders are always easier to come by and require less supervision, I set up a recorder in each office to tape the group's discussions. I connect the machine to a long extension cord and have the play button always on, so that by just plugging in the cord from outside their office, I can activate the recorder without participants conscious of when it is going or not.
3. Debriefing. The ultimate success of this simulation hinges on the efficacy of a wrap-up or debriefing session (together with the self-evaluation checklist and recordings) where students and the controller can openly discuss behaviors, outcomes, general language difficulties, and the contextual appropriateness of their language discourse. Because I, as the controller, do not take part in the simulation, I am able to look in as an observer without inhibiting students from assuming their roles.
Although there are several different approaches to debriefing (see Bullard, 1992), I hold a two-hour session the next class period, giving me time to reflect back on the simulation and organize my comments regarding students' behavioral or linguistic errors that were most apparent---and giving students a needed respite from such an intensive experience. Furthermore, as Bullard puts it, "the teacher has the chance to analyze the errors and to develop strategies for dealing with them at leisure rather than having to operate on the spur of the moment" (p. 64). Pedagogically speaking, this break has allowed me to view or listen to the tapes, record my observations, and prepare follow-up classroom lessons in the form of short role plays to reinforce areas that need improvement.
One simple technique for using the recordings in the debriefing is to write a checklist of listening or observation tasks. For example, I give students a checklist of the expressions studied in class for asking and expressing opinions in business settings, ask the students to watch the video, and check off the ones they hear, or see (in the case of certain non-verbal communication, e.g., gestures, facial expressions, paralanguage, etc.). Then, we come up with a group impression of how well students did.
What do you think about . . . ? What's your opinion on . . . ? If you ask me, . . . In my opinion, . . . You're exactly right. Yeah. That's how I feel! I agree. I don't see it that way. I don't agree. I see what you're saying, but . . .
Figure 1: Observation Task Sheet As the debriefing continues, I ask the participants to look at the remarks they made on the job appraisal form and critique their performance accordingly, checking to see if their own assessments concur with what they view on tape.
Measuring the Product: The Proposal The second part of the evaluation deals with the product: the written proposal. I assign grades by looking at several specific criteria: (a) layout of the proposal (introduction, rationale, design, etc.), (b) mechanics (punctuation, spelling, and capitalization as studied in class), (c), content (organization, depth and breadth of arguments, and presentation of ideas), and (d) language usage (business terminology). I collect these proposals at the end of the simulation, and then score and return them. Each member of the group receives the same grade.
To determine both the effectiveness of the simulation and the value of the assessment tools used as viewed by the participants, I administered a short, written questionnaire comprised of four open-ended questions to 15 students in Japanese (to elicit more detailed comments), and these responses were then translated into English. [Those responses of particular interest have been cited here.]
The first question asked students to compare this simulation with other language activities in their other classes (e.g., dictation, skits, pair work, oral interviews, written tests, etc.). Eleven of the 15 students (S) regarded this technique more productive than other exercises they had experienced before:
S3: It [the simulation] was fun because the students were in control of the business rather than the teacher telling us what we should do next.
S5: It was a useful experience because the parts of the simulation didn't come straight out of a textbook.
S7: This activity combined what we practiced all year and what we will later need on the job.
The second question asked students whether they felt they had ample opportunities within the simulation to practice the skills studied in class:
S2: I like it because the phone conversations were not scripted by the teacher, but were created by the students out of a real need to communicate.
S11: Each thing we did was related to the next, so I had the chance to try many things at once.
S15: It simulated the pressures of the real thing and allowed me to see whether I had mastered my English or not.
The third question focused on whether the skills-assessment methods (checklist, videotaping, debriefing session, and proposals) were helpful in measuring students' abilities and provided enough diagnostic feedback to assist them in seeing their strengths and weaknesses for improvement.
S2: Talking to all the students together at the final meeting was good because I could see that other students had similar concerns and problems in English, and we could learn from each other.The final question dealt with the overall design of the simulation and asked students how it could be improved. Of the 15 students, seven suggested no specific changes. The other eight students recommended modifications in format, timing, role allocation, and formal feedback. Some of these suggestions include:
S5: The evaluation sheet was useful because it helped me learn how to check my own ability.
S9: I enjoyed watching the video of the simulation because I could see myself using English. I always wondered if others could understand what I was saying.
S1: The first day was exciting, but as the simulation continued on over several classes, it lost some of its momentum.
S10: I wish more cultural issues in working with foreign companies would have been introduced.
S15: It would have been nice if there had been some foreign teachers acting as members of the staff to motivate and force us to communicate more in English.
The results of the survey and my own observations have helped me chart a new course using simulations as the cornerstone of our program. One might question the plausibility of carrying out such elaborate simulations, considering the limitations of time and space, for example, while dealing simultaneously with weighty demands of classroom requirements already. Finding myself under the same constraints, I have slowly progressed from simple skits, to detailed role plays, to more involved productions over some time, giving myself time to digest and process this unique method of teaching and testing while gaining converts along the way. . . and the reward has encouraged me to push on.
Whatever the obstacles, the comments in the questionnaire have shown me that once students had tasted the benefits of simulation, their desires to learn improved considerably. Furthermore, the extent to which the students praised our efforts not only reflects how radically different this kind of approach still is in Japan, but how little simulations have permeated into the classroom although they have been the focus of discussion for many years in teacher-training circles. Finally, the students' responses seem to mirror the current state of affairs in many language-teaching settings: traditional methods of assessing oral proficiency do little to prepare the trainee for the realities and demands of life.
Since initiating the use of simulations as a pedagogical learning and testing tool in the classroom, my students and I have found a great sense of fulfillment and satisfaction in taking part in activities that innovative, pragmatic in nature, and fun. What Jones observed several years ago is just as, if not more, significant today: "The time seems to be ripe for extending their [simulations] use . . . particularly in the field of language assessment" (1982, p. 77).
Black, M. C. (1995). Entrepreneurial English: Teaching business English through simulation. English Teaching Forum, 33 (2), 2-9.
Bullard, N. (1992). Briefing and debriefing. In D. Crookall & R. L. Oxford (Eds.), Simulation, gaming and language learning (pp. 55-66). New York: Newbury House.
Crookall, D., & Oxford, R. L. (Eds.). (1990). Simulation, gaming, and language learning. New York: Newbury House. Dulay, H., Burt, M., & Karashen, S. (1982). Language two. New York: Oxford University Press.
Hughes. A. (1989). Testing for language teachers. Cambridge: Cambridge University Press.
Jones, K. (1982). Simulations in language teaching. Cambridge: Cambridge University Press.
Jones, K. (1985). Designing your own simulations. London: Methane.
Jones, K. (1987). Simulations: A handbook for teachers and trainers (2nd ed.). New York: Nichols Publishing.
Krashen, S. D. (1982). Principles and practice in second language acquisition. Oxford: Pergamon.
Littlejohn, A. (1990). Testing: The use of simulation/games as a language testing device. In D. Crookall & R. L. Oxford (Eds.), Simulation, gaming and language learning (pp. 125-133). New York: Newbury House.
Littlewood, W. T. (1981). Communicative language teaching: An introduction. Cambridge: Cambridge University Press.
Madsen, H. (1983). Techniques in testing. New York: Oxford University Press.
McArthur, T. (1983). A foundation course for language teachers. Cambridge: Cambridge University Press.
McClean, J. (1995). Negotiating a spoken-English scheme with Japanese university students. In J. D. Brown & S. O. Yamashita (Eds.), Language testing in Japan (pp. 136-148). Tokyo: The Japan Association for Language Teaching.
Onoda, S. (1995, September 18). Good testing methods a prerequisite for teaching. The Daily Yomiuri, p. 9.
Oxford, R. (Ed.). (1990). Using and learning language through simulation / gaming. Newbury Prk, CA: Sage.
Sharrock, W. W., & Watson, D. R. (1985). Reality construction in L2 simulations. In D. Crookall (Ed.), Simulation applications in L2 education and research. Oxford: Pergamon.
Stern, S. L. (1980). Drama in second language learning from a psycholinguistic perspective. Language Learning, 30, 77-97.
Taylor, J. L., & Walford, R. (1978). Learning and the simulation game. Milton Keynes: Open University Press.
This assessment is based on the list of responsibilities and skills needed as a member of Fly Company. Use the following list to judge your own abilities and write other comments.
3 = Well done 2 = Fair - Needs improvement 1 = Unable to finish the work satisfactorily
1. I can use the computer to write letters/faxes/memos: . . . . 3 2 1
(format, addresses, punctuation, spelling, greetings and closings, envelope format, fax layout, abbreviations, speed, etc.)
2. I am able to answer the phone and take messages in English: . . . . 3 2 1
(answering the phone, asking for additional information, recording message correctly, responding quickly, etc.)
3. I work well with other employees in the office: . . . . 3 2 1
(helping others as a team and eager to do extra work when needed, etc.)
4. I am able to express my opinions clearly on important decisions: . . . . . 3 2 1
(agreeing, disagreeing, persuading, asking questions, etc.) ______________________________________________________________________
5. I complete my assigned work on time: . . . . . . . . 3 2 1
6. I come to work on time: . . . . . . . . . 3 2 1
__________________________ _____________________ _____________________ Employee's Signature Position Date
_________________________ ______________________ _____________________ Employer's Signature Position Date
Copyright © 1998-2006 by Randall S. Davis, All rights reserved.