“The challenge actually made the experience more fun.”
Dr. Nathan Carr distinctly remembers the secure test development environment known as the Bunker
in Baku, Azerbaijan, where he was sequestered for five days with a small group of English language test
writers from that country’s State Examination Center (SEC). His 2019 English Language Specialist assignment: Assist in revising the English language portion of that country’s national 9th and 11th grade school-leaving examinations. Treated with all the stealth of a top-secret mission, life in the Bunker – ironically located on the top floor of an office building– included thorough bag searches to ensure no electronics or outside teaching materials were on the premises, sealed light switches and outlets to prevent possible installation of cameras or microphones, and cell phone jamming equipment to disrupt potential outside communication.
Once “sealed in,” as Carr puts it, he and the other test writers were there for the duration – working,
eating, and sleeping – with doors to the outside opening only after tests had been securely removed
from the building and distributed nationwide, and students had taken the exam that Sunday. “Except for
the TV in the dining area that showed Azerbaijani shows, we were completely cut off from the outside
world,” says Carr, the first foreign educator allowed in the Bunker.
Carr did not enter the Bunker immediately upon arrival in Azerbaijan. First, he met with SEC staff
members for an overview of what they did and required, the initial step in what would be a three-phase
project to revamp and develop a complete set of national test specifications. While impressed by the
technical quality of their statistical analyses of SEC test data and rigorous scoring procedures for test
items, he also recognized a lack of consistent and coherent test parameters. “There were some
guidelines, but nothing else. It was all very random,” he says. Indeed, uniformity was needed – for
example, how many main idea, detail, and inferences items should be included with each reading
passage? – but that would be the focus of later project phases. Next on the schedule was the Bunker.
Test writers for all major subjects, not just English language, were housed in the Bunker, but Carr worked
exclusively with two English language item writers, tweaking or eliminating old questions and adding
new ones. While that would have been challenge enough in such a short period of time, everything had to be accomplished without the use of computers or reference books. “It was an interesting exercise to see what I remembered without the ability to look anything up,” he recalls. “The challenge actually made the experience more fun.” In addition, Carr’s personal notes for future reference and all test changes had to be handwritten – typed changes were done by staff in an off-limits room in the building.
Despite the absence of technology and contact with the outside world, as well as 12-hour workdays, Carr
found his time in the Bunker “illuminating,” helping him clarify ideas for how to better train item writers
in the future. “I realized that what I take for granted as common sense is not always obvious to those
writing or revising items,” he says. “Our work in the Bunker helped me anticipate situations that could
come up during training sessions.” And while he admits that living under such tight
security was intense, he appreciates the SEC’s focus on maintaining the integrity of the test and testing
process. “The objective was to make sure everything was very straightforward and on the up-and-up,
with no hint of impropriety.”
Over the next 10 months, Carr returned to Azerbaijan to complete the second and third phases of the
project. It was then that he was able to put what he had learned in the Bunker into action, working with
item writers on skill-specific questions and with item reviewers on best practices for critiquing items. By
the end of the project, Carr had also helped them develop a coherent, unified test specifications
document for the school leaving exams. “My overall goal was to train them to improve the usefulness of
these tests, so they were more effective measures of their students’ ability to communicate in English,”
says Carr.
“That’s one of the reasons I feel so fortunate to be part of this program – it’s both professionally and life enriching.”
Carr undertook a similar three-phase Specialist assignment in Vietnam in 2016 and 2017 with the Center
for Language Testing and Assessment at the University of Languages and International Studies, part of
Vietnam National University. Here, objectives were highly specific. The country already had an
established national assessment for English language proficiency – the Vietnam Standardized Test of
English Proficiency (VSTEP) – but test developers wanted particular help with such assessment
components as calculating reliability for speaking and listening, analyzing test data, estimating task
difficulty, interpreting results, and setting cut scores. According to Carr, “Ultimately their goal was to improve the test, so there would be positive washback on the curriculum. They were well aware of the impact an assessment has on teaching and learning — if it’s not on the test, it won’t become part of the curriculum. They had a lot of foresight: improving the English language test leads to an improved curriculum, and that leads to improved English language proficiency, which gives the country a competitive advantage.”
As often happens with Specialists during repeat assignments in a country, a common theme beyond the
task at hand emerged for Carr, namely, a growing admiration for Vietnamese food and coffee. “I like
spicy dishes and strong coffee, and Vietnam excels in both,” says Carr. His Vietnamese colleagues
regularly took him out to restaurants, where he sampled Vietnamese wraps, DIY barbeque, egg coffee,
and more. Even during the workday, lunch in the cafeteria often took center stage. If Carr attempted to
skip the midday meal, his Vietnamese coworkers would not allow it. “The young lecturers insisted on
mothering me, and I would gladly eat whatever ‘light meal’ they placed before me – typically a lunch
tray laden with a wide variety of Vietnamese dishes,” he recalls. During those hour-plus lunches, Carr
and his coworkers would bond over life’s ordinary moments. “‘How bad is the traffic?’ ‘Do your children
do that?’ ‘Why is it so hard to keep electronics charged?’ All those common international threads of
conversation.” Carr recollects. “That’s one of the reasons I feel so fortunate to be part of this program –
it’s both professionally and life enriching.”
Nathan Carr is Professor of TESOL at California State University, Fullerton. He was born and raised in Oklahoma, earned a B.A. in Russian and Russian Studies at the University of Oklahoma, and started his career in TESOL by teaching English in Taiwan for three years. He earned an M.Ed. in TESOL at Oklahoma City University and a Ph.D. in Applied Linguistics at UCLA with an emphasis in language assessment. His research interests are eclectic but generally focus on language assessment, particularly automated scoring in web-based testing, validation, and teacher training in language assessment. He regularly teaches courses on assessment, curriculum development, and teaching methodology. He has been a Fulbright Scholar in Kazakhstan, and an English Language Specialist in Azerbaijan, Indonesia, and Vietnam.