qwertyuiopasdfghjklzxcvbnmqwertyui

opasdfghjklzxcvbnmqwertyuiopasdfgh
jklzxcvbnmqwertyuiopasdfghjklzxcvb
nmqwertyuiopasdfghjklzxcvbnmqwer
Standard Setting: The Canadian
tyuiopasdfghjklzxcvbnmqwertyuiopas
Academic English Language
dfghjklzxcvbnmqwertyuiopasdfghjklzx
(CAEL) Assessment and the
cvbnmqwertyuiopasdfghjklzxcvbnmq
International English Language
Competency Assessment
wertyuiopasdfghjklzxcvbnmqwertyuio
(IELCA)
pasdfghjklzxcvbnmqwertyuiopasdfghj
klzxcvbnmqwertyuiopasdfghjklzxcvbn
Summary Report and
mqwertyuiopasdfghjklzxcvbnmqwerty
Recommendations
uiopasdfghjklzxcvbnmqwertyuiopasdf
ghjklzxcvbnmqwertyuiopasdfghjklzxc
Submitted by:

vbnmqwertyuiopasdfghjklzxcvbnmrty
Wendy Fraser, Consultant,

uiopasdfghjklzxcvbnmqwertyuiopasdf
Testing Manager, Centre for English Language Assessment and Support
(CELAS), Carleton University

ghjklzxcvbnmqwertyuiopasdfghjklzxc
Angel Arias, Consultant
PhD Student, Faculty of Education, Montreal University

vbnmqertyuioasdfghjklzxcvbnmqwery
uiopasdfghjklzxcvbnmqwertyuiopasdf
 BACKGROUND

Policy makers must be aware of the considerations that drive the assessment system.
They should be aware of what the assessment and the cut scores are intended to
accomplish. They must be aware of the effects of the operational cut scores on
students, faculties, schools and other stakeholders. (Zieky & Perie, 2004, p. 7)

The purpose of this study was to establish equivalencies between the International English
Proficiency Test (IELCA) and the Canadian Academic English Language (CAEL) Assessment
(i.e., in reading, listening, speaking and writing). The study responded to the need for formal
standard setting (Cizek, 2001; Cizek & Bunch, 2007) by evaluating the relationship between the
IELCA and the CAEL Assessment. The CAEL is an English proficiency test used across Canada
by Canadian institutions for admission and accreditation purposes. According to Learning
Resource Network (LRN), the IELCA also operationalizes the construct of academic English.

At the time of the study, no formal standard setting research had been conducted in relation to
Canadian standards, and, as Cizek and Bunch (2007) point out, “…if categorical decisions must
be made, they will be fairer, wiser, more open, more valid, more efficient, and more defensible
when they utilize established, systematic processes that result in cut-scores that are based on
non-arbitrary, explicit criteria” (p. 8). Thus, the purpose of the present study was to relate IELCA
test item/task difficulty (in reading and listening) and test performance (in speaking and writing)
to criteria defined by the CAEL Assessment in order to make recommendations on test
equivalencies.
Seventeen panelists were recruited to participate in the standard setting study. This report
summarizes the methods, procedures, data collection, analysis and findings leading to the
proposed recommendations.

OVERVIEW OF THE IELCA AND THE CAEL ASSESSMENT

THE IELCA is intended to assess the English language skills of students, who want to study or work in
an English language environment. The IELCA assesses four aspects of English language that include
reading, listening, speaking, and writing. The IELCA is offered in two types of modules academic and
general training. The former module assesses the students' ability to study in undergraduate or
postgraduate level in any educational institutions, colleges, and universities of English speaking countries.
The latter module assesses the basic essential English language skills in a wide range of frameworks of
the social and educational environment. The general training component is also appropriate for people
who want to study in secondary education, immigration purposes, work experience and other training
purposes. Both the academic and general training modules assess four components of the language ability
i.e., reading, listening, speaking and writing.

Table 1. IELCA by skill focus, item/task type and time

SKILL FOCUS ITEM/TASK TYPE TIME

Reading Multiple-choice (single answer) 80 minutes
True/False
Listening Multiple-choice (single answer) 30 minutes
Fill in the blanks
Speaking Face to face interview 12-15 minutes
General questions about home, family,
hobbies, work, and studies
Question short answer
Writing Task 1: description of a chart, graph, 60 minutes
table or diagram
Task 2: an essay reflecting candidate’s
own opinions, views, arguments,
problems or analysis on a specific
writing prompt

DESCRIPTION AND SPECIFICATIONS OF THE FOUR LANGUAGE SKILLS AS

MEASURED BY IELCA

Reading
In the Reading Test, candidates are asked a range of questions to assess their reading ability. The main
components of the questions include a central theme, general ideas, skim reading, detail reading, main
arguments, opinions, views, attitude and substance of the text. The reading passage has three long texts
between 700 to 800 words. These are factual, critical, narrative, discursive and analytical. The passages
are extracted from newspapers, journals and books. The passage is easy to get to a non-specialist audience
which is appropriate for enrolling to undergraduate, postgraduate course, immigration purposes and
professional accreditation.

Listening
the listening section assesses the ability to understand the central ideas, factual information, purpose,
viewpoint and development of the ideas and information. Candidates need to listen carefully the main
ideas and answer multiple choice questions within a set time frame. The audio recordings feature speakers
from various English-speaking countries; they have different accents and dialects reflecting their
international environment.

Speaking
The speaking test is a face to face oral interview with an examiner. This test is recorded. There are three
parts in this test. In the first part, candidates are asked general questions about themselves, their home,
family, hobbies, work and study. In the second part, candidates are given a specific topic in which they
have to speak about 2 minutes like speaking about a topic in the classroom. There is one minute time to
prepare a note prior to speaking on the topic. In the last part of the speaking test, candidates are asked
questions about the topic related to part 2. These are short questions and the candidates have the
opportunity to clarify and add some more information on the topic.

Writing
The writing section of the test has two tasks of 120 words and 220 words. The candidates are assessed on
the basis of their ability to write in a clear manner including appropriate language, content, vocabulary
and analytical ideas.

Academic Writing
In Academic Writing, candidates have to write a description of a chart, graph, table or diagram. In task 1
candidates are required to write clearly and in their own words. Task 2 is an essay and candidate have to
include their own opinions, views, arguments, problems or analysis. It is expected that the tone of the
writing responds to formal language maxims as it is required in academia.

General Writing
The General Training module is also composed of two tasks. In the first task candidates need to write a
letter about either an explanation of a given situation or writing a letter requesting information. The letter
may be in any format: formal, informal or personal. It should be written in at least 120 words. In the
second task, candidates are asked to write an essay in any form in at least 220 words.

THE CAEL ASSESSMENT is a criterion-referenced, topic-based performance test, comprised

of an integrated set of language activities. The language tasks and activities in the CAEL Assessment
are systematically sampled from those that are commonly undertaken within the university academic
community. The content for the tasks on the CAEL Assessment is drawn from introductory
university courses at times when professors are introducing new topics to their students with the
expectation that the students know little or nothing about the content. The test is comprised of
representative tasks and performances that characterize academic study (see Table 2, below), for
example:

• speaking about academic experience, information, or understanding,

• listening to, taking notes, and transferring or applying information on a topic introduced or
extended by an academic lecture,
• reading and selectively applying information from academic articles and texts about a topic
introduced or extended by a lecture, and
• incorporating what has been learned from the lecture and readings in writing a formal,
academic response to an academic task.

Table 2. CAEL Assessment by skill focus, item/task type and time

SKILL ITEM/TASK TYPE TIME

FOCUS
Reading Multiple-choice (single answer)
Multiple-choice (multiple answers)
Short answer
Fill Charts and tables 55 minutes
 Fill in the blanks
Label diagrams
Listening Summarize spoken text 25 minutes
Multiple-choice (single answer)
Multiple-choice (multiple answers)
Fill in the blanks
Short answer response
Fill in tables or charts/information transfer
Take notes on spoken text
Extended response
Speaking Personal Introduction
Lecture re-tell
Question short answer 25 minutes
Text read aloud
Impromptu mini presentation
Writing Write essay based on information in reading
and listening lecture 45 minutes

Similarities between the two tests are important to establish equivalencies and relating test scores
on the IELCA to the CAEL Assessment. Both tests operationalize a construct of English for
academic purposes (EAP) at the level of undergraduate/first-year university/college admission.
Both tests feature task topics pertinent to academic genre. Both tests report proficiency on the
four language skills and all speaking and writing performances are marked by human raters

There are, however, important differences between the two tests. Although both CAEL and
IELCA tasks are academic related topics, they differ in that IELCA items/tasks sample from a
wide range of topics and contexts; CAEL items/tasks are fully integrated within a single topic.
CAEL test takers are provided with the essay prompt for the writing sub-test at the beginning of
the test, introduced to the topic through the readings (two-three) with items/tasks that are used
for the reading sub-test scores. The listening sub-test consists of an extended lecture on the same
topic with items/tasks that are used for the listening sub-test scores. Test takers use the
information from the readings and lecture in responding to the prompt in the writing sub-test at
the end of the test.

IELCA reading and listening texts and tasks are shorter with mainly multiple choice or fill-in the
blanks responses; CAEL reading and listening texts are longer and involve extended reading and
listening with such tasks such as written summaries, information transfer, and short answer
responses.

All performance on the CAEL Assessment (within sub-skills and overall score) is defined by
criterion-referenced band scores ranging from 10 to 90 (see Appendix 3). These criteria served
as performance level descriptions (Cizek & Bunch, 2007, p. 46) for categorizing IELCA
performance/proficiency and/or item/task difficulty in the standard setting sessions.
Although CAEL Assessment proficiency standards are set internally by tertiary institutions in
relation to their own programs, in general most institutions in Canada require a band 70 on
the CAEL Assessment for admission; a number of institutions accept band 60. Only two
institutions in Canada require proficiency above band 70 for admission to their first-year,
undergraduate programs.

COMMITTEE SELECTION FOR THE PTE ACADEMIC STANDARDS

SETTING PANEL

When planning standard settings it is of paramount importance to ensure that the panel members
understand the procedures being used to set the standards and have expertise in the field and
context where tests are implemented. In fact, “participants in the standard-setting process are
critical to the success of the endeavor and are a source of variability of standard setting results”
(Cizek & Bunch, 2007, p. 49).

As a result, (N=17) panelists were recruited for the standard setting. They were strategically
selected to provide a representative sample of adequate expertise in language proficiency
required for university studies and included panelist drawn from the groups below.

1) Certified raters, (CAEL, CELPIP, IELTS, CLB Exit Assessment)

1) EAP specialists, with extensive teaching experience at the academic level.

2) Undergraduate students, majoring in Applied Linguistics and enrolled in the fourth year
undergraduate course in language testing and assessment (ALDS 4201) at Carleton University.

3) Graduate Students in Applied Linguistics, with expertise in language testing and

assessment, teaching in universities in Canada or abroad, who were specializing in language
assessment at the MA or PhD level.

4) Professional test developers/researchers, continually conducting research on test construct

and validation at Carleton University.

Overview of panelists’ characteristics

Gender
Male: 17.64%
Female: 82.36%

Years of teaching EAP/ESL EFL (abroad) Countries

Canada
0 5 (29.41%)

1-5 5 (29.41%) yes no

yes no
yes no
yes no China and Mexico
yes yes

6-10 4 (23.53%) yes no

yes no
yes no
yes no

11-15 3 (17.65%) yes yes Dominican Republic

yes yes Indonesia and Singapore
yes yes Ecuador, Peru, Qatar

Language Testing Experience

Rating of high stakes tests 6 (CAEL, CELPIP, IELTS)

Test Development 9 (CAEL, IELTS)

Test Research 4 (CAEL, CELPIP, CLB)

Standard Setting Experience 4 (PTE Academic; Canadian Language

Benchmarks; CELPIP-G)

Other relevant experience in contexts where cut-points for language proficiency have
played a role

8 panelists (CAEL, TOEFL iBT, Pearson Academic, CLB)

In summary, the standard setting panel was composed of participants who had expertise in
language testing research, ESL teaching, and participants who were relevant to the context of the
standard setting study (e.g., students enrolled in an forth year language testing and assessment
undergraduate course (ALDS 4201) at Carleton University. In all, the standard setting panel
represented key expertise in the evaluation of English language proficiency relevant to the main
selection context of both the IELCA and the CAEL Assessment.