Testing Writing
Arman Argynbayev
MA in TEFL
Suleyman Demirel University
[Link]@[Link]
Why is writing an important skill?
Is it easy to teach and test? Why (not)?
What kind of operations and functions do we measure
in writing?
e.g. Expressing: writing a complaint letter
Operations / Functions
• Expressing: thanks, requirements, opinion, attitude,
complaints, reasons, etc.
• Directing: ordering, instructing, persuading, advising, warning
• Describing: actions, events, objects, people, processes
• Narration: sequence of events
• Reporting: description, comment, decisions
Texts
• Type: form, letter, message, fax, notice, postcard, recipe,
report, set of instructions, etc.
• Topics: a common theme or depends on the test
specifications.
• Length: depends on the test specifications
Types of writing
• Controlled writing
• Guided writing
Types of writing
• Controlled writing
• Guided writing
Controlled writing
• The outcome is totally predictable
• Highly reliable (either right or wrong)
• Easy and quick to correct even by students
• Lack of original expression
• Time-consuming
• Can be assigned for homework
Controlled writing
• Jigsaw sentences
- Students match sentence halves
A B
Teachers and lecturers get to visit many countries.
Ski enthusiasts profit from long holidays.
Animal lovers often go to beach resorts.
Bird watchers enjoy game park safaris.
Tourists enjoy the countryside.
Hotel owners prefer winter holidays.
Art crews work through holiday periods.
Controlled writing
• Copying with corrections
• Transformation tasks
• Dictation
• Dicto-Comp (аудирование)
• Dictoglos
Controlled writing
• Transformation Tasks
- change the tenses in the paragraph (e.g. present to past)
- change full forms to reduced forms (contractions)
- reducing a passage (deleting unnecessary words and phrases)
- combine two/three sentences into one
- change from active to passive voice
- change questions into statements
Guided writing
Students are given words or pictures to help them write.
• Picture description
• Picture sequence essay
• Writing a summary
• Key word essay
• Replying to letters
• Replying to advertisements
• Newspaper headlines
• Story completion (The first or the last sentence is given)
Direct testing vs. Indirect
• Always use direct testing
• Even professionals cannot produce indirect writing tests
Set as many separate tasks as possible
• Candidates must have several ‘fresh starts’
• A greater reliability and validity are achieved
• Balance practicality and desirability
Test only writing ability, and nothing else
• Do not test creativity, imagination, intelligence, or general
knowledge.
• Poor samples:
1) Write a conversation you have with a friend about the holiday
you plan to have together.
2) Write a speech and describe your country to foreign students.
3) ‘Envy is the sin which most harms the sinner.’ Discuss.
4) Discuss the advantages and disadvantages of being born into a
wealthy family.
Restrict candidates
• Give no choice of tasks
• Tasks should be well defined
• Provide information in the form of notes or pictures
Use multiple scoring
• Use at least two scorers
• Reliability coefficient of a paper scored only once is 0.25
(Harris, 1968)
• Two (multiple) tasks scored twice received 0.9 reliability
coefficient.
Sample task: Narrative
Sample task: Describing
Sample task: A letter
Sample task: Essay
Holistic vs. Analytic Scoring
Holistic scoring
• A single score is assigned
• Scoring is rapid
Analytic scoring
• A separate score is assigned for each aspect
• More reliable but time-consuming
• Candidates are compelled to consider all aspects of evaluation
Evaluation rubric
• Content (development of thesis, relevant, knowledgeable)
• Organisation (clearly stated ideas, supported, well-
organised, logical sequencing, cohesive)
• Vocabulary (range, word mastery, etc.)
• Language use (complex constructions, agreement,
articles, etc.)
• Mechanics (punctuation, spelling, paragraphing)
• Task completion
Variations on Traditional Assessment
• Use a green pen instead of a red one.
• Discuss the marking criteria with students.
• Write the correct answers in the margin.
• Use correction codes in the margin.
• Underline all errors of one type.
• Write a letter in reply.
• Write nothing. Discuss the work with the individual students.
• Only write a comment about the meaning and message of the piece.
• Devise an exercise, quiz, game from errors.
• Give a dictation based on sentences from students’ work.
Writing correction code
• Code Use Example
• WW Wrong word As our plane flew on the mountains we saw snow.
• WT Wrong time As our plane flew over the mountains we see snow.
• WF Wrong form As our plane flew over the mountains we was seeing snow.
• WO Wrong order As our plane over the mountain flew we saw snow.
• SP Spelling As our plane flue over the mountains we saw snow.
• P Punctuation As our plane flew over the mountains; we saw snow.
• X Extra word As our plane flew over to the mountains we saw snow.
• M Missing word As our plane flew over the mountains saw snow.
• R Register As our plane flew over the mountains we observed snow.
• ? Not clear As our plane flew over the mountains we saw snow.
• ! Silly mistake! As our plane flew over the mountains we seed snow.
• RW Try re-writing Our vehicle flies, we snow find, over mountains you saw it.
Can you correct these mistakes using the error correction code?
Assignment
1) Work in groups of 3.
2) Calibrate scales by finding various essays representing different proficiency
levels (an A essay, a B essay, a C essay, etc.)
3) Retrieve an essay of one of the team members from the previous years.
4) Correct the essay, using the correction code.
5) Evaluate the essay, using analytic rubric. Give it a mark individually. Do not
show the results to your team members.
6) Submit your scoring results to the team leader who must calculate the average.
7) Compare your scoring with your team members’.
8) Present results in our seminar.
9) You can use a ready-made evaluation rubric, such as in IELTS or TOEFL.
10) Do not display the name of the essay writer.
