TEST OF ORAL ABILITY

Tests of Oral Ability

The speaking ability is one of the most difficult language skills to measure. Speaking is such a common everyday activity and by putting it into a testing context, it immediately loses some of its authenticity.

Brown and Yule (1983) principles for testing speaking :

a)         Elicit speech which has a purpose.

b)         Elicit extended chunks of speech.

c)         Elicit structured or organised speech.

d)         Quantify the notion of communicative effectiveness.

Hughes (2002), test oral ability need to consider issues related to interactivity, authentic conditions for speech, and different spoken genres.

Grading of tests, therefore, must decide whether these micro skills will be assessed. The genre of the speech or conversation will also affect oral performance.

Test formats for testing speaking

 Weir (1990) has listed and discussed several of these formats as follows :

1.             Verbal essays
2.             Oral presentations
3.             Free interview
4.             Controlled interview
5.             Information transfer : Description of a picture sequence
6.             Information transfer : Questions on a single picture
7.             Information task
8.             Information gap - Student to teacher
9.             Information gap - Student to student
10.             Role Play

Oral presentations are somewhat similar to the verbal essay except that the candidates are allowed to prepare for the presentation.

 Both the free and controlled interview involve some form of conversation where examiners elicit spoken responses especially through a question and answer session. 

The contention that interviews are similar to authentic situations involving  employer and employee relations is somewhat  inaccurate as different profession are reported to emphasise different oral abilities.

Weir also describe two types of information transfer activities :

            a) Involves responding to a picture sequence

            b) Answer questions on a single picture.

Usage of pictures can be helpful as there is no language presented in the visual stimulus.

Candidates will therefore rely on their own knowledge of and proficiency in the language in order to communicate.

Weir’s description : 
- Student to student interaction : Alleviate the issue of unequal power and different levels of proficiency between interlocutors. 
- Student to teacher interaction : Weaker students might be disadvantaged especially if the conversation monopolized by the more proficient student.                     

In role play, candidates are expected to “play one of the roles in an interaction which might be reasonably expected of him in the real world” (Weir 1990: 79)

Role plays are easier if they present a plausible situation, Bailey (1998).

Figure 1 : Plausibility and experience as factors in performance on role play tasks (Bailey,1998)                         

Issues in assessing oral ability

Cohen (1994 : 278 - 279) cautions examiners involved in assessing oral ability of the need to establish several crucial dimensions of oral communication. These include:

            - The topic of the interaction

            - The level of formality

            - The number of participants

            - The relative status of the participants

            - The familiarity of the participants to each other

            - The gender of the participants

Cohen suggested that oral assessment should include a section that provides candidates with a chance to warm-up rehears.

Other suggestion, use of multiple measures of speaking ability and even the collection of taped data of the student’ oral ability in an oral portfolio.

Cohen also suggest that, oral assessments formats may be used to specify tasks  that  candidates are expected to perform such as describing objects, giving instructions and narrating a story.

Problems with item

It should be emphasised that the interview needs to be carefully structured so that  the aspects of the test which are considered important are covered with each students, and each student is tested in a similar way. 

It is not fair to the students if some of them are only required to make simple but   appropriate comments, while other equally good ones are forced to use complex language which betrays their inadequacies.

Interviewers also need to be trained to put candidates at ease, to get genuine conversation going without saying much themselves, to manage to appear interested in each interview and to know how to ask questions which will elicit the language required.

DICTATION TEST

The Dictation Test

Similar to cloze test

Both dictation and cloze test are able to predict overall language ability.

Dictation has the ability of assessing listening. (Hughes, 1972)

Teacher begins by selecting an appropriate passage:

·   Short and not longer than one paragraph.

·   Appropriate to the students language ability and cultural background.

The structure of the dictation

a) Read out at least three times

b) First time, normal rate of reading.

c) Students listen for general idea of the passage.

d) Second reading, little slower and expect students to write down what is read.

e) Teacher pauses to break the passage into meaningful chunks referred to as bursts.

f)  Third reading, students are expected to check their work, editing it for errors.

What makes a dictation difficult?

a)      The length of the phrase or burst

      - Longer burst, more difficult.
      - Students will have to retain more in short term memory while taking down burst.

b)      The length of the pauses between bursts

      - Longer pause, easier for students.
      - How long to pause? As long as the burst itself.

c)       The content of the dictation passage

- Familiar content makes the dictation easier.
- Jargon and highly technical words are difficult for some people.

d)      The syntactic and structural properties of the sentences in the passage
  
      Complex structures make dictation more difficult.

(1)    Ali caught the woman who stole the money

(2)    The woman that Ali caught stole the money

Both convey same meaning. Second sentence is more complicated because its structure is unfamiliar. The woman which is the subject of the verb stoledoes not appear immediately before the verb. The student is therefore required to make a “long distance” connection between the subject and the verb.

e)      Clarity of voice, expression and pace or tempo.

- Clarity of voice is important in dictations.
- Include facial expressions.
- The more animated the person dictating, the more cues are provided to the students.

Scoring a dictation

Two general methods. 

1.  Traditional method
2.  Pragmatic method

Both differ in treatment of spelling error.

Let’s consider the sentences “The boy is a pessimist. He never sees the silver lining in a dark cloud” which are read out to the students as part of a dictation passage. How would you grade the following responses?

     A. The boy is a pessimis. He never sees the silver lining in a dark cloud.

     B. The boy is a pacifist. He never sees the silver lining in a dark cloud.

Sentence A contains a spelling mistake – pessimis – while sentence B does not. However, in sentence B, the word pacifist is another meaningful word in the English language that simply is not appropriate in this context.

According to pragmatic method, the student will not be penalized for sentence A, but will have points deducted for producing sentence B.

The reasoning:

–   Sentence A is merely a spelling error which at least retains phonetics similarity to the original.

– Sentence B introduces a new word which makes the entire sentence inaccurate and in this case, unacceptable as well.

Argument:

–  Sentence B, there is a clear mistake which should be treated as such is a test.

– Sentence A, the absence of the letter “t”may well be due to the student’s haste to note down what he has heard and is therefore not as serious an error in a test.

Variants of the diction

A.      Graded or graduated dictation

·        Dictation passage becomes progressively more difficult. Gradually increasing the number of words in a burst.

·      Dictation may begin with a burst consisting of two words and the number of words slowly increases until there can be up to thirteen or fourteen words in a burst.

·    Normally the processing load becomes too high when a burst exceeds seven words. However, better and more proficient students will be able to handle seven words and perhaps even more by chunking words that collocate naturally.

B.      Partial dictation

• Similar to listening cloze activity. Students are provided the passage with some words or phrases deleted.
• They are expected to listen to a passage and fill in words or phrases.
• It is commonplace to have partial dictations in which single words or even short phrases are deleted.

C.      Dictocomp

• The students are expected to use the information they hear to construct a coherent piece of composition instead of taking down the passage exactly as it was dictated.
• The teacher will determine the key elements of the original passage which the students is expected to include in the composition.
• Dictocomp can be said to be a test listening comprehension in a very specific way in that the student has to decide what pieces of information are important and should be included. This is reminiscent of summaries.
• Dictocomp also tests writing ability as well because the students are expected to write a cohesive piece based on the passage that was dictated to them.

THE CLOZE TEST

The Cloze Test

What is cloze test?

The cloze test is a test that is often associated with language proficiency testing. It is more than simply filling in the blanks in a passage as it has a theoretical basis.

Structure

a) Consist passage with blanks

            - First sentence left intact without any blanks.

               + To ensure that the test takers have some context to work with.

            - Normally, not more than 20 blanks.

          - When constructing the cloze test, the format may not be needed to follow too rigidly.

What makes a cloze text difficult?

For students learning English as a second language, a passage written in English is often already difficult to read.

Among the factors that affect the difficulty level of the cloze procedure include the  followig:

- Length of the text

- Familiarity of vocabulary and structures.

- Length and complexity of the sentences.

- Familiarity with topic and discourse genre.

- Frequency of which blanks are spaced.

Grading a cloze test

There are two methods to how a cloze test can be graded. These methods are the exact word method and acceptable word method.

Exact word method :

a) Only one answer is accepted for each blank.

b) More objective therefore more reliable.

Acceptable word method :

a) Accept any suitable answers.

b) More subjective and therefore less reliable.

Variants of the cloze test

The C-test

- Highly correlated to the cloze test.

- Claimed that the C-test can provide the same benefits and advantages.

 - Providing half of the word

 - More reliable and objective.

Disadvantage / Criticism of the C-test.

- Looks complicated and may therefore confuse the students.

- Some believe that the C-test merely tests spelling and not an overall understanding of the English language.

Bachman (1985) Claims the kind of information that is needed to fill in the blank in a cloze test can categorized into four types :

a) Information located within the clause.

b) Information spread across clauses but within a single sentence.

c) Information distributed across sentences but within the text.

d) Extra-textual information.

Problems

a) A high proportion of function words deleted, which may be fairly easy for competent language users to restore, and which may distinguish between students of different    levels of ability, whereas other versions may have lost a high proportion of content        words which may prove to be irretrievable even for native speakers.

b) Marking cloze texts can be difficult since there may be many possible answer for any one gap, and there is often disagreement as to what answer are acceptable.

c) Such tests may be a wasteful way of testing (Unless the aim of the cloze test is to test overall language proficiency)

d) C-test : Instructions are too complicated.

The items must be thoroughly pretested. 

ESSAY FORMAT

Different forms of essay format
short response answer to extended response answer.
 Essay writing at SPM level are divided into two parts:

ü  Directed writing    - provides structure to the students and expects the students to write according to the prompts given

ü  Continous writing - expects the students to draw upon their experience and past knowledge as well as knowledge of writing conventions.

Advantages of the essay format

·         can assess higher order skills

ü  students are able to analyse,synthesis and evaluate through topics that require students to argue or express their opinions.

·         emphasises communication skills

ü  important aspect in social relations

·         eliminates guessing

ü  unlikely to occur in essay because students need to provide more ideas on their own.

·         relatively easy to construct

ü  within few minutes,the questions are easy to construct rather than constructing other  test  formats.

Disadvantages of the essay format

·         can only test a limited amount of content if compared to multiple choice type questions.

·         prone to bluffing by the students.

ü  repeating the same idea so that the teacher will be impressed by the length of essay.

·         essays tend to be less reliable than other test format.

ü  different graders have different impressions of a single essay.

ü  help the students by providing clear guidelines.

·         essays are difficult to score.

ü  consume more times to grade the students essay as you need to decide which scoring approach to use.

Approaches to scoring essay

·         there are three major approaches to scoring essays which is :

ü  holistic scoring

v  readers react to the students' composition as a whole and and a single score is awarded to the writing

v  The rank would be from 1-6.

v  example:

1	2	3	4	5	6
Vocab is precise,vivid and varied	Vocab is adequate for grade level	Vocab is simple	Vocab is limited and repetitions	Respond with few isolated words	No response
Meaning is conveyed easily	Meaning is conveyed but break down at times	Meaning is frequently not clear	Meaning is unclear	No complete sentences are written	No response
Transition from one idea to another idea is smooth and provides readers with clear understanding.	Some transitions of idea is evident	There are few transition markers or repetitive transition markers	No transitions markers	No evidence of concepts of  writing	No response
A few mechanical errors are present but there are no disrupts in communication	Mechanical errors are present but does not disrupt the communication	Mechanical errors disrupt communication	Mechanical errors cause serious disruption in communication.	-	No response

v  primary trait scoring are related to holistic scoring because primary trait scoring focuses on certain functional criteria based on the purpose of the writing and the grade will be given based on how well the students are able to express that function.

ü  analytical scoring

v  raters access students'performance on a variety of categorieswhich are hypothesised to make up the skill of writing.

v  some possible contents used in assessing writing ability using an analytic scoring:

      Content

      organisation

      language use

      vocabulary

      mechanics

ü  objective scoring

v  this approach relies on the quantified methods of evaluating students writing.

v  this scoring will be conducted by :

      establish standardization by limiting the length of the assessment.

      identify the elements to be assessed-go through the essay up to 250 words and highlight all the mistakes from spelling to mechanics.

      operationalise the assessment-assign a weight score for each mistake range from 1-3.

      Quantify the assessment- 250 (words) = 3.52 correctness score

                                               71 (sum of errors)

Comparison of three approaches

Holistic Scoring	Advantages	Disadvantages
	·         quickly graded ·         provide a public standard that is understood by the teachers and students alike ·         relatively higher degree of rater reliability ·         applicable to the assessment of many different topics. ·         emphasise the students strength rather than their weakness.	·         the single score can actually mask the differences across students compositions. ·         does not provide a lot of diagnostic feedback.
Analytical Scoring	·         provides clear guidelines in grading in the form of various components. ·         allow the graders to consciously address important aspects of  writing.	·         writing ability is unmaturally split up into components.
Objective in scoring	·         objectivity in scoring ·         grading a paper can be stopped and resumed with ease even after a period of time as the consistency or reliability of scoring is maintained.	·         still some degree of subjectivity is involved. ·         accentuates negative aspects of the learners writing without giving credit for what they can do well.

Ensuring accurate scoring of essays

Familiarisation with scale

·         familiarisation is important to achieve a valid and accurate scores.

·         graders should be familiar with  the scoring criteria so that the if bands are involved,they would know what does each band requires.

Identifying benchmark papers

·         this step is to provide a clear and representative example of students work according to the grading criteria.

·         Bands can only provide general description of what is expected meanwhile benchmark papers provide concrete examples and help ensure fairness in grading.

Grading

·         Norming

ü  papers are stacked together and divided according to the band

ü  read each paper more closely and state our first impression

·         Final Grades

ü  when we need to assign more precise numerical scores

ü  grade all students on one essay first before moving on to the next essay in order to ensure more consistent grading.