UP Open University EDSC 311

7
Formative Evaluation

INTRODUCTION
Congratulations, again, for completing the first draft of your module. You
reviewed the accuracy, clarity, appeal and originality of your module. Would the
evaluation be the same if it were an external evaluation, .that is, done by other
people? You will find out in this· module. Who are in a position to evaluate your
module?

What is formative evaluation? How is it different from external evaluation?

External evaluation is done by people not involved in the production of the
material being evaluated. Formative evaluation (Bloom et al. 1971) in curriculum
development refers to the evaluation when the material is being formed or written.
Formative evaluation results are, therefore, used in the revision of a curriculum
material before it is published or printed for dissemination.

WHAT TO EXPECT
In this module, you will undertake formative and external evaluation of your
module. You will construct instruments for this purpose and administer them.

Module 7: Formative Evaluation 65

 EDSC 311 UP Open University

� OBJECTIVES

At the end of this module, you should be able to:

Construct and administer instruments for collecting feedback on the module from
colleagues and target users;

1. Analyze formative evaluation data 1n terms of:

1.1. congruence of curriculum material with its objectives;

1.2. student involvement in the module;
1.3. cognitive demand of the module;
1.4. readability of the module.

2. Use formative evaluation data to revise one's module; and

3. Demonstrate patience and determination in constructing and administering

instruments, and objectivity and openness in analyzing evaluation data.

WHAT I WANT YOU TO DO

Examine the first draft of your module. Recall the criteria that guided you in
writing it. You will decide who can evaluate your module and provide the
feedback you need to revise the module. You will also decide what information
about your module will help you improve it. It is expected that you will construct
and administer at least one instrument, and analyze and use the data that you will
gather to revise your module.

Evaluation of a module can be quantitative, qualitative or both. Quantitative

evaluation yields numbers like scores and ratings. Qualitative evaluation is
expressed in words. The trend in curriculum evaluation is to have both
quantitative and qualitative evaluation. For curriculum materials, favorable
quantitative evaluation may lift the spirit of the author. However, qualitative
evaluation in the form of specific suggestions is more useful to the author in
revising the curriculum material.

66 Module 7: Formative Evaluation

 UP Open University EDSC 311

An essential evaluation criterion for a curriculum material is the attainment of its

objectives. How will you know that the objectives of your module have been
attained? If one objective is for the user or student to be able to explain a concept,
for example, energy conservation, will a correct answer on an appropriate test
item be your indicator for the attainment of the objective? Your indicator may be
a class mean score on a test addressing the objectives of your module. This test
may be the mastery test, if you made one for your module.

You may use other measures such as interview questions to probe student's
understanding of a concept. A class or group project can be another measure of
the attainment of an instructional objective. For example, ah energy conservation
device made by students can be an evidence of a group's application of the energy
conservation principle.

Do YOU KNOW THAT •.•

Curriculum development requires a complex set of skills? Besides mastery of
subject matter, you need skills in teaching, writing, editing, art layout and
evaluation.

Content analysis by a science education expert or a colleague of the learning

opportunities provided in your module is another way of determining whether or
not the module as written enables the learner to attain its objectives. It is possible
that your module expected students to apply a concept and yet you did not have
activities or examples of applications of that concept in your module.

Another equally important criterion for a curriculum material that you studied in
Unit 6 is accuracy of concepts and skills presented in the module. For science or
mathematics curriculum materials, accuracy of content is best evaluated by a
scientist or mathematician.

Another criterion is originality of the curriculum material, measured mainly in

terms of innovations in presenting concepts and developing skills. A
science/mathematics educator or teacher is in a position to evaluate this criterion.
How about the clarity and appeal of your module? Can a science/mathematics
educator or teacher evaluate your module on these criteria? Do Activity 1 for the
evaluation of your module by a subject-matter expert and/or colleague.

This unit has four activities. You can do them in any sequence.

Module 7: Formative Evaluation 67

 EDSC 311 UP Open University

ACTIVITY 7-1

1. If a subject-matter expert for your module is not available, request the most
senior or experienced colleague or one with the highest degree in the subject
area of your module to evaluate your module on the five criteria: attainment
of objectives, accuracy of content, originality, clarity and appeal.
For important curriculum projects or an evaluation study, at least three
evaluators are needed. The average or intercorrelation of the ratings of
evaluators may be computed.
2. Make an instrument to include directions for your evaluator. The instrument
can be a rating scale.
A scale can be a four-point scale, for example: very much, much, not much,
not at all. An example of a five-point scale is Likert scale: strongly agree,
agree, uncertain, disagree, strongly disagree. If you decide on a Likert scale,
have about 20 statements with equal numbers of positive and negative
statements focusing on feelings and not facts/ideas. You can have a five­
point, seven-point, or ten-point scale where you label only the end points, for
example, poor-excellent. Refer, if needed, to books you used in tests and
measurement in your master's degree or undergraduate courses in education.
3. Give a copy of your module to your evaluator. Include in your directions to
the evaluator the writing of marginal notes -- corrections, suggestions,
reactions -- on the module itself. Set a realistic deadline (for example, one day
may be too short, and one month may be too long for a module) for your
evaluator to submit his/her evaluation.

Extensive marginal notes may be a blow to one's ego, but you will find these
most useful in revising your module. More notes mean more opportunities to
improve your module.

68 Module 7: Formative Evaluation

 UP Open University EDSC 311

/sAQ 1-1
I ?
1. Who is your evaluator and what is his/her background or expertise?
2. What instrument did you make and why?
3. Which marginal notes and other comments of your evaluator did you accept
and incorporate in your module? List three of these. (Delay answering this
question until your evaluator submits accomplished instrument and marginal
notes.)

You were the first to review your module in Unit 6. You can further evaluate the
appeal and clarity of your module without asking anyone else. Find out in
Activities 2 and 3. Activity 2 uses the student involvement index of Romey
(1968).

ACTIVITY 7-2

1. Randomly select ten pages (or use all pages if less than 10) of your module.
2. Starting with the first paragraph on each page, classify the first ten sentences
according to the categories in Table 1.
3. For each page, from page 1 to page 10 (pl to plO in Table 1), count the total
number of sentences for each category. Enter in Table 1.
4. Compute the Student Involvement Index, as follows:

Total No. of Sentences in Category II

Student Involvement Index =
Total No. of Sentences in Category I

Module 7: Formative Evaluation 69

 EDSC 311 UP Open University

Table 1. Student Involvement Index of a Module

No. of Sentences Total

Category
pl p2 p3 p4 p5 p6 p7 p8 p9 plO
I.
1. Facts
2. Stated Conclusion
3. Definitions
4. Questions answered
.
immediately
Total for I
II.
5. Questions requiring
student to analyze
data
6. Statement requiring
student to formulate
conclusion
7. Directions to student
to perform and
analyze some activity
and solve problems
8. Questions to arouse
student answer and
not answered
immediately.
Total for II

Total for II
Student Involvement Index of your Module =
Total for I

In Step 2, Romey' s procedure (1986) calls for 25 sentences. The number is

reduced to ten sentences for your convenience. In all, you analyzed one hundred
sentences of your module for student involvement.

70 Module 7: Formative Evaluation

 UP Open University EDSC 311

/sAQ 1-2
I ?
1. Compare the items in Categories I and II. Which category of items involves
students?
2. Why is student involvement, as defined by Romey (1968), a measure of the
appeal of a module or curriculum material?
3. Is it better to have a higher student involvement index? Why?
4. Compare Categories I and II when the student involveJl!ent index is high.
5. Based on the formula for student involvement index, what should be its value
for a module?

As in Activity 2, you will do the next activity on your own. Activity 3 aims to
find out the readability or reading ease of your module. In this activity, you will
use a common readability procedure, that of Fry (1968).

ACTIVITY 7-3
1. Skipping proper nouns, count 100 words in the first page, middle page and last
page (Table 1). If the first or middle page has less than 100 words, proceed to
the next page. If the last page has less than 100 words, start counting in the
previous page. Count as one word the article, "a", the pronoun, "I", and
contractions like "you're."
2. Count the number of sentences in each of the three 100-word samples. Enter
these numbers in Table 2.

3. Count the number of syllables in each of the three 100-word samples. Enter
these numbers in Table 2.
4. Compute the average number of sentences and the average number of
syllables of the three samples. Enter these averages in Table 2.

Module 7: Formative Evaluation 71

 EDSC 311 UP Open University

Table 2. Number of Sentences and Syllables Per Hundred Words

· ·i�;��· ;:;�· · · · · · · · · ···l·············��'....°.�.�.���=���.5.· · · · · ·+· · · · · · ��:. ��.�=-�!.��1.��·· · · · · · · I

1· · · · · �=�::�r· · · · ·-1 -· · · · · · · · · ·-· · · · · · +· · · · · · · · · · · · · · · · · I

!······-··-·····

[:: : : : : : ������:: : : : : : : :r: : : : : : : : : : : : : : : : : : : : : : r: : ·: : : : : : : : : : : : : : : : : : : : : i

5. Plot on the graph in Figure 1 the point with the average number of sentences
as y-coordinate and the average number of syllables as the x-coordinate. Read
the approximate grade level of the area where the point lies. Subtract two
grade levels.
The grade levels in the readability graph (Fig. 1) are based on norms in the
United States at the time the data were collected. The local norm is two
grade levels lower.

72 Module 7: Formative Evaluation

 UP Open University EDSC 311

FIGURE I. GRAPH FOR ESTIMATING READABILITY*

1-\vcragc Number of Syllables per 100 Words

Short Words Lon!J Words

1?.'1
__o ___
108 17.0 132 13G 14� "14 MU ...!_fi2 -�G 160 1Gtt lGU 172
2�.o
;;20.0
� IG.7
E 1'1.3
-t9
f 12.5
... 011.1
0 .c.
;: UJ1Q.O

8 9.2
... 0.3
a. 7.5

e
§ 7.1
6.7
� 6.3
0 5.8
ci 5.5
z 5.3
5.0
4.0
<{
4.5
g4.3
� 4.2
� 4.0
g' 3.0
.3
3.7
3.6
*by Edwerd Fry, Rutgers University Reading Center(1968).

Thus far, the evaluation of your module has not considered feedback from its
target users, the students. But who can better evaluate the clarity and appeal of a
curriculum material than its intended reader? Activity 4 shows some ways of
obtaining and utilizing student feedback.

/? /sAQ 7-3
1. Compare the intended grade level of your module and the one obtained from
the readability graph. If the grade levels are different, what are the
· implications for the revision of your module?
2. Examine the vertical and horizontal scales of the graph (Fig. 1) and
corresponding grade levels. Write one inference relating the sentence length,
word length, and grade level.
3. For reading ease, what is one recommendation for sentence length and word
length, based on the graph?

Module 7: Formative Evaluation 73

 EDSC 311 UP Open University

ACTIVITY 7-4

1. Reproduce as many copies of your module as the number of students in a

target class. Let the students read your module and perform the activities and
experiments in the module. (If reproducing 30 or more copies is expensive,
have only five copies and request five groups, one group at a time, to try out
the module.)
2. Construct a Student Questionnaire where students list words, figures and
tables in the module that they find unclear as they go through the modules.
Remind them to do this every now and then.
3. Include items in the questionnaire on their overall evaluation of the clarity and
difficulty of the module.
Clarity of a module is a measure of its difficulty. An unclear part of a module
will be difficult to read and understand. Other factors contributing to the
difficulty of a curriculum material are the level of abstraction of the concept
and the preparedness of the reader, in terms of prerequisite concepts and other
background information needed, for the new concepts and skills presented.
4. Ask in the questionnaire for parts in the module that students find boring.
Have an item on their overall evaluation of the appeal of your module to them.
The difficulty of a curriculum material is a measure of its cognitive demand -­
the set of concepts and skills required of a reader to attain the cognitive
objectives of the material. Curriculum writing requires constant awareness of
the cognitive demand, as well as the affective effect of the written material on
the reader. This will mean great empathy -- placing yourself, the author, in
the minds and hearts of your readers.
5. Include in the questionnaire an item on students' suggestions for the
improvement of the module. Remind your students to give their suggestions.
You will find these very useful in revising your module.
6. Observe how students perform the activities/experiments in the module,
particularly, errors in laboratory skills or science process skills. As you go
around, informally interview some students. Ask in a non-threatening
manner, why they are doing certain steps in the procedure. This is to find out
if they understand what they are doing and why they are doing some steps that
may be different from those in the module. Gently probe students' thinking
skills as they do the experiment or activity.

74 Module 7: Formative Evaluation

 UP Open University EDSC 311

7. After students have finished the module and accomplished the questionnaire,
formally interview five of them, selected randomly. You can write their
responses as marginal notes in their accomplished questionnaire. Their
answers in the questionnaire can be probed as follows:
7.1 Ask why they find certain words, tables and figures unclear.
7.2 Ask for their definitions of unclear words that they listed.
7.3 Ask for their definitions of words which they did not list but which you or
others think may be unclear to them.
7.4 Do the same for figures or tables which they did not list but which may be
difficult for students to understand.
7.5 Ask why they find some parts boring and how these can be made more
interesting.
8. Process data collected in No. 2, page 80 as follows (Talisayon 1983):
8.1 Use your three 100-word samples in Activity 3. From the listed unclear
words, use only those found in these samples. Count the number of
readers or tryout students who found a word in these samples unclear. Do
these for all unclear words in the samples. The total number of words,
Nx, is 300 words (3 samples x 100 words per sample). Also count the
total number of readers, Nr.
8.2 The feedback-based readability or communication index, CI, for words of
the module (Talisayon 1983):

Sum of (fX)
CI =
(Nr)(Nx)

where N = no. of readers indicating a given unclear word

f = no. of times unclear word appears in the samples of words
Nr = total number of readers
Nx = total number of words in the samples

Module 7: Formative Evaluation 75

 EDSC 311 UP Open University

For example, five words -- Xl, X2, X3, X4, XS -- are found unclear out
of 300 words in the three samples. Suppose that Xl appears twice and X3
appears thrice in the samples. This makes the total number of unclear
words, Nx, eight. Suppose, further, that the number of students who
found these words unclear are: XI - 7, X2 - 3, X3 - 1, X4 - 2, and X5 - 10.
Then the CI is:

CI= (2)(7)(1)+ (3)(1)+ (3)(1)(1)+ (2)(1)+ (10)(1) = O.Ol

(300) (8)

Note the multiplier, 2, for Xl and 3 for X3.

8.3 The range of acceptable values for CI is:

0 �CI� 0.01

In the example in Item 8.2, the CI has reached the maximum acceptable
value of 0.01. Two reasons for the high value of CI is that one word
found unclear by seven readers appeared twice and another unclear word
appeared thrice in the sample of words.

8.4 The same procedure, formula and range of acceptable values can be used
for sentences, paragraphs, tables, figures and equations.

You may wish to refer to my readability module in your reader for

clarification and other details. This is a long activity indeed, but I hope you
found each step worthwhile for getting feedback for your module.

76 Module 7: Formative Evaluation

 UP Open University EDSC 311

J ? JsAQ 7-4
1. List at least one error in procedure or misconception of the students as they
went through your module. How will you revise your module to address this
error or misconception?
2. Write one word, and if any, one sentence, paragraph, table, figure and
equation that one or more of your tryout students indicated to be unclear.
How will you make the unclear item(s) clear?
3. Show or explain why the communication index, CI, has a minimum value of 0
and a maximum value of 1.

If you have done all the activities at this point, CONGRATULATIONS! It

shows your interest in the course and determination to improve your module.
The four activities are one way of a thorough and systematic formative
evaluation of a curriculum material. You may have found the activities time­
consuming and tedious at times, but I hope that with each one, you gained
valuable information for revising your module. Hold on. You have one more
unit to complete the evaluation component of the curriculum development
process.

SUMMARY
You did four activities to gather comments from an expert/colleague and tryout
students and to estimate the student involvement, grade level and communication
index of your module. All these provided you data to help you revise your
module and improve its content accuracy, congruence with instructional
objectives, readability and clarity, and appeal to your target students.

Module 7: Formative Evaluation 77

 EDSC 311 UP Open University

REFERENCES
Bloom, B., Hastings, J.T. & Madaus, G. Handbook of Summative and Formative
Evaluation of Student Learning (1971). New York: McGraw-Hill (Chapters
4, 6, 9, 10, 18, 19).
Fry, Edward (1968). A Readability Formula that Saves Time. Journal of
Reading. April 1968. pp. 513-578.
Romey, William D. (1965). Inquiry Techniques for Teaching Science. New
Jersey: Prentice-Hall. pp. 44-51.
Talisayon, V. M. (1983). A Feedback-based Readability Formula for Science and
Mathematics Curriculum Materials. UPISMED Monograph No. 30.

� ASAQ 7-1

1. Answers here will vary. I hope that you were able to get the cooperation of a
friend or colleague who has the best subject-matter background for your
module.
2. Different kinds of instruments are possible, such as an interview schedule, a
short questionnaire, a rating scale, or simply a set of directions for marginal
notes on your module. A five-point rating scale, for example, may be used
for the expert's overall rating of the module, where "5" may be "excellent"
and "1" is "poor". Whatever instrument you use, it is helpful to add an item
asking for the expert's suggestions to improve the module.
3. You may disagree with some comments of your evaluator. However, this is
an opportunity to demonstrate the scientific value of openness to ideas.
Matters of accuracy of content are not debatable unless the concepts or writing
style depends on the author. If you feel strongly about a strategy or a way of
expressing your ideas, you, as the author, normally have the last say.

78 Module 7: Formative Evaluation

 UP Open University EDSC 311

� ASAQ 7-2

1. Category II involves students, for example, requiring them to analyze,

conclude, and perform an activity.
2. The appeal of a curriculum material is high when it involves the senses, for
instance, sight, sound and touch, of the reader.
3. Yes. A higher student involvement index means an active reader.
Involvement of the reader in the material makes it interesting. It is widely
believed that activities also facilitate learning, that is, learn by doing.
4. High student involvement index means a high numerator, II, and a low
numerator, I. For a high student involvement index, II> I.
5. A high student involvement index (II / I) means II> I or a ratio greater
than 1.

� ASAQ 7-3

1. If your intended grade level is lower than the level you got from the graph,
simplify complex words, sentences, paragraphs, figures and
experiments/activities. If your intended level is higher than the graph level,
you can introduce more difficult and abstract concepts, higher thinking skills,
and higher-level experiments/activities.
2. One inference is the greater the word length and sentence length, the higher
the grade level.
3. Whenever possible, shorten words and sentences. Everything else being
equal, short words and sentences are often clearer.

Module 7: Formative Evaluation 79

 EDSC 311 UP Open University

� ASAQ 7-4

1. Answers here will vary. For example, a student's error in procedure may be
addressed by replacing a confusing word or by putting a picture or diagram
· illustrating a step. Just a word of caution: from my experience and those of
others, students tend to rely on pictures of procedural steps, instead of reading
the text.
2. Answers here will vary, too. Suppose the unclear word is "synthesize". This
can be replaced perhaps by "put together", "combine" or "integrate"
depending on the context in which the word is used.
3. The communication index is zero if the numerator is zero, that is no one
indicated any unclear word. The maximum value is 1 if the numerator equals
the denominator, that is, all of the readers indicated all of the words in the
samples unclear.

80 Module 7: Formative Evaluation