il

740 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

Knowledge-Based Classification and Tissue

Labeling of MR Images of Human Brain
Chunlin Li, Dmitry B. Goldgof, and Lawrence 0. Hall

Absfruct- This paper presents a knowledge-based approach Image features of abnormal tissues may be very close to
to automatic classification and tissue labeling of 2-D magnetic those of their neighbor normal tissues. Therefore, abnormal
resonance (MR) images of the human brain. The system con- tissues may be found in a normal tissue component after
sists of two components: an unsupervised clustering algorithm
and an expert system. MR brain data is initially segmented a segmentation. Abnormal tissues may also deform the ge-
by the unsupervised algorithm, then the expert system locates ometry of normal tissues. In the literature most systems on
a landmark tissue or cluster and analyzes it by matching it segmenting MR brain images were demonstrated on either
with a model or searching in it for an expected feature. The normal or abnormal slices [3]-[6]. Second, a tissue may have
landmark tissue location and its analysis are repeated until a varied image features. The effect on segmentation is that
tumor is found or all tissues are labeled. The knowledge base
contains information on cluster distribution in feature space and several segmented components which have different labels
tissue models. Since tissue shapes are irregular, their models correspond to a single tissue. These problems motivate us
and matching are specially designed: 1) qualitative tissue models to take a knowledge-based approach. Domain knowledge is
are defined for brain tissues such as white matter; 2) default applied to an initial segmentation to progressively identify
reasoning is used to match a model with an MR image; that
is, if there is no mismatch between a model and an image, landmarks, split tissues, and determine normality or abnor-
they are taken as matched. The system has been tested with mality. We chose to use a clustering algorithm to obtain
fifty-three slices of MR images acquired at different times by the initial segmentation, because cluster analysis can offer
two different scanners. It accurately identifies abnormal slices important domain knowledge, and clusters have better se-
and provides a partial labeling of the tissues. It provides an mantic meanings than edges [3] regarding knowledge-based
accurate complete labeling of all normal tissues in the absence of
large amounts of data non-uniformity, as verified by radiologists. analysis.
Thus the system can be used to provide automatic screening of Among early reports of knowledge-based segmentation are
slices for abnormality. It also provides a first step toward the an expert system for segmentation using domain-independent
complete description of abnormal images for use in automatic heuristics [7] and knowledge-based identification of artery
tumor volume determination. branches [SI. Recent such work in medical images includes
the segmentation of coronary vessels [9], 3-D MR brain data
[4], and 2-D MR brain data [5]. In [lo] a symbolic model
I. INTRODUCTION utilizing fuzzy sets for object representation in a top down

A UTOMATIC segmentation of medical images has the

potential to facilitate an imaging-based diagnosis. De-
spite a variety of techniques proposed in the past decades
manner is shown to provide a feasible method of segmenting
2-D magnetic resonance images. Edge and/or region operators
are applied before knowledge-based processing in [71-[91.
[I], automatic segmentation of non-trivial images is still at Feature distribution and intensity distributionsare used directly
the stage of laboratory research. Combined strategies have for segmentation of brain data in [4] and [5] respectively.
been proposed [2] to treat the segmentation process itself as Other approaches for segmenting brain images include an
a complex task. This paper presents such an approach for unsupervised clustering of MR data [6], the use of syntactic
segmenting 2-D magnetic resonance (MR) images of human rules to analyze CT brain images [ 1I], and a graph-theoretic
brain. method to segment MR brain data [3].
There are two classes of MR slices from patients: normal Given a slice position in an imaging plane of the brain,
and abnormal. Normal slices contain normal brain tissues such tissue models can be defined. Shape information has not been
as white matter, gray matter, and cerebro-spinal fluid (CSF). adequately used in brain image segmentation such as [4], [5].
Abnormal slices contain both normal and abnormal brain The main contribution of our work is to use tissue models in
tissues. The segmentation of MR brain images is complicated labeling and classifying MR brain images. Early results of the
by two factors. One is the abnormality in a given slice. research were reported in 1121.
Manuscript received August 27, 1992; revised May 30, 1993. This research The system is organized as follows. An MR slice is ini-
was partially supported by Whitaker Foundation Grants, a grant from Siemens tially segmented by an unsupervised fuzzy c-means clustering
Corporation, the National Cancer Institute (CA59 42541). and a grant from
the Florida High Technology and Industry Council Software Section. The algorithm (UFCM) [13], [14], then an expert system uses
associate editor responsible for coordinating the review of this paper and model-based recognition techniques [ 151 to locate a land-
recommending its publication was Dr. R. Leahy. mark, called a focus-of-attention tissue. Qualitative models
The authors are with the Department of Computer Science and Engineering,
University of South Florida, Tampa, FL 33620. of brain tissues are defined and matched with their instances
IEEE Log Number 9213397. from images (discussed in the next section). If a significant
0278-0062/93$03.00 0 1993 IEEE
L1 et al.: KNOWLEDGE-BASED CLASSIFICATION AND TISSUE LABELING 74 1

deformation is detected in a tissue, the slice is classified to

be abnormal. Otherwise, the expert system locates the next
focus-of-attention tissue according to known expected tissues.
This process is repeated until either a classification decision
is reached that the slice is abnormal or all tissues of the
slice are labeled. Section VI1 discusses the reason for such an
organization. The sequential stages of classification and tissue
labeling are summarized in the following:
(a) (b) (C)
1 . Separate skull tissues from tissues of interest, namely,
Fig. I . Feature images of different slices. (a) T1-weighted. (b) Proton
the white matter, gray matter, cerebro-spinal fluid (CSF), density. (c) T2-weighted.
and abnormal tissues.
2. Locate white matter. The shape of white matter is usually
significantly deformed by large abnormal tissues.
3. Use white matter to locate CSF, if white matter is not
significantly deformed. A symmetric measure of CSF
indicates whether it contains an abnormal tissue.
4. Locate gray matter, if both white matter and CSF are
believed to be normal. If there are deformations less
significant in white matter, a final analysis is done at
this stage with the knowledge of white matter and CSF. Fig. 2 . A normal slice of interest
A rule-based expert system tool, CLIPS [ 161, [ 171, is used to
organize the system. Low-level modules for image processing
and high-level modules for image analysis are all written in C
and called as actions from the right-hand sides of the rules.
This system provides an accurate and complete automatic
(no human intervention on a per subject basis) segmentation
and labeling of normal volunteer brains. It also suggests
the possibility of automatic screening of MR slices for ab- Tum
normality. Such capability is contingent upon the untested
ability to detect small tumors and, that available, effective
Fig. 3. An abnormal slice of interest.
guiding knowledge for all slices. By partially labeling tissues
in abnormal issues, the system may be used as a basis
for a complete automatic segmentation scheme for abnormal [18]. These slices reveal a gross anatomical structure and
images. Some additional work along these lines is discussed in their CSF looks loosely like an “X”. Each slice consists of
our last section. A complete accurate automatic segmentation three feature images: T1-weighted (Tl), proton density (PD),
of all tissues into their respective tissue types would be and TZweighted (T2). Figure 1 shows the feature images of
useful in tumor volume determination for treatment plan- different slices of interest. Slices of Fig. l(b) and (c) contain
ning. tumors, which deform their neighbor tissues. The effects of
The remainder of the paper is divided into seven sections. tumors can be demonstrated more clearly by labeled slices.
Section I1 introduces the tissue models and UFCM behavior. Figure 2 shows a normal slice. The labeled tissues are white
Sections 111, IV, V, and VI describe the techniques used matter (white), gray matter (black), cerebro-spinal fluid (CSF)
for the listed four stages. The last two sections present the (inner gray), and ventricle area (enclosed by white matter).
experimental results and a discussion of them and future Figure 3 shows an abnormal (tumor) slice. The tumor (light
directions. gray) occupies an area which belongs to white matter and CSF
otherwise. Due to the fact that brain tissues vary with patients
11. TWO SOURCES OF KNOWLEDGE and slice positions within the neighborhood of our interest, it
is a nontrivial effort to model the shape of white matter and
There are two sources of knowledge: 1) anatomical structure
CSF. The models we defined for normal white matter and CSF
of tissues; 2) distribution of class centers in feature space. are qualitative. They have some flexibility shown by the fact
The features are different image intensities. Both kinds of that the system can process multiple slices in a row instead
knowledge are acquired from imaging anatomy [18] and of a single slice in the axial plane. Section VI1 discusses the
experimental observation. tolerance of the qualitative models. Sections IV and V describe
the details of the models and techniques for implementation. A
A. Slices of Interest for the Study default logic [ 191 is followed to match the qualitative models:
The anatomical structure of the brain varies with different unless there are expected deformations or a mismatch in an
planes of imaging and with different positions in the same instance of a model, classify the instance into a normal tissue.
plane. We focus on slices at a neighborhood of a position of If a deformation exists but is small, the classification decision
the axial plane (about 7 to 8 cm from the top of the head) is delayed until adequate evidence is collected.
142 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

Fig. 7. (a) Typical skull class image. (b) A synthesized image with the lowest
Fig. 4. Class centers in TI, PD, and T2 space. three classes in T2 spectrum as the background (white). (c) A quadrangle.

3. CSF takes the highest T2 class in the absence of tumors,

and the highest or second highest T2 class in the presence
of tumors.
4. Skull tissues (fat, skin, and bone) are scattered into three
to four classes. Two to three classes of the smallest T2
values, next to air, are skull tissues.
5. White matter is clustered into one to two classes. It is
smaller in T2 than gray matter and sometimes smaller
Fig. 5. Corresponding class centers in TI, T2 space than one to two skull classes.
6. Gray matter is clustered into one to three classes which
are higher in T2 than white matter and skull tissues.
3 classes 4 classes 3 classes The observation of class center distribution in the T2
spectrum is motivated by the semi-automatic segmentation
Air- of T2-weighted MR brain scans [20]. Another observation
Fat white Gray Tumor
Skin Matter Matter CSF
was recently reported [6], which is a special case of this
Bone observation.

Fig. 6. Class center distribution in the T2 spectrum. 111. SEPARATING SKULL TISSUES
Skull tissues are not of interest; they will be separated so
we can focus on the brain itself. Some skull classes may
B. Behavior of UFCM be between white matter and gray matter classes in the T2
Since slices are processed without a priori knowledge of spectrum. Figure 7(a) shows a typical skull class (black pixels).
their being normal or abnormal, ten classes are chosen for They can be separated by combining the previous observations
each slice. This may lead to over-segmentation of tissues in and the anatomical knowledge that they form an exterior band
normal slices, but reduces the chance that tumors are clustered and enclose the other brain tissues. An image is created from
into classes that contain normal tissues. The UFCM clustering the ten classes by making the lowest three classes in T2
algorithm takes a slice consisting of three images as input and spectrum the background (white) and the other seven classes
clusters them into ten classes. We refer to an image consisting the foreground (black) (Fig. 7(b)). The lowest three classes
of the ten classes or regions as a clustered image. Each class form a visible gap between the skull tissues and the other brain
has a cluster center (7'1, P D , T 2 ) . Figure 4 shows the ten tissues. The center of the brain is calculated as the first moment
class centers of two slices in the T1, PD, and T2 space. Six of the binary image [21]. A Cartesian coordinate system is set
characters are used to represent, from left to right, classes up in the center of the brain. A point ( X r , 0) is found by
of air (A), skull tissues (B), white matter (W), gray matter moving right along the X axis from the origin until a point
(G), tumor (T), and CSF (C), respectively. The class centers such that an 8 x 8 window from (X,, 0) to (X, 7, 0) and +
in feature space can be better illustrated by projecting them centered around X axis has K background pixels in a row
into TI and T2 space. Figure 5 is obtained by projecting the and L background pixels in the window ( K 2 5 and L 2 25).
two plots in Fig. 4 into the TI and T2 space respectively. Refer to (X,, 0) as the right inner point. The left inner point
The knowledge obtained from the class center distribution is can be found similarly. The upper and lower inner points can
used in locating focus-of-attention tissues. It consists of the also be calculated the same way except that they are in the Y
following, as illustrated by the T2 spectrum (Fig. 6): direction and K background pixels in a row are replaced by
K pixels in a column. These four points give a quadrangle as
1. Air always appears in the lowest one or two classes as shown in Fig. 7(c). A class image is a binary one in which
separate classes. only the pixels belonging to the class are made the foreground
2. Tumor always appears as the highest or the second (black); all other pixels are made the background (white). The
highest T2 class. It may be in a class of its own or analysis, hereafter, is in class images with the brain-centered
in the class of CSF. Cartesian coordinate system.
LI et aL: KNOWLEDGE-BASED CLASSIFICATION AND TISSUE LABELING 743

Since we have known that the lowest three classes are air
and skull tissues and the highest three are gray matter, CSF,
and tumors, only four class images are classified by a rule:

IF (The number of the foreground pixels within the

quadrangle is small)
THEN (It is a skull class)
ELSE (It is a class of white or gray matter) ... ...,

A typical skull class has from 0 to 200 pixels in the

quadrangle area while a white matter or gray matter class has
from 400 to several thousand foreground pixels in the area.
The difference may be further amplified if a polygon with more
than four angles is used to approximate the round background
band between the skull tissues and the other inner tissues. A
threshold of 300 is used in the current implementation. Fifty-

&
three clustered images are successfully processed. Each image
is divided into a group of air and skull tissue classes and a
group of the other tissue classes. In the latter group, tissues
of interest are the white matter, gray matter, CSF, or tumor
in order of increasing T2.
f
I I

IV. WHITEMATER
After separating skull classes, the lowest class in the T2 . .

spectrum is white matter. If white matter is split, the second ...

.\. " . .J

smallest class is also white matter. Both global and structural

methods are used to detect white matter splitting. A shape Fig. 8. Illustration of two neighbor classes. (a) One white matter class. (b)
analysis technique is presented to match white matter and The neighbor white matter class. (c) A white matter class. (d) The neighbor
its qualitative model. If the match is acceptable, a polygon gray matter class. (e) A white matter class. (f) The neighbor gray matter class.
approximating the ventricle area is generated to locate CSF,
the next focus-of-attention tissue.

Y
A. White Matter Splitting
White matter is sometimes, although not frequently, clus-
tered into two neighbor classes. Figures 8(a) and (b) show a
white matter split into two neighbor classes, Figs. 8(c) and
(d) are a white matter class and its neighbor class of gray
matter. Global methods are more appropriate for detecting the
splitting than structural and relational methods [15]. It is noted
@ wm
BT=d BT=w BT = wl
that a white matter class is more compact than its neighbor
gray matter class, if white matter is not split (Figs. 8(c) and
Fig. 9. The BTs of three objects.
(d)); otherwise, two white matter classes have similar density
values (Figs. 8(a) and (b)). However, if white matter and gray
matter are clustered into one class each as shown in Figs. 8(e)
and (0, they also have similar density values. of a component or of all components in an image. The BT
Hence, a two-level binary decision tree is used to detect of a circular object is its diameter, and of a rectangular
white matter splitting. In the first level, two neighbor classes area is the lesser of its width and length, which are shown
of similar density values (as measured by the biorthogonal in Fig. 9 along with the BT of an irregular object. A BT
thickness (BT) and density d which are defined below) are gives a rough estimate of how big a component is in an
tentatively classified as resulting from white matter splitting. image. A morphological erosion operator with a 5 x 5 square
On the second level, shape information is used to make a final as a structural element [22], [23] was used to measure the
classification. compactness of a class image. A density, P,/Po, is calculated
A biorthogonal thickness, BT (z,y), is defined to be the for each class, where Po is the number of object pixels in
lesser of the vertical (Y) and horizontal (X) metrics of a the class image, P, is the number of object pixels left after
component at point (z,9). Considering the shape irregularity an erosion. The class known to be white matter is used as a
of brain tissues, BT is defined to be the maximum BT (z, y) template to decide if the other class is also white matter. A
I
744 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

density normalization is used to scale the two density values

to make the value of the known white matter class ( d ( W M ) )
not less than 0.1. This is done by repeated multiplication by
10 until 0.1 5 d ( W M ) < 1. For example, if densities before
scaling are 0.02 for the class known to be white matter and
0.003 for the other class, then after scaling they become 0.2
and 0.03.
The rule below examines the normalized density values of
two neighbor class images. First, biorthogonal thickness (BT)
is used to detect the simple case when the unknown class is not
gray matter, since biorthogonal thickness of the gray matter is
never much larger than the BT of the white matter. Second,
the “ELSE’ and second “IF” clause is invoked if BT is not
giving a clear indication. The main idea here is that if white
matter is split into two classes-the densities of both classes
would be similar; otherwise the density of the gray matter is
smaller.

IF (The BT of the unknown class is much bigger than

the known white matter class)
THEN (The unknown class is white matter)
ELSE I
IF (Both density values are bigger than 0.1)
THEN (The unknown class might be a white matter)
ELSE (No white matter splitting)

Fifty-three slices were tested with the rule, of which eight

slices have white matter splitting. The rule classified seven
of the eight splitting slices as splitting (one slice by BT, six
slices by density values), eight other slices as splitting (all by
density values), and other remaining slices as non-splitting.
I I
The misclassified splitting slice suffers from significant data
non-uniformity. The fourteen slices classified by the density
feature are further classified in the second level of the decision .:.:.
tree. Two other global methods, which failed to provide (€9 (h)
such a strong discrimination, have been tried: 1) A measure Fig. 10. Shape analysis to detect white matter splitting. (a) A binary image.
of compactness [23] calculated from two neighbor classes (b) A binary image. (c) A boundary bond. (d) A boundary bond. (e) A merged
respectively; 2) Texture of first- and second-order statistics image. (0 A merged image. (g) A boundary band. (h) A boundary band.
1241 calculated from intensity images (Tl, PD, and T2) with
two neighbor classes as masks.
subtractingthe eroded image from the merged image, as shown
In the second level of the binary decision tree, only slices
in Figs. 1O(g) and (h). A ratio is calculated for each slice
tentatively classified as white matter splitting are processed.
by dividing the number of pixels in a merged band (Fig.
For such a slice, a binary image is created by merging all lO(g)) by the number of pixels in the original band (Fig.
tissue classes except skull ones recognized in Section 111.
lO(c)). The ratio is used to decide if a slice has white matter
Figures 10(a) and (b) show the binary images of two slices splitting:
corresponding to Figs. 8(a), (b) and (e), (f) respectively. An
enclosed round band is calculated from a binary image by IF (The ratio is big (> 0.5))
erosion and set subtraction operations. A binary image is first THEN (No white matter splitting)
eroded with a 7 x 7 structural element; a band is formed by ELSE (White matter splitting)
subtracting the eroded image from the binary image. Figures
lO(c) and (d) show the bands corresponding to (a) and (b) The ratio of a typical slice with white matter splitting is well
respectively. Since gray matter occupies much more boundary below 0.5 (0.085 for the slice of Fig. 1O(g)) and the ratio is
area of the binary images than white matter as shown in Fig. bigger than 0.5 otherwise (0.89 for the slice of Fig. 10(h)).
8, the number of pixels within a band is used to decide if The eight slices, which have no white matter splitting but
gray matter is merged with white matter. Another image is were tentatively classified as having split white matter, were
created for each slice by merging its two neighbor classes. classified as non-splitting. The other six slices which actually
Figures 10(e) and (f) show the merged images of Figs. 8(a), have white matter splitting were still classified as containing
(b) and (e), (f) respectively. A band is then generated by split white matter.
L1 er al.: KNOWLEDGE-BASED CLASSIFICATION AND TISSUE LABELING 745

Fig. 11. Class images of white matter. Fig. 13. Emitting lines from the origin of the Cartesian system.

B. Model Matching
Fig. 14. Abnormal white matter images.
Figure 11 shows three class images of white matter. An
important characteristic of the white matter is that if there are
no tumors or the tumors are small, the central background or from the origin to the boundary of the image, but b o h its
ventricle area is enclosed by the white matter (Fig. ll(a)). upper and lower neighbor lines, Li-1 and Li+l, reach white
Tumors may occupy the white matter area and break its matter, a Pi is interpolated by connecting the two components
continuity (Fig. ll(c)). Small gaps between the white matter in which Pi-1 and Pi+l reside, if the distance between the
may exist around the ventricle area and gray matter intrudes two components is small (less than 5). The distance between
(Fig. 1l(b)), depending on slice positions. Similarly, the gray two components is defined in Section IV-C. Otherwise Di
matter may occupy the white matter area along the Y axis is set to be -1. For the purpose of continuity, compute the
(Fig. ll(c)). The qualitative model for white matter defines connected components of the white matter in the left half
normal white matter as contiguous along both sides of the (a: 5 0) and right half (a: 2 0) of the image respectively.
ventricle area; the continuity may be slightly interrupted by The number of gaps in the left half of the image, GapLeft,
gray matter. The lateral continuity of the white matter around is calculated from L1 to L7, and GapRight from Lg to L I S .
the ventricle area is used to classify the white matter as being The gaps from L1 to L3 and from Lg to L11 may be induced
normal or abnormal. by gray matter intrusion. The following rule classifies white
To make the classification rule more abstract and reliable, matter:
we first enhance the white matter images. A morphological
opening operator [23], [22] with a 3 x 3 structural element is IF (There is a significant gap OR
applied to the background of white matter images to connect GapLeft > 1 OR GapRight > 1 OR
small gaps between the white matter. The resulting images are a gap is not between L1 to L3 or Lg to L11)
then smoothed by a 3 x 3 median filter. To suppress small iso- THEN (The slice is abnormal)
lated strips of white matter, the connected components of the ELSE (Connect gaps AND form a polygon to
white matter are computed and a component is made the back- approximate the ventricle area)
ground, if its number of pixels is smaller than a number S ( S =
70). Figure 12 shows the enhanced images from Fig. 11. A gap with a width of 25 pixels or over is taken as
A coarse to fine analysis of the white matter is used to deal significant. For the white matter with gaps less than TF3
with possible deformations. Lines are emitted outward from (TB = 25), the decision is delayed after the location and
the origin of the Cartesian system (Fig. 13), each of the lines analysis of CSF, as is the case of Fig. ll(c). Figure 14
is 45/N degrees apart from its immediate neighbors (N = shows six white matter images classified as abnormal. Nine
2 in current implementation). If a line L, reaches the white white matter images are shown in Fig. 15 which are clas-
matter, record the position Pi of the first white matter pixel sified as normal at this point. Two slices of Figs. 15(a)
it meets and calculate the distance D; between Pi and the and (b) suffer from data nonuniformity, but are correctly
origin. If a line Li does not reach the white matter on its way classified.
746 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

BtmLeft
BtmRight

Fig. 16. A polygonal approximation to the ventricle area.

Fig. 15. Normal white matter images

31
I '

C. Generating a Polygon
Local extremes of the boundary are used to calculate the
distance between components and to form a polygon. We use
KI -
chain code [25], [26], both clockwise and counterclockwise, (a) (b)
to follow the boundary and to search for extremes. If lines Fig. 17. Two abnormal CSF classes.
L;+l and L; in the left half of the image do not belong to one
component, start a counterclockwise chaining from L,+1 and V. CSF
follow the white matter boundary upward to search for the first The CSF class is located by calculating the number of pixels
local maximum of y. Let the (2,y) be the first local maximum, within the polygonal area in the top two classes of the T2
the smallest distance between the (z,y) and the component spectrum. The one with a larger number of pixels is the CSF
in which Li resides is defined to be the distance between class. If CSF class is not the top one in the T2 spectrum, the
the two components. For the right half, start a clockwise top class is tumor, and we can achieve a complete labeling
chaining and follow the white matter boundary upward. The by labeling all classes between CSF and white matter as gray
remaining operation is the same as in the left half. To form matter. However, abnormal tissues are often clustered into CSF
a polygonal approximation of the ventricle area, the chain classes. For the slices of interest, normal CSF is symmetrical
code is to follow the boundary of the ventricle area, or the on a vertical axis through its center. The Y axis of the
background boundary (white). The first local extremes in the Cartesian coordinate system setup in Section 111 is used as the
Y axis are used. The following rule guides the chaining to vertical axis. Abnormal tissues in the CSF class can be detected
look for the local maximum y's of the upper two ventricle by measuring the symmetry of the CSF image. A simple way
horns (TopRight and TopLeft): is to compare the number of pixels in the left and the right
half of the image. We define a fuzzy symmetric measure [27],
IF ((LOmeets white matter (DOis not -1)) 1
THEN (Start a clockwise chaining from POtoward
the right for TopRight) 1 +
(Start a counter clockwise chaining from POtoward where L and R are the numbers of foreground (black)
the left for TopLeft, POis the upper point) pixels in the left and right half of the CSF class respectively.
ELSE (Start a clockwise chaining from P4 upward for A 3 x 3 median filter is applied before calculating L and
TopLeft) R to eliminate the effect of small strips of CSF around
(Start a counter clockwise chaining from Plzupward intracranial boundary. It has been found that a threshold
for TopRight) of the symmetric value discriminates well between normal
(Interpolate the upper point POfrom TopLeft and and abnormal CSF images, so no effort in deriving linguistic
TopRight) variables of fuzzy symmetry has been extended.
IF (The symmetric measure is very small (< 0.1))
A similar rule guides the chaining to search for the local
THEN (There are abnormal tissues in the CSF class)
minimum y's of the lower two ventricle horns (BtmLeft and
BtmRight). Figure 16 shows a polygonal approximation to the The values calculated from normal CSF classes are bigger
ventricle area with P4 and Plz as the left and right point than 0.1, and the ones from CSF classes containing abnormal
respectively. tissues are much smaller than 0.1. Figure 17 shows two
LI er al.: KNOWLEDGE-BASEDCLASSIFICATION AND TISSUE LABELING 747

abnormal CSF classes, their fuzzy symmetric values are 0.028

and 0.021 for (a) and (b) respectively.

VI. GRAYMATER
- 53 slices

According to the distribution of class centers in the T2

spectrum, gray matter classes reside in between white matter
class and CSF. So for normal slices, we can label all classes
between white matter and CSF in the T2 spectrum (without
counting skull tissues) as gray matter. On the other hand, for Normal
an abnormal slice, one of these classes which lies next to CSF
21 slices labeled
in the T2 spectrum may contain a tumor. This class is labeled
only when CSF is symmetric and the gaps in white matter Fig. 18. Hierarchical classification and tissue labeling.
along two sides of the ventricle area are neither negligible nor
significant enough to label white matter as abnormal. Such
gaps may be caused by either tumor or islands of gray matter
on both sides of the ventricle area near the two upper horns.
In the class next to CSF, if there is a big mass which happens
to be within the gap of white matter, the mass is labeled as a
tumor, because the two islands of gray matter do not appear in
this class. Our current implementation is a simplified version
of the conceptual strategy. The suspected image class of gray
matter is preprocessed by an opening operator with a 3 x 3
structural element. The bi-orthogonal thickness (BT) is used
to measure the size of a mass. Only connected components of
at least 100 pixels are classified by the following rule:
IF (The BT of a component is big (> 10 pixels))
THEN (Label the component as a tumor)
If there are no masses being labeled as tumors, the gaps in Fig. 19. Tissue labeling of normal slices. White: white matter. Black: gray
white matter are due to gray matter intrusion and all classes matter. Gray: CSF.
between white matter and CSF are labeled as gray matter.
not affect the classification of the other dozen slices which
VII. RESULTS were acquired before and after the enhancement. One normal
Fifty-three slices of MR brain data have been collected from slice was classified as abnormal in white matter matching.
GE Advantage 1.5 Tesla and Siemens 1.5 Tesla Magnetom. It suffered from significant data non-uniformity and white
For each patient the scanners generate from twenty-six to matter splitting cannot be detected. A binary tree in Fig.
thirty slices in the axial plane. The thickness of each slice 18 shows the hierarchical classification and tissue labeling
is 5 mm. We originally selected a viable slice as the slice of the tested slices. Normal classified slices are completely
closest to the center of the head in the axial plane, where labeled as shown in Fig. 19, where white matter is labeled as
the ventricles are clearly visible in images from volunteers, white, gray matter as black, and CSF as gray. Most abnormal
and later extended the selection to up to five consecutive classified slices are labeled partially depending on when a slice
slices from the same subject. The number of consecutive is classified as abnormal, as shown in Fig. 20.
slices fitting our qualitative model has been found to be up The results were evaluated by radiology experts. Each
to five depending upon the subject orientation. This is true labeled slice was randomly chosen to be shown to experts
for both patients and volunteers. The 3 to 5 slices that fit together with its three intensity images (Tl, PD, and T2).
our qualitative models cover a thickness of 1.5 to 2.5 cm. Therefore, different slices from a patient were evaluated at
Regardless of scanner parameters and time of scanning, we different times. Six (5 normal and 1 abnormal) of the fifty-three
randomly chose thirty-seven slices from seventeen patients and slices were identified to suffer from either data non-uniformity
sixteen slices from ten normal volunteers. There are from one or improper registration. In the remaining 47 slices, 17 were
to five consecutive slices from each patient or volunteer. normal classified and 30 abnormal classified. Except the
After running these slices through the system, thirty-two abnormal slice misclassified as normal, 16 normal classified
slices were classified as abnormal and twenty-one normal. slices were all rated as good representationsof intensity images
One abnormal slice with a small tumor was classified as for their labeling of white matter, gray matter, and CSF. For
normal. The small tumor did not lead to deformation in the four normal slices which suffered from data non-uniformity
white matter and was clustered into a gray matter class. The and were correctly classified as normal, the labeling of CSF
same slice acquired after gadolinium injection was correctly was good, but the labeling of white matter and gray matter was
classified as abnormal. But the gadolinium enhancement did easily identified as mixed. Since most of the abnormal slices
748 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

classified as abnormal. In the nine cases that two slices were

from a patient, seven of them were correctly classified as
abnormal, two of them were normal (as abnormal tissues did
I not appear in them) and correctly labeled.

VIII. DISCUSSION
Automatic segmentation of 2-D MR images of the human
brain is a complex task. This paper presents a hybrid approach.
An initial segmentation of MR data by the UFCM produces
ten classes or regions. The regions have much better semantic
~
meanings in MR brain images than edges [3], and knowledge-
based analysis can be effectively applied.
The class center distribution in the T2 spectrum and tissue
/ models are kemel knowledge of the system. An expert system
.I uses model-based recognition and image processing techniques
-. ._- on the ten classes to locate a landmark tissue and to look for
(C) (d) expected features. The process is repeated until a tumor is
Fig. 20. Tissue labeling of abnormal slices. (a) Complete labeling. (b) CSF
detected or all the tissues are labeled. The sequential stages
and tumor not labeled. (c) CSF and tumor not labeled. (d) Only white matter of locating one landmark after another make the system more
labeled reliable by placing priority on easily identified tissues. As long
as a landmark tissue is believed to be deformed significantly,
TABLE I the model-based recognition process is terminated, since there
1 slice 2 slices 3 slices 4 slices 5 slices Total is no way to model a tumor and to predict how a following
Volunteers 6 1 1 11 landmark tissue will appear. Furthermore, abnormal tissues
Patients 9 9 1 1 36 like edema may be clustered into classes of CSF or gray
matter, so model-based recognition techniques are effective
for labeling normal slices, but other non-model techniques are
were partially labeled, we manually labeled the remaining necessary to continue the labeling of abnormal slices.
classes not classified into gray matter, CSF, edema, and tumor Qualitative tissue models are appropriate for brain tissues
(Skull and white matter classes had already been labeled). whose shapes are too irregular to allow a quantitative model-
The thirty labeled abnormal slices were evaluated along with ing. The matching schema follows a default logic and looks
normal slices. The white matter labeling in all these slices, for an expected set of mismatches. If there is no expected
which was done by the expert system, was rated as a good mismatch between a model and its instance, a match is
representation. But in several slices abnormal tissues like assumed. This matching schema allows flexible extension.
edema were labeled as gray matter or CSF. The reason is When a new anomaly of a model is observed, rules can be
that gray matter, CSF, and abnormal tissues are closer to each added to capture the mismatch. The models and their match
other in intensity than to white matter. So an edema may be have been implemented by a set of thresholds, for instance,
clustered into a class of gray matter or CSF when a slice how wide a gap is in white matter, how symmetrical CSF is
is clustered into ten classes. This suggests that a class-based and how big a mass is in a particular class. These thresholds
labeling is not sufficient for abnormal slices. Therefore, the are reliable only for the slices of interest defined in this paper.
class-based labeling process terminates as soon as a significant For slices of other positions their qualitative models will differ
deformation is detected. However, the knowledge accumulated and dictate another set of thresholds.
in the process can be used by other approaches to further the The advantages of the proposed approach are demonstrated
analysis. One possible such method is to apply the UFCM on by the successful performance of the system on multiple slices
the image areas excluding skull tissues and white matter to from different volunteers and patients. Three to five slices
achieve a refined clustering which may help extract abnormal in the axial plane can be processed with the current model.
tissues from gray matter and CSF. Complete tissue labeling has been achieved in normal slices,
The qualitative models are insensitive to the exact slice but only partial tissue labeling in most abnormal slices. We
locations and work well on multiple slices from a patient. expect to label abnormal slices completely by applying the
Table I shows this information from the forty-seven slices UFCM to re-cluster regions of unclassified classes with the
which were evaluated by expert radiologists. Each entry is knowledge from previous analysis. One approach is to give
the number of K slices from a patient. the information about the known clusters (labeled tissues) to
Three slices from one volunteer and two slices from another UFCM, essentially letting the expert system provide some
were all correctly labeled. Four slices were taken from one supervision. This can be done in two ways. The first is to
patient without gadolinium enhancement; one of them was use the cluster centers of the known tissues for initialization
misclassified as normal. Five slices were taken from the same of clusters. Initialization has major effects on how fast FCM
patient with gadolinium enhancement; they were all correctly converges and how accurate a segmentation it provides. This
LI et al.: KNOWLEDGE-BASED CLASSIFICATION AND TISSUE LABELING 149

will guide the clustering to a different local minimum and it suggestions have improved the presentation of several parts of
has been found to often provide a good separating local mini- the paper. We thank Dr. M. L. Silbiger for his comments on
mum, if we focus on the mixed classes. The second possibility the labeled results.
is to use an algorithm like semi-supervised FCM (SFCM) with
a subset of the pixels near the cluster centers of the identified
tissues used as training data. SFCM [28] integrates the use of REFERENCES
some training data into an essentially unsupervised clustering
K. S. Fu and J. K. Mui, “A survey on image segmentation,” Pattern
algorithm. It has been shown to provide better segmentations. Recognition, vol. 13, pp. 3-16, 1981.
In fact, we have done some initial work on the use of P. Suetens, P. Fua, and A. J. Hanson, “Computational strategies for
what we call knowledge-based re-clustering 1291, which is object recognition,” ACM Computing Surveys, vol. 24, no. 1, pp. 5-61,
1992.
applicable to other types of problems also. Our initial work Zhenyu Wu and Richard Leahy, “Image segmentation via edge contour
has shown that the application of focus of attention with some finding: A graph theoretic approach,” in IEEE Computer Vision and
training or initialization data from identified tissues, where Pattern Recognition, pp. 613-619, 1992.
S. P. Raya, “Low level segmentation of 3-D magnetic resonance brain
the tissues are identified as in the work presented here, can images-a rule-based system,” IEEE Trans. Medical Imaging, vol. 9,
provide complete accurate labeling of abnormal MR images no. 3, pp. 327-337, 1990.
of the brain. S. Dellepiane, G. Venturi, and G. Vemazza, “A fuzzy model for the
processing and recognition of MR pathological images,” in IPMI’91,
Since intra-slice data nonuniformity is negligible for many pp. 444-457, 1991.
slices and inter-slice nonuniformity is significant, it is attrac- G. Gerig, I. Martin, and R. Kikinis et al., “Automating segmentation of
dual-echo MR head data,” in 12th Int. Con$ Information Processing in
tive to explore the possibility of qualitative models and similar Med. Imaging (IPMI’91). 1991.
expert systems for other small clusters of MR slices in the A. M. N a i f and M. D. Levine, “Low level image segmentation: An
axial plane. It has been pointed out by radiologists that a expert system,” IEEE Trans. Pattern Analysis and Machine Intelligence,
vol. 6, no. 5, pp. 555-577, 1984.
qualitative model for slices far from the top of the head may S. Tsuji and H. Nakano, “Knowledge-based identification of artery
be quite complex. Spatial knowledge of tissue regions may be branches in cine-angiograms.” in Int. Joint Conf Artificial Intell., pp.
propagated from slices in which segmentation has been done 710-718, 1981.
S. S. Stansfield, “Angy: A rule-based expert system for automatic
to slices under consideration. Where non-uniformity exists in segmentation of coronary vessels from digital subtracted angiograms,”
a slice in the axial plane, as slices are processed that are far IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 2,
from the center of the coil, it may be necessary to cluster pp. 188-199, 1986.
W. Menhardt, “Fuzzy clustering and components analysis,” in Ninth
on sub-regions where non-uniformity is less problematic. We SMRM, 1990.
will explore relying less on human knowledge and more J. Winter, “Automated computer tomography image analysis using
contour map topology,” IEEE Trans. Medical Imaging, vol. 3, no. 4,
on clustering characteristic knowledge for slices with fewer pp. 163-169, 1984.
clear anatomical features. If successful knowledge guided C. Li, D. B. Goldgof, and L. 0. Hall, “Towards automatic classification
segmentations of a full set of slices can be achieved, slice- and tissue labeling of MR brain images,” in Proc. Int. Workshop on
Structural and Syntactic Pattern Recogn., 1992.
based evidence and results from different expert systems can R. L. Cannon, J. V. Dave, and J. C. Bezdek, “Efficient implementation of
then be combined to obtain a 3-D tissue labeling without the fuzzy c-mean clustering algorithms,” IEEE Trans. Pattern Analysis
suffering inter-slice nonuniformity. The fact that the results and Machine Intelligence, vol. 8, no. 2, pp. 248-255, 1986.
L. 0. Hall and A. M. Bensaid et al., “A comparison of neural network
of this work can be used for a knowledge-based re-clustering and fuzzy clustering techniques in segmenting magnetic resonance
scheme holds the future possibility of completely labeled images of the brain,” IEEE Trans. Neural Networks, vol. 3 , no. 5, pp.
tissues in abnormal slices. The current system accurately labels 672482.
R. T. Chin and C. R. Dyer, “Model-based recognition in robot vision,”
normal images for screening, provided there is not large non- ACM Computing Surveys, vol. 18, no. I , pp. 67-108, 1986.
uniformity within a single slice, and provides a base for Version 4.3 Clips reference manual, tech, rep. JSC-22948, Artificial
Intelligence Section, Lyndon B. Johnson Space Center, 1989.
labeling tumor, edema and necrosis. J. Giarratano and G. Riley, Expert Systems: Principles and Program-
The expert system is organized around CLIPS [ 161, [ 171, ming. Boston: PWS-Kent, 1989.
a rule-based tool. Both the low-level image processing and H. N. Schnitzlein and F. Reed Murtagh, Imaging Anatomy of the Head
and Spine: A Photographic Color Atlas of MRI, CT, gross, and Mi-
high-level image analysis modules are written in C. Since croscopic Anatomy in Axial, Coronal, and Sagittal Planes. Baltimore:
CLIPS provides a convenient interface protocol with C, all Urban k Schwarzenberg, 2nd ed., 1990.
the modules are used as rule actions. It serves naturally as R. Reiter, “A logic for default reasoning,” Artificial Intelligence, vol.
13, no. 1 and 2, pp. 81-132, 1980.
an efficient rule-based shell for various image processing and A. P. Dhawan and L. Arata, “Knowledge-based 3D analysis from 2D
analysis purposes. Based on the pool of routines provided by medical images,” IEEE Engineering in Medicine and Biology, vol. IO,
such a shell, one can design sets of rules as shell scripts to no. 4, pp. 30-37, 1991.
B. Horn, Robof Vision. New York: McGraw-Hill, 1986.
form a specific image processing package as discussed in [30] J. Serra, Image Analysis and Mathematical Morphology. New York:
or to build an expert system around the pool of routines [31]. Academic Press, 1982.
A. K. Jain, Fundamentals of Digital Image Processing. Englewood
Cliffs, NJ: Pi-entice Hall, 1989.
R. M. Haralick, K. Shanmugam, and I. Dinstein, “Textual features for
ACKNOWLEDGMENT image classification,” IEEE Trans. Systems, Man, and Cybernetics, vol.
3, no. 6, pp. 610-621, 1973.
The authors thank Robert Velthuizen, Amine Bensaid, and H. Freeman, “Computer processing of line-drawing images,” ACM
Larry Clarke for providing MR brain data and helpful dis- Computing Surveys, vol. 6, no. 1, pp. 57-97, 1974.
G . Gallus and P. W. Neurath, “Improved computer chromosome analysis
cpssions. We thank Horst Bunke of University of Bern and incorporating preprocessing and boundary analysis,” Phys. in Med. and
Abe Kandel for valuable suggestions, and David Eggert whose Biol., vol. 15, no. 3, pp. 435-545, 1970.
750 IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. 12, NO. 4, DECEMBER 1993

[271 H.-J. Zimmermann, F u u y Set Theory and Its Application, 2nd edition. Image Processing and Biomed. Visualization, 1993. San Jose, CA.
Boston: Kluwer Academic, 1991. [30] F. Grimm and H. Bunke. “An expert system for software configuration
1281 A. Bensaid, L. 0. Hall, and J. C. Bezdek, “A partially supervised fuzzy and its application to computer vision,” in SPIE Applicarions of Artificial
c-means algorithm for segmentation of MR images,” in SPIE Conf: on Intelligence VII, pp. 172-183, 1989. Orlando, Florida.
the Science of Neural Networks Proc., 1992. Orlando, lX. [31] Z. [Link],T. Pun, and C. Pellegrini, “An expert system for guiding image
[291 C. Li, L. 0. Hall, and [Link], “Knowledge-based classification and segmentation,” Computerized Medical Imaging and Graphics, vol. 14,
tissue labeling of MR images of human brain,” in SPIE Conf: on Biomed. no. 1, pp. 13-24, 1990.