CHAPTER VII
Semi-supervised Learning
Dr. K. Rajkumar,
Dean of International Relations,
Associate Professor,
Department of Computer Science
Bishop Heber College
Semi-supervised Learning
• Supervised methods use labeled data for training, unsupervised
methods are applied on unlabeled data and have to determine feature
importance by themselves.
• Semi-supervised methods follow a hybrid approach using a small
number of labeled instances and a large number of unlabeled
instances.
Why Semi-supervised Learning?
• Labeling data is a time consuming and costly approach, is often
performed by a data scientist.
• Manual labeling can also introduce human bias.
Dr. K. Rajkumar, Dept of Computer Science 2
Semi-supervised Learning
• Semi-supervised methods perform considerably better using unlabeled
data during training which improves accuracy than unsupervised
methods.
• However, using semi-supervised methods is not always possible and
often it is hard to know the distribution of the unlabeled data.
• There are two basic semi-supervised approaches: Transductive and
inductive learning.
• The transductive learning tries to use the labeled data in order to infer
the labels of the unlabeled data.
• The inductive learning tries to deduce rules from the labeled data that
can then be applied to the unlabeled data set.
Dr. K. Rajkumar, Dept of Computer Science 3
Semi-supervised Learning
• Transductive approach steps:
➢First the learner is trained, such as na¨ıve Bayes classifier, to learn the classes.
➢Applying the trained learner to assign class probabilities to the unlabeled data.
This step is called the “expectation” step.
➢Then, training a new learner using the labels of all the data. This step is called
the “maximization” step.
➢The steps are repeated until the model does not produce a different estimate
anymore.
➢This procedure is called the expectation maximization algorithm or EM
algorithm. Each expectation maximization iteration generalizes the model
more.
➢The expectation maximization procedure guarantees finding model parameters
that have equal or greater likelihood at each iteration.
Dr. K. Rajkumar, Dept of Computer Science 4
Expectation Maximization
• Expectation maximization (EM), also called expectation
maximization cluster analysis, is a method to solve maximum
likelihood function when some of the variables in the model cannot be
directly observed, i.e., latent variables.
• Expectation maximization assumes that the data is composed of
multiple multivariate normal distributions, which is a strong
assumption.
• It iteratively tries to find an optimal model by alternatingly improving
the model and the object assignment to the model.
Dr. K. Rajkumar, Dept of Computer Science 5
Expectation Maximization
• Under certain conditions we might still be able to determine the means
for each group by iteratively applying expectation maximization.
• The expectation maximization is used to determine the mean and
standard deviation parameters for each group.
• To verify whether there is some structure in the class and not just
random data, the algorithm can be applied to single classes.
• The expectation maximization algorithm tends to be very slow.
• Especially with high dimensional data, the expectation step can be
very slow.
• Also, the algorithm can get stuck in a local maxima that is far from the
global maxima.
Dr. K. Rajkumar, Dept of Computer Science 6
Pseudo Labeling
• Pseudo labeling is a simple and efficient technique used for semi-
supervised learning.
• It is also used in deep learning.
• In fact, pseudo labeling can be used for most neural networks and
training methods.
• In semi-supervised learning, the features are learned from the labeled
data.
• It its an advantage of the information in the unlabeled data to get a
better understanding of the structure of the data.
• Pseudo labeling can be used to learn from the unlabeled data.
Dr. K. Rajkumar, Dept of Computer Science 7
Pseudo Labeling
• Pseudo labeling goes through following steps:
i. Train a model or several models using the labeled data set. The
training data set might have to be manually labeled.
ii. The model which performed best is then used on the unlabeled
data to predict the class.
iii. Combine the training set with the labels and the one with the
pseudo labels.
iv. Train the model like before but with the combined data sets.
Dr. K. Rajkumar, Dept of Computer Science 8
Pseudo Labeling
• Deep learners often use unsupervised methods for pre-training.
• The initial weights of the deep neural network are initialized by
applying layerwise unsupervised training.
• After the weights are initialized, they are fine-tuned using labeled data
and the backpropagation algorithm in a supervised fashion.
• This also works using semi-supervised methods.
• In many cases, even when using older approaches, such as the na¨ıve
Bayes classifier, we can obtain superior performance by adding
unlabeled data and using semi-supervised learning.
Dr. K. Rajkumar, Dept of Computer Science 9
