Computation and Cognition Assignment 3
Abhishek Mandlik
November 15, 2024
1 Introduction
In the course of this study, I concentrated on basic principles of machine
learning, particularly supervised learning, binary classification, and linear
models. While researching topics such as linearly separated data from the
delta rule and the perceptron learning rule, I acquired a clearer picture of
the processes models apply to learn how to classify the various classes in a
given dataset. This process expanded my understanding of the concepts and
processes that underpin machine learning, its code in practice, the visual
apparatus, and visual methods for evaluating the qualities of the models.
2 Understanding Linearly Separable Data and
Dataset Generation
The learning process required the creation of a linearly separable dataset,
which enabled the use of linear classifying algorithms such as the delta and
perceptron learning rules. Creating linearly separable data is important be-
cause it helps a linear model create a clear dividing line between classes which
means the model will be able to converge to an optimal solution. A linear
model may not be able to perform adequately in situations where the classes
are highly overlapping in which case, the classes can be barely separated,
and will lead to poor performance and lack of convergence.
Example of Linearly Separate Data-set
1
Figure 1: DataSet
3 Delta Learning Rule and Weight Adjust-
ment
The formation of the delta rule, an essential part of supervised learning, was
about the most important aspects of this particular project. I saw how itera-
tively updating model weights allows one to reduce the discrepancy between
actual and expected outputs. This procedure strengthened my knowledge of
the gradient descent algorithm generally used in the field of machine learn-
ing, which is consistent with how the delt rule works. It adjusts the weights
by moving in the opposite direction of the error gradient.
One also needs a learning rate to make use of the delta rule effectively, for
that affects how much each weight is altered, therefore it serves to enhance
speed while simultaneously providing stability. Using this idea in code en-
abled me to see how different rates of learning alter the convergence and the
stability of the learning cycle. This concept has a wider bearing in neural
networks which have similar gradient based learning schemes.
2
Figure 2: Final Output: Delta Rule
Figure 3: Weights in Delta
3
4 Perceptron Learning Rule and Binary Clas-
sification
I implemented the perceptron learning rule to classify the dataset, building
on the delta rule. The perceptron is one of the simplest and earliest binary
classifiers, making it an excellent starting point for understanding linear clas-
sification. Unlike the delta rule, which gradually adjusts weights to minimize
error, the perceptron uses a binary decision threshold. It makes predictions
by assigning a label of either -1 or +1 based on whether the weighted sum
of the inputs surpasses a certain threshold.
By using the perceptron learning rule, I gained valuable insights into
how adjusting the weights impacts the decision boundary. It reinforced the
importance of separating data in a linear manner for the model to make
accurate predictions. However, this exercise also revealed a key limitation of
the perceptron: it only works when the data is linearly separable. When the
data is not linearly separable, the perceptron will fail to find a solution, as
it is unable to handle more complex patterns.
This limitation helped me understand why more advanced algorithms,
like support vector machines (SVMs) and neural networks, are necessary for
non-linear problems. These algorithms are designed to handle the complexity
of real-world data, where linear separation is not always possible.
4
Figure 4: Perceptron
Figure 5: Weights in Perceptron
5
5 Weight Initialization, Convergence, and Learn-
ing Rate
A key takeaway from this process was understanding the impact of weight
initialization and convergence. By starting with random weights, I observed
how different initializations could affect the number of iterations required for
the model to converge. Convergence occurs when the weight updates become
minimal, signaling that the model has stabilized and can generalize well.
I also gained a deeper understanding of the learning rate (n). When the
learning rate was set too high, the weights overshot the optimal solution,
causing them to oscillate without converging. On the other hand, a lower
learning rate led to more gradual and stable updates, but required more it-
erations to reach the solution. This trade-off between the learning rate and
convergence speed emphasized the importance of carefully tuning hyperpa-
rameters to ensure both effective and efficient model training.
6 Visualization and Decision Boundary In-
terpretation
The visualization of the dataset and decision boundaries provides a clear
understanding of how different learning rules influence the separation of lin-
early separable data. By plotting the initial dataset and observing the deci-
sion boundaries learned by both the Delta Rule and the Perceptron Rule, we
can interpret how each algorithm classifies the data. The decision boundary
represents the threshold where the model switches between the two classes,
essentially distinguishing one class from the other. The Delta Rule produces
a more gradual adjustment of weights, leading to a smooth decision bound-
ary, while the Perceptron Rule only makes adjustments when errors occur,
resulting in a more step-like boundary. Both algorithms successfully sepa-
rate the data in this case, but the way they adjust their boundaries reveals
their respective learning approaches and strengths. This visualization helps
to grasp the relationship between the learning process and the boundary that
ultimately classifies new data points.
6
7 Conclusion
Overall, this project was a valuable exercise in understanding machine learn-
ing fundamentals. From data generation to model training and visualization,
I learned the principles behind supervised learning and how linear classifiers
operate. Working with the delta and perceptron learning rules provided prac-
tical insights into weight adjustment and optimization, which will be useful
as I explore more advanced machine learning algorithms. This project high-
lighted machine learning development’s iterative and modular nature, where
each part of the codebase serves a critical role in model performance and
interpretability.
