n deep learning, "tangent distance" refers to a technique often used for
comparing images or high-dimensional data by measuring the similarity
between their underlying manifold structures. This method was
introduced in the context of nearest-neighbor classification and involves
computing distances between images based on their "tangents" or local
deformations, rather than pixel-wise differences.
Here's a breakdown of the concept:
1. Manifold Hypothesis:
The manifold hypothesis in machine learning suggests that high-
dimensional data (like images) lie on a much lower-dimensional
manifold. In simpler terms, though the data lives in a high-dimensional
space, it can be represented using fewer dimensions due to inherent
structure (e.g., rotations, translations, and scaling).
2. Tangent Vectors:
For a given data point (such as an image), the idea is that small
transformations (like rotation, scaling, or slight shifts) can be represented
by vectors tangent to the manifold that the data lies on. These
transformations lead to what is called "tangent vectors."
3. Tangent Distance:
Tangent distance is the distance between two points (e.g., images) in the
direction of their tangent spaces, rather than just the Euclidean or pixel-
wise distance. This is more effective for tasks where the relevant
information is in how the images can transform (e.g., slight rotations of
handwritten digits) rather than in exact pixel values.
4. How It Works:
First, compute the tangent space (set of possible local
deformations) for each image or data point.
Then, instead of directly computing the distance between the two
data points, compute the distance between their corresponding
tangent spaces.
This allows the distance measure to be invariant to small
transformations, making it more robust to variations in the data.
5. Applications:
Tangent distance has been applied in tasks like handwritten digit
recognition (e.g., MNIST), where small distortions (like rotations and
scale changes) in images shouldn't affect classification results. It can also
be useful in other pattern recognition tasks where the underlying
transformations of the data are important.
In deep learning, although tangent distance is not as common as other
methods like convolutional neural networks (CNNs), it is still a useful
concept when dealing with tasks where invariance to small
transformations is important.
Tangent Prop and Manifold Tangent Classifier are both techniques
used in machine learning, particularly for improving generalization in
neural networks by incorporating geometric properties of the data.
1. Tangent Prop:
Tangent Prop was introduced by Yann LeCun et al. in the paper “Tangent
Prop – A Formalism for Specifying Selected Invariances in an Adaptive
Network” (1992). The main idea behind Tangent Prop is to train neural
networks to be invariant to small transformations of the input data (such
as translation, rotation, scaling, etc.) by explicitly penalizing the
network’s sensitivity to these changes.
Key Concepts:
Invariances: Many real-world tasks involve inputs that undergo
transformations. For example, an image of a digit may be shifted or
rotated slightly, but we want the neural network to classify the digit
correctly regardless of these small changes.
Tangent Vector: For each data point, a tangent vector is
computed. This vector represents the direction in which the data
point changes due to small transformations.
Regularization: Tangent Prop adds a regularization term to the
loss function, which penalizes the network for having a high
derivative along the direction of the tangent vector. This
encourages the network to be less sensitive to small perturbations
in the input, leading to better generalization.
How it Works:
1. Compute tangent vectors for small transformations of the input
data.
2. The neural network is trained not only to minimize the primary loss
(such as classification error), but also to ensure that its output
doesn’t change significantly in the direction of the tangent vectors.
This is done by adding a term to the loss function that penalizes
large gradients in those directions.
Benefits:
Encourages the network to learn representations that are invariant
to certain transformations, improving robustness and
generalization.
Particularly useful in tasks like image recognition where the input
data may undergo small changes.
2. Manifold Tangent Classifier (MTC):
The Manifold Tangent Classifier is a method that builds on the idea of
Tangent Prop but emphasizes learning the structure of the data manifold
in high-dimensional spaces. It was introduced in the paper "Learning
Invariant Features by Harnessing the Freedom of Transformation" by
Goodfellow et al. (2013).
Key Concepts:
Data Manifold: In high-dimensional spaces, data often lies on a
lower-dimensional manifold. For example, images of digits may
exist in a high-dimensional space, but the space of valid images
(e.g., digit "2" written in different styles) is much smaller and
constrained.
Autoencoders: MTC often uses autoencoders to help learn the
structure of the data manifold. An autoencoder is a neural network
trained to compress data into a low-dimensional latent space and
then reconstruct it, implicitly learning the structure of the data.
Manifold Tangent: MTC computes tangent vectors to the data
manifold (using autoencoders) and ensures that the classifier is
invariant along these tangent directions.
How it Works:
1. Use an autoencoder to learn a latent representation of the data and
compute tangent vectors along the manifold.
2. Train the classifier to be invariant to perturbations in the tangent
directions of the manifold, similar to how Tangent Prop works.
3. This results in a classifier that is robust to variations within the data
manifold (e.g., different styles of writing the same digit) but
sensitive to variations off the manifold (e.g., noise or invalid data
points).
Benefits:
Improves generalization by explicitly leveraging the structure of
the data manifold.
More effective than simple regularization techniques like L2
regularization or dropout because it directly incorporates the
geometry of the data.
Particularly useful in unsupervised or semi-supervised learning,
where the amount of labeled data is limited but the underlying
structure of the data can be learned from unlabeled examples.
Summary:
Tangent Prop trains neural networks to be invariant to small
transformations by penalizing sensitivity to these transformations.
Manifold Tangent Classifier extends this idea by learning the
underlying data manifold and ensuring invariance along its tangent
directions, often using autoencoders to model the manifold.
Both methods improve generalization by incorporating geometric
information into the training process.
