FACIAL EXPRESSION RECOGNITION

Subhash Chandra Gupta, Raj Aryan

Galgotias University

1. ABSTRACT- Emotion recognition is receiving lots of attention from researchers

from the last decade for improving the interaction between humans and machines.
Emotion recognition is the technique used for facial expression extraction. We
proposed a system to define different types of features to automatically detect
emotion from facial images using action units (AUs). The proposed system
automatically detects faces from the captured image and codes them with respect
to seven dimensions like Happy, Sad, surprise, fear, disgust, Anger and Neutral. In
this paper, we will explain facial expression and difference between expression and
emotions. This paper also focuses on previous research done on facial expression
recognition, different techniques for facial expression recognition and emotion
detection.
This paper also explains why facial expression recognition is important and what
can be future works or improvements for this system. This paper concludes every
fact about facial expression recognition from its definition to how it works to its
improvements.
Keywords: Facial expression, Emotion Recognition, Action Units, Facial Expression
Extraction, Neutral.

2. INTRODUCTION:
Human communication has two main aspects; Verbal and Non-Verbal.
FER systems are mainly based on Non-Verbal aspects of
communication. Non-verbal communication is expressed through
facial expressions. Facial expression, body movement and
physiological reactions are the basic units of non-verbal
communication. Human Facial Expressions are extremely essential in
social communication. Using mathematical algorithms plays a huge
role and can be used to describe the interpretation of human facial
characteristics. Recent advancement in Machine learning, Artificial
intelligence, automated face analysis, and pattern recognition have
made it possible to develop automatic face recognition systems to
address these applications.
Facial expression is one of the basic and powerful means for
humans to communicate their ideology and emotions. Facial
expression recognition (FER) has emerged as an important research
area over the last two decades. The FER system can be used in many
important applications such as driver safety, health care, maintaining
history, video conferencing, virtual reality, artificial reality and
cognitive science etc.
The facial expression recognition system presented in this
research work contributes a tough face recognition algorithm based
on the mapping of behavioral characteristics with the physiological
biometric characteristics. The physiological characteristics of the
human face with common to various expressions such as happiness,
sadness, fear, anger, surprise and disgust are associated with
geometrical structures which are restored as base matching
templates for the recognition system.
The behavioral features of this system associate the point of
view behind different expressions as classified base. The classified
bases are isolated as exposed and hidden categories in genetic
algorithmic genes. The gene training set assesses the expressional
uniqueness of individual faces and provides a strong expressional
recognition model.
There are a total 7 types of universal expression considered in
this system.
1. Anger: involves three main features- teeth revealing, eyebrows
down and inner side tightening, squinting eyes. The function is clear-
preparing for attack. The teeth are ready to bite and threaten
enemies, eyes and eyebrows squinting to protect the eyes, but not
closing entirely in order to see the enemy.
2. Disgust: involves wrinkled nose and mouth. Sometimes it even
involves tongue coming out. This expression mimics a person that
tasted bad food and wants to spit it out, or smelling foul smell.
3. Fear: involves widened eyes and sometimes open mouth. The
function- opening the eyes so wide is supposed to help increase the
visual field (though studies show that it doesn’t actually do so) and
the fast eye movement, which can assist finding threats. Opening the
mouth enables to breathe quietly and by that not being revealed by
the enemy.
4. Surprise: very similar to the expression of fear. Maybe because a
surprising situation can frighten us for a brief moment, and then it
depends whether the surprise is a good or a bad one. Therefore, the
function is similar.
[Link]: involves a slight pulling down of lip corners, inner side of
eyebrows is rising. Darwin explained this expression by suppressing
the will to cry. The control over the upper lip is greater than the
control over the lower lip, and so the lower lip drops. When a person
screams during a cry, the eyes are closed in order to protect them
from blood pressure that accumulates in the face. So, when we have
the urge to cry and we want to stop it, the eyebrows are rising to
prevent the eyes from closing.
6. Contempt: involves lip corner to rise only on one side of the face.
Sometimes only one eyebrow rises. This expression might look like
half surprise, half happiness. This can imply a person who receives
this look is surprised by what he said or did (not in a good way) and
that we are amused by it. This is obviously an offensive expression
that leaves the impression that a person is superior to another
person.
7. Happiness: usually involves a smile- both corners of the mouth
rising, the eyes are squinting and wrinkles appear at eyes corners.
The initial functional role of the smile, which represents happiness,
remains a mystery. Some biologists believe that a smile was initially a
sign of fear. Monkeys and apes clenched teeth in order to show
predators that they are harmless. A smile encourages the brain to
release endorphins that assist lessening pain and resemble a feeling
of well-being. Those good feelings that one smile can produce can
help dealing with the fear. A smile can also produce positive feelings
for someone who is witness to the smile, and might even get him to
smile too. Newborn babies have been observed to smile involuntarily,
or without any external stimuli while they are sleeping. A baby’s
smile helps his parents to connect with him and get attached to him.
It makes sense that for evolutionary reasons, an involuntary smile of
a baby helps creating positive feelings for the parents, so they
wouldn’t abandon their offspring. The advantage of this
representation is that it is possible to express as numbers the
continuous Scale from “mildly irritated” to “incandescent with rage”
and also to capture the shades of grey between related pairs of
emotions.
One major thanks to detect emotions is by analyzing the voice
by setting parameters in areas like Tone, pitch, pace, volume etc. but
this is often a posh algorithm to figure out and takes tons of
Computing time also at cost. This system demands an Interactive
Voice Response (IVR) System which helps only to detect emotions
while on talk.
Back within the 80's, Ekman was already referred to as a
specialist for study of facial expressions, when approached by a
psychiatrist, asking if Ekman has the power to detect liars. The
psychiatrist wanted to detect if a patient is lying by threatening to
suicide. Ekman watched a tape of a patient over and once again,
trying to find a clue until he found a blink of an eye of desperation,
meaning that the patient's threat wasn't empty. Since then, Ekman
has found those critical split seconds in almost every liar's
documentation. The leading character within the TV series "Lie to
me" is predicated on Paul Ekman himself, the person who dedicated
his life to reading people's expressions- the "human polygraph". The
research of facial expressions and emotions began a few years before
Ekman's work. Charles Darwin published his book, called "The
Expression of the Emotions in Man and Animals" in 1872. This book
was dedicated to nonverbal patterns in humans and animals and to
the source of expressions.
Darwin's two former books- "The Descent of Man, and Selection
in reference to Sex" and "On the Origin of Species" represented the
thought that man didn't came into existence in his present condition,
but during a gradual process- Evolution. This was, of course, a
revolutionary theory since in the middle of the 19th century no one
believed that man and animal "obeyed the same rules of nature".
Darwin's work attempted to find parallels between behaviors and
expressions in animals and humans. Ekman's work supports Darwin's
theory about universality of facial expressions, even across cultures.
The main idea of "The Expression of the Emotions in Man and
Animals" is that the source of nonverbal expressions of man and
animals are functional, and not communicative, as we may have
thought. This means that facial expressions creation wasn't for
communication purposes, except for something else. An important
observation was that individuals who were born blind had similar
facial expressions to individuals who were born with the power to
ascertain. This observation was intended to contradict Sir Charles
Bell's idea (a Scottish surgeon, anatomist, neurologist and
philosophical theologian, who influenced Darwin's work), who
claimed that human facial muscles were created to provide humans
the unique option to express emotions, meaning, for communicative
reasons.
And from here all the work for facial expression recognition is
recognized and is carried further. Recent years there are a lot more
ways to discover which one of them is explained in my paper. there
are lots of facts about which we will talk about in the coming
sections.
3. DIFFERENCE BETWEEN EXPRESSION (mood) AND
EMOTION:
Moods can last for hours while emotions last anywhere from seconds
to minutes, at most. This is why it’s typically easier to identify
emotional triggers but difficult to pinpoint the trigger for our moods.
Moods also don’t have their own unique facial expressions whereas
the universal emotions do.
3.1 EXPRESSION:
● We can’t trigger expressions as true expressions are hard to
identify.
● Expression can be faked.
● expression depends on mood. It has a duration of hours to
minutes till psychological state changes.
● Expressions are important for communication between humans.
● Expressions increase understanding among each other.
● Expressions can be provoked by a better level of understanding.
● The eyes viewed as the most common feature of understanding
expression as eyes express a lot. Eye contact is the most
common interpersonal communication.
3.2 EMOTION:
● A strong feeling comes from a particular state of situation,
moment, mood or relationship.
● The affective aspect of consciousness.
● Emotion is a state of pleasure or displeasure.
 ● Emotion is a complex state of feeling which affects our physical
or psychological behavior.
● Emotion has universal face expression. It has a duration of
seconds or minutes till our behavioral state changes.
● Emotions are easily identifiable. It can be triggered anytime if
we know weak points.
● Emotion can’t be faked.
4. EMOTION RECOGNITION TECHNIQUES:
The emotion recognition is basically a way to identify human
feelings. the accuracy of emotion recognition improved widely while
combining the estimation of facial expression and feature extraction.
There are mainly 4 subfields of emotion recognition:
● emotion recognition in audio
● emotion recognition in video
● emotion detection in text
● emotion detection in facial expression
These are subfields by which we can detect human emotions.
Emotion detection with facial expression is the field in which we
will carry forward our paper. Facial expression is the best way
to detect human emotion, that's the reason facial expression
recognition is the important subfield for emotion detection.
There are three categories in which emotion detection
techniques are classified to tell which type of emotion it is:
4.1 knowledge-based technique:
This is basically the utilization of domain for semantic and
syntactic features of language in order to recognize emotion
type. It is basically the process in which we use all our
knowledge to gain similarity to determine emotion type. This is
of two types:
4.1.1 Dictionary based approach.
4.2.2 corpus-based approach.
4.2 statistical based technique:
This technique uses some of the machine learning algorithms.
Some of the algorithms used for this technique are support
vector machine, naive Bayes, maximum entropy, deep learning,
artificial neural network, convolutional neural network, long
short-term memory, extreme machine learning, computer
vision, speech recognition and natural learning processing.
4.3 hybrid approach:
This is the combination of statistical based technique and
knowledge-based technique. It shows computational complexity
during the classification process.
5. FACIAL EXPRESSION RECOGNITION ALGORITHMS:
ALGORITHMS FOR FACIAL EXPRESSION RECOGNITION
There are various methods in which facial expression recognition
systems work, generally facial features are compared from a given
image with faces within a database. Different algorithms are used
to achieve different facial expression recognition levels, generally
with the use of machine learning and Bio-metric Artificial Intelligence
are used to analyze patterns based on the person's facial textures and
shape.
Some of the commonly used Algorithms are-
5.1 3-Dimensional recognition
The use of 3D sensing has the potential for greater recognition
accuracy than 2D. Three-dimensional face recognition techniques use
3D sensors to capture information about the shape of a face. This
information is then used to identify distinctive features on the surface
of a face, such as the contour of the eye sockets, nose, and chin. 3-D
has much Greater flexibility in capturing an image also it can create
more accurate images, generating a faster database match.
5.2 Skin texture analysis
Another emerging trend uses the visual details of the skin, as
captured in standard digital or scanned images. This technique,
called Skin Texture Analysis, turns the unique lines, patterns, and
spots apparent in a person's skin into a mathematical space
Surface Texture Analysis works much the same way facial recognition
does. A picture is taken of a patch of skin, called a skin print That
patch is then broken up into smaller blocks. Using algorithms to turn
the patch into a mathematical, measurable space, the system will
then distinguish any lines, pores and the actual skin texture.
5.3 Thermal cameras
A different form of taking input data for face recognition is by using
thermal cameras, by this procedure the cameras will only detect the
shape of the head and it will ignore the subject accessories such as
glasses, hats, or makeup. Unlike conventional cameras, thermal
cameras can capture facial imagery even in low-light and night-time
conditions without using a flash and exposing the position of the
camera However, a problem with using thermal pictures for face
recognition is that the databases for face recognition is limited.

5.4 Facial recognition combining different techniques

As every method has its advantages and disadvantages, technology
companies have amalgamated the traditional, 3D recognition and
Skin Textual Analysis, to create recognition systems that have higher
rates of success. Combined techniques have an advantage over other
systems. It is relatively insensitive to changes in expression, including
blinking, frowning or smiling and has the ability to compensate for
moustache or beard growth and the appearance of eyeglasses. The
system is also uniform with respect to race and gender.
 According to Darwin, there are three "chief principles", which
are three general principles of expression:
1. The first one is called the "principle of serviceable habits". He
described it as a habit that was reinforced at the beginning and then
inherited by offspring. For example: he noticed a serviceable habit of
raising the eyebrows in order to increase the vision field. He
connected it to a person who is trying to remember something, while
performing those actions, as though he could "see" what he is trying
to remember.
2. The second principle is called "antithesis". Darwin suggested that
some actions or habits might not be serviceable themselves, but
carried out only because they are opposite in nature to a serviceable
habit.
3. The third principle is called "The principle of actions due to the
constitution of the Nervous System". This principle is independent
from will or a certain extent of habit. For example: Darwin noticed
that animals rarely make noises, but in special circumstances, like
fear or pain they respond by making involuntary noises.

6. WHY FACIAL EXPRESSION RECOGNITION IS IMPORTANT:

Human facial expressions are nonverbal aspects of communication
and a good way to understand the other people. facial expression
recognition extracts and analyzes audio, video, feed, image etc. to
understand the and give outcome data of emotion.
● It is very important for computer-based programs to
understand human expression and emotional state to respond
to them in that emotion only.
● It is very because of its ability to mimic human emotion and
give it a way to understand humans in a better way.
● This approach is helpful for making human friendly robots in
the future.
● It helps computer-based programs to set interpersonal
relations with other people.
● It makes our life independent of human responses as we can
get a good friend in our computer program.
● It is very important in fields of computer vision and artificial
intelligence.
● It is also important on human level as well as it increases the
understanding level of human to human.
● Understanding the human facial expressions and the study of
expressions has many aspects, from computer analysis, emotion
recognition, lie detectors, airport security, nonverbal
communication and even the role of expressions in art.

7. HOW FACIAL EXPRESSION RECOGNITION WORK?

Facial expression recognition (FER) is important for social
interactions and effective communication. The importance of facial
expression systems is widely recognized in social interaction and
social intelligence. Utilizing the data from faces of humans has
always been a tough task. The FER system has been an active
research topic since the 19th century. The facial expression
recognition system was introduced in 1978. The main issue of
building a facial expression recognition system is face detection and
alignment, image normalization, feature extraction, and classification.
following is the flow chart how facial expression recognition work:

fig. 1. flowchart of facial expression recognition system.

The FER system includes the major stages such as face image pre-
processing, feature extraction and classification.
7.1 Preprocessing
Pre-processing is a process which can be used to improve the
performance of the FER system and it can be carried out before the
feature extraction process. Image pre-processing includes different
types of processes such as image clarity and scaling, contrast
adjustment, and additional enhancement processes to improve the
expression frames.
7.2 Feature extraction
Feature extraction is finding and depicting positive features of
concern within an image for further processing. Feature extraction
includes two types and they are geometric based and appearance
based. The geometrically based feature extraction comprises eye,
mouth, nose, eyebrow, other facial components and the appearance-
based feature extraction comprises the exact section of the face.
Feature extraction transforms the pixel image data into a higher-level
representation of motion, appearance of inner structures. Feature
extraction methods extract holistic features from the initial
representation features of face. They map an input representation
onto a lower dimensional space to discover a delegate structure from
the representation
7.3 Classification
The classification is also one of the important processes in which the
above-mentioned expressions such as smile, sad, anger, disgust,
surprise, and fear are categorized. Expression categorization is a
process of assigning observed data to one of predefined facial
expression categories by a classifier.
8. FUTURE SCOPE AND CONCLUSION:
There are many areas where facial expression recognition is
gaining interest day by day. There are many areas which still need to
be improvised and updated.
Here are the challenges which faced in current system:
● Illumination variation: illumination variation leads by change in
lightning, camera position and face position. by changing the
lightning condition in every stage of feature extraction can
improvise this issue.
● Subject-dependence: subject-dependence means that the
system is only able to recognize expressions of pre-trained
human faces. This is a major and most critical problem which
needs to be improved to make the system more reliable.
● Head pose: as in most lab experiments, main view to detect
expression in frontal view but in real there is not always front
view available so it is hard to detect expression in those
conditions. This is a problem which can be improved in future
by increasing large pose selection methods where classifier are
designed not to generalize the features.
● It can be improved on the basis of accuracy, reliability and
security.

Here is the future of facial expression recognition:

● Large Pose Variation: this is the future technique used to
minimize the issue of head pose. In this we try to increase the
pose area to train the system about different views of faces and
extraction of features from that view.
● Multimodal Sensor Data Collection: in this we try to examine
different lightning angles with different sensors so that all the
data to be collected and combined for feature extraction.
● Facial Emotion Classifier: this is an approach to use a facia
classifier to classify the features more accurately so that we can
reduce generalization of features which create subject
dependence.
● Fusion Method Selection: in this we detect features which
undergo a different process of fusion method selection so that
extraction can be accurately done.
● There are many evolution genetic frameworks which suit the
future generation application of facial expression recognition
 such as criminal detection, in hostage retrieval and negotiation
department.

This paper now concludes every fact about facial expression

recognition systems. There is a brief discussion about facial
expression recognition and its future. What is facial expression
recognition and how does it advance our ever-growing technological
world. There is also mentioned why this technology is needed for our
future. This technology promises the best yet reliable source to
recognize facial expression and understanding of expression. In this
paper there was a difference between expression and emotion which
is very important to understand so that the system will know how to
react to what. Here are mentioned techniques of facial expression
recognition and how it can be done. This system promises a reliable
future to AI by fulfilling its dream of making robots like humans. By
implementing this system inside robots increases the dependency of
humans on robots because this can be a big step to make humans and
robots best friends or can say they both can co-exist in nature without
any disaster. Lastly, there is also mentioned the areas of
improvement for the current system and future of this facial
expression recognition system.
9. ACKNOWLEDGEMENT:
I hereby thanks to my co-author MR. Shubhas Chandra Gupta
who is my mentor and guide as well for this research paper. Thank
you Mr. Shubham Kumar and Mr. Ashutosh Upadhyay for being
there while surveying every part of this paper. Special thanks to
Galgotias University for providing me every possible help to complete
this paper. I am very glad that I got to be the part of such great work
and my contribution was appreciated by teachers and mentors.
10. REFERENCES:
[1] Nithya Roopa S.; (2019); “ Emotion Recognition from Facial Expression using Deep
Learning” ; In: International Journal of Engineering and Advanced Technology (IJEAT)
ISSN: 2249 – 8958, Volume-8 Issue-6S, August 2019;
[2] Jyoti Kumari, R. Rajesh, K.M. Kumari;(2015) “ Facial Expression Recognition: A
Survey” In: Second International Symposium on Computer Vision and the Internet of
Procedia Computer Science Volume 58, 2015, Pages 486-491;
[3] Yunxin Huang , Fei Chen, Shaohe Lv and Xiaodong Wang;(2019); “Facial
Expression Recognition: A Survey” ; In: Symmetry 2019, 11, 1189;
doi:10.3390/sym11101189;Available at : [Link]
8994/11/10/1189/htm#cite
[4] I. Michael, W.R. Sam Emmanuel;(2018); “ A Survey on Human Face Expression
Recognition Techniques” ; In: Journal of King Saud University - Computer and
Information Sciences ,
[Link]
[5] “Emotion Recognition” by wikipedia online available at
[Link]
[6] “ How to Measure Emotions and Feelings (And the Difference Between Them)” by
Bryn Fransworth ; Available online at : [Link]
emotions/
[7] “ Mood vs. Emotion: Differences & Traits” by Paul Ekman ; Available at :
[Link]
[8] Najmeh Samadiani, Guangyan Huang, Borui Cai, Wei Luo, Chi-Hung Chi, Yong
Xiang, Jing He; “ A Review on Automatic Facial Expression Recognition Systems
Assisted by Multimodal Sensor Data” In: Sensors (Basel) 2019 Apr; 19(8): 1863.
Published online 2019 Apr 18. doi: 10.3390/s19081863;
[9] Article from book from Shodhganga page no. 139-140 ; Available at :
[Link]
[10] Krcadinac, U., Pasquier, P., Jovanovic, J., & Devedzic, V. ;(2013); “Synesketch: An
open source library for sentence-based emotion recognition”;. In: IEEE Transactions on
Affective Computing, 4(3), 312-325;
[11] Russell, James (1994); "Is There Universal Recognition of Emotion From Facial
Expression? A Review of the Cross-Cultural Studies"; Psychological Bulletin. 115 (1):
102–41;
[12] Thayer, Stephen; (1980); "The effect of expression sequence and expressor identity
on judgments of the intensity of facial expression". Journal of Nonverbal Behavior. 5
(2): 71–79.