Affective Brain–Computer

Interfaces (aBCIs): A Tutorial

This tutorial guides the reader into the basics of brain–computer interfaces (BCIs)
leading into “closed-loop” affective BCIs that include the chain from analysis to
brain stimulation.
By D ONGRUI W U , Fellow IEEE, B AO -L IANG L U , Fellow IEEE, B IN H U , Fellow IEEE,
AND Z HIGANG Z ENG , Fellow IEEE

ABSTRACT | A brain–computer interface (BCI) enables a user I. I N T R O D U C T I O N

to communicate directly with a computer using only the cen- Affects, including moods and emotions, are pervasive in
tral nervous system. An affective BCI (aBCI) monitors and/or our everyday life and essential in human cognition, com-
regulates the emotional state of the brain, which could facil- munication, and decision-making [1]. They are also very
itate human cognition, communication, decision-making, and important in human–machine interactions. For example,
health. The last decade has witnessed rapid progress in aBCI one of the founding fathers of artificial intelligence, Prof.
research and applications, but there does not exist a com- Alan Minsky [2], argued early in 1988 in his seminal
prehensive and up-to-date tutorial on aBCIs. This tutorial fills book The Society of Mind that “the question is not whether
the gap. It introduces first the basic concepts of BCIs and intelligent machines can have any emotions, but whether
then, in detail, the individual components in a closed-loop machines can be intelligent without any emotions.”
aBCI system, including signal acquisition, signal processing, Affective computing, first proposed by Prof. Rosalind
feature extraction, emotion recognition, and brain stimulation. Picard in 1995 [3], is “computing that relates to, arises
Next, it describes three representative applications of aBCIs, from, or influences emotions.” It quickly became an inter-
i.e., cognitive workload recognition, fatigue estimation, and disciplinary research area of computer science, psychology,
depression diagnosis and treatment. Several challenges and cognitive science, and so on. As shown in Fig. 1, the
opportunities in aBCI research and applications, including brain biannual International Conference on Affective Computing
signal acquisition, emotion labeling, diversity and size of aBCI and Intelligent Interaction (ACII) was initiated in 2005 and
datasets, algorithm comparison, negative transfer in emotion the affiliated aBCI Workshop in 2009. A dedicated journal,
recognition, and privacy protection and security of aBCIs, are IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, was established in
also explained. 2010. The Association for the Advancement of Affective
Computing (AAAC) was founded in 2011. As of April 20,
KEYWORDS | Affective computing; brain–computer inter-
2023, on Google Scholar, about 1200 researchers choose
face (BCI); emotion recognition; emotion regulation; machine
affective computing as one of their areas of interest, and
learning.
about 82 000 publications included “affective computing”
in their titles or texts.
Manuscript received 4 October 2022; revised 21 April 2023; accepted 11 May Humans can recognize and display emotions through
2023. Date of publication 31 May 2023; date of current version 11 October 2023.
facial expressions [4], speech [5], gestures [6], texts [7],
This work was supported in part by the STI 2030—Major Project 2021ZD0201300
and in part by the Hubei Province Funds for Distinguished Young Scholars under and their combinations [8]. These are also important
Grant 2020CFA050. (Corresponding authors: Bin Hu; Zhigang Zeng.)
modalities in affective computing. Physiological signals,
Dongrui Wu and Zhigang Zeng are with the Key Laboratory of the Ministry of
Education for Image Processing and Intelligent Control, School of Artificial e.g., electroencephalogram (EEG), functional magnetic
Intelligence and Automation, Huazhong University of Science and Technology, resonance imaging (fMRI), functional near-infrared spec-
Wuhan 430074, China (e-mail: [email protected]; [email protected]).
Bao-Liang Lu is with the Department of Computer Science and Engineering, troscopy (fNIRS), electrocardiogram (ECG), eye move-
Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: ment, blood pressure, skin temperature, and respiration,
[email protected]).
Bin Hu is with the Institute of Engineering Medicine, Beijing Institute of have also been used in affective computing [9], [10].
Technology, Beijing 100811, China (e-mail: [email protected]). Compared with other modalities, physiological signals are
Digital Object Identifier 10.1109/JPROC.2023.3277471 more difficult to disguise and, hence, may reflect more

0018-9219 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

1314 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

Fig. 1. Important events in the history of aBCIs.

authentic emotions. Brain signals, particularly EEG, are sensory and reward feedback to the user,” an aBCI has two
popular in physiological signal-based affective computing, important characteristics.
perhaps because they directly measure the state of the 1) Multimodal inputs: In addition to EEG, physiological
brain, where emotions originate. signals, such as eye movements, and nonphysiolog-
Brain signals are also the input signals in brain– ical signals, such as facial expressions and speech,
computer interfaces (BCIs) [11], which enable a user to are also frequently used in aBCIs for more accurate
communicate directly with a computer using only the emotion recognition. In contrast, classical noninva-
central nervous system. BCIs have found applications in sive motor imagery-based BCIs [28] usually use solely
neural rehabilitation [12], text input [13], gaming [14], EEG signals.
emotion recognition [15], mental fatigue evaluation [16], 2) More factors to be considered, e.g., the subject’s age,
vigilance estimation [17], speech synthesis [18], sentence gender, and/or cultural/education background. Oth-
decoding [19], movement and touch functionality recov- erwise, it may be difficult to effectively induce the
ery [20], [21], robot control [22], [23], and so on. intended emotions. For example, the video clips used
According to the input signal source, there are three in aBCI experiments should ideally be different for
types of BCIs [24], [25]. teenagers and the elderly. These additional factors are
1) Noninvasive BCIs, where brain signals are recorded on generally not very important for motor BCIs.
the scalp. EEG is the most popular input to noninva- Rapid progress has been made in each aBCI block in
sive BCIs due to its safety, low cost, and convenience. Fig. 3 in the past few decades, but there does not exist
2) Invasive BCIs, which surgically implant sensor arrays a comprehensive and up-to-date tutorial on them. This
or electrodes into the cortex to record and decode tutorial fills the gap by introducing these individual com-
brain signals, and/or stimulate the brain. ponents in detail. Among the three input brain signals
3) Partially invasive BCIs, which surgically place the sen- (EEG, fMRI, and fNIRS), fMRI and fNIRS, perhaps due to
sors between the skull and the cortex.
An affective BCI (aBCI) [26] monitors and/or regulates
the emotional state of the brain. It has been attracting
rapidly increasing research interests recently, as demon-
strated in Fig. 2, which shows the number of publications
per year returned by query Emotion OR Affect “brain com-
puter interface” on Google Scholar.
The flowchart of a closed-loop aBCI system is depicted
in Fig. 3. It consists of signal acquisition, signal processing,
feature extraction, emotion recognition, and/or brain stim-
ulation. Most aBCIs so far are noninvasive BCIs, especially
when brain stimulation is not included.
Compared with a traditional motor BCI, in which
Shanechi [27] “uses a mathematical algorithm termed a
“decoder” to estimate the user’s intended movement state Fig. 2. Number of publications per year, returned by query Emotion
from neural activity, uses the decoded movement to con- OR Affect “brain computer interface” on Google Scholar on April 15,
trol an external actuator (prosthetic device), and provides 2023.

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1315
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

settings to elicit specific emotions of the subjects, and

recording the brain and other physiological signals simul-
taneously. For example, a “happy” movie clip rated by
multiple evaluators is supposed to elicit a happy emotion
from the subject.
Unlike other classical BCI paradigms, e.g., motor
imagery [32], event-related potentials [33], and steady-
state visual evoked potentials [34], aBCI experiments are
more difficult to perform due to the following reasons.
Fig. 3. Flowchart of a closed-loop aBCI system.
1) Music or videos are frequently used to elicit specific
emotions; thus, the emotions of the music or videos
need to be evaluated first. Since emotion is very
subtle and has large individual differences, multiple
their high cost, larger sizes, and shorter history, are used
evaluators are needed for each music or video, which
less frequently than EEG in aBCIs. For example, a Google
is labor-intensive. For example, in the popular DEAP
Scholar search of “EEG emotion” in the title on April
dataset [35], 14–16 volunteers were recruited to eval-
20, 2023, returned 2260 articles, whereas “fMRI emo-
uate the arousal, valence, and dominance values of
tion” returned 523 articles, and “fNIRS emotion” returned
each music video.
28 results. In addition, many studies in the latter two
2) It takes some time for music or video to influence the
categories are on the evaluation of brain activities in
emotion of a user, so an aBCI data trial is generally
different emotional states, instead of emotion recognition.
longer than a trial in other BCI paradigms. For exam-
A Web of Science search on April 20, 2023, revealed
ple, a steady-state visual evoked potential [13] trial
that, among all 733 IEEE TRANSACTIONS ON AFFECTIVE COMPUTING
may be less than a second, and motor imagery [32]
publications so far, 79 are on EEG, two on fMRI, and none
or event-related potential [33] trial may take only
on fNIRS. Thus, fMRI- and fNIRS-based aBCIs are only
a few seconds, whereas an aBCI trial may take a
briefly introduced in this tutorial. The focus is on EEG-
couple of minutes. Thus, collecting aBCI data is time-
based aBCIs.
consuming.
The remainder of this tutorial is organized as follows.
3) Usually, aBCI experiments require the subjects to con-
Sections II–VI describe in detail the individual components
centrate on the emotional media to elicit the intended
in Fig. 3, i.e., signal acquisition, signal processing, feature
emotion, which may quickly result in fatigue. Thus,
extraction, emotion recognition, and brain stimulation,
an aBCI experiment cannot last too long. As each
respectively. Section VII introduces applications of aBCIs.
aBCI trial may take a couple of minutes, this further
Section VIII points out challenges and future research
reduces the number of trials per subject in an aBCI
directions in aBCIs. Finally, Section IX draws conclusions.
experiment.
II. S I G N A L A C Q U I S I T I O N I N a B C I s
Different brain signals have been used in aBCIs, e.g., C. Input Signals
EEG, fMRI, and fNIRS. They are also frequently recorded Different brain signals, e.g., EEG, fMRI, and fNIRS, have
together with other physiological signals, e.g., ECG and been used in aBCIs, with EEG being the most popular.
eye movement, and video and speech, for multimodal EEG signals can be acquired by an EEG headset from the
and more accurate aBCIs. This section introduces how scalp [36]. A typical EEG headset includes tens of channels
typical aBCI experiments are performed, and how the brain (electrodes), labeled according to the international 10-20
signals are acquired. or 10-10 system, as shown in Fig. 5. The letters on a

A. Emotion Representation
Emotions can be represented using categories, e.g.,
Ekman’s six basic emotions [29] (anger, disgust, fear,
happiness, sadness, and surprise), or continuously in the
2-D space of arousal and pleasure (or valence) [30],
or the 3-D space of arousal, pleasure (or valence), and
dominance [31], as shown in Fig. 4. Both categorical and
continuous emotions have been used in aBCIs.

B. Emotion Elicitation
Most aBCI experiments were performed in controlled Fig. 4. Ekman’s six basic emotions in the 3-D space of arousal,
laboratory environments, using deliberately designed pleasure, and dominance.

1316 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

Instead of measuring the neural activities by detecting

their electrical signals (as in EEG), fMRI [43] measures the
neural activities by detecting changes associated with the
brain blood flow, and fNIRS [44] uses near-infrared light to
estimate cortical hemodynamic activities associated with
the neural activities. fMRI and fNIRS have also found
applications in aBCIs, as will be introduced in Section V-F.

D. Public aBCI Datasets

Though it is not easy to record aBCI datasets, there have
been more than ten public aBCI datasets in the community,
as summarized in Table 2 and sorted according to the
year the corresponding paper was published. We also list
the number of Google Scholar citations to the papers (as
of April 15, 2023) as an indicator of their popularity.
The three most popular ones are DEAP [35], MAHNOB-
HCI [45], and SEED [46].
The Database for Emotion Analysis using Physiolog-
ical signals (DEAP) [35] includes EEG and peripheral
Fig. 5. Electrode labels and locations of an EEG headset according
to the international 10-10 system. Nz: nose; A: ear lobe; F: frontal;
physiological signals (EOG, EMG, GSR, and so on) of
C: central; P: parietal; O: occipital; T: temporal; Fp: frontal polar; 32 participants, while they were watching 40 1-min-long
and z: zero. music video excerpts. 22 of the 32 participants also had
frontal face video recorded. Each video excerpt was rated
label indicate the lobe location, and the number (index) by 14–16 volunteers based on arousal, valence, and dom-
indicates how far away the electrode is from the center of inance, and also by each participant based on arousal,
the brain: the index increases as the electrode moves away valence, like/dislike, dominance, and familiarity.
from the central. The odd indices are on the left, and the MAHNOB-HCI [45] is one of the three databases in
even indices are on the right. the MAHNOB family (laughter, HCI-tagging, and MHI-
The typical amplitude of EEG is between 5 and mimicry). It includes video, audio, eye tracking, and phys-
300 µV, and the highest useful frequency is usually iological signals (EEG, ECG, RSP, and ST) of 30 partici-
below 100 Hz [37]. EEG signals can be partitioned into pants, while they were watching 20 emotional movie clips.
different frequency bands, reflecting different emotions or Each participant self-reported his/her ratings of valence,
brain states, as summarized in Table 1. arousal, dominance, and predictability, and also emotional
Not all EEG frequency bands and channels contribute keywords (anger, anxiety, fear, sadness, disgust, neutrality,
equally to emotion recognition. Ray and Cole [40] found surprise, amusement, and joy).
that EEG alpha activity reflects attentional demands, The Shanghai Jiao Tong University Emotion EEG dataset
whereas beta activity reflects emotional and cognitive pro- (SEED) [46], collected and released by the second author
cesses. Furthermore, the beta activities of the two hemi- of this tutorial, has been developed into a family of
spheres are different in the temporal areas for positive or datasets.
negative emotions and in the parietal areas for cognitive 1) SEED: It includes both EEG (62-channel ESI Neu-
tasks. Zheng et al. [41] found that EEG patterns for emo- roScan System) and eye movement (SMI eye-tracking
tion recognition are relatively stable over time and cross- glasses) data of 12 subjects and EEG data only of
sessions. Particularly, for positive emotions, the lateral another three subjects. Data were collected when the
temporal lobes have higher beta and gamma activations;
Table 1 EEG Frequency Bands and Emotions [26], [38], [39]
for neutral emotions, the parietal and occipital lobes have
higher alpha responses; and, for negative emotions, the
parietal and occipital lobes have significantly higher delta
responses, and the prefrontal lobes have higher gamma
responses. Peng et al. [42] proposed an approach to
automatically identify important features, and critical fre-
quency bands and channels, in EEG-based emotion recog-
nition. They found that the gamma band, and prefrontal
and left/right central channels are important. Results from
these studies may be used to select the few most useful
EEG channel locations for specific aBCIs, reducing the cost
and increasing user-friendliness.

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1317
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

Table 2 Public aBCI Datasets

subjects were watching film clips, carefully selected 6) SEED-VIG: Different from the above five emotion
to elicit different emotions (positive, negative, and datasets, SEED-VIG targets vigilance estimation. Sub-
neutral). jects played a driving game in a virtual driving system,
2) SEED-IV: As an evolution of SEED, the number of with an enormous screen placed in front of a real
emotion categories in SEED-IV changes to four: car. Their EEG and EOG data were collected, and the
happy, sad, fear, and neutral. EEG and eye movement vigilance level was labeled with the PERCLOS [57]
data are provided. indicator by an eye tracker.
3) SEED-V: As an evolution of SEED-IV, the number of
emotion categories in SEED-V increases to five: happy, III. S I G N A L P R O C E S S I N G I N a B C I s
sad, fear, disgust, and neutral. EEG and eye move- EEG signals are very weak (typical magnitude: 5–300 µV)
ment data are provided. and easily contaminated by artifacts (such as eye blinks
4) SEED-FRA: This dataset contains EEG and eye move- and muscle movements) and noise. Thus, it is very impor-
ment data of eight French subjects with positive, tant to perform signal processing to increase their signal-
negative, and neutral emotions. to-noise ratio.
5) SEED-GER: This dataset contains EEG and eye move- Signal processing in aBCIs typically includes sequen-
ment data of eight German subjects with positive, tially temporal filtering, rereferencing, artifact removal,
negative, and neutral emotions. resampling, and epoching.

1318 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

A. Temporal Filtering data. A certain signal source may be eye blinks and, hence,
Not all frequencies in EEG are useful for emotion recog- can be removed.
nition, e.g., the very low frequencies may be dc drifts, ICA has two main differences from PCA [64].
and very high frequencies may be noise. Thus, usually, 1) Each successive PCA component accounts for as much
EEG signals need to be bandpass filtered. A commonly as possible of the remaining activity not accounted
used passband is 4–45 Hz [58]. However, recent research for by previous PCA components, so different PCA
has shown that the delta band (0.5–4 Hz) power may components may have dramatically different con-
be very useful in characterizing negative emotions [41], tributions, with the first the maximum (could be
and high-pass filtering EEG at 1 Hz was recommended for more than 50%). ICA components have much more
independent component analysis (ICA) [59]; thus, a better homogeneous contributions, ranging from roughly
passband choice may be 1–45 Hz. 5% down to ∼0%, because ICA tries to identify maxi-
In addition, notch filtering may be used to remove the mally independent activity sources.
50-/60-Hz powerline interference. 2) PCA components of EEG data are spatially or tem-
porally orthogonal, depending on which dimension
B. Rereferencing PCA is applied to. ICA components are maximally
temporally independent, but there are no spatial con-
EEG records the voltages with respect to a specific
straints.
reference. Typical reference electrodes [60] include one
mastoid (e.g., TP10), linked mastoids, the vertex electrode Recently, end-to-end deep learning approaches [65]
(Cz), single or linked earlobes, or the nose tip. Headsets have also been proposed for EEG artifact removal.
with active electrodes may record EEG reference-free.
Rereferencing is usually performed after data recoding and D. Resampling
filtering to increase the signal-to-noise ratio; particularly, EEG signals are typically recorded at a much higher
rereferencing post hoc can remove 40-dB unnecessary sampling rate (e.g., 1024 Hz) than the useful frequencies
noise of active headsets [60]. in emotion recognition, so usually downsampling is used to
The most frequently used rereferencing approach is reduce the memory and computational cost. Many studies
the common average reference [58], which removes the downsampled EEG data to 128 Hz [58].
mean of all channels from each individual channel. Special
attention should be paid when the original EEG signals E. Epoching
were recorded with reference [60]
EEG signals are usually recorded continuously for each
1) If the data were recorded with reference to the nose
stimulus, which may last several minutes. Each such piece
tip or ear lobe, then these reference electrodes should
of EEG data may be viewed as a block, and usually,
be excluded from computing the average reference.
each block is further partitioned into many overlapping or
2) If N -channel EEG data were recorded with reference
nonoverlapping shorter (e.g., 10 s) epochs to increase the
to a particular electrode, e.g., TP10, then the signal of
number of trials in analysis.
TP10 can be recovered from the N -channel data first,
i.e., TP10 = (Sum of N electrode activities)/(N + 1).
Now, there are N + 1 electrodes, and their average
F. Signal Processing for fMRI-/fNIRS-Based aBCIs
can be computed and removed from each individual An fMRI image consists of multiple sequentially sam-
electrode. pled slices. Its processing typically involves the following
steps [66].
C. Artifact Removal 1) Slice timing correction, which interpolates the slices so
The next step is usually artifact (e.g., eye blinks and that they can be viewed as being sampled at exactly
muscle movements) removal. Many researchers remove the same time [67].
artifacts manually, but semiautomatic approaches, such 2) Realignment, which performs motion correction to
as ICA [61], blind source separation [62], and principal eliminate the effect of head movements [68].
component analysis (PCA), are also popular. They aim 3) Coregistration, which registers low spatial resolution
to find some spatial filters to transform the original EEG fMRI images to a high-resolution structural MRI
channels into some (usually fewer) “virtual channels,” image of the same subject [69].
some of which may be artifacts/noise and, hence, could 4) Normalization, which registers a subject’s anatomical
be removed. structure to a standardized stereotaxic template [70].
Due to volume conduction effects [63], each EEG chan- Currently, there is no standard fNIRS signal processing
nel may measure the compound from multiple underlying procedure. Some studies [71], [72] included the following
primary sources in the brain. ICA is one of the most popular steps, among others:
approaches to separate these sources. It identifies multiple 1) artifact removal, which identifies and removes noisy
independent component filters to produce maximally tem- channels and incorrect trials, and removes/corrects
porally independent signal sources available in the original motion artifacts;

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1319
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

Table 3 Representative EEG Features in aBCIs 12) Fractal dimension [78] that is also a measure of sig-
nal complexity: First, rewrite x(t) as {x(m), x(m +
k), . . . , x(m + k⌊(T − m/k)⌋)}, where k is the time
interval, m ∈ {1, . . . , k} is the initial time, and ⌊·⌋ is
the floor operation. Then, compute

P⌊ T −m ⌋
k=1
k
|x(m + ik) − x(m + (i − 1)k)|
Lm (k) = T −1
.
k2 ⌊ T −m
k
⌋

The fractal dimension is finally computed as the neg-

ative slope of the log-log plot of (1/k) km=1 Lm (k)
P

against k.
13) Higher order crossings (HOCs) [79] that captures the
2) filtering, which uses a finite impulse response band- EEG oscillatory pattern: Apply k different high-pass
pass filter to remove both very-low-frequency and filters to a zero-mean time series to obtain k filtered
high-frequency noises; time series, and extract the k HOC features as the
3) epoching, which partitions the fNIRS signal into short number of zero-crossings of them.
trials. The first seven time-domain features are generic, i.e.,
they can be computed for any time series, not necessarily
IV. F E A T U R E E X T R A C T I O N I N a B C I s specific to EEG. Hjorth’s three measures were specifically
EEG is multichannel time series, so time-domain, proposed for EEG signal analysis.
frequency-domain, time–frequency-domain, and brain
connectivity features [73], [74] could be extracted for B. Frequency-Domain Features
aBCIs, as summarized in Table 3. Different EEG frequency bands may reflect different
emotions, as shown in Table 1. Thus, frequency-domain
A. Time-Domain Features features [73] usually include band power derivatives (e.g.,
Let x(t) ∈ RT be the time series of a single EEG channel, mean, minimum, maximum, variance, ratio of mean pow-
where T is the number of time samples. Time-domain ers of different bands, and differential entropy (DE) [80])
features could include [73]: and higher order spectra (e.g., bispectra and bicoherence
1) Mean: µx = (1/T ) Tt=1 x(t). magnitudes [81]).
P

2) Standard deviation: σx = ((1/T ) Tt=1 (x(t) − An effective and popular frequency-domain feature in
P
2 1/2 EEG-based emotion recognition is DE [80]
µx ) ) .
3) Power: Px = (1/T ) Tt=1 x2 (t).
P
(x−µx )2
4) First difference [9] that approximates the gradient: Z +∞ − 2
2σx
2
(x−µx )
!
e 1 −
δx = (1/T − 1) Tt=−11 |x(t + 1) − x(t)|.
P 2
DEx = − √ log √ e 2σx
dx
−∞ 2πσx2 2
2πσx
5) Normalized first difference [9] or normalized length
density [75] that captures self-similarities of the time 1
= log 2πeσx2 .


series: δ̄x = δx /σx . 2
6) Second difference [9]: γx = (1/T − 2) Tt=−12 |x(t+ 2)−
P
For a fixed-length EEG sequence, DE is equivalent to
x(t)|.
the logarithmic energy spectrum in a certain frequency
7) Normalized second difference [9]: γ¯x = γx /σx .
band [82].
8) Hjorth’s activity [76] that is the variance or mean
Differential asymmetry (DASM) and rational asymmetry
power of the EEG signal: Ax = σx2 .
(RASM), which are the difference and ratio between the
9) Hjorth’s mobility [76] that may be considered as a
DEs of a pair of hemispherically symmetric electrodes (e.g.,
mean frequency: Mx = σẋ /σx , where ẋ is the first
O1 and O2 in Fig. 5, denoted as xleft and xright ), respec-
derivative of x.
tively, can further be extracted to describe the hemispheric
10) Hjorth’s complexity [76] that measures more details of
asymmetry [80]
the EEG signal with reference to the sine wave: Cx =
Mẋ /Mx .
11) Normalized nonstationarity index (NSI) [77] that uses DASM = DExleft − DExright
the variation of the local average over time to represent DExleft
RASM = .
the signal complexity: x(t) is first normalized by the DExright
standard deviation and then divided into multiple
small segments. The mean of each segment is com- Similar differential spectral asymmetry features, DLAT
puted, and the NSI is the standard deviation of these and DCAU, were also proposed in [83]. DLAT consists
means. of the differential spectral band powers (delta, theta,

1320 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

alpha, beta, and gamma) for 12 left-right electrode pairs, where cov(x, y) is the covariance. PCC has a range of
e.g., Fp1-Fp2 and F7-F8. The latter consists of those [−1, 1], and a PCC of zero means that there is no linear
for 12 fronto-posterior electrode pairs, e.g., Fp1-O1 and relationship between x(t) and y(t).
√
F7-P7. Define z(t) = x(t) + j x̃(t), where j = −1, and
The bispectrum Bis is the Fourier transform of the
∞
third-order moment of x(t), i.e., [73], [81]
Z
PV x(τ )
x̃(t) = dτ
π −∞ t−τ
Bis(f1 , f2 ) = E [FT(f1 ) · FT(f2 ) · FT∗ (f1 + f2 )]
is the Hilbert transform of x(t), in which PV is the Cauchy
principal value. The instantaneous phase ϕ(t) of x(t) is
where E is expectation, FT is the Fourier transform of x(t),
then
and ∗ is the complex conjugate.
Bicoherence Bic is the normalized Bis [73], [81]
x̃(t)
ϕ(t) = arctan .
x(t)
Bis(f1 , f2 )
Bic(f1 , f2 ) = p
P (f1 ) · P (f2 ) · P (f1 + f2 )
The PLV [96] computes the phase synchronization between
Channels i and k, by taking the absolute average of phase
where P (f ) = E[FT(f ) · FT∗ (f )] is the power spectrum. differences over temporal windows
Several other entropy features, e.g., approximate
entropy [84], sample entropy [84], permutation entropy T
[85], and dispersion entropy [86], have also been pro- 1 X j(ϕi (t)−ϕk (t))
PLV(i, k) = e .
posed and used in EEG-based emotion recognition [87]. T
t=1
Giannakakis et al. [88] gave a review on biosignal-based
psychological stress detection and concluded that EEG The PLI [97] is another way to compute the phase synchro-
alpha asymmetry index (e.g., the natural logarithm of nization between channels i and k
the Alpha power of F3 minus that of F4) is consistently
reduced under stress. T
1 X
PLI(i, k) = sign (ϕi (t) − ϕk (t)) .
T
t=1

C. Time–Frequency-Domain Features
Both PLV and PLI take values in [0, 1]. A larger value
Time–frequency-domain features are usually 2-D spec- indicates better phase locking, and a value of zero means
tral representations of EEG signals in simultaneously time of two channels are independent.
and frequency domains [89], including short-time Fourier Moon et al. [95] performed binary arousal classifica-
transform (STFT), spectrograms computed from STFT tion on DEAP and found that the PCC and PLV features
[89], discrete wavelet transform [90], Cohen’s class [91], always outperformed power spectrum density (PSD) fea-
Zhao–Atlas–Marks (ZAM) transform [92], empirical mode tures when fed into convolutional neural networks, and
decomposition (also known as Hilbert–Huang spectrum PLV always outperformed PLI.
(HHS) [93], [94]), and so on.
Hadjidimitriou and Hadjileontiadis [89] compared
E. Feature Combinations
STFT, ZAM, and HHS features in music like/dislike classifi-
cation and found that, generally, ZAM features performed Different feature extraction approaches could also be
the best due to their high resolution in both time and assembled into a pipeline to extract new features.
frequency domains. A frequently used approach is to extract more fea-
tures from different frequency bands. Hadjidimitriou and
Hadjileontiadis [89] computed the spectrograms of STFT,
D. Brain Connectivity ZAM transform, and Hilbert–Huang spectrum in both beta
(13–30 Hz) and gamma (31–49 Hz) bands and then con-
Connectivities between different brain regions (elec- catenated them as features for music appraisal responses
trodes) have also been used as features in emotion recogni- (like and dislike) classification. Zheng et al. [41] used PSD,
tion. Frequently used brain connectivity measures [95] are DE, DASM, RASM, and so on in five different frequency
the Pearson correlation coefficient (PCC), phase-locking bands for emotion classification on DEAP and SEED.
value (PLV) [96], and phase lag index (PLI) [97]. Another idea is to extract more features from different
The PCC measures the linear correlation between two intrinsic mode functions in empirical mode decomposition.
time series x(t) and y(t) Liu et al. [98] first performed empirical mode decomposi-
tion of the original EEG signals to obtain five intrinsic mode
cov(x, y) functions and then computed the DE for each intrinsic
PCCx,y =
σx σy mode function and each channel. The dynamic DE features

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1321
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

were then used by a convolutional neural network for distribution map features [112], a deep canonical corre-
emotion classification. lation analysis model fusing EEG functional connectivity
Kroupi et al. [75] extracted power ratio features network features and eye movement features [113], and a
from theta, alpha, low beta (13–16 Hz), middle beta spiking neural network model [114].
(17–20 Hz), high beta (21–29 Hz), and low gamma We should note that, for within-subject emotion recog-
(30-47 Hz) bands, by dividing a subject-specific baseline nition, it is very important to perform cross-block data
power from the trial powers. Additional features included partition and evaluations.
the Wasserstein distance between trial and baseline pow- For example, SEED includes 15 4-min film clips for each
ers, and the normalized length density index. subject. Each 4-min EEG data recording for a film clip can
be viewed as a block, and each block can be partitioned
F. Feature Extraction for fMRI-/fNIRS-Based aBCIs into multiple shorter nonoverlapping trials (e.g., 10 s) to
increase the number of training and test samples. Thus,
Functional connectivities between brain regions of inter-
each block consists of 24 trials with the same emotion
est (ROIs) are common fMRI features in aBCIs [99],
label, and there are a total of 24 × 15 = 360 trials.
[100]. The ROIs could be defined according to the atlas
For within-subject emotion recognition, each subject is
of automated anatomical labeling [101], dense individu-
considered individually. It is very important that the train-
alized and common connectivity-based cortical landmarks
ing and test sets should come from different blocks, e.g.,
(DICCCOLs) [102], and so on. Then, measures such as
Blocks 1–12 for training and Blocks 13–15 for test. If we
small-world parameters [103], network efficiency [104],
mix the 360 trials altogether and randomly select 80%
and nodal centrality metrics [105] could be computed.
of them for training and the remaining 20% for testing,
For fNIRS feature extraction [106], the raw fNIRS light
then there is a block-design pitfall [115]: “the block design
intensities could be converted to relative changes in hemo-
leads to the classification of arbitrary brain states based on
dynamic responses in terms of oxy-hemoglobin (Hbo)
block-level temporal correlations that are known to exist in
and deoxy-hemoglobin (Hbr), and then, Hbo + Hbr and
all EEG data, rather than stimulus-related activity. Because
Hbo-Hbr, which estimate the total blood volume and the
every trial in their test sets comes from the same block
oxygenation change, can be calculated for each optode.
as many trials in the corresponding training sets, their
Finally, their statistics such as mean, median, standard
block design, thus, leads to classifying arbitrary temporal
deviation, maximum, minimum, and maximum–minimum
artifacts of the data instead of a stimulus-related activity.”
could be extracted as features.
The block-design pitfall becomes more significant if each
4-min EEG block is partitioned into multiple overlapping
V. E M O T I O N R E C O G N I T I O N I N a B C I s
trials. For example, if the [0, 10] second trial in a certain
Once the features are extracted, traditional machine learn-
block is used in training, and the subsequent [5], [15]
ing algorithms, e.g., k-nearest neighbors [107], decision
second trial from the same block is used in the test, then
tree [107], and support vector machine (SVM) [108],
there is data leakage since part of the test data have
can be used for emotion recognition. When deep learn-
been seen in training, and hence, the test results will be
ing [109] is used, feature extraction and classifica-
overoptimistic.
tion/regression can be integrated into a single end-to-end
neural network, so manual feature extraction may not be
necessary. B. Transfer Learning for Cross-Subject/
Cross-Session Emotion Recognition
A. Within-Subject Emotion Recognition A machine learning model may work well when there
Many aBCI studies performed within-subject emotion are adequate training data, and the training and test
recognition, where each subject is considered individually. data have the same distribution. Unfortunately, these two
Both traditional machine learning approaches [110] and assumptions are not always satisfied in aBCIs: 1) for fast
deep learning have been considered. calibration to improve system utility and user-friendliness,
For example, Huang et al. [15] performed within- it is desirable to collect as few calibration trials from a
subject EEG-based binary (positive/negative) emotion new subject as possible, i.e., the subject-specific training
classification for both healthy subjects and disorder-of- data is usually not enough and 2) due to large individual
consciousness patients, using manually extracted PSD fea- differences (data distributions from different subjects are
tures and an SVM classifier. The ten healthy subjects usually significantly different), generally, it is not feasible
achieved on average over 90% classification accuracy, to use data from existing subjects directly for the calibra-
and some disorder-of-consciousness patients also achieved tion of a new subject.
about 70% accuracy. Liu et al. [111] proposed a 3-D Transfer learning [116] has been widely used in
convolution attention neural network for both within- BCIs [117], including aBCIs, to reduce the subject-specific
and cross-subject EEG emotion classifications on SEED. calibration effort. Transfer learning uses data or knowledge
In within-subject classification, it outperformed a deep from some source domains (existing subjects) to facilitate
convolutional neural network using electrode-frequency the model training in a target domain (a new subject).

1322 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

To reduce the domain discrepancy and, hence, overcome Li et al. [130] reviewed transfer learning for EEG-based
negative transfer [118], transfer learning may weigh the emotion recognition and observed that the research inter-
source domain samples so that their distribution is more est is gradually shifting from instance transfer, to fea-
similar to that of the target domain (instance transfer) or ture transfer, and then to parameter transfer (their
perform feature transformations so that the feature distri- Table 2 summarizes about 20 representative transfer learn-
butions of the source and target domains are more similar ing approaches in aBCIs). So far, feature transfer has
in the new feature space (feature transfer) or use the the most investigations. They also gave a summary of
source models to regularize the target model (parameter the performance of some representative transfer learn-
transfer) and so on. ing approaches, in cross-session, cross-subject, and cross-
For instance transfer, Zhang et al. [119] proposed indi- database transfers. The state-of-the-art cross-subject trans-
vidual similarity-guided transfer learning for EEG-based fer learning classification accuracies on SEED are around
emotion recognition. They used first maximum mean dis- 90% for three-class valence classification [112], [131].
crepancy (MMD) to quantify the similarities between the
source subjects and the target subject, then TrAdaBoost C. Deep Transfer Learning
to further weight the trials from the top few most similar
Deep learning [132], which has achieved great success
source subjects to make the axillary data distribution more
in many other domains, has also been gaining popularity
resemble the target distribution, and, finally, an SVM for
in aBCIs. In deep learning [109], feature extraction and
classification. Lin [120] first used the Riemannian dis-
classification/regression are integrated into a single neural
tances of MESH features (concatenation of DLAT, DCAU,
network and simultaneously optimized.
and PSD features) [83] between multiple source sessions
Deep learning-based transfer learning, or deep transfer
and the target session to select the most similar few source
learning, can be achieved in different ways. The most
sessions and then augmented their data with the target
popular and straightforward approach [112] is parameter
session data for cross-session transfer. Particularly, matrix
transfer, i.e., to train a deep learning model using data
factorization of robust PCA was further used to reweight
from multiple auxiliary subjects and then adapt it to the
the samples for instance transfer.
new subject by fixing the first few feature extraction layers
For feature transfer, Zheng et al. [121] demonstrated
and then using the subject-specific data to fine-tune the last
the promising performance of two classical feature dimen-
few classification layers. Another idea is to use adversarial
sionality reduction approaches, transfer component analy-
learning to bring the data distributions of the auxiliary and
sis [122], and kernel principal analysis [123] in EEG-based
new subjects closer. For example, Luo et al. [133] proposed
emotion classification on SEED. Both aim to learn a
the Wasserstein generative adversarial network domain
set of common transferrable components between the
adaptation to achieve an 87% cross-subject emotion classi-
source and target domains in a latent feature space.
fication accuracy on SEED. It uses first pretraining to map
Chai et al. [124] proposed adaptive subspace feature
source and target domains to common latent feature space
matching, which uses first PCA in the source and tar-
and then adversarial training to bring the source and target
get domains, respectively, to map the DE features into
domain latent features together.
lower dimensional subspaces, then a linear transformation
matrix to match their marginal distributions, and, finally,
conditional distribution adaptation to further reduce the
D. Multimodal Learning
distribution discrepancy. Multimodal signals, e.g., EEG, ECG, eye movement, and
EEG data alignment approaches, e.g., Riemannian align- facial expressions, may be used together for more reliable
ment [125] and Euclidean alignment [126], may also emotion recognition. An important question is how to
be used to transform the EEG data from different sub- effectively fuse the information from different modalities.
jects so that their distributions are more consistent (and Generally, there are two popular fusion strategies in
hence to facilitate transfer learning); thus, they may also multimodal emotion recognition.
be viewed as feature transfer approaches. For example, 1) Feature-level fusion, where features of each modal-
Wang et al. [127] demonstrated the effectiveness of the ity are extracted individually and independently,
Riemannian Alignment-Minimum Distance to Riemannian and then concatenated into a single larger feature
Mean (RA-MDRM) [125] on DEAP, and Jiang et al. [128] vector for classification or regression. For exam-
demonstrated the effectiveness of Euclidean alignment on ple, Wu et al. [108] concatenated features from
both DEAP and SEED. four physiology signals (EEG, ECG, skin conductance
For parameter transfer, Zheng and Lu [129] used source level, and respiration) and then used an SVM for
domain data to train multiple SVM classifiers, one for each subject-dependent classification of three arousal lev-
domain, and then learned a regression function to describe els, reaching an average accuracy of 96.5% on 18 sub-
the relationship between the feature distribution and SVM jects. Cai et al. [107] performed feature-level fusion
parameters. The parameters of the target domain SVM for multimodal depression recognition. They collected
classifier were then computed by applying such a function EEG signals, while the subjects were listening to
to the unlabeled target data. audio with different emotions (neutral, negative, and

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1323
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

positive), each emotion being viewed as a modality. Yan et al. [139] investigated the same problem using a
For each subject, EEG features from each modality different approach. They first used EEG and eye movement
were computed, and features from the three modal- features to train a multimodal fusion network and an emo-
ities were linearly combined to form a new feature tion classifier and then trained a conditional generative
vector, which was used to classify if the subject had adversarial network to learn the relationship between eye
depression or not. movements and the multimodal features. In the test phase,
2) Decision-level fusion, which builds a classifier for eye movement features were used to regress multimodal
each modality and then aggregates their results. Usu- features for emotion classification.
ally, the outputs of each classifier are the classification Eye movements should not play a role in some
confidence of different classes, and there are differ- BCI paradigms, e.g., motor imagery; thus, they are
ent approaches to aggregate this confidence [134], called artifacts there and should be removed. How-
e.g., (weighted) summation, product, Dempster– ever, multiple studies [137], [138], [139] verified that
Shafer, and Bayesian belief integration. For example, they could be very useful in aBCIs. For example,
Huang et al. [135] used neural networks to classify Soleymani et al. [137] showed that eye gaze features out-
four emotions (happiness, neutral, sadness, and fear) performed EEG in video-elicited emotion classification,
from facial expressions, and two SVMs to classify and their decision-level fusion performed the best. This
these four emotions and three intensities (strong, should not diminish the usefulness of EEG in aBCIs, as,
ordinary, and weak) from EEG, respectively. Sum- in real-world applications, vision is not the only input to
mation and product decision-level fusion approaches elicit emotions; other emotional inputs, such as sound and
were then used for multimodal emotion detection, smell, may be reflected by EEG but not eye movements.
both outperforming single-modal detection.
There is no universal conclusion on which fusion F. fMRI-/fNIRS-Based Emotion Recognition
approach is better. Zheng et al. [136] compared both
fusion strategies in EEG and eye-tracking data-based emo- In addition to EEG, fMRI and fNIRS have also been used
tion classification on SEED and found that, on aver- in emotion recognition.
age, feature-level fusion outperformed two decision-level Han et al. [100] combined fMRI-derived features (func-
fusion approaches (73.59% versus 72.98% and 68.90%). tional connectivities between ROIs, computed by the
Soleymani et al. [137] used EEG, pupil diameter, eye wavelet transform coherence) and low-level audio–visual
blinks, and gaze distance for subject-independent classifi- features (lighting key, color energy, visual excitement, Mel-
cation of three arousal levels (calm, medium aroused, and frequency cepstral coefficients, and so on) for binary video
activated) and three valence classes (unpleasant, neutral, arousal classification (low and high), and showed that
and pleasant). They found that decision-level fusion out- fMRI-derived features can increase the subject-dependent
performed feature-level fusion, reaching 68.5% and 76.4% classification accuracy by more than 10%. fMRI-derived
average classification accuracies on arousal and valence, features themselves achieved over 92% cross-subject clas-
respectively, on 24 subjects. sification accuracies on three subjects.
Wang et al. [140] studied intersession instability in
fNIRS-based emotion recognition and found a 22.2% aver-
age deterioration of binary emotion classification (negative
E. Cross-Modal Learning
and neutral) accuracy between two sessions with three-
In real-world applications of aBCIs, maybe not all input week apart. As the change of the distributions of fNIRS
signals used in training are available in the test, e.g., features may be the cause of the performance decline,
both EEG and eye movement signals are used in training, they proposed a feature selection approach that considers
whereas only eye movement signals are available in the both the feature separability and their stability over time,
test. In this case, using all available modalities in training which gave a 5% accuracy improvement in cross-session
may still be more beneficial than using only one of them. classification.
Zheng et al. [138] considered modality deficiency in Sun et al. [106] showed that using EEG and fNIRS
heterogeneous transfer learning, i.e., the source subjects simultaneously for positive/negative arousal classification
have both EEG and eye movement signals, whereas the tar- outperformed every single modality. The logarithmic PSD
get subject has only eye movement signals. They showed for theta, slow alpha (8–10 Hz), alpha, and beta bands
that with only eye tracking data for the target subject, was extracted from each electrode as EEG features. The
TL can still make use of the discriminative information mean, median, standard deviation, maximum, minimum,
in EEG from the source subjects, achieving comparable and maximum–minimum of Hbo + Hbr and Hbo-Hbr
performance with the model based on EEG data in emotion from each optode were extracted as fNIRS features.
recognition. This greatly extends the application scenarios In addition, brain activity asymmetry features, i.e., the
of aBCIs, as eye trackers may be easier to wear, and eye difference between the spectral powers of symmetrical
tracking data may be less subjective to body movement EEG electrode pairs on the two hemispheres, were also
artifacts. extracted.

1324 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

VI. B R A I N S T I M U L A T I O N demanded by a task and those resources available to be

Brain stimulation in aBCI is used to achieve emotion regu- supplied by the human.” Maintaining an appropriate work-
lation [141], which “refers broadly to implementation of a load helps improve the operator’s safety and efficiency.
conscious or nonconscious goal to start, stop, or otherwise For example, using a Virtual Reality Stroop Task with
modulate the trajectory of an emotion.” Proper emotion three difficulty levels and the user’s reaction time as the
regulation is important in mental and physical health, and performance measure, Wu et al. [108] empirically verified
social interactions [142]. the Yerkes–Dodson law [151], which states that the per-
Multiple EEG-based emotion biomarkers have been formance in mental tasks is a nonmonotonic function of
proposed in the literature. For example, Hardt and arousal (stress or workload): as arousal increases from low
Kamiya [143] found that, for high anxiety subjects, to high, the performance first increases and then decreases.
increasing EEG Alpha activity reduced anxiety and vice There may be associations between workload and EEG
versa. Thus, they suggested that long-term alpha feed- frequency band power. For example, the increasing work-
back training (>5 h) may be useful in anxiety therapy. load may lead to alpha band power decrease and theta
The asymmetry in frontal alpha power also indicates dif- band power increase [152], [153]. As a result, EEG-based
ferent emotions [144]: greater right-lobe alpha activity cognitive workload recognition has been used in many dif-
than left-lobe activity is associated with negative emo- ferent working environments [154], e.g., education [148],
tions and vice versa. These findings have been used to air traffic control [155], and autism intervention [156].
guide the design of EEG-based neurofeedback training Zhou et al. [154] presented a comprehensive review on
to regulate [142] “single-frequency band EEG activity in EEG-based cognitive workload recognition, focusing on
specific brain regions or EEG power differences between machine learning approaches. Both traditional machine
hemispheres.” Huang et al. [142] further developed an learning and deep learning approaches have been exten-
EEG-based BCI for emotion regulation, which uses video sively used.
clips to regulate the user’s emotion directly to a certain
state (positive, neutral, and negative), instead of regulat- B. Fatigue Estimation
ing the activity of a certain EEG frequency band.
Similar noninvasive emotion regulation studies have Fatigue (drowsiness) is an important contributor to
also been performed using fMRI [145] and fNIRS [146]. many accidents, particularly traffic accidents. The U.S.
However, instead of regulating the activity of a certain EEG National Highway Traffic Safety Administration [157]
frequency band, they regulate the activity of a certain ROI, estimated that, in 2017, drowsy driving resulted in
e.g., amygdala or dorsolateral prefrontal cortex. 91 000 police-reported crashes, 50 000 injuries, and nearly
Brain stimulation using electroconvulsive therapy 800 deaths. These numbers may be underestimated.
(ECT), transcranial magnetic stimulation (TMS), vagal Different strategies, e.g., computer vision-based [158]
nerve stimulation (VNS), and deep brain stimulation and driving behavior-based [159], have been used
(DBS) have also been used in treatment-resistant depres- for driver fatigue estimation. EEG-based driver fatigue
sion (TRD) therapies. Cusin and Dougherty [147] sum- (drowsiness) estimation, which is less dependent on overt
marized their efficacy, contraindications, and side effects: behavior and less susceptible to deception [26], belongs to
“ECT is widely available and its effects are relatively the latter. Another advantage of EEG-based driver fatigue
rapid in severe TRD, but its cognitive adverse effects may estimation is that fatigue may be detected from EEG signals
be cumbersome. TMS is safe and well tolerated, and it earlier than from other modalities like facial expressions,
has been approved by FDA for adults who have failed as fatigue originates deep in the brain.
to respond to one antidepressant, but its use in TRD Many different approaches, particularly transfer learn-
is still controversial as it is not supported by rigorous ing, have been used in EEG-based driver drowsiness esti-
double-blind randomized clinical trials. The options requir- mation [159], [160], [161]. Commercial products, e.g.,
ing a surgical approach are VNS and DBS. VNS has been SmartCap,1 have also been developed.
FDA-approved for TRD; however, it is not indicated for
the management of acute illness. DBS for TRD is still C. Depression Diagnosis and Treatment
an experimental area of investigation, and double-blind As indicated by the World Health Organization [162],
clinical trials are underway.” “depression is a common mental disorder. Globally, it is
estimated that 5% of adults suffer from depression.
VII. a B C I A P P L I C A T I O N S Depression is a leading cause of disability worldwide and
aBCIs have found applications in education [148], enter- is a major contributor to the overall global burden of
tainment [149], healthcare [15], and so on. Some repre- disease.”
sentative ones are introduced next. Many studies have investigated the use of aBCIs in
depression diagnosis, and multiple EEG-based biomarkers
A. Cognitive Workload Recognition
were proposed. For example, Allen et al. [163] found that
Cognitive workload may be defined as [150] “the rela-
tion between the function relating the mental resources 1 https://www.smartcaptech.com

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1325
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

prefrontal cortex EEG Alpha asymmetry may be a good increase the conductivity between the scalp and the elec-
indicator of depression. Putnam and McSweeney [164] trodes. They seem to be a good compromise between
further found that specific depression symptoms are gel-based electrodes and dry electrodes in terms of signal
uniquely associated with prefrontal cortex EEG alpha activ- quality and convenience.
ity patterns: lower bilateral prefrontal cortex alpha activity
for higher levels of rumination and lower right bilateral B. Emotion Labeling
prefrontal cortex alpha activity for higher levels of self-
Unlike videos or music, whose emotions can be labeled
esteem. Li et al. [165] found that major depression disor-
by human evaluators directly, it is almost impossible to
der patients have lower N2 amplitudes for all stimuli and
directly label the emotions of EEG trials. Thus, many
reduced anterior cingulate cortex activation for incongru-
aBCI studies used videos or music to elicit emotions and
ent stimuli. de Aguiar Neto and Rosa [166] gave a review
assumed that the subject’s EEG has the same emotion label
on EEG-based depression biomarkers.
as the stimulating video or music.
aBCIs have also been extensively used in TRD ther-
For example, a “happy” movie clip rated by multiple
apies [147]. One of the latest progress is DBS [167],
evaluators is supposed to elicit a happy emotion from
which implants electrodes within certain areas of the
the subject. However, there are several limitations of this
brain to generate electrical impulses to regulate emo-
approach: 1) because of individual differences, the subject
tion. Using multisite intracranial electrodes implanted
may not feel happy when watching the movie; 2) even
in a severely depressed patient, Scangos et al. [168]
though the subject may feel happy, the activation level may
found “an elaborate repertoire of distinctive emotional
be too low to be reflected in his/her EEG signals; and 3) the
responses that were rapid in onset, reproducible, and
subject may feel happy for a short duration of the movie,
context and state-dependent” to focal electrical neuromod-
but it is difficult to know which part it is, so it is usually
ulation. Rao et al. [169] showed that lateral orbitofrontal
assumed that the subject has a happy emotion during the
cortex stimulation-induced neural features associated with
entire duration of the movie. Because of these reasons, the
positive mood states and, hence, improved mood state in
emotion label for EEG signals may not be accurate.
depression subjects. These pilot studies provided proof of
In addition, people may exhibit multiple emotions simul-
concept for personalized and effective DBS-based treat-
taneously, e.g., a graduate may feel both happy and sad
ment of emotion disorders, e.g., severe depression.
at the graduation ceremony, and a viewer may feel both
sad and angry when seeing a good person killed in a
VIII. C H A L L E N G E S A N D
war movie. Unfortunately, currently, most aBCI experi-
OPPORTUNITIES IN aBCIs
ments assign only one emotion label or rating to each
Though we have witnessed rapid progress in aBCI research
movie/music, which may be too simplified.
and applications in the last few decades, multiple chal-
Thus, more accurate and realistic emotion labeling
lenges still need to be overcome before their broad real-
approaches are needed in aBCIs.
world applications.

A. Brain Signal Acquisition C. Diversity and Size of aBCI Datasets

Common brain signals used in aBCIs include EEG, fMRI, A closer look at the datasets in Table 2 reveals at least
and fNIRS. The latter two require more expensive equip- two limitations.
ment to acquire, so EEG is currently the most popular aBCI 1) The dataset size is very small: the number of subjects
input signal. ranges from 8 to 58 (most below 30), and the num-
There are three types of EEG electrodes: gel-based, ber of stimuli ranges from 13 to 40. Huge datasets
water-based, and dry. in other application domains have greatly boosted
Gel-based electrodes use conductive gel to increase their breakthroughs, e.g., ImageNet [171] (more than
the conductivity between the scalp and the electrodes, 14 million images and 20 000 categories) for deep
and hence, the EEG signal quality is usually much bet- learning in image recognition and more than 680 bil-
ter than that of dry electrodes. However, gel injection is lion tokens for Generative Pretrained Transformer 3
time-consuming and user-unfriendly, which may hinder (GPT-3) [172] in natural language processing. It is
the acceptance of gel-based EEG electrode-based aBCIs to expected that larger aBCI datasets will also facilitate
consumers. more rapid progress of aBCIs.
Dry electrodes, which do not need conductive gel, are 2) The demography of subjects is not diverse enough:
much more convenient to use. Unfortunately, their sig- the mean age is between 21 and 35, so children,
nal quality still needs improvements [170]. For broad teenagers, and the elderly are not adequately repre-
real-world applications, it is very important to develop sented. However, these underrepresented groups are
cheap, convenient, and high-fidelity brain signal acquisi- also huge markets for aBCI technologies. For example,
tion devices. the U.S. Centers for Disease Control and Preven-
Water-based electrodes, or semidry electrodes, use tion (CDC) estimates that [173], between 2016 and
tap/saline water or slow-releasing electrolyte liquid to 2019, for children and teenagers aged 3–17 years,

1326 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

9.4% (approximately 5.8 million) had anxiety, and 4) Inappropriate transfer learning algorithm. Each trans-
4.4% (approximately 2.7 million) had depression. fer learning algorithm has its assumptions and spe-
The World Health Organization estimates that 5.7% cific application scenarios. Choosing an inappropriate
of adults older than 60 have depression. These large one may result in a negative transfer. For example,
populations should not be overlooked. Due to large transfer component analysis [122], a classic transfer
individual differences and the subtle nature of emo- learning algorithm, assumes that the source and tar-
tions, models and strategies developed from datasets get domains have the same conditional probability
of young adults may not be directly applicable to distribution, but different marginal distributions, so it
these groups, and dedicated datasets and models focuses on reducing the latter. It may result in a
should be created for them. negative transfer if the conditional distributions are
significantly different.
There are also different strategies to mitigate or avoid
D. Comparison of Different Approaches negative transfer [118], which have not been extensively
New machine learning algorithms are constantly pro- investigated in aBCIs yet.
posed for aBCIs, and their effectiveness is demonstrated 1) domain similarity estimation, which is particularly
in the corresponding publications. Unfortunately, it is chal- useful in selecting the most similar source domains
lenging to directly compare different algorithms due to the from multiple ones, i.e., to reduce the domain diver-
following reasons. gence;
1) Different datasets are used, e.g., researchers may 2) safe transfer, which includes deliberately designed
collect their own datasets for performance evaluation. algorithms that can avoid negative transfer with the-
It is desirable to publicize these datasets so that oretical guarantees, regardless of how the source and
further comparisons can be made, or, if there are target subjects are different from each other;
restrictions to release the self-collected datasets, then 3) negative transfer mitigation, which alleviates
experimental results on some public datasets should negative transfer using data/model transferability
also be reported. enhancements, training process enhancements,
2) Different experimental settings, e.g., different split- and/or target prediction enhancements.
tings of blocks into trials and different partitions of
training/validation/test sets. It is recommended to F. Privacy-Preserving aBCIs
introduce these details as much as possible in the Many aBCI studies used transfer learning to facilitate
publications. the calibration, making use of EEG data from the source
3) Tricks in algorithm design and optimization, e.g., the subjects. In addition to emotions, EEG signals also con-
structure of the neural network, setting of hyperpa- tain other private information [174], e.g., user identity,
rameters, and the number of training epochs. It is health status, and psychological state, which may be eas-
recommended to share the code with the publications ily revealed. For example, Kong et al. [175] performed
or at least specify these details as much as possible. EEG-based user identification on SEED (and three other
EEG datasets from different BCI paradigms), achieving
over 99% accuracy using only seconds of EEG data.
E. Negative Transfer in Emotion Recognition As a result, user privacy protection in BCIs has become
Transfer learning is now prevailing in cross- very important [176], [177]. Several recent laws and
subject/cross-session calibration of emotion recognition regulations, e.g., the European General Data Protection
algorithms in aBCIs. However, transfer learning may Regulation2 (GDPR; effective since May 25, 2018) and
not guarantee improved performance, i.e., negative the China Personal Information Protection Law (effective
transfer [118] could occur. since January 11, 2021), also enforce strict user privacy
Zhang et al. [118] pointed out four possible reasons for protection.
the negative transfer. Xia et al. [174] summarized three different strategies to
implement privacy-preserving BCIs.
1) Large domain divergence, e.g., negative transfer in
1) Cryptography, which includes homomorphic encryp-
emotion recognition may occur when the source and
tion, secure multiparty computation, and secure pro-
target subjects have significantly different cultural
cessors. For example, Agarwal et al. [178] used secure
backgrounds.
multiparty computation for user privacy protection
2) Poor source data quality, e.g., the emotion recognition
in EEG-based driver drowsiness estimation. At a rea-
accuracy of the source subject is too low to be useful
sonable computational cost, they obtained identical
in helping the target subject.
results as in the unencrypted case.
3) Poor target data quality, e.g., EEG data from the target
2) Perturbation, which transforms or adds noise to the
subject contain too many artifacts/noise, or the EEG
original EEG data while maintaining their utility for
electrodes may be placed at scalp locations that are
not responsible for emotions. 2 https://gdpr-info.eu/

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1327
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

emotion recognition. It can be implemented through Though very important, there has not been research
differential privacy or data reconstruction. on adversarial defenses for aBCIs. We [188] performed a
3) Machine learning-aided systems, which may be used systematic review on adversarial attacks and defenses in
to [174] “help people better understand privacy poli- physiological computing, including BCIs, and pointed out
cies and inform them about the privacy risks when several potential adversarial defense strategies:
making privacy decisions.”
Recently, Zhang et al. [179] proposed unsuper- 1) data modification, which improves the adversar-
vised multisource decentralized transfer for offline ial robustness by modifying the training data, e.g.,
privacy-preserving EEG classification, which implements through adversarial training [189], or the test data,
transfer learning using the source model parameters or e.g., through data compression [190] or randomiza-
predictions, instead of the source data. Experiments on tion [191];
SEED demonstrated that it can achieve simultaneously 2) model modification, which improves the adversarial
privacy protection and higher classification accuracy. robustness by modifying the target model directly,
Nevertheless, privacy protection of aBCIs has not e.g., through regularization [192] and defensive dis-
received enough attention compared with accurate emo- tillation [193];
tion recognition in aBCIs. 3) auxiliary tools, which improve the adversarial robust-
ness by using auxiliary machine learning modules,
G. Secure aBCIs e.g., adversarial detection [194].
Recent research found that many machine learning
Their effectiveness has yet to be validated in aBCIs.
models, including both traditional machine learning and
deep learning, are vulnerable to adversarial attacks [180],
where deliberately designed tiny perturbations, which may IX. C O N C L U S I O N
be too small to be noticed by human eyes or detected by Affects, including moods and emotions, are pervasive in
computer algorithms, are used to fool a machine learning our everyday life and are essential in human cognition,
algorithm. communication, and decision-making. An aBCI monitors
Both classification and regression models in EEG-based and/or regulates the emotional state of the brain, which
BCIs are subject to adversarial attacks [181], [182], [183], can be used in education, entertainment, healthcare, and
[184], [185]. For example, Meng et al. [186] proposed so on. This tutorial on aBCIs introduces first the basic
two adversarial attack approaches for EEG-based regres- concepts of BCIs and then, in detail, the individual com-
sion problems, which can generate small perturbations to ponents in a closed-loop aBCI system, including signal
change the estimated driver drowsiness level by a prede- acquisition, signal processing, feature extraction, emo-
termined amount, at nearly 100% success rate. tion recognition, and brain stimulation. It also describes
Adversarial attacks on BCIs could lead to serious security three representative applications of aBCIs, i.e., cognitive
and safety problems, as pointed out by a recent report workload recognition, fatigue estimation, and depression
by the RAND Corporation [187]: “hacking BCI capabilities diagnosis and treatment. Finally, several challenges and
could theoretically provide adversaries with direct path- opportunities in aBCI research and applications, includ-
ways into the emotional and cognitive centers of opera- ing brain signal acquisition, emotion labeling, diversity
tors’ brains to sow confusion or emotional distress. In the and size of aBCI datasets, algorithm comparison, negative
extreme, adversary hacking into BCI devices that influence transfer in emotion recognition, and privacy protection and
the motor cortex of human operators could theoretically security of aBCIs, are pointed out.
send false directions or elicit unintended actions, such as To the best of our knowledge, this is the most compre-
friendly fire.” hensive and up-to-date aBCI tutorial in the literature. .

REFERENCES
[1] J. S. Lerner, Y. Li, P. Valdesolo, and K. S. Kassam, Netherlands, Sep. 2009, pp. 1–8. Sinica, vol. 47, no. 8, pp. 1769–1784, 2021.
“Emotion and decision making,” Annu. Rev. [7] C. O. Alm, D. Roth, and R. Sproat, “Emotions from [11] B. J. Lance, S. E. Kerick, A. J. Ries, K. S. Oie, and
Psychol., vol. 66, no. 1, pp. 799–823, 2015. text: Machine learning for text-based emotion K. McDowell, “Brain–computer interface
[2] M. Minsky, The Society of Mind. New York, NY, prediction,” in Proc. Conf. Hum. Lang. Technol. technologies in the coming decades,” Proc. IEEE,
USA: Simon and Schuster, 1988. Empirical Methods Natural Lang. Process., vol. 100, no. 2, pp. 1585–1599, May 2012.
[3] R. Picard, Affective Computing. Cambridge, MA, Vancouver, BC, Canada, 2005, pp. 579–586. [12] J. J. Daly and J. R. Wolpaw, “Brain–computer
USA: MIT Media Lab., 1995. [8] L. Kessous, G. Castellano, and G. Caridakis, interfaces in neurological rehabilitation,” Lancet
[4] P. Ekman et al., “Universals and cultural “Multimodal emotion recognition in speech-based Neurol., vol. 7, no. 11, pp. 1032–1043, 2008.
differences in the judgments of facial expressions interaction using facial expression, body gesture [13] X. Chen, Y. Wang, M. Nakanishi, X. Gao, T.-P. Jung,
of emotion,” J. Personality Social Psychol., vol. 53, and acoustic analysis,” J. Multimodal User and S. Gao, “High-speed spelling with a
no. 4, pp. 712–717, 1987. Interfaces, vol. 3, nos. 1–2, pp. 33–48, Mar. 2010. noninvasive brain–computer interface,” Proc. Nat.
[5] M. El Ayadi, M. S. Kamel, and F. Karray, “Survey [9] R. W. Picard, E. Vyzas, and J. Healey, “Toward Acad. Sci. USA, vol. 112, no. 44, pp.
on speech emotion recognition: Features, machine emotional intelligence: Analysis of E6058–E6067, Nov. 2015.
classification schemes, and databases,” Pattern affective physiological state,” IEEE Trans. Pattern [14] D. Marshall, D. Coyle, S. Wilson, and
Recognit., vol. 44, no. 3, pp. 572–587, Mar. 2011. Anal. Mach. Intell., vol. 23, no. 10, M. Callaghan, “Games, gameplay, and BCI: The
[6] M. Kipp and J.-C. Martin, “Gesture and emotion: pp. 1175–1191, Oct. 2001. state of the art,” IEEE Trans. Comput. Intell. AI
Can basic gestural form features discriminate [10] X. Quan, Z. Zeng, J. Jiang, Y. Zhang, B.-L. Lv, and Games, vol. 5, no. 2, pp. 82–99, Jun. 2013.
emotions?” in Proc. 3rd Int. Conf. Affect. Comput. D. Wu, “Physiological signals based affective [15] H. Huang et al., “An EEG-based brain computer
Intell. Interact. Workshops, Amsterdam, The computing: A systematic review,” Acta Autom. interface for emotion recognition and its

1328 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

application in patients with disorder of augmented brain–computer interfaces in the next [55] W. Liu, W.-L. Zheng, Z. Li, S.-Y. Wu, L. Gan, and
consciousness,” IEEE Trans. Affect. Comput., decades,” Proc. IEEE, vol. 100, no. 2, B.-L. Lu, “Identifying similarities and differences
vol. 12, no. 4, pp. 832–842, Oct. 2021. pp. 1553–1566, May 2012. in emotion recognition with EEG and eye
[16] T. G. Monteiro, C. Skourup, and H. Zhang, “Using [37] P. Fattahi, G. Yang, G. Kim, and M. R. Abidian, movements among Chinese, German, and French
EEG for mental fatigue assessment: A “A review of organic and inorganic biomaterials people,” J. Neural Eng., vol. 19, no. 2, Apr. 2022,
comprehensive look into the current state of the for neural interfaces,” Adv. Mater., vol. 26, no. 12, Art. no. 026012.
art,” IEEE Trans. Hum.-Mach. Syst., vol. 49, no. 6, pp. 1846–1885, Mar. 2014. [56] J. Xue, J. Wang, S. Hu, N. Bi, and Z. Lv, “OVPD:
pp. 599–610, Dec. 2019. [38] Electroencephalography. Accessed: May 25, 2025. Odor-video elicited physiological signal database
[17] W.-L. Zheng and B.-L. Lu, “A multimodal approach [Online]. Available: https://en.wikipedia.org/ for emotion recognition,” IEEE Trans. Instrum.
to estimating vigilance using EEG and forehead wiki/Electroencephalography Meas., vol. 71, pp. 1–12, 2022.
EOG,” J. Neural Eng., vol. 14, no. 2, Apr. 2017, [39] B.-L. Lu, Y. Zhang, and W.-L. Zheng, “A survey of [57] D. Dinges and R. Grace, “PERCLOS: A valid
Art. no. 026017. affective brain–computer interface,” Chin. J. Intell. psychophysiological measure of alertness as
[18] G. K. Anumanchipalli, J. Chartier, and E. F. Chang, Sci. Technol., vol. 3, no. 1, pp. 36–48, 2021. assessed by psychomotor vigilance,” U.S. Dept.
“Speech synthesis from neural decoding of spoken [40] W. J. Ray and H. W. Cole, “EEG alpha activity Transp., Federal Highway Admin., Oklahoma City,
sentences,” Nature, vol. 568, no. 7753, reflects attentional demands, and beta activity OK, USA, Tech. Rep., FHWA-MCRT-98-006, 1998.
pp. 493–498, Apr. 2019. reflects emotional and cognitive processes,” [58] S. M. Alarcão and M. J. Fonseca, “Emotions
[19] D. A. Moses et al., “Neuroprosthesis for decoding Science, vol. 228, no. 4700, pp. 750–752, recognition using EEG signals: A survey,” IEEE
speech in a paralyzed person with anarthria,” New May 1985. Trans. Affect. Comput., vol. 10, no. 3, pp. 374–393,
England J. Med., vol. 385, no. 3, pp. 217–227, [41] W. Zheng, J. Zhu, and B. Lu, “Identifying stable Jul. 2019.
Jul. 2021. patterns over time for emotion recognition from [59] M. Klug and K. Gramann, “Identifying key factors
[20] M. Capogrosso et al., “A brain–spine interface EEG,” IEEE Trans. Affect. Comput., vol. 10, no. 3, for improving ICA-based decomposition of EEG
alleviating gait deficits after spinal cord injury in pp. 417–429, Jul. 2019. data in mobile and stationary experiments,” Eur. J.
primates,” Nature, vol. 539, no. 7628, [42] Y. Peng, F. Qin, W. Kong, Y. Ge, F. Nie, and Neurosci., vol. 54, no. 12, pp. 8406–8420, 2021.
pp. 284–288, 2016. A. Cichocki, “GFIL: A unified framework for the [60] Backgound on EEG References. Accessed:
[21] P. D. Ganzer et al., “Restoring the sense of touch importance analysis of features, frequency bands, Sep. 25, 2022. [Online]. Available: https://eeglab.
using a sensorimotor demultiplexing neural and channels in EEG-based emotion recognition,” org/tutorials/ConceptsGuide/rereferencing_
interface,” Cell, vol. 181, no. 4, pp. 763–773, IEEE Trans. Cognit. Develop. Syst., vol. 14, no. 3, background.html
May 2020. pp. 935–947, Sep. 2022. [61] S. Makeig, A. J. Bell, T.-P. Jung, and
[22] M. Velliste, S. Perel, M. C. Spalding, [43] S. A. Huettel et al., Functional Magnetic Resonance T. J. Sejnowski, “Independent component analysis
A. S. Whitford, and A. B. Schwartz, “Cortical Imaging, vol. 1. Sunderland, MA, USA: Sinauer of electroencephalographic data,” in Proc. Adv.
control of a prosthetic arm for self-feeding,” Associates, 2004. Neural Inf. Process. Syst., Denver, CO, USA,
Nature, vol. 453, no. 7198, pp. 1098–1101, [44] M. Ferrari and V. Quaresima, “A brief review on Dec. 1996, pp. 145–151.
Jun. 2008. the history of human functional near-infrared [62] T.-P. Jung et al., “Removing
[23] L. R. Hochberg et al., “Reach and grasp by people spectroscopy (fNIRS) development and fields of electroencephalographic artifacts by blind source
with tetraplegia using a neurally controlled application,” NeuroImage, vol. 63, no. 2, separation,” Psychophysiology, vol. 37, no. 2,
robotic arm,” Nature, vol. 485, no. 7398, pp. 921–935, Nov. 2012. pp. 163–178, Mar. 2000.
pp. 372–375, May 2012. [45] M. Soleymani, J. Lichtenauer, T. Pun, and [63] S. B. Rutkove, Introduction to Volume Conduction.
[24] R. P. Rao, Brain–Computer Interfacing: An M. Pantic, “A multimodal database for affect Totowa, NJ, USA: Humana Press, 2007,
Introduction. New York, NY, USA: Cambridge Univ. recognition and implicit tagging,” IEEE Trans. pp. 43–53.
Press, 2013. Affect. Comput., vol. 3, no. 1, pp. 42–55, [64] Background on Independent Component Analysis
[25] M. L. Martini, E. K. Oermann, N. L. Opie, F. Panov, Mar. 2012. Applied to EEG. Accessed: Sep. 25, 2022. [Online].
T. Oxley, and K. Yaeger, “Sensor modalities for [46] W. Zheng and B. Lu, “Investigating critical Available: https://eeglab.org/tutorials/
brain–computer interface technology: A frequency bands and channels for EEG-based ConceptsGuide/ICA_background.html
comprehensive literature review,” Neurosurgery, emotion recognition with deep neural networks,” [65] J. Yu, C. Li, K. Lou, C. Wei, and Q. Liu,
vol. 86, no. 2, pp. E108–E117, 2020. IEEE Trans. Auto. Mental Develop., vol. 7, no. 3, “Embedding decomposition for artifacts removal
[26] C. Muhl, B. Allison, A. Nijholt, and G. Chanel, pp. 162–175, Sep. 2015. in EEG signals,” J. Neural Eng., vol. 19, no. 2,
“A survey of affective brain computer interfaces: [47] R. Subramanian, J. Wache, M. K. Abadi, Apr. 2022, Art. no. 026052.
Principles, state-of-the-art, and challenges,” R. L. Vieriu, S. Winkler, and N. Sebe, “ASCERTAIN: [66] Y. Wang and D. Wu, “Real-time fMRI based brain
Brain-Comput. Interfaces, vol. 1, no. 2, pp. 66–84, Emotion and personality recognition using computer interface: A review,” in Proc. 24th Int.
Apr. 2014. commercial sensors,” IEEE Trans. Affect. Comput., Conf. Neural Inf. Process., Guangzhou, China,
[27] M. M. Shanechi, “Brain–machine interfaces from vol. 9, no. 2, pp. 147–160, Apr. 2018. Nov. 2017, pp. 833–842.
motor to mood,” Nature Neurosci., vol. 22, no. 10, [48] S. Katsigiannis and N. Ramzan, “DREAMER: A [67] R. Sladky, K. J. Friston, J. Tröstl, R. Cunnington,
pp. 1554–1564, Oct. 2019. database for emotion recognition through EEG E. Moser, and C. Windischberger, “Slice-timing
[28] G. Pfurtscheller and C. Neuper, “Motor imagery and ECG signals from wireless low-cost effects and their correction in functional MRI,”
and direct brain–computer communication,” Proc. off-the-shelf devices,” IEEE J. Biomed. Health NeuroImage, vol. 58, no. 2, pp. 588–594,
IEEE, vol. 89, no. 7, pp. 1123–1134, Jul. 2001. Informat., vol. 22, no. 1, pp. 98–107, Jan. 2018. Sep. 2011.
[29] P. Ekman and W. V. Friesen, “Constants across [49] J. A. Miranda-Correa, M. K. Abadi, N. Sebe, and [68] K. J. Friston, C. D. Frith, R. S. J. Frackowiak, and
cultures in the face and emotion.,” J. Personality I. Patras, “AMIGOS: A dataset for affect, R. Turner, “Characterizing dynamic brain
Social Psychol., vol. 17, no. 2, pp. 124–129, 1971. personality and mood research on individuals and responses with fMRI: A multivariate approach,”
[30] J. A. Russell, “A circumplex model of affect,” groups,” IEEE Trans. Affect. Comput., vol. 12, NeuroImage, vol. 2, no. 2, pp. 166–172, Jun. 1995.
J. Personality Social Psychol., vol. 39, no. 6, no. 2, pp. 479–493, Apr. 2021. [69] W. M. Wells, P. Viola, H. Atsumi, S. Nakajima, and
pp. 1161–1178, Dec. 1980. [50] Y. Li, W. Zheng, Z. Cui, Y. Zong, and S. Ge, “EEG R. Kikinis, “Multi-modal volume registration by
[31] A. Mehrabian and J. A. Russell, An Approach to emotion recognition based on graph regularized maximization of mutual information,” Med. Image
Environmental Psychology. Cambridge, MA, USA: sparse linear regression,” Neural Process. Lett., Anal., vol. 1, no. 1, pp. 35–51, Mar. 1996.
MIT Press, 1974. vol. 49, no. 2, pp. 555–571, Apr. 2019. [70] J. Ashburner and K. Friston, “The role of
[32] G. Pfurtscheller, D. Flotzinger, and J. Kalcher, [51] T. Song, W. Zheng, C. Lu, Y. Zong, X. Zhang, and registration and spatial normalisation in detecting
“Brain–computer interface—A new Z. Cui, “MPED: A multi-modal physiological activations in functional imaging,” Clin. MRI,
communication device for handicapped persons,” emotion database for discrete emotion vol. 7, no. 1, pp. 26–27, 1997.
J. Microcomput. Appl., vol. 16, no. 3, pp. 293–299, recognition,” IEEE Access, vol. 7, [71] P. Pinti, F. Scholkmann, A. Hamilton, P. Burgess,
Jul. 1993. pp. 12177–12191, 2019. and I. Tachtsidis, “Current status and issues
[33] L. A. Farwell and E. Donchin, “Talking off the top [52] W. Zheng, W. Liu, Y. Lu, B. Lu, and A. Cichocki, regarding pre-processing of fNIRS neuroimaging
of your head: Toward a mental prosthesis utilizing “EmotionMeter: A multimodal framework for data: An investigation of diverse signal filtering
event-related brain potentials,” Electroencephalogr. recognizing human emotions,” IEEE Trans. methods within a general linear model
Clin. Neurophysiol., vol. 70, no. 6, pp. 510–523, Cybern., vol. 49, no. 3, pp. 1110–1122, Mar. 2019. framework,” Frontiers Hum. Neurosci., vol. 12,
Dec. 1988. [53] H. Becker, J. Fleureau, P. Guillotel, F. Wendling, p. 505, Jan. 2019.
[34] O. Friman, I. Volosyak, and A. Graser, “Multiple I. Merlet, and L. Albera, “Emotion recognition [72] M. L. Filippetti, J. Andreu-Perez, C. De Klerk,
channel detection of steady-state visual evoked based on high-resolution EEG recordings and C. Richmond, and S. Rigato, “Are advanced
potentials for brain–computer interfaces,” IEEE reconstructed brain sources,” IEEE Trans. Affect. methods necessary to improve infant fNIRS data
Trans. Biomed. Eng., vol. 54, no. 4, pp. 742–750, Comput., vol. 11, no. 2, pp. 244–257, Apr. 2020. analysis? An assessment of baseline-corrected
Apr. 2007. [54] T. Li, W. Liu, W. Zheng, and B. Lu, “Classification averaging, general linear model (GLM) and
[35] S. Koelstra et al., “DEAP: A database for emotion of five emotions from EEG and eye movement multivariate pattern analysis (MVPA) based
analysis; using physiological signals,” IEEE Trans. signals: Discrimination ability and stability over approaches,” NeuroImage, vol. 265, Nov. 2023,
Affect. Comput., vol. 3, no. 1, pp. 18–31, time,” in Proc. 9th Int. IEEE/EMBS Conf. Neural Art. no. 119756.
Mar. 2012. Eng. (NER). San Francisco, CA, USA: IEEE, [73] R. Jenke, A. Peer, and M. Buss, “Feature extraction
[36] L.-D. Liao et al., “Biosensor technologies for Mar. 2019, pp. 607–610. and selection for emotion recognition from EEG,”

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1329
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

IEEE Trans. Affect. Comput., vol. 5, no. 3, [92] Y. Zhao, L. E. Atlas, and R. J. Marks, “The use of recognition,” IEEE Trans. Affect. Comput., vol. 12,
pp. 327–339, Sep. 2014. cone-shaped kernels for generalized no. 3, pp. 801–820, Jul. 2021.
[74] Z. Mohammadi, J. Frounchi, and M. Amiri, time-frequency representations of nonstationary [111] S. Liu et al., “3DCANN: A spatio-temporal
“Wavelet-based emotion recognition system using signals,” IEEE Trans. Acoust., Speech, Signal convolution attention neural network for EEG
EEG signal,” Neural Comput. Appl., vol. 28, no. 8, Process., vol. 38, no. 7, pp. 1084–1091, Jul. 1990. emotion recognition,” IEEE J. Biomed. Health
pp. 1985–1990, Aug. 2017. [93] N. Huang et al., “The empirical Informat., vol. 26, no. 11, pp. 5321–5331,
[75] E. Kroupi, J. Vesin, and T. Ebrahimi, “Subject- modedecomposition and the Hilbert spectrum for Nov. 2022.
independent odor pleasantness classification using nonlinear and non-stationary time series analysis,” [112] F. Wang et al., “Emotion recognition with
brain and peripheral signals,” IEEE Trans. Affect. Proc. Roy. Soc. London. Ser. A, Math., Phys. Eng. convolutional neural network and EEG-based
Comput., vol. 7, no. 4, pp. 422–434, Oct. 2016. Sci., vol. 454, no. 1971, pp. 903–995, 1998. EFDMs,” Neuropsychologia, vol. 146, Sep. 2020,
[76] B. Hjorth, “EEG analysis based on time domain [94] P. C. Petrantonakis and L. J. Hadjileontiadis, Art. no. 107506.
properties,” Electroencephalogr. Clin. “Emotion recognition from brain signals using [113] X. Wu, W.-L. Zheng, Z. Li, and B.-L. Lu,
Neurophysiol., vol. 29, no. 3, pp. 306–310, hybrid adaptive filtering and higher order “Investigating EEG-based functional connectivity
Sep. 1970. crossings analysis,” IEEE Trans. Affect. Comput., patterns for multimodal emotion recognition,”
[77] J. M. Hausdorff, A. Lertratanakul, vol. 1, no. 2, pp. 81–97, Dec. 2010. J. Neural Eng., vol. 19, no. 1, Feb. 2022,
M. E. Cudkowicz, A. L. Peterson, D. Kaliton, and [95] S. Moon, S. Jang, and J. Lee, “Convolutional Art. no. 016012.
A. L. Goldberger, “Dynamic markers of altered neural network approach for EEG-based emotion [114] Y. Luo et al., “EEG-based emotion classification
gait rhythm in amyotrophic lateral sclerosis,” recognition using brain connectivity and its spatial using spiking neural networks,” IEEE Access,
J. Appl. Physiol., vol. 88, no. 6, pp. 2045–2053, information,” in Proc. IEEE Int. Conf. Acoust., vol. 8, pp. 46007–46016, 2020.
Jun. 2000. Speech Signal Process. (ICASSP), Calgary, AB, [115] R. Li et al., “The perils and pitfalls of block design
[78] T. Higuchi, “Approach to an irregular time series Canada, Apr. 2018, pp. 2556–2560. for EEG classification experiments,” IEEE Trans.
on the basis of the fractal theory,” Phys. D, [96] J.-P. Lachaux, E. Rodriguez, J. Martinerie, and Pattern Anal. Mach. Intell., vol. 43, no. 1,
Nonlinear Phenomena, vol. 31, no. 2, pp. 277–283, F. J. Varela, “Measuring phase synchrony in brain pp. 316–333, Jan. 2021.
Jun. 1988. signals,” Hum. Brain Mapping, vol. 8, no. 4, [116] F. Zhuang et al., “A comprehensive survey on
[79] P. C. Petrantonakis and L. J. Hadjileontiadis, pp. 194–208, 1999. transfer learning,” Proc. IEEE, vol. 109, no. 1,
“Emotion recognition from EEG using higher [97] C. J. Stam, G. Nolte, and A. Daffertshofer, “Phase pp. 1–34, Jul. 2020.
order crossings,” IEEE Trans. Inf. Technol. Biomed., lag index: Assessment of functional connectivity [117] D. Wu, Y. Xu, and B. Lu, “Transfer learning for
vol. 14, no. 2, pp. 186–197, Mar. 2010. from multi channel EEG and MEG with EEG-based brain–computer interfaces: A review of
[80] R.-N. Duan, J.-Y. Zhu, and B.-L. Lu, “Differential diminished bias from common sources,” Hum. progress made since 2016,” IEEE Trans. Cognit.
entropy feature for EEG-based emotion Brain Mapping, vol. 28, no. 11, pp. 1178–1193, Develop. Syst., vol. 14, no. 1, pp. 4–19, Mar. 2022.
classification,” in Proc. 6th Int. IEEE/EMBS Conf. 2007. [118] W. Zhang, L. Deng, L. Zhang, and D. Wu, “A survey
Neural Eng. (NER), San Diego, CA, USA, [98] S. Liu, X. Wang, L. Zhao, J. Zhao, Q. Xin, and on negative transfer,” IEEE/CAA J. Autom. Sinica,
Nov. 2013, pp. 81–84. S. Wang, “Subject-independent emotion vol. 10, no. 2, pp. 305–329, Feb. 2023.
[81] S. A. Hosseini, M. A. Khalilzadeh, recognition of EEG signals based on dynamic [119] X. Zhang et al., “Individual similarity guided
M. B. Naghibi-Sistani, and V. Niazmand, “Higher empirical convolutional neural network,” transfer modeling for EEG-based emotion
order spectra analysis of EEG signals in emotional IEEE/ACM Trans. Comput. Biol. Bioinf., vol. 18, recognition,” in Proc. IEEE Int. Conf. Bioinf.
stress states,” in Proc. 2nd Int. Conf. Inf. Technol. no. 5, pp. 1710–1721, Sep. 2021. Biomed. (BIBM), San Diego, CA, USA, Nov. 2019,
Comput. Sci., Kiev, Ukraine, Jul. 2010, pp. 60–63. [99] J. Zhang et al., “Disrupted brain connectivity pp. 1156–1161.
[82] L.-C. Shi, Y.-Y. Jiao, and B.-L. Lu, “Differential networks in drug-naive, first-episode major [120] Y. Lin, “Constructing a personalized cross-day
entropy feature for EEG-based vigilance depressive disorder,” Biol. Psychiatry, vol. 70, EEG-based emotion-classification model using
estimation,” in Proc. 35th Annu. Int. Conf. IEEE no. 4, pp. 334–342, Aug. 2011. transfer learning,” IEEE J. Biomed. Health
Eng. Med. Biol. Soc. (EMBC), Osaka, Japan, [100] J. Han, X. Ji, X. Hu, L. Guo, and T. Liu, “Arousal Informat., vol. 24, no. 5, pp. 1255–1264,
Jul. 2013, pp. 6627–6630. recognition using audio-visual features and May 2020.
[83] Y.-P. Lin, Y.-H. Yang, and T.-P. Jung, “Fusion of FMRI-based brain response,” IEEE Trans. Affect. [121] W. Zheng, Y. Zhang, J. Zhu, and B. Lu, “Transfer
electroencephalographic dynamics and musical Comput., vol. 6, no. 4, pp. 337–347, Oct. 2015. components between subjects for EEG-based
contents for estimating emotional responses in [101] N. Tzourio-Mazoyer et al., “Automated anatomical emotion recognition,” in Proc. Int. Conf. Affect.
music listening,” Frontiers Neurosci., vol. 8, p. 94, labeling of activations in SPM using a macroscopic Comput. Intell. Interact. (ACII), Xian, China,
May 2014. anatomical parcellation of the MNI MRI Sep. 2015, pp. 917–922.
[84] J. S. Richman and J. R. Moorman, “Physiological single-subject brain,” NeuroImage, vol. 15, no. 1, [122] S. Jialin Pan, I. W. Tsang, J. T. Kwok, and Q. Yang,
time-series analysis using approximate entropy pp. 273–289, Jan. 2002. “Domain adaptation via transfer component
and sample entropy,” Amer. J. Physiol.-Heart [102] D. Zhu et al., “DICCCOL: Dense individualizedand analysis,” IEEE Trans. Neural Netw., vol. 22, no. 2,
Circulatory Physiol., vol. 278, no. 6, common connectivity-based cortical landmarks,” pp. 199–210, Feb. 2011.
pp. H2039–H2049, Jun. 2000. Cerebral cortex, vol. 23, no. 4, pp. 786–800, 2013. [123] B. Schölkopf, A. Smola, and K.-R. Müller,
[85] C. Bandt and B. Pompe, “Permutation entropy: A [103] D. Watts and S. Strogatz, “Collective dynamics of “Nonlinear component analysis as a kernel
natural complexity measure for time series,” Phys. ‘small-world’ networks,” Nature, vol. 393, eigenvalue problem,” Neural Comput., vol. 10,
Rev. Lett., vol. 88, no. 17, Apr. 2002, pp. 440–442, Jun. 1998. no. 5, pp. 1299–1319, Jul. 1998.
Art. no. 174102. [104] V. Latora and M. Marchiori, “Efficient behavior of [124] X. Chai et al., “A fast, efficient domain adaptation
[86] M. Rostaghi and H. Azami, “Dispersion entropy: A small-world networks,” Phys. Rev. Lett., vol. 87, technique for cross-domain
measure for time-series analysis,” IEEE Signal Oct. 2001, Art. no. 198701. electroencephalography(EEG)-based emotion
Process. Lett., vol. 23, no. 5, pp. 610–614, [105] M. Rubinov and O. Sporns, “Complex network recognition,” Sensors, vol. 17, no. 5, p. 1014,
May 2016. measures of brain connectivity: Uses and May 2017.
[87] X. Ding, Z. Liu, D. Li, Y. He, and M. Wu, interpretations,” NeuroImage, vol. 52, no. 3, [125] P. Zanini, M. Congedo, C. Jutten, S. Said, and
“Electroencephalogram emotion recognition pp. 1059–1069, Sep. 2010. Y. Berthoumieu, “Transfer learning: A Riemannian
based on dispersion entropy feature extraction [106] Y. Sun, H. Ayaz, and A. N. Akansu, “Multimodal geometry framework with applications to
using random oversampling imbalanced data affective state assessment using fNIRS + EEG and brain–computer interfaces,” IEEE Trans. Biomed.
processing,” IEEE Trans. Cognit. Develop. Syst., spontaneous facial expression,” Brain Sci., vol. 10, Eng., vol. 65, no. 5, pp. 1107–1116, May 2018.
vol. 14, no. 3, pp. 882–891, Sep. 2022. no. 2, p. 85, Feb. 2020. [126] H. He and D. Wu, “Transfer learning for
[88] G. Giannakakis, D. Grigoriadis, K. Giannakaki, [107] H. Cai, Z. Qu, Z. Li, Y. Zhang, X. Hu, and B. Hu, brain–computer interfaces: A Euclidean space
O. Simantiraki, A. Roniotis, and M. Tsiknakis, “Feature-level fusion approaches based on data alignment approach,” IEEE Trans. Biomed.
“Review on psychological stress detection using multimodal EEG data for depression recognition,” Eng., vol. 67, no. 2, pp. 399–410, Feb. 2020.
biosignals,” IEEE Trans. Affect. Comput., vol. 13, Inf. Fusion, vol. 59, pp. 127–138, Jul. 2020. [127] Y. Wang, J. Liu, Q. Ruan, S. Wang, and C. Wang,
no. 1, pp. 440–460, Jan. 2022. [108] D. Wu et al., “Optimal arousal identification and “Cross-subject EEG emotion classification based
[89] S. K. Hadjidimitriou and L. J. Hadjileontiadis, classification for affective computing using on few-label adversarial domain adaption,” Exp.
“EEG-based classification of music appraisal physiological signals: Virtual reality Stroop task,” Syst. Appl., vol. 185, Dec. 2021, Art. no. 115581.
responses using time-frequency analysis and IEEE Trans. Affect. Comput., vol. 1, no. 2, [128] Y. Jiang, S. Xie, X. Xie, Y. Cui, and H. Tang,
familiarity ratings,” IEEE Trans. Affect. Comput., pp. 109–118, Jul. 2010. “Emotion recognition via multiscale feature fusion
vol. 4, no. 2, pp. 161–172, Jun. 2013. [109] Z. Liang et al., “EEGFuseNet: Hybrid unsupervised network and attention mechanism,” IEEE Sensors
[90] M. Murugappan, M. Rizon, R. Nagarajan, deep feature characterization and fusion for J., vol. 23, no. 10, pp. 10790–10800, May 2023.
S. Yaacob, I. Zunaidi, and D. Hazry, “EEG feature high-dimensional EEG with an application to [129] W.-L. Zheng and B.-L. Lu, “Personalizing
extraction for classifying emotions using FCM and emotion recognition,” IEEE Trans. Neural Syst. EEG-based affective models with transfer
FKM,” Int. J. Comput. Commun., vol. 1, no. 2, Rehabil. Eng., vol. 29, pp. 1913–1925, 2021. learning,” in Proc. 25th Int. Joint Conf. Artif. Intell.,
pp. 21–25, 2007. [110] B. García-Martínez, A. Martínez-Rodrigo, New York, NY, USA, Jul. 2016, pp. 2732–2738.
[91] L. Cohen, “Time-frequency distributions—A R. Alcaraz, and A. Fernández-Caballero, “A review [130] W. Li, W. Huan, B. Hou, Y. Tian, Z. Zhang, and
review,” Proc. IEEE, vol. 77, no. 7, pp. 941–981, on nonlinear methods using A. Song, “Can emotion be transferred?—A review
Jul. 1989. electroencephalographic recordings for emotion on transfer learning for EEG-based emotion

1330 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

recognition,” IEEE Trans. Cognit. Develop. Syst., performance prediction,” Theor. Issues Ergonom. Current Biol., vol. 28, no. 24, pp. 3893–3902,
vol. 14, no. 3, pp. 833–846, Jul. 2022. Sci., vol. 3, no. 2, pp. 159–177, Jan. 2002. Dec. 2018.
[131] J. Li, S. Qiu, Y. Shen, C. Liu, and H. He, [151] R. M. Yerkes and J. D. Dodson, “The relation of [170] F. Iacopi and C.-T. Lin, “A perspective on
“Multisource transfer learning for cross-subject strength of stimulus to rapidity of electroencephalography sensors for
EEG emotion recognition,” IEEE Trans. Cybern., habit-formation,” J. Comparative Neurol. Psychol., brain–computer interfaces,” Prog. Biomed. Eng.,
vol. 50, no. 7, pp. 3281–3293, Jul. 2020. vol. 18, no. 5, pp. 459–482, Nov. 1908. vol. 4, no. 4, Oct. 2022, Art. no. 043002.
[132] I. Goodfellow, Y. Bengio, and A. Courville, Deep [152] A.-M. Brouwer, M. A. Hogervorst, J. B. F. van Erp, [171] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and
Learning. Boston, MA, USA: MIT Press, 2016. T. Heffelaar, P. H. Zimmerman, and R. Oostenveld, L. Fei-Fei, “ImageNet: A large-scale hierarchical
[Online]. Available: “Estimating workload using EEG spectral power image database,” in Proc. IEEE Conf. Comput. Vis.
http://www.deeplearningbook.org and ERPs in the n-back task,” J. Neural Eng., Pattern Recognit., Miami Beach, FL, USA,
[133] Y. Luo, S.-Y. Zhang, W.-L. Zheng, and B.-L. Lu, vol. 9, no. 4, Aug. 2012, Art. no. 045008. Jun. 2009, pp. 248–255.
“WGAN domain adaptation for EEG-based [153] C. Muhl, C. Jeunet, and F. Lotte, “EEG-based [172] GPT-3. Accessed: Oct. 1, 2022. [Online].
emotion recognition,” in Proc. Int. Conf. Neural Inf. workload estimation across affective contexts,” Available: https://en.wikipedia.org/wiki/GPT-3
Process., Siem Reap, Cambodia, Dec. 2018, Frontiers Neurosci., vol. 8, p. 114, Jun. 2014. [173] Data and Statistics on Children’s Mental Health.
pp. 275–286. [154] Y. Zhou, S. Huang, Z. Xu, P. Wang, X. Wu, and Accessed: Oct. 1, 2022. [Online]. Available:
[134] D. Ruta and B. Gabrys, “An overview of classifier D. Zhang, “Cognitive workload recognition using https://www.cdc.gov/childrensmentalhealth/
fusion methods,” Comput. Inf. Syst., vol. 7, no. 1, EEG signals and machine learning: A review,” IEEE data.html
pp. 1–10, 2000. Trans. Cognit. Develop. Syst., vol. 14, no. 3, [174] K. Xia et al., “Privacy-preserving brain–computer
[135] Y. Huang, J. Yang, P. Liao, and J. Pan, “Fusion of pp. 799–818, Sep. 2022. interfaces: A systematic review,” IEEE Trans.
facial expressions and EEG for multimodal [155] P. Aricò et al., “Adaptive automation triggered by Computat. Social Syst., early access,
emotion recognition,” Comput. Intell. Neurosci., EEG-based mental workload index: A passive Jun. 27, 2023, doi: 10.1109/TCSS.2022.3184818.
vol. 2017, Sep. 2017, Art. no. 2107451. brain–computer interface application in realistic [175] X. Kong, W. Kong, Q. Fan, Q. Zhao, and
[136] W.-L. Zheng, B.-N. Dong, and B.-L. Lu, air traffic control environment,” Frontiers Hum. A. Cichocki, “Task-independent EEG identification
“Multimodal emotion recognition using EEG and Neurosci., vol. 10, p. 539, Oct. 2016. via low-rank matrix decomposition,” in Proc. IEEE
eye tracking data,” in Proc. 36th Annu. Int. Conf. [156] L. Zhang et al., “Cognitive load measurement in a Int. Conf. Bioinf. Biomed. (BIBM), Dec. 2018,
IEEE Eng. Med. Biol. Soc., Chicago, IL, USA, virtual reality-based driving system for autism pp. 412–419.
Aug. 2014, pp. 5040–5043. intervention,” IEEE Trans. Affect. Comput., vol. 8, [176] R. Yuste et al., “Four ethical piorities for
[137] M. Soleymani, M. Pantic, and T. Pun, “Multimodal no. 2, pp. 176–189, Apr. 2017. neurotechnologies and AI,” Nature, vol. 551,
emotion recognition in response to videos,” IEEE [157] Fatality Analysis Reporting System Encyclopedia. no. 7679, pp. 159–163, 2017.
Trans. Affect. Comput., vol. 3, no. 2, pp. 211–223, National Highway Traffic Safety Administration. [177] M. Ienca, P. Haselager, and E. J. Emanuel, “Brain
Jun. 2012. Accessed: Feb. 2, 2017. [Online]. Available: leaks and consumer neurotechnology,” Nature
[138] W.-L. Zheng, Z.-F. Shi, and B.-L. Lu, “Building https://www-fars.nhtsa.dot.gov/Main/index.aspx Biotechnol., vol. 36, no. 9, pp. 805–810,
cross-subject EEG-based affective models using [158] L. Bergasa, J. Nuevo, M. Sotelo, R. Barea, and Oct. 2018.
heterogeneous transfer learning,” Chin. J. M. Lopez, “Real-time system for monitoring driver [178] A. Agarwal et al., “Protecting privacy of users in
Comput., vol. 43, no. 2, pp. 177–189, 2020, in vigilance,” IEEE Trans. Intell. Transp. Syst., vol. 7, brain–computer interface applications,” IEEE
Chinese. no. 1, pp. 63–67, Mar. 2006. Trans. Neural Syst. Rehabil. Eng., vol. 27, no. 8,
[139] X. Yan, L.-M. Zhao, and B.-L. Lu, “Simplifying [159] Y. Cui, Y. Xu, and D. Wu, “EEG-based driver pp. 1546–1555, Aug. 2019.
multimodal emotion recognition with single eye drowsiness estimation using feature weighted [179] W. Zhang, Z. Wang, and D. Wu, “Multi-source
movement modality,” in Proc. 29th ACM Int. Conf. episodic training,” IEEE Trans. Neural Syst. decentralized transfer for privacy-preserving
Multimedia, Oct. 2021, pp. 1057–1063. Rehabil. Eng., vol. 27, no. 11, pp. 2263–2273, BCIs,” IEEE Trans. Neural Syst. Rehabil. Eng.,
[140] F. Wang, M. Mao, L. Duan, Y. Huang, Z. Li, and Nov. 2019. vol. 30, pp. 2710–2720, 2022.
C. Zhu, “Intersession instability in fNIRS-based [160] D. Wu, V. J. Lawhern, S. Gordon, B. J. Lance, and [180] D. J. Miller, Z. Xiang, and G. Kesidis, “Adversarial
emotion recognition,” IEEE Trans. Neural Syst. C. Lin, “Driver drowsiness estimation from EEG learning targeting deep neural network
Rehabil. Eng., vol. 26, no. 7, pp. 1324–1333, signals using online weighted adaptation classification: A comprehensive review of defenses
Jul. 2018. regularization for regression (OwARR),” IEEE against attacks,” Proc. IEEE, vol. 108, no. 3,
[141] A. Etkin, C. Büchel, and J. J. Gross, “The neural Trans. Fuzzy Syst., vol. 25, no. 6, pp. 1522–1535, pp. 402–433, Mar. 2020.
bases of emotion regulation,” Nature Rev. Dec. 2017. [181] X. Zhang and D. Wu, “On the vulnerability of CNN
Neurosci., vol. 16, no. 11, pp. 693–700, Nov. 2015. [161] C.-S. Wei, Y.-P. Lin, Y.-T. Wang, C.-T. Lin, and classifiers in EEG-based BCIs,” IEEE Trans. Neural
[142] W. Huang, W. Wu, M. V. Lucas, H. Huang, Z. Wen, T.-P. Jung, “A subject-transfer framework for Syst. Rehabil. Eng., vol. 27, no. 5, pp. 814–825,
and Y. Li, “Neurofeedback training with an obviating inter- and intra-subject variability in May 2019.
electroencephalogram-based brain–computer EEG-based drowsiness detection,” NeuroImage, [182] Z. Liu, L. Meng, X. Zhang, W. Fang, and D. Wu,
interface enhances emotion regulation,” IEEE vol. 174, pp. 407–419, 2018. “Universal adversarial perturbations for CNN
Trans. Affect. Comput., early access, Dec. 13, 2021, [162] Depression. Accessed: Oct. 1, 2022. [Online]. classifiers in EEG-based BCIs,” J. Neural Eng.,
doi: 10.1109/TAFFC.2021.3134183. Available: https://www.who.int/news-room/fact- vol. 18, no. 4, Aug. 2021, Art. no. 0460a4.
[143] J. V. Hardt and J. Kamiya, “Anxiety change sheets/detail/depression [183] X. Zhang et al., “Tiny noise, big mistakes:
through electroencephalographic alpha feedback [163] J. J. Allen, W. G. Iacono, R. A. Depue, and P. Arbisi, Adversarial perturbations induce errors in
seen only in high anxiety subjects,” Science, “Regional electroencephalographic asymmetries in brain–computer interface spellers,” Nat. Sci. Rev.,
vol. 201, no. 4350, pp. 79–81, Jul. 1978. bipolar seasonal affective disorder before and vol. 8, no. 4, Apr. 2021, Art. no. nwaa233.
[144] R. Mennella, E. Patron, and D. Palomba, “Frontal after exposure to bright light,” Biol. Psychiatry, [184] R. Bian, L. Meng, and D. Wu, “SSVEP-based
alpha asymmetry neurofeedback for the reduction vol. 33, nos. 8–9, pp. 642–646, Apr. 1993. brain–computer interfaces are vulnerable to
of negative affect and anxiety,” Behaviour Res. [164] K. M. Putnam and L. B. McSweeney, “Depressive square wave attacks,” Sci. China Inf. Sci., vol. 65,
Therapy, vol. 92, pp. 32–40, May 2017. symptoms and baseline prefrontal EEG alpha no. 4, Apr. 2022, Art. no. 140406.
[145] P. Linhartová, A. Látalová, B. Kósa, T. Kaspárek, activity: A study utilizing ecological momentary [185] X. Jiang, L. Meng, S. Li, and D. Wu, “Active
C. Schmahl, and C. Paret, “FMRI neurofeedback in assessment,” Biol. Psychol., vol. 77, no. 2, poisoning: Efficient backdoor attacks to transfer
emotion regulation: A literature review,” pp. 237–240, Feb. 2008. learning based BCIs,” Sci. China Inf. Sci., 2023.
NeuroImage, vol. 193, pp. 75–92, Jun. 2019. [165] J. Li, Y. Hao, W. Zhang, X. Li, and B. Hu, “Effective [Online]. Available: https://www.sciengine.com/
[146] A.-C.-S. Kimmig et al., “Feasibility of NIRS-based connectivity based EEG revealing the inhibitory SCIS/doi/10.1007/s11432-022-3548-2
neurofeedback training in social anxiety disorder: deficits for distracting stimuli in major depression [186] L. Meng, C.-T. Lin, T.-P. Jung, and D. Wu,
Behavioral and neural correlates,” J. Neural disorders,” IEEE Trans. Affect. Comput., vol. 14, “White-box target attack for EEG-based BCI
Transmiss., vol. 126, no. 9, pp. 1175–1185, no. 1, pp. 694–705, Jan. 2023. regression problems,” in Proc. Int. Conf. Neural Inf.
Sep. 2019. [166] F. S. De Aguiar Neto and J. L. G. Rosa, “Depression Process., Sydney, NSW, Australia, Dec. 2019,
[147] C. Cusin and D. D. Dougherty, “Somatic therapies biomarkers using non-invasive EEG: A review,” pp. 476–488.
for treatment-resistant depression: ECT, TMS, Neurosci. Biobehavioral Rev., vol. 105, pp. 83–93, [187] A. Binnendijk, T. Marler, and E. M. Bartels,
VNS, DBS,” Biol. Mood Anxiety Disorders, vol. 2, Oct. 2019. Brain–Computer Interfaces: U.S. Military
no. 1, pp. 1–9, Dec. 2012. [167] J. S. Perlmutter and J. W. Mink, “Deep brain Applications and Implications, An Initial
[148] P. Antonenko, F. Paas, R. Grabner, and T. van Gog, stimulation,” Annu. Rev. Neurosci., vol. 29, p. 229, Assessment. Santa Monica, CA, USA: RAND
“Using electroencephalography to measure Jul. 2006. Corporation, 2020.
cognitive load,” Educ. Psychol. Rev., vol. 22, no. 4, [168] K. W. Scangos, G. S. Makhoul, L. P. Sugrue, [188] D. Wu et al., “Adversarial attacks and defenses in
pp. 425–438, Dec. 2010. E. F. Chang, and A. D. Krystal, “State-dependent physiological computing: A systematic review,”
[149] G. Du, W. Zhou, C. Li, D. Li, and P. X. Liu, responses to intracranial brain stimulation in a Nat. Sci. Open, vol. 2, no. 1, Jan. 2023,
“An emotion recognition method for game patient with depression,” Nature Med., vol. 27, Art. no. 20220023.
evaluation based on electroencephalogram,” IEEE no. 2, pp. 229–231, Feb. 2021. [189] C. Szegedy et al., “Intriguing properties of neural
Trans. Affect. Comput., vol. 14, no. 1, pp. 591–602, [169] V. R. Rao et al., “Direct electrical stimulation of networks,” in Proc. Int. Conf. Learn. Represent.,
Jan. 2023. lateral orbitofrontal cortex acutely improves mood Apr. 2014, pp. 1–10.
[150] C. D. Wickens, “Multiple resources and in individuals with symptoms of depression,” [190] N. Das et al., “Keeping the bad guys out:

Vol. 111, No. 10, October 2023 | P ROCEEDINGS OF THE IEEE 1331
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.
 Wu et al.: aBCIs: A Tutorial

Protecting and vaccinating deep learning with pp. 1369–1378. perturbations against deep neural networks,” in
JPEG compression,” 2017, arXiv:1705. [192] B. Biggio, B. Nelson, and P. Laskov, “Support Proc. IEEE Symp. Secur. Privacy (SP), San Jose, CA,
02900. vector machines under adversarial label noise,” in USA, May 2016, pp. 582–597.
[191] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and Proc. Asian Conf. Mach. Learn., Taipei, Taiwan, [194] A. Qayyum, J. Qadir, M. Bilal, and A. Al-Fuqaha,
A. Yuille, “Adversarial examples for semantic Nov. 2011, pp. 97–112. “Secure and robust machine learning for
segmentation and object detection,” in Proc. IEEE [193] N. Papernot, P. McDaniel, X. Wu, S. Jha, and healthcare: A survey,” IEEE Rev. Biomed. Eng.,
Int. Conf. Comput. Vis. (ICCV), Oct. 2017, A. Swami, “Distillation as a defense to adversarial vol. 14, pp. 156–180, 2021.

ABOUT THE AUTHORS

Dongrui Wu (Fellow, IEEE) received the B.E. Bin Hu (Fellow, IEEE) received the Ph.D.
degree in automatic control from the Uni- degree in computer science from the
versity of Science and Technology of China, Institute of Computing Technology, Chinese
Hefei, China, in 2003, the M.E. degree in Academy of Sciences, Beijing, China,
electrical engineering from the National Uni- in 1998.
versity of Singapore, Singapore, in 2005, He is currently a Professor and the Dean
and the Ph.D. degree in electrical engi- of the School of Medical Technology, Beijing
neering from the University of Southern Institute of Technology, Beijing. He has
California, Los Angeles, CA, USA, in 2009. more than 400 publications (more than
He is currently a Professor with the School of Artificial Intelligence 10 000 citations; H-index: 51).
and Automation, Huazhong University of Science and Technology, Prof. Hu is a member of the National Computer Science
Wuhan, China. He has more than 200 publications. His research Teaching and Steering Committee and the Science and Technology
interests include affective computing, brain–computer interface, Committee. He is a member of the Steering Council of Association
computational intelligence, and machine learning. for Computing Machinery (ACM) China. His awards include the 2014
Prof. Wu received the NAFIPS Early Career Award in 2014, the IEEE China Overseas Innovation Talent Award, the 2016 Chinese Ministry
SMC Society Early Career Award in 2017, the USERN Prize in Formal of Education Technology Invention Award, the 2018 Chinese
Sciences in 2020, the Chinese Association of Automation Early National Technology Invention Award, and the 2019 WIPO-CNIPA
Career Award in 2021, the Ministry of Education Young Scientist Award for Chinese Outstanding Patented Invention. He is a National
Award in 2022, and six Outstanding Paper Awards. He has been the Distinguished Expert and a 973 Chief Scientist, and was awarded
Editor-in-Chief of IEEE TRANSACTIONS ON FUZZY SYSTEMS since 2023. National Advanced Worker in 2020. He is a Principal Investigator
of large grants, such as the National Transformative Technology
“Early Recognition and Intervention Technology of Mental Disorders
Bao-Liang Lu (Fellow, IEEE) received the Based on Psychophysiological Multimodal Information,” which
B.S. degree in instrument and control engi- has greatly promoted the development of objective, quantitative
neering from the Qingdao University of Sci- diagnosis, and nondrug interventions for mental disorders. He is
ence and Technology, Qingdao, China, in the Vice-Chair of the China Committee of the International Society
1982, the M.S. degree in computer science for Social Neuroscience and the Co-Chair of the Computational
and technology from Northwestern Polytech- Psychophysiology and Cognitive Computing Technical Committees
nical University, Xi’an, China, in 1989, and of the IEEE Systems, Man and Cybernetics Society. He also serves
the Dr.Eng. degree in electrical engineering as the Editor-in-Chief of IEEE TRANSACTIONS ON COMPUTATIONAL
from Kyoto University, Kyoto, Japan, in 1994. SOCIAL SYSTEMS.
He was with the Qingdao University of Science and Technol-
ogy from 1982 to 1986. From 1994 to 1999, he was a Frontier
Researcher with the Bio-Mimetic Control Research Center, Institute
of Physical and Chemical Research (RIKEN), Nagoya, Japan, and a
Research Scientist with the RIKEN Brain Science Institute, Wako, Zhigang Zeng (Fellow, IEEE) received the
Japan, from 1999 to 2002. Since 2002, he has been a Full Professor Ph.D. degree in systems analysis and inte-
with the Department of Computer Science and Engineering, Shang- gration from the Huazhong University of
hai Jiao Tong University, Shanghai, China. He is an Executive Dean Science and Technology, Wuhan, China,
of the Qing Yuan Research Institute, Shanghai Jiao Tong University, in 2003.
where he is also the Director of the Center for Brain-Like Computing He is currently a Professor with the School
and Machine Intelligence, the Key Laboratory of Shanghai Educa- of Artificial Intelligence and Automation,
tion Commission Intelligent Interaction and Cognitive Engineering, Huazhong University of Science and Tech-
and the Ruijin-Mihoyo Laboratory. His research interests include nology, and the Key Laboratory of Image
brain-like computing, deep learning, emotion artificial intelligence, Processing and Intelligent Control of the Education Ministry of
and affective brain–computer interface. China, Wuhan. He has published over 300 international journal arti-
Prof. Lu received the 2018 IEEE TRANSACTIONS ON AUTONOMOUS cles. His current research interests include the theory of functional
MENTAL DEVELOPMENT Outstanding Paper Award, the First Prize of differential equations and differential equations with discontinuous
the 10th Wu Wenjun Artificial Intelligence Natural Science Award, right-hand sides, and their applications to the dynamics of neural
the 2021 Best of IEEE TRANSACTIONS ON AFFECTIVE COMPUTING Paper networks, memristive systems, and associative memories.
Collection, and the 2022 ANNS (Asia Pacific Neural Network Soci- Dr. Zeng has been a member of the Editorial Board of Neu-
ety) Outstanding Achievement Award. He was the Past President ral Networks since 2012, Cognitive Computation since 2010, and
of the Asia Pacific Neural Network Assembly and the General Applied Soft Computing since 2013. He was an Associate Editor of
Chair of the 18th International Conference on Neural Informa- IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS from
tion Processing. He is an Associate Editor of IEEE TRANSACTIONS 2010 to 2011 and IEEE TRANSACTIONS ON FUZZY SYSTEMS from 2016
ON COGNITIVE AND DEVELOPMENTAL SYSTEMS, IEEE TRANSACTIONS ON to 2021. He has been an Associate Editor of IEEE TRANSACTIONS ON
AFFECTIVE COMPUTING, and the Journal of Neural Engineering. CYBERNETICS since 2014.

1332 P ROCEEDINGS OF THE IEEE | Vol. 111, No. 10, October 2023
Authorized licensed use limited to: UNIVERSIDADE DE SAO PAULO. Downloaded on March 19,2025 at 13:41:25 UTC from IEEE Xplore. Restrictions apply.