SAMI 2021 • IEEE 19th World Symposium on Applied Machine Intelligence and Informatics • January 21–23 • Herl’any, Slovakia

Explaining Deep Neural Network using Layer-wise

Relevance Propagation and Integrated Gradients
Ivan Čı́k∗ , Andrindrasana David Rasamoelina∗ , Marián Mach∗ , Peter Sinčák∗
∗ Departmentof Cybernetics
and Artificial Intelligence
Technical University of Košice
Slovakia

Abstract—Machine learning has become an integral part of increased attention. It is because of these areas that the field
technology in today’s world. The field of artificial intelligence of EXPLAINABLE AI, was born [7]. In this paper, we focus
is the subject of research by a wide scientific community. In on decisions of Deep Convolutional Neural Networks using
particular, through improved methodology, the availability of big
data, and increased computing power, today’s machine learning EfficientNet [8] neural network architecture on datasets like
algorithms can achieve excellent performance that sometimes MNIST[3], MNIST-Fashion dataset[4], Imagenette and Image-
even exceeds the human level. However, due to their nested woof which are subsets of ImageNet [5]. Explainable Artificial
nonlinear structure, these models are generally considered to Intelligence techniques used in experiments were Integrated
be ”Black boxes” that do not provide any information about Gradients [1] and Layer-wise Relevance Propagation [2].
what exactly leads them to provide a specific output. This
raised the need to interpret these algorithms and understand II. E XPLAINABLE A RTIFICIAL I NTELLIGENCE -
how they work as they are applied even in areas where they
can cause critical damage. This article describes Integrated I NTERPRETABILITY
Gradients [1] and Layer-wise Relevance Propagation [2] methods Artificial Intelligence approaches are now days used in
and presents individual experiments with. In experiments we every field spanning from e-commerce [9] to computer games
2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI) | 978-1-7281-8053-3/21/$31.00 ©2021 IEEE | DOI: 10.1109/SAMI50585.2021.9378686

have used well-known datasets like MNIST[3], MNIST-Fashion

dataset[4], Imagenette and Imagewoof which are subsets of [10] and including applications in healthcare [11, 12, 13],
ImageNet [5]. criminal justice [14, 15], 3D human action analysis [16],
point cloud classification [17], battlefield simulations [18], and
I. I NTRODUCTION more. When such machine learning algorithms are used as
The increase in computational power in recent years has solutions, we need to provide why and how outputs from these
made it possible to use increasingly computationally intensive algorithms are produced. Hence, providing the interpretability
algorithms achieving superhuman performance in many tasks in e.g., quantum mechanics could be complicated and perfor-
that were previously thought to be computationally unattain- mance costly [19].
able [6]. However, these algorithms became complex and There are applications and systems where explainability
it is not clear how they compute or predict their output. is not necessary, such as in the field of online translation
Due to these problems, model interpretability methods have services, object recognition in the image, and more. Although
become increasingly important. Model understanding is both the models of such systems achieve excellent performance,
an active area of research as well as an area of focus for their nonlinear structure makes them highly non-transparent,
practical applications across industries using machine learning. so it is not clear what led to the determination of the decision
Probably the most important area in this regard is healthcare reached by the system. Explanation of the model can differ
where mistakes can affect human life. Until now there has depending on the task and needs of the end-user. For example,
been no robust solution to these problems. The current problem a developer creating an artificial intelligence model may
is that we do not see nor understand in-depth deep learning prefer explanations that describe the insights of the model to
algorithms, therefore we do not understand the outcome. Lack facilitate debugging. On the other hand, the consumer may
of transparency and interpretability is the issue with many prefer explanations regarding his desired input and output. By
state-of-the-art models. In some cases, it would be unbearable explaining the model, users can understand the process that
for us to know what is happening in these algorithms, such as led to the system’s decision or prediction [20, 21] and after
medical diagnosis, where a confidence criterion is a basis for all, build trustworthiness to the model, [22]. Because today’s
a model decision. The key solution would be to know why the AI systems are trained on large numbers of data, in many
algorithm output is as trustworthy as obtaining this information cases even millions of examples, they can observe patterns in
would not reduce the performance of the algorithm. Due to data that people are unable to observe or apprehend properly.
these problems, explainable artificial intelligence (XAI) has Using explainable AI, it is possible to extract this knowledge
become an area of interest in the research community. The from the system to some extent to gain new knowledge [7].
recent development of methods for visualization, explanation, There are several definitions for explainable AI. One ap-
and interpretation of in-depth learning models has attracted proach to achieving Explainable AI is to develop models

978-1-7281-8053-3/21/$31.00 ©2021 IEEE 000381

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
I. Cık et al. • Explaining Deep Neural Network using Layer-wise Relevance Propagation and Integrated Gradients

that are understandable to its users and developers but to gather more data to enhance model performance. Others
develop methods to obtain some form of explanation from listed explainability as a debugging solution, as it may
complex models that are difficult for users to understand or be a powerful tool to debug deep neural networks.
even impossible to understand [23]. The goals that Explainable • Explain to improve - A model that can be clarified and
Artificial approaches should provide differ in the research understood is one that can be boosted easier.
community. Providing explainable models while maintaining • Explain to discover - Asking for answers is a valuable
a high level of learning performance and enabling the end- tool for learning new facts, gathering information, and
users to fully understand what leads to building trust are thus gaining knowledge.
the main goals in [24]. Instead of creating the methods for Knowledge and demand for explanation are growing in
explaining black boxes [25], we should develop models that different domains, hence the question as ”why the use of XAI
are inherently interpretable as it is stated by Rudin [15]. is not systematic?” or ”why is XAI not being used in every AI
Interpretability can differ from the task [26, 27, 28] however, system?”. It is currently a very difficult technical issue to add
there are also attempts to introduce general definitions [29]. interpretability to AI systems. In some cases expert systems
Usually, a human can comprehend 7+-2 pieces of information are explainable but inflexible and hard to use, sometimes we
at a time [30]. use Deep Neural Networks as a solution. The main advantage
At the Fairness, Accountability, and Transparency in Ma- of these algorithms is that they are effective but on the other
chine Learning, the main goal of explainability in machine hand it is virtually impossible to see inside. Further solution
learning, ” is to ensure that algorithmic decisions, as well as [35] were proposed to explain those family of the model, but
any data driving those decisions, can be explained to end-users again using the non-interpretable model to interpret or explain
and other stakeholders in non-technical terms” [31]. the non-interpretable model. The problem, remains the same.
Responsible AI (RAI) is about societal values, moral and Advanced machine learning algorithms go to the opposite
ethical considerations. Responsible AI has three main pillars end of the spectrum, generating systems able to function solely
[32]: from observations and construct their world models on which
• Accountability refers to the need for its owners, con- to base their predictions. Although, the complexity which
sumers and others with whom the program communicates gives ML algorithms exceptional predictive abilities often
to explain, and justify its decisions and actions. makes the results of the algorithms difficult to understand.
• Responsibility refers to the position of individuals them- Nevertheless, because of their structure and how they operate,
selves and the capacity of AI systems to react to their own ML algorithms are difficult to interpret. Intrinsically, ML
decisions and recognize errors or unexpected results. algorithms consider high-degree interactions between features
• Transparency refers to the need to describe, inspect and of inputs which make it difficult to disaggregate such functions
reproduce the mechanisms through which AI systems into humanly understandable forms. We take the most popular
make decisions and learns to adapt to its environment, contemporary ML model, the DNN as an example. DNN has
and to the governance of the data used created [31].In a common nonlinear multi-layer structure consisting of many
[23] transparency is one of the properties that can enable hidden layers and several neurons per layer, this architecture
interpretability. allows to generate high-level prediction by multiple levels
of linear transformations and nonlinear activations. While a
A. Why do we need XAI?
single linear transformation can be represented by looking at
There are at least four reasons, based on the explored the weights from the input features to each of the output
literature and usage nowadays: groups, multiple layers and non-linear correlations in each
• Explain to justify - There were several controversies layer suggest that a super-complicated hierarchical structure is
over AI / ML-powered systems that yielded biased or separated, which is a complex and theoretically problematic
discriminatory results over the last several years [33]. The process [36].
author showed an example from the field in which an AI
system trained to predict the risk of human pneumonia III. I NTEGRATED G RADIENTS
reached a completely wrong conclusion. Applying this Integrated Gradient (IG) [1] is a deep neural network
model in a black box manner will not reduce but increase explainability technique that visualizes the significance of its
the number of deaths associated with pneumonia [34]. input function that contributes to the prediction of the model. It
• Explain to control - Understanding system behavior pro- computes the integral of the gradients of the output prediction
vides greater visibility over unknown bugs and defects for the class with respect to the input image pixels and requires
and helps identify and correct errors in low critical no modification to the original deep neural network. One of
situations. Most data scientists are frustrated with the the main advantages of the integrated gradients method is
low or inconsistent model performance. Understanding that it can be used on various models such as image, text,
why the model performs so poorly on certain inputs, or structured data. IG can be used for:
and also to identify regions of the input space with • Understanding feature importance by extracting rules
lower average output. Besides, they seek advice on how from the network.
to design new features, remove redundant features, and • Debugging deep learning models performance.

000382

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 SAMI 2021 • IEEE 19th World Symposium on Applied Machine Intelligence and Informatics • January 21–23 • Herl’any, Slovakia

The algorithm does not use transitions and is therefore

applicable to general architectures (including networks with
discontinuous units). The LRP globally explains the classifi-
cation decision and the heatmap score has a clear interpretation
as evidence for or against the category [39].
LRP is a method of identifying important pixels by inverting
a neural network. The reverse pass is a conservative redistri-
bution process, in which the neurons that contribute the most
to the previous layer are the most important.

Fig. 1. Today’s systems without XAI against today’s systems with XAI [24]

• Identifying data skew by understanding the important

Fig. 2. Diagram of the LRP process. Each neuron is redistributed to lower
features contributing to the prediction. layers as much as it receives from higher layers. [40]
Integrated Gradients computes the average gradient while the
input varies along a linear path from a baseline x to x, where x Mathematically, it redistributes the prediction f (x) back-
is provided input. It is possible to apply this method to image ward, using redistribution rules, until it assigns a relevance
models or text models. This method can be applied in image score Ri to each variable. An important feature of the redis-
models, some text models, and other various tasks, the aim is tribution process is called global conservation property and is
to get the insights of our model and to enable users to engage given as:
with models better. The baseline is defined by the user and is X X X
usually selected as zero. Ri = ... = Rj = Rk = ... = f (x) (1)
i j k
A. CNN
The described property says that in each step of the re-
Convolutional Neural Networks have achieved great suc-
distribution process the overall relevance is maintained. The
cesses in a number of different learning tasks including image
relevance score Ri of each input variable determines the extent
recognition [37, 38]. Compared to other computer vision meth-
to which this variable has contributed to the prediction [2].
ods, they can be trained fully autonomously with only little
The global conservation principle is that it preserves the
pre-processing. The ability to allocate computational processes
propagated quantity between the neurons of two adjacent
to a graphics processing unit has accelerated the training (l)
layers. When we denote Ri as the significance associated
process and brought an explosion of use due to a speed-up (l+1)
in training time through parallelization. Convolutional Neural with the i-th neuron of layer l and Rj as the significance
Network architectures are providing good performance on associated with the j-th neuron in the next layer, the conser-
various tasks such as object detection, face recognition, and vation principle requires that
more. If resources are available, models can be scaled up for X (l)
X (l+1)
better accuracy. A study of model efficient model scaling came Ri = Rj (2)
i j
up with a solution that carefully balancing network depth,
width, and resolution can lead to better performance[8]. A new where the sums pass through all the neurons of the layers of
family of models, called EfficientNets was first introduced in the given neural model. Applying this rule repeatedly for all
[8]. layers will ensure that the heatmap resulting from the LRP
P (1)
IV. LRP F RAMEWORK complies p hp = f (x), where hp = Rp [2].
The LRP method has various modifications such as LRP2 ,
In this section, we describe the layer-wise relevance prop- LRPα1 , GB, PA, LRPCM P , and others. The methods and rules
agation (LRP) technique introduced in [2]. LRP method ex- that apply to them are described in [41]. For our experiments
plains deep neural network prediction and is one of the rele- we used -LRP which proceeds layer by layer, redistributing
vance propagation methods so that the prediction propagates prediction scores until it reaches the input layer. Starting at the
backward without the use of gradients. The algorithm starts output layer L and assigns the relevance of the target neuron
from the output layer L of the deep neural network, and c equal to the output of the neuron itself and the relevance of
then moves in the opposite direction in the graph, gradually all other neurons to zero.
redistributing the prediction score until the input is reached.
Each neuron receives a share in the output of the network and (
further redistributes it to its predecessors in the same amount Si (x), if unit i is the target unit of interest
riL = (3)
until the input variables are reached [2, 39]. 0, otherwise

000383

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
I. Cık et al. • Explaining Deep Neural Network using Layer-wise Relevance Propagation and Integrated Gradients

A recursive rule that redistributes the relevance of one layer First, we took all the test images that belong to the ”Coat”
to the next layer is the -rule class. For each image in this class, we used the LRP method
and added the individual values at the same positions ,and then
(l)
X zji (l+1) divided them by the number of images.
ri = P P rj
0 (zji0 + bj ) +  ∗ sign( 0 (zji0 + bj ))
j i i
(4)
where we defined zji to be the weighted activation of a
neuron i onto neuron j in the next layer and bj the additive
bias of unit j. A small amount of  is added to the denominator
of Equation 2 to avoid numerical instability [42].
V. E XPERIMENTS
In this section, we describe the experiments we performed
on various data. Our goal was to interpret the outputs from
neural networks. We tried to identify which areas were crucial
for classification at the input. Subsequently, we tried to create Fig. 4. Cumulative heatmap of the ”Coats” class.
a cumulative map for the whole group of inputs (class) and
not just for one specific input. We consider this work to be The output is the average heatmap for a given class, which
important, as artificial intelligence is also used in critical areas should visualize which area of the image most influences
where it can affect human life, so it is important to understand the classification decision for a given class in general. The
what is happening in individual algorithms. problem of misclassification comes when we have similar
heatmaps for two classes, e.g. dog breed. If our accuracy is not
A. LRP Experiments high enough, we would look at classes that have very similar
In the first experiment, we used -LRP method. In our cumulative heatmaps, and in the training pictures, we would
experiments, we used the well-known MNIST dataset [3], try to cover the areas on which they learn. In this way, we
which consists of handwritten numbers from 0 to 9, so it could get our model to consider other areas than identifying
contains 10 classes. Images measure 28 x 28 x 1 pixels. the class.
The dataset contains 60,000 training and 10,000 test images. B. Integrated Gradients Experiments
The neural network model had 2 hidden layers, each with
In the first experiment, we used the pre-trained EfficientNet-
256 neurons. On testing dataset, we achieved an accuracy of
B0 model, which was trained on ImageNet [5]. Transfer
96.75%.
learning was used to change the last layer to 10 classes output
and train the model on the Imagenette dataset. Imagenette is
a subset of 10 easily classified classes from ImageNet (tench,
English springer, cassette player, chain saw, church, French
horn, garbage truck, gas pump, golf ball, and parachute). For
each class there are 1000 images ± 40 images with resolution
of 320px x 320px. On validation test data we achieved an
accuracy of 90.05% after 20 epochs.

Fig. 3. Visualization of output and heatmap by -LRP method. Pixels are red
if their value is positive, or blue if their value is negative. The intensity of
the color depends on the distance of value from zero. Dark red pixels had the
greatest influence on the neural network decision when choosing a class. [40]

For further experiments with this method, we decided to

use a more complex dataset. We used the MNIST-Fashion
dataset[4], which has 10 classes, each class including a certain
type of clothing (T-Shirt/Top, Trouser, Pullover, Dress, Coat, Fig. 5. Overlayed Integrated Gradients on Tench
Sandals, Shirt, Sneaker, Bag, and Ankle boots). The amount of
data, architecture, number of epochs, and all parameter settings We took an example from tench breed, compute integrated
for training remained the same as in the first experiment, but gradients, and then overlay them on the original image. From
the accuracy was 86.14%, which was caused by the higher the output image, we can observe that the gradients are located
complexity of the MNIST-fashion dataset. on the tench.

000384

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
 SAMI 2021 • IEEE 19th World Symposium on Applied Machine Intelligence and Informatics • January 21–23 • Herl’any, Slovakia

In the second experiment, we used a sample from the R EFERENCES

Imagewoof dataset. Imagewoof is a subset of 10 classes from
Imagenet that are not so easy to classify since they are all dog [1] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic
breeds. We used a pretrained EfficientNet-B0 model trained attribution for deep networks,” arXiv preprint
on ImageNet data. arXiv:1703.01365, 2017.
In the second experiment, we used the pre-trained [2] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-
EfficientNet-B0 model, which was trained on ImageNet [5]. R. Müller, and W. Samek, “On pixel-wise explanations
Transfer learning was used to change the last layer to 10 for non-linear classifier decisions by layer-wise relevance
classes output and train the model on the Imagewoof dataset. propagation,” PloS one, vol. 10, no. 7, p. e0130140, 2015.
Imagewoof is a subset of 10 classes from Imagenet that are not [3] L. Deng, “The mnist database of handwritten digit im-
so easy to classify since they are all dog breeds. On validation ages for machine learning research [best of the web],”
test data we achieved an accuracy of 85.15% after 20 epochs. IEEE Signal Processing Magazine, vol. 29, no. 6, pp.
From the accuracy value, we can observe that it was more 141–142, 2012.
difficult to train. [4] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a
novel image dataset for benchmarking machine learning
algorithms,” arXiv preprint arXiv:1708.07747, 2017.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and
L. Fei-Fei, “Imagenet: A large-scale hierarchical image
database,” in 2009 IEEE conference on computer vision
and pattern recognition. Ieee, 2009, pp. 248–255.
[6] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,”
nature, vol. 521, no. 7553, pp. 436–444, 2015.
[7] W. Samek, T. Wiegand, and K.-R. Müller, “Explain-
able artificial intelligence: Understanding, visualizing
and interpreting deep learning models,” arXiv preprint
arXiv:1708.08296, 2017.
[8] M. Tan and Q. V. Le, “Efficientnet: Rethinking
Fig. 6. Overlayed Integrated Gradients on Maltese dog model scaling for convolutional neural networks,” arXiv
preprint arXiv:1905.11946, 2019.
We took an example from the Maltese dog breed, compute [9] W. Wang and I. Benbasat, “Recommendation agents for
integrated gradients, and then overlay them on top of the electronic commerce: Effects of explanation facilities on
original image. The gradients are noisy and are not just on trusting beliefs,” J. of Management Information Systems,
the object but are distributed throughout the image. This may vol. 23, pp. 217–246, 05 2007.
be due to the high similarity of classes. The model cannot find [10] S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon,
unique features as all classes of dog breeds have similar ones. W. Samek, and K.-R. Müller, “Unmasking clever hans
predictors and assessing what machines really learn,” 02
VI. C ONCLUSION
2019.
In the experiments we used MNIST and MNIST-Fashion [11] J.-M. Fellous, G. Sapiro, A. Rossi, H. S. Mayberg,
data, our model was able to recognize numbers (96.75% ) and M. Ferrante, “Explainable artificial intelligence for
and clothing (86.14% ) with quite a high accuracy. Using neuroscience: Behavioral neurostimulation,” Frontiers in
the individual LRP and IG methods, we were able to detect Neuroscience, vol. 13, p. 1346, 2019.
the areas that were most important in the model’s decisions. [12] E. Tjoa and C. Guan, “A survey on explainable artificial
These techniques worked satisfactorily, but on the ImageWoof intelligence (xai): towards medical xai,” arXiv preprint
dataset, where we achieved an accuracy of 85.15% after arXiv:1907.07374, 2019.
20 epochs, these areas were noisy and ambiguous. Artificial [13] X. Zhang, L. Han, W. Zhu, L. Sun, and D. Zhang,
intelligence is applied in every area of the world. With such “An explainable 3d residual self-attention deep neural
a wide range of coverage comes great responsibility. It is network for joint atrophy localization and alzheimer’s
necessary to understand machine learning algorithms, whether disease diagnosis using structural mri,” arXiv preprint
by the developer or the user because their decisions can arXiv:2008.04024, 2020.
have critical impacts. This article provides an overview of the [14] C. Rudin, “Algorithms for interpretable machine learn-
theory of explainable artificial intelligence, describes selected ing,” in Proceedings of the 20th ACM SIGKDD inter-
methods - Layer-wise Relevance Propagation and Integrated national conference on Knowledge discovery and data
Gradients. The article also includes experiments that defined mining, 2014, pp. 1519–1519.
the problems we encountered. In the future, we will to further [15] ——, “Stop explaining black box machine learning mod-
develop the methods of explainable artificial intelligence and els for high stakes decisions and use interpretable models
contribute to this area. instead,” Nature Machine Intelligence, vol. 1, no. 5, pp.

000385

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.
I. Cık et al. • Explaining Deep Neural Network using Layer-wise Relevance Propagation and Integrated Gradients

206–215, 2019. [33] A. Howard, C. Zhang, and E. Horvitz, “Addressing

[16] T. S. Kim and A. Reiter, “Interpretable 3d human action bias in machine learning algorithms: A pilot study on
analysis with temporal convolutional networks,” in 2017 emotion recognition for intelligent systems,” in 2017
IEEE conference on computer vision and pattern recogni- IEEE Workshop on Advanced Robotics and its Social
tion workshops (CVPRW). IEEE, 2017, pp. 1623–1631. Impacts (ARSO). IEEE, 2017, pp. 1–7.
[17] M. Zhang, H. You, P. Kadam, S. Liu, and C.-C. J. Kuo, [34] R. Caruana, Y. Lou, J. Gehrke, P. Koch, M. Sturm, and
“Pointhop: An explainable machine learning method for N. Elhadad, “Intelligible models for healthcare: Predict-
point cloud classification,” IEEE Transactions on Multi- ing pneumonia risk and hospital 30-day readmission,”
media, 2020. in Proceedings of the 21th ACM SIGKDD international
[18] M. Fox, D. Long, and D. Magazzeni, “Explainable plan- conference on knowledge discovery and data mining,
ning,” arXiv preprint arXiv:1709.10256, 2017. 2015, pp. 1721–1730.
[19] G. Shmueli et al., “To explain or to predict?” Statistical [35] A. Rai, “Explainable ai: From black box to glass box,”
science, vol. 25, no. 3, pp. 289–310, 2010. Journal of the Academy of Marketing Science, vol. 48,
[20] F. Morelli, S. Geschwill, K. Zerr, and C. Los- no. 1, pp. 137–141, 2020.
sos, “Rationalität maschineller entscheidungen im un- [36] S. Tan, K. C. Sim, and M. Gales, “Improving the
ternehmen durch die einbindung von explainable artifi- interpretability of deep neural networks with stimulated
cial intelligence “(xai),” Angewandte Forschung in der learning,” in 2015 IEEE Workshop on Automatic Speech
Wirtschaftsinformatik, p. 8. Recognition and Understanding (ASRU). IEEE, 2015,
[21] W. Wang and I. Benbasat, “Recommendation agents for pp. 617–623.
electronic commerce: Effects of explanation facilities on [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet
trusting beliefs,” Journal of Management Information classification with deep convolutional neural networks,”
Systems, vol. 23, no. 4, pp. 217–246, 2007. in Advances in neural information processing systems,
[22] A. Das and P. Rad, “Opportunities and challenges in 2012, pp. 1097–1105.
explainable artificial intelligence (xai): A survey,” arXiv [38] N. J. Nilsson, “Stuart russell and peter norvig, artificial
preprint arXiv:2006.11371, 2020. intelligence: A modern approach,” Artificial intelligence,
[23] Z. C. Lipton, “The mythos of model interpretability,” vol. 82, no. 1-2, pp. 369–380, 1996.
Queue, vol. 16, no. 3, pp. 31–57, 2018. [39] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and
[24] D. Gunning, “Explainable artificial intelligence (xai),” K.-R. Müller, “Evaluating the visualization of what a
Defense Advanced Research Projects Agency (DARPA), deep neural network has learned,” IEEE transactions on
nd Web, vol. 2, no. 2, 2017. neural networks and learning systems, vol. 28, no. 11,
[25] A. Adadi and M. Berrada, “Peeking inside the black- pp. 2660–2673, 2016.
box: A survey on explainable artificial intelligence (xai),” [40] G. Montavon, A. Binder, S. Lapuschkin, W. Samek,
IEEE Access, vol. 6, pp. 52 138–52 160, 2018. and K.-R. Müller, “Layer-wise relevance propagation:
[26] A. Freitas, “Comprehensible classification models: A an overview,” in Explainable AI: interpreting, explaining
position paper,” ACM SIGKDD Explorations Newsletter, and visualizing deep learning. Springer, 2019, pp. 193–
vol. 15, pp. 1–10, 03 2014. 209.
[27] J. Huysmans, K. Dejaeger, C. Mues, J. Vanthienen, and [41] M. Kohlbrenner, A. Bauer, S. Nakajima, A. Binder,
B. Baesens, “An empirical evaluation of the comprehen- W. Samek, and S. Lapuschkin, “Towards best practice
sibility of decision table, tree and rule based predictive in explaining neural network decisions with lrp,” in
models,” Decision Support Systems, vol. 51, pp. 141–154, 2020 International Joint Conference on Neural Networks
04 2011. (IJCNN). IEEE, 2020, pp. 1–7.
[28] S. Rüping, “Learning interpretable models,” 10 2006. [42] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross,
[29] D. A. Melis and T. Jaakkola, “Towards robust inter- “Towards better understanding of gradient-based attribu-
pretability with self-explaining neural networks,” in Ad- tion methods for deep neural networks,” arXiv preprint
vances in Neural Information Processing Systems, 2018, arXiv:1711.06104, 2017.
pp. 7775–7784.
[30] G. A. Miller, “The magical number seven, plus or minus
two: Some limits on our capacity for processing informa-
tion.” Psychological review, vol. 63, no. 2, p. 81, 1956.
[31] M. FAT, “Fairness, accountability, and transparency in
machine learning,” Retrieved December, vol. 24, p. 2018,
2018.
[32] K. J. Danjuma, “Performance evaluation of ma-
chine learning algorithms in post-operative life ex-
pectancy in the lung cancer patients,” arXiv preprint
arXiv:1504.04646, 2015.

000386

Authorized licensed use limited to: University of New South Wales. Downloaded on May 18,2021 at [Link] UTC from IEEE Xplore. Restrictions apply.