{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T05:13:03Z","timestamp":1771564383452,"version":"3.50.1"},"reference-count":97,"publisher":"MDPI AG","issue":"14","license":[{"start":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T00:00:00Z","timestamp":1658102400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"DFG Research Training Group","doi-asserted-by":"publisher","award":["WisPerMed"],"award-info":[{"award-number":["WisPerMed"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"name":"University of Duisburg-Essen","award":["WisPerMed"],"award-info":[{"award-number":["WisPerMed"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Digital histopathology poses several challenges such as label noise, class imbalance, limited availability of labelled data, and several latent biases to deep learning, negatively influencing transparency, reproducibility, and classification performance. In particular, biases are well known to cause poor generalization. Proposed tools from explainable artificial intelligence (XAI), bias detection, and bias discovery suffer from technical challenges, complexity, unintuitive usage, inherent biases, or a semantic gap. A promising XAI method, not studied in the context of digital histopathology is automated concept-based explanation (ACE). It automatically extracts visual concepts from image data. Our objective is to evaluate ACE\u2019s technical validity following design science principals and to compare it to Guided Gradient-weighted Class Activation Mapping (Grad-CAM), a conventional pixel-wise explanation method. To that extent, we created and studied five convolutional neural networks (CNNs) in four different skin cancer settings. Our results demonstrate that ACE is a valid tool for gaining insights into the decision process of histopathological CNNs that can go beyond explanations from the control method. ACE validly visualized a class sampling ratio bias, measurement bias, sampling bias, and class-correlated bias. Furthermore, the complementary use with Guided Grad-CAM offers several benefits. Finally, we propose practical solutions for several technical challenges. In contradiction to results from the literature, we noticed lower intuitiveness in some dermatopathology scenarios as compared to concept-based explanations on real-world images.<\/jats:p>","DOI":"10.3390\/s22145346","type":"journal-article","created":{"date-parts":[[2022,7,19]],"date-time":"2022-07-19T00:19:21Z","timestamp":1658189961000},"page":"5346","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":21,"title":["Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1595-5155","authenticated-orcid":false,"given":"Daniel","family":"Sauter","sequence":"first","affiliation":[{"name":"Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany"}]},{"given":"Georg","family":"Lodde","sequence":"additional","affiliation":[{"name":"Department of Dermatology, University Hospital Essen, 45147 Essen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5811-7100","authenticated-orcid":false,"given":"Felix","family":"Nensa","sequence":"additional","affiliation":[{"name":"Institute for AI in Medicine (IKIM), University Hospital Essen, 45131 Essen, Germany"},{"name":"Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, 45147 Essen, Germany"}]},{"given":"Dirk","family":"Schadendorf","sequence":"additional","affiliation":[{"name":"Department of Dermatology, University Hospital Essen, 45147 Essen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8279-9239","authenticated-orcid":false,"given":"Elisabeth","family":"Livingstone","sequence":"additional","affiliation":[{"name":"Department of Dermatology, University Hospital Essen, 45147 Essen, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6358-1824","authenticated-orcid":false,"given":"Markus","family":"Kukuk","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Fachhochschule Dortmund, 44227 Dortmund, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2022,7,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"341ps12","DOI":"10.1126\/scitranslmed.aaf5027","article-title":"What does research reproducibility mean?","volume":"8","author":"Goodman","year":"2016","journal-title":"Sci. Transl. Med."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1037\/a0015108","article-title":"Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences","volume":"13","author":"Schmidt","year":"2009","journal-title":"Rev. Gen. Psychol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1038\/515009a","article-title":"Metascience could rescue the \u2018replication crisis\u2019","volume":"515","author":"Schooler","year":"2014","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ioannidis, J.P.A. (2005). Why Most Published Research Findings Are False. PLoS Med., 2.","DOI":"10.1371\/journal.pmed.0020124"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"E14","DOI":"10.1038\/s41586-020-2766-y","article-title":"Transparency and reproducibility in artificial intelligence","volume":"586","author":"Adam","year":"2020","journal-title":"Nature"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e23436","DOI":"10.2196\/23436","article-title":"Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study","volume":"23","author":"Schmitt","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6423","DOI":"10.1038\/s41598-020-62724-2","article-title":"Resolving challenges in deep learning-based analyses of histopathological images using explanation methods","volume":"10","author":"Seegerer","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Murchan, P., \u00d3\u2019Brien, C., O\u2019Connell, S., McNevin, C.S., Baird, A.-M., Sheils, O., \u00d3 Broin, P., and Finn, S.P. (2021). Deep Learning of Histopathological Features for the Prediction of Tumour Molecular Genetics. Diagnostics, 11.","DOI":"10.3390\/diagnostics11081406"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"101813","DOI":"10.1016\/j.media.2020.101813","article-title":"Deep neural network models for computational histopathology: A survey","volume":"67","author":"Srinidhi","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1328","DOI":"10.1038\/s41591-021-01461-z","article-title":"AI in medicine must be explainable","volume":"27","author":"Kundu","year":"2021","journal-title":"Nat. Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1002\/path.5921","article-title":"Developing image analysis methods for digital pathology","volume":"257","author":"Bankhead","year":"2022","journal-title":"J. Pathol."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Li, Z., and Xu, C. (2021, January 11\u201317). Discover the Unknown Biased Attribute of an Image Classifier. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01470"},{"key":"ref_13","unstructured":"Denton, E., Hutchinson, B., Mitchell, M., Gebru, T., and Zaldivar, A. (2019, January 16\u201317). Detecting Bias with Generative Counterfactual Face Attribute Augmentation. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA."},{"key":"ref_14","unstructured":"Denton, E., Hutchinson, B., Mitchell, M., Gebru, T., and Zaldivar, A. (2019, January 16\u201317). Image Counterfactual Sensitivity Analysis for Detecting Unintended Bias. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Balakrishnan, G., Xiong, Y., Xia, W., and Perona, P. (2020, January 23\u201328). Towards Causal Benchmarking of Bias in Face Analysis Algorithms. Proceedings of the Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58523-5_32"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., and Vetter, T. (2018, January 18\u201322). Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems. Proceedings of the 2018 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPRW.2018.00283"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., and Vetter, T. (2019, January 16\u201317). Analyzing and Reducing the Damage of Dataset Bias to Face Recognition with Synthetic Data. Proceedings of the 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00279"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"42370","DOI":"10.1109\/ACCESS.2021.3065456","article-title":"Bias Analysis on Public X-Ray Image Datasets of Pneumonia and COVID-19 Patients","volume":"9","author":"Igual","year":"2021","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Wang, W., and Zhu, S.-C. (2018, January 2\u20137). Examining CNN Representations with Respect to Dataset Bias. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11833"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Bissoto, A., Fornaciali, M., Valle, E., and Avila, S. (2019, January 16\u201317). (De)Constructing Bias on Skin Lesion Datasets. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Long Beach, CA, USA.","DOI":"10.1109\/CVPRW.2019.00335"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Pacini, M., Nesti, F., Biondi, A., and Buttazzo, G. (2021, January 26\u201328). X-BaD: A Flexible Tool for Explanation-Based Bias Detection. Proceedings of the 2021 IEEE International Conference on Cyber Security and Resilience (CSR), Rhodes, Greece.","DOI":"10.1109\/CSR51186.2021.9527956"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"51","DOI":"10.2478\/jaiscr-2021-0004","article-title":"Towards Explainable Classifiers Using the Counterfactual Approach\u2013Global Explanations for Discovering Bias in Data","volume":"11","author":"Grochowski","year":"2021","journal-title":"J. Artif. Intell. Soft Comput. Res."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1016\/j.inffus.2021.07.015","article-title":"Finding and removing Clever Hans: Using explanation methods to debug and improve deep models","volume":"77","author":"Anders","year":"2022","journal-title":"Inf. Fusion"},{"key":"ref_24","unstructured":"Anders, C.J., Neumann, D., Marinc, T., Samek, W., M\u00fcller, K.-R., and Lapuschkin, S. (2020, January 17). XAI for Analyzing and Unlearning Spurious Correlations in ImageNet. Proceedings of the 2020 ICML Workshop on Extending Explainable AI Beyond Deep Models and Classifiers (XXAI), Vienna, Austria."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1096","DOI":"10.1038\/s41467-019-08987-4","article-title":"Unmasking Clever Hans predictors and assessing what machines really learn","volume":"10","author":"Lapuschkin","year":"2019","journal-title":"Nat. Commun."},{"key":"ref_26","unstructured":"Krishnakumar, A., Prabhu, V., Sudhakar, S., and Hoffman, J. (2021, January 22\u201325). UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models. Proceedings of the 32nd British Machine Vision Conference, Virtual Event."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Reimers, C., Penzel, N., Bodesheim, P., Runge, J., and Denzler, J. (2021, January 19\u201325). Conditional dependence tests reveal the usage of ABCD rule features and bias variables in automatic skin lesion classification. Proceedings of the 2021 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).","DOI":"10.1109\/CVPRW53098.2021.00200"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Reimers, C., Runge, J., and Denzler, J. (2020, January 23\u201328). Determining the Relevance of Features for Deep Neural Networks. Proceedings of the Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58574-7_20"},{"key":"ref_29","unstructured":"Tian, Y., Zhong, Z., Ordonez, V., Kaiser, G., and Ray, B. (July, January 27). Testing DNN Image Classifiers for Confusion & Bias Errors. Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering, Seoul, Korea."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Serna, I., Pe\u00f1a, A., Morales, A., and Fierrez, J. (2021, January 10\u201315). InsideBias: Measuring Bias in Deep Networks and Application to Face Gender Biometrics. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italiy.","DOI":"10.1109\/ICPR48806.2021.9412443"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Wang, A., Narayanan, A., and Russakovsky, O. (2020, January 23\u201328). REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets. Proceedings of the 16th European Conference on Computer Vision, Computer Vision\u2014ECCV 2020, Glasgow, UK.","DOI":"10.1007\/978-3-030-58580-8_43"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Phillips, P.J., Hahn, C.A., Fontana, P.C., Yates, A.N., Greene, K., Broniatowski, D.A., and Przybocki, M.A. (2021, December 14). Four Principles of Explainable Artificial Intelligence. Available online: https:\/\/doi.org\/10.6028\/NIST.IR.8312.","DOI":"10.6028\/NIST.IR.8312"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Bach, S., Binder, A., Montavon, G., Klauschen, F., M\u00fcller, K.-R., and Samek, W. (2015). On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0130140"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13\u201317). \u201cWhy Should I Trust You?\u201d: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.","DOI":"10.1145\/2939672.2939778"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Stock, P., and Cisse, M. (2018, January 8\u201314). ConvNets and ImageNet beyond Accuracy: Understanding Mistakes and Uncovering Biases. Proceedings of the Computer Vision\u2014ECCV 2018, Munich, Germany.","DOI":"10.1007\/978-3-030-01231-1_31"},{"key":"ref_37","unstructured":"Tong, S., and Kagal, L. (2020, January 12\u201318). Investigating Bias in Image Classification using Model Explanations. Proceedings of the 2020 ICML Workshop on Human Interpretability in Machine Learning (WHI 2020), Virtual Event."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Young, K., Booth, G., Simpson, B., Dutton, R., and Shrapnel, S. (2019, January 17). Deep Neural Network or Dermatologist?. Proceedings of the Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, Shenzhen, China.","DOI":"10.1007\/978-3-030-33850-3_6"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"de Sousa, I.P., Vellasco, M.M.B.R., and Da Costa Silva, E. (2021). Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers. Sensors, 21.","DOI":"10.3390\/s21165657"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Korbar, B., Olofson, A.M., Miraflor, A.P., Nicka, C.M., Suriawinata, M.A., Torresani, L., Suriawinata, A.A., and Hassanpour, S. (2017, January 21\u201326). Looking Under the Hood: Deep Neural Network Visualization to Interpret Whole-Slide Image Analysis Outcomes for Colorectal Polyps. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.","DOI":"10.1109\/CVPRW.2017.114"},{"key":"ref_41","unstructured":"Yona, G., and Greenfeld, D. (2021, January 14). Revisiting Sanity Checks for Saliency Maps. Proceedings of the 1st Workshop on eXplainable AI Approaches for Debugging and Diagnosis (XAI4Debugging@NeurIPS2021), Virtual Event."},{"key":"ref_42","unstructured":"Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018, January 3\u20138). Sanity Checks for Saliency Maps. Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montr\u00e9al, QC, Canada."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., and Preece, A. (2020, January 7\u201312). Sanity Checks for Saliency Metrics. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i04.6064"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Graziani, M., de Sousa, I.P., Vellasco, M.M.B.R., Da Costa Silva, E., M\u00fcller, H., and Andrearczyk, V. (October, January 27). Sharpening Local Interpretable Model-Agnostic Explanations for Histopathology: Improved Understandability and Reliability. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2021, Strasbourg, France.","DOI":"10.1007\/978-3-030-87199-4_51"},{"key":"ref_45","unstructured":"Graziani, M., Lompech, T., M\u00fcller, H., and Andrearczyk, V. (2021, January 8\u20139). Evaluation and Comparison of CNN Visual Explanations for Histopathology. Proceedings of the AAAI Conference on Artificial Intelligence Workshops (XAI-AAAI-21), Virtual Event."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"103865","DOI":"10.1016\/j.compbiomed.2020.103865","article-title":"Concept attribution: Explaining CNN decisions to physicians","volume":"123","author":"Graziani","year":"2020","journal-title":"Comput. Biol. Med."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"e1424","DOI":"10.1002\/widm.1424","article-title":"Explainable artificial intelligence: An analytical review","volume":"11","author":"Angelov","year":"2021","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"ref_48","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., and Sayres, R. (2018, January 10\u201315). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_49","unstructured":"Graziani, M., Andrearczyk, V., and M\u00fcller, H. (2020\u201320, January 16). Regression Concept Vectors for Bidirectional Explanations in Histopathology. Proceedings of the Understanding and Interpreting Machine Learning in Medical Image Computing Applications, Granada, Spain."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Yeche, H., Harrison, J., and Berthier, T. (2019, January 17). UBS: A Dimension-Agnostic Metric for Concept Vector Interpretability Applied to Radiomics. Proceedings of the Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, Shenzhen, China.","DOI":"10.1007\/978-3-030-33850-3_2"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Khaleel, M., Tavanapong, W., Wong, J., Oh, J., and de Groen, P. (2021, January 7\u20139). Hierarchical Visual Concept Interpretation for Medical Image Classification. Proceedings of the 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), Aveiro, Portugal.","DOI":"10.1109\/CBMS52027.2021.00012"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Ge, Y., Xiao, Y., Xu, Z., Zheng, M., Karanam, S., Chen, T., Itti, L., and Wu, Z. (2021, January 19\u201325). A Peek into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00223"},{"key":"ref_53","unstructured":"Bahadori, M.T., and Heckerman, D. (2021, January 3\u20137). Debiasing Concept-based Explanations with Causal Analysis. Proceedings of the International Conference on Learning Representations, Virtual Event."},{"key":"ref_54","unstructured":"Kazhdan, D., Dimanov, B., Jamnik, M., Li\u00f2, P., and Weller, A. (2020, January 19\u201323). Now You See Me (CME): Concept-based Model Extraction. Proceedings of the CIKM 2020 Workshops, Virtual Event."},{"key":"ref_55","unstructured":"Ghorbani, A., Wexler, J., Zou, J.Y., and Kim, B. (2019, January 8\u201314). Towards Automatic Concept-based Explanations. Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Kamakshi, V., Gupta, U., and Krishnan, N.C. (2021, January 18\u201322). PACE: Posthoc Architecture-Agnostic Concept Extractor for Explaining CNNs. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.","DOI":"10.1109\/IJCNN52387.2021.9534369"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Zhang, R., Madumal, P., Miller, T., Ehinger, K.A., and Rubinstein, B.I.P. (2021, January 2\u20139). Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event.","DOI":"10.1609\/aaai.v35i13.17389"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Wu, W., Su, Y., Chen, X., Zhao, S., King, I., Lyu, M.R., and Tai, Y.-W. (2020, January 14\u201319). Towards Global Explanations of Convolutional Neural Networks with Concept Attribution. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00868"},{"key":"ref_59","unstructured":"Yeh, C.-K., Kim, B., Arik, S., Li, C.-L., Pfister, T., and Ravikumar, P. (2020, January 6\u201312). On Completeness-aware Concept-Based Explanations in Deep Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Virtual Event."},{"key":"ref_60","doi-asserted-by":"crossref","unstructured":"Lucieri, A., Bajwa, M.N., Braun, S.A., Malik, M.I., Dengel, A., and Ahmed, S. (2020, January 19\u201324). On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.","DOI":"10.1109\/IJCNN48605.2020.9206946"},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Fang, Z., Kuang, K., Lin, Y., Wu, F., and Yao, Y.-F. (2020, January 12\u201316). Concept-based Explanation for Fine-Grained Images and Its Application in Infectious Keratitis Classification. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.","DOI":"10.1145\/3394171.3413557"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Janik, A., Dodd, J., Ifrim, G., Sankaran, K., and Curran, K. (2021, January 15\u201320). Interpretability of a Deep Learning Model in the Application of Cardiac MRI Segmentation with an ACDC Challenge Dataset. Proceedings of the Medical Imaging 2021: Image Processing, Virtual Event.","DOI":"10.1117\/12.2582227"},{"key":"ref_63","unstructured":"Graziani, M., Brown, J.M., Andrearczyk, V., Yildiz, V., Campbell, J.P., Erdogmus, D., Ioannidis, S., Chiang, M.F., Kalpathy-Cramer, J., and M\u00fcller, H. (2019, January 13). Improved interpretability for computer-aided severity assessment of retinopathy of prematurity. Proceedings of the Medical Imaging 2019: Computer-Aided Diagnosis, San Diego, CA, USA."},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"2456","DOI":"10.1109\/TBME.2020.3043215","article-title":"Robust and Interpretable Convolutional Neural Networks to Detect Glaucoma in Optical Coherence Tomography Images","volume":"68","author":"Thakoor","year":"2021","journal-title":"IEEE. Trans. Biomed. Eng."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Clough, J.R., Oksuz, I., Puyol-Ant\u00f3n, E., Ruijsink, B., King, A.P., and Schnabel, J.A. (2019, January 13\u201317). Global and Local Interpretability for Cardiac MRI Classification. Proceedings of the Medical Image Computing and Computer Assisted Intervention\u2014MICCAI 2019, Shenzhen, China.","DOI":"10.1007\/978-3-030-32251-9_72"},{"key":"ref_66","doi-asserted-by":"crossref","unstructured":"Mincu, D., Loreaux, E., Hou, S., Baur, S., Protsyuk, I., Seneviratne, M., Mottram, A., Tomasev, N., Karthikesalingam, A., and Schrouff, J. (2021, January 8\u201310). Concept-based model explanations for Electronic Health Records. Proceedings of the Conference on Health, Inference, and Learning, Virtual Event.","DOI":"10.1145\/3450439.3451858"},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1038\/s43856-021-00013-3","article-title":"Determining breast cancer biomarker status and associated morphological features using deep learning","volume":"1","author":"Gamble","year":"2021","journal-title":"Commun. Med."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"Cai, C.J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G.S., and Stumpe, M.C. (2019, January 4\u20139). Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.","DOI":"10.1145\/3290605.3300234"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1038\/s41746-019-0131-z","article-title":"Similar image search for histopathology: SMILY","volume":"2","author":"Hegde","year":"2019","journal-title":"npj Digit. Med."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1093\/iwc\/iwv001","article-title":"An Experience Perspective on Intuitive Interaction: Central Components and the Special Effect of Domain Transfer Distance","volume":"27","author":"Diefenbach","year":"2015","journal-title":"Interact. Comput."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"75","DOI":"10.2307\/25148625","article-title":"Design Science in Information Systems Research","volume":"28","author":"Hevner","year":"2004","journal-title":"MIS Q."},{"key":"ref_72","doi-asserted-by":"crossref","first-page":"337","DOI":"10.25300\/MISQ\/2013\/37.2.01","article-title":"Positioning and Presenting Design Science Research for Maximum Impact","volume":"37","author":"Gregor","year":"2013","journal-title":"MIS Q."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Hevner, A., and Chatterjee, S. (2010). Design Research in Information Systems, Springer.","DOI":"10.1007\/978-1-4419-5653-8"},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Vaishnavi, V.K., and Kuechler, W. (2015). Design Science Research Methods and Patterns: Innovating Information and Communication Technology, CRC Press Taylor & Francis Group. [2nd ed.].","DOI":"10.1201\/b18448"},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Johannesson, P., and Perjons, E. (2021). An Introduction to Design Science, Springer International Publishing. [2nd ed.].","DOI":"10.1007\/978-3-030-78132-3"},{"key":"ref_76","doi-asserted-by":"crossref","first-page":"2274","DOI":"10.1109\/TPAMI.2012.120","article-title":"SLIC Superpixels Compared to State-of-the-Art Superpixel Methods","volume":"34","author":"Achanta","year":"2012","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_77","doi-asserted-by":"crossref","unstructured":"Linardatos, P., Papastefanopoulos, V., and Kotsiantis, S. (2021). Explainable AI: A Review of Machine Learning Interpretability Methods. Entropy, 23.","DOI":"10.3390\/e23010018"},{"key":"ref_78","unstructured":"Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. (2015, January 7\u20139). Striving for Simplicity: The All Convolutional Net. Proceedings of the 2015 International Conference on Learning Representations (ICLR) Workshop, San Diego, CA, USA."},{"key":"ref_79","unstructured":"National Cancer Institute (2022, January 24). The Cancer Genome Atlas Program, Available online: https:\/\/www.cancer.gov\/about-nci\/organization\/ccg\/research\/structural-genomics\/tcga."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Thomas, S., and Hamilton, N. (2022, May 18). Histopathology Non-Melanoma Skin Cancer Segmentation Dataset: Data Collection. Available online: https:\/\/espace.library.uq.edu.au\/view\/UQ:8be4bd0.","DOI":"10.14264\/8be4bd0"},{"key":"ref_81","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/3464903","article-title":"Biases in AI Systems","volume":"64","author":"Srinivasan","year":"2021","journal-title":"Commun. ACM"},{"key":"ref_82","doi-asserted-by":"crossref","first-page":"101915","DOI":"10.1016\/j.media.2020.101915","article-title":"Interpretable Deep Learning Systems for Multi-Class Segmentation and Classification of Non-Melanoma Skin Cancer","volume":"68","author":"Thomas","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_83","doi-asserted-by":"crossref","first-page":"1681","DOI":"10.1016\/j.cell.2015.05.044","article-title":"Genomic Classification of Cutaneous Melanoma","volume":"161","author":"Akbani","year":"2015","journal-title":"Cell"},{"key":"ref_84","doi-asserted-by":"crossref","unstructured":"Macenko, M., Niethammer, M., Marron, J.S., Borland, D., Woosley, J.T., Guan, X., Schmitt, C., and Thomas, N.E. (July, January 28). A method for normalizing histology slides for quantitative analysis. Proceedings of the 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Boston, MA, USA.","DOI":"10.1109\/ISBI.2009.5193250"},{"key":"ref_85","doi-asserted-by":"crossref","first-page":"101544","DOI":"10.1016\/j.media.2019.101544","article-title":"Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology","volume":"58","author":"Tellez","year":"2019","journal-title":"Med. Image Anal."},{"key":"ref_86","doi-asserted-by":"crossref","unstructured":"Taylor, L., and Nitschke, G. (2018, January 18\u201321). Improving Deep Learning with Generic Data Augmentation. Proceedings of the 2018 IEEE Symposium Series on Computational Intelligence (SSCI), Bangalore, India.","DOI":"10.1109\/SSCI.2018.8628742"},{"key":"ref_87","unstructured":"Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2020, January 7\u201312). Random Erasing Data Augmentation. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), New York, NY, USA."},{"key":"ref_88","doi-asserted-by":"crossref","unstructured":"Li, X., and Plataniotis, K.N. (2020). How much off-the-shelf knowledge is transferable from natural images to pathology images?. PLoS ONE, 15.","DOI":"10.1371\/journal.pone.0240530"},{"key":"ref_89","doi-asserted-by":"crossref","first-page":"21899","DOI":"10.1038\/s41598-020-78129-0","article-title":"Effectiveness of transfer learning for enhancing tumor classification with a convolutional neural network on frozen sections","volume":"10","author":"Kim","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_90","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_91","unstructured":"Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16\u201321). Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA."},{"key":"ref_92","unstructured":"Ramachandran, P., Zoph, B., and Le, Q.V. (May, January 30). Searching for Activation Functions. Proceedings of the 2018 International Conference on Learning Representations (ICLR) Workshop, Vancouver, BC, Canada."},{"key":"ref_93","unstructured":"Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., and Devin, M. (2021, December 16). TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Available online: https:\/\/www.tensorflow.org\/."},{"key":"ref_94","unstructured":"Google LLC. (2022, April 11). tf.keras.Model|TensorFlow Core v2.8.0. Available online: https:\/\/www.tensorflow.org\/api_docs\/python\/tf\/keras\/Model#fit."},{"key":"ref_95","unstructured":"Kempf, W., Hantschke, M., Kutzner, H., and Burgdorf, W.H. (2008). Dermatopathology, Steinkopff-Verlag Heidelberg."},{"key":"ref_96","doi-asserted-by":"crossref","unstructured":"Jin, X., An, H., Wang, J., Wen, K., and Wu, Z. (2021, January 22\u201324). Reducing the Annotation Cost of Whole Slide Histology Images Using Active Learning. Proceedings of the 2021 3rd International Conference on Image Processing and Machine Vision (IPMV), Hong Kong, China.","DOI":"10.1145\/3469951.3469960"},{"key":"ref_97","first-page":"1253","article-title":"Data-Driven Meets Theory-Driven Research in the Era of Big Data: Opportunities and Challenges for Information Systems Research","volume":"19","author":"Maass","year":"2018","journal-title":"J. Assoc. Inf. Syst."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5346\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:52:45Z","timestamp":1760140365000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/14\/5346"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,18]]},"references-count":97,"journal-issue":{"issue":"14","published-online":{"date-parts":[[2022,7]]}},"alternative-id":["s22145346"],"URL":"https:\/\/doi.org\/10.3390\/s22145346","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,18]]}}}