The Performance of Machine and Deep Learning

Classifiers in Detecting Zero-Day Vulnerabilities

Faranak Abri1 , Sima Siami-Namini2 , Mahdi Adl Khanghah3 , Fahimeh Mirza Soltani3 , and Akbar Siami Namin1
1
Department of Computer Science, Texas Tech University, USA
2 Department of Mathematics and Statistics, Texas Tech University University, USA
3 Department of Computer Science, University of Debrecen, Hungary

{faranak.abri, sima.siami-namini, akbar.namin}@ttu.edu; {adl.mahdi1365, soltani.fahimeh1364}@gmail.com

analytical techniques should be employed. Approaches such

arXiv:1911.09586v1 [cs.CR] 21 Nov 2019

Abstract—The detection of zero-day attacks and vulnerabilities

is a challenging problem. It is of utmost importance for network as behavioral-based [2], data mining-based [3], learning-based
administrators to identify them with high accuracy. The higher [4], machine learning [5], and more notably deep learning-
the accuracy is, the more robust the defense mechanism will be.
In an ideal scenario (i.e., 100% accuracy) the system can detect based [6] have already been proposed to detect malware.
zero-day malware without being concerned about mistakenly The conventional machine learning algorithms developed
tagging benign files as malware or enabling disruptive malicious for addressing clustering problems employ various forms of
code running as none-malicious ones. This paper investigates classification techniques to group a given dataset into distinct
different machine learning algorithms to find out how well they sub-classes. For instance, regression-based approaches parti-
can detect zero-day malware. Through the examination of 34
machine/deep learning classifiers, we found that the random tion the given data into two or more clusters and then fit
forest classifier offered the best accuracy. The paper poses several multiple linear regression lines on the data within each cluster;
research questions regarding the performance of machine and whereas, the Support Vector Machine (SVM) algorithms in-
deep learning algorithms when detecting zero-day malware with tend to partition the given data into clusters by fitting a hyper-
zero rates for false positive and false negative. plane into the data with acceptable margins. Furthermore,
Index Terms—zero-day vulnerability, machine learning
ensemble learning-based approaches such as random forest
build multiple decision trees and then take the majority votes
I. I NTRODUCTION
of these decision trees and then perform clustering.
Malware is a malicious application, specifically developed As an extension to conventional machine learning algo-
to perform malicious activities including disruption, damaging rithms, the deep learning-based approaches [7]–[9] take the
the underlying computer system and data, or gaining autho- clustering problem into a deeper level by exploring the hidden
rized access to the computer systems. A common approach relationships between the features of the dataset and then
to identify malicious applications is through signature-based building a hierarchical model to perform clustering. While the
matching, where the profile of a suspicious application is time complexity of building such models is high, the deep
compared against the reported ones. The popular intrusion exploration of data helps in building more accurate clustering.
detection systems often employ such pattern-matching tech- Given the recent advancement in deep learning-based ap-
niques and tools such as YARA [1] to implement rule-based proaches to model prediction problems, a major challenging
detection mechanisms. These pattern matching approaches to question is whether these learning-based techniques and in
the malware detection problem, however, are static and thus particular machine and deep learning algorithms are effective
not capable of detecting all types of malware. Broadly speak- in detecting zero-day malicious software. This paper inves-
ing, pattern and signature-based malware detection mecha- tigates the performance of the conventional machine learn-
nisms have the ability of detecting only a small subset of ing techniques in comparison with their counterparts, deep
all classes of malware software and thus these approaches learning-based algorithms. The goal is to empirically observe
are less effective in detecting more sophisticated, obfuscated, whether these learning-based approaches are a suitable model-
unknown, or newly developed malware software. A special ing approach for detecting zero-day vulnerabilities. The paper
case of interest is zero-day, a type of malware with no history conducts several experiments with respect to four learning-
or clear remediation strategy. based clustering techniques:
The software vulnerability, which is unknown to system 1) Conventional machine learning
administrators and thus there is no known security patches or 2) Simple neural network learner (i.e., Single layer)
remedies to confront it, is called a zero-day vulnerability. Due 3) Deep learning with multiple layers
to the absent of a structured remediation strategy and mitiga- 4) Deep(er) learning with multiple hidden layers
tion plan, such vulnerabilities are exploitable by adversaries. While it is expected that these learning-based algorithms
The signature-based approaches to malware detection can- demonstrate acceptable performance and accuracy, it is crucial
not be utilized to catch zero-day malware applications. As a that the malware detection systems learned and developed
result, instead of syntactical approaches (i.e., signature-based) for catching zero-day vulnerabilities do not label legitimate
applications as malware (i.e., zero false positive), and do III. S TATE - OF - THE -A RT: Z ERO -DAY ATTACKS
not label malicious applications as legitimate (i.e., zero false
negative). An elevated false positive and false negative rates Bileg and Dumitras [10] conducted a study on zero-day
will cause either eliminating important files or enabling ma- attacks to identify their characteristics before and after dis-
licious applications to be installed and then activated on the closure. Based on their study, a zero-day attack lasts ap-
underlying operating systems and thus it leads to serious loss proximately for 312 days and at most 30 months. It was
or damage to the assets. Hence, it is of utmost importance also reported that once these vulnerabilities are revealed, the
to achieve extremely high accuracy and extremely low false number of their exploitation increases five times. They also
positive/negative predictions when building the model. suggested a heuristic approach for identifying zero-day attacks.
This paper investigates the performance of the aforemen- Miller [11] investigated several research questions about
tioned learning-based classes of algorithms for a given dataset efficient machine learning systems for detecting zero-day mal-
and measures the accuracy of the models built using popular ware including the application of feature selection techniques
cross validation approaches (k-fold). The key contributions of for this problem. It was suggested that the best approach
this paper are as follows: for feature selection is using a combination of features but
at the same time reducing the number of features based
1) Conduct several experimental studies to compare the
on the classifier type. Miller also suggested that combining
performance of various types of machine and deep
classification and clustering techniques and taking advantage
learning algorithms in detecting zero-day vulnerabilities.
of using different classifiers would lead to better results and a
2) Report that no individual learning algorithm was able to
safety net against false positives.
achieve 100% accuracy when fitting the models.
Sharma et al. [12] proposed a prevention strategy, called
3) Report that the conventional machine learning algo-
Distributed Diagnosis System (DDS), based on the context
rithms are as good as deep learning-based models.
graph strategy. The DDS system consists of three parts: 1)
4) Report that random forest outperforms other machine
Central Diagnosis System (CDS), 2) Local Diagnosis System
and deep learning algorithms in providing the best
(LDS), and 3) Semi Diagnosis System (SDS). The DDS
models for detecting zero-day vulnerabilities.
system was developed to guarantee the protection of IoT
The paper is organized as follows: Section II highlights devices once a possible zero-day vulnerability is identified.
the research questions to address. A background on zero-day Their strategy considers a protocol for sharing sensitive data
malware and vulnerability detection is presented in Section among IoT devices and other network entities. As a general
III. The experimental setup is provided in Section IV. Section prevention strategy, if an IoT device is infected, it will be
V lists the classifiers along with their parameters studied removed from the network before exploiting other entities.
in this paper. Section VI reports the results and compares Alazab et al. [13] developed a machine learning framework
the performance of different machine and deep learning al- to detect zero-day vulnerability. They trained eight different
gorithms. Section VII discusses some implications observed classifiers which worked based on the API call sequences and
while analyzing the experimental data. Section VIII concludes frequencies, as the feature set. Their framework first retrieves
the paper and highlights the future research directions. the API call sequences from the disassembled executable
and then updates the signature database based on these API
II. C HALLENGES AND R ESEARCH Q UESTIONS
calls and then reports the similarity value. The classification
This paper empirically investigates whether it is possible to is applied using a supervised learning algorithm with eight
train learning-based algorithms and then employ the developed classifiers. They achieved an acceptable true positive and false
model to predict whether a zero-day vulnerability or malware positive rates using Support Vector Machine (SVM) to detect
with no prior historical data can be detected. More specifically, unknown malware samples with obfuscated code.
the experiments conducted in this paper intend to address the Comar et al. [14] developed a two-level framework for zero-
following research questions: day malware detection. The framework consists of six parts:
RQ. 1 Does standardization of data help in enhancing the 1) a data capture module, 2) an intrusion detection/prevention
accuracy? system (IDS/IPS), 3) an information storage, 4) the feature ex-
RQ. 2 Is it possible to train a machine/deep learning algo- traction and transformation module, 5) a supervised classifier,
rithm and achieve 100% accuracy for prediction and and 6) an UI portal. Using a class-based profiling technique,
thus reduce the false positive/negative ratios to zero their framework was able to separate unknown malware form
for detecting zero-day vulnerabilities/malware? the known ones and then using network traffic features for
RQ. 3 Which machine/deep learning algorithm performs the their SVM classifier it could detect zero-day malware.
best for detecting zero-day malware? Zho and Pezaros [15] evaluated six different machine learn-
RQ. 4 Does the batch size, i.e., the size of the next batch ing models for zero-day intrusion detection. The train dataset
of data for training a model, influence the accuracy consists of fourteen different intrusions and their test dataset
of predictions? consists of eight different types of zero-day attacks. Using a
RQ. 5 Does over-training a machine/deep learning model decision tree classifier, they achieved 96% accuracy, 5% false
increase the accuracy of detecting zero-day malware? positive rate with acceptable overhead.
 Xiao et al. [16] designed an IoT attack model and discussed TABLE I
different learning techniques for IoT security. The super- T HE SELECTED FEATURES OF THE DATASET.
vised, unsupervised, and reinforcement learning algorithms Features
were employed for different security aspects such as malware Characteristics MajorLinkerVersion
detection, authentication, and access control. They provided MinorLinkerVersion SizeOfCode
SizeOfInitializedData SizeOfUninitializedData
a few backup security solutions with the machine learning- AddressOfEntryPoint BaseOfCode
based security schemes to enable reliable and secure IoT BaseOfData ImageBase
services. They also recommended some solutions to avoid SectionAlignment FileAlignment
MajorOperatingSystemVersion MinorOperatingSystemVersion
problems such as oversampling, insufficient training data, and MajorImageVersion MinorImageVersion
bad feature extraction in the learning methods. MajorSubsystemVersion MinorSubsystemVersion
SizeOfImage SizeOfHeaders
IV. E XPERIMENTAL S ETUP CheckSum Subsystem
DllCharacteristics SizeOfStackReserve
A. Dataset and Features SizeOfStackCommit SizeOfHeapReserve
SizeOfHeapCommit LoaderFlags
We obtained the data from the Kaggle Website [17]. Ac- NumberOfRvaAndSizes SectionsNb
cording to the hosting Website, the raw dataset had been SectionsMeanEntropy SectionsMinEntropy
obtained from the malware security partner of Meraz’18, the SectionsMaxEntropy SectionsMeanRawsize
SectionsMinRawsize SectionMaxRawsize
annual techno-cultural festival of IIT Bhiali. According to the SectionsMeanVirtualsize SectionsMinVirtualsize
contributor, the raw dataset included features for both malware SectionMaxVirtualsize ImportsNbDLL
and legitimate files. The dataset is offered in two pieces: The ImportsNb ImportsNbOrdinal
ExportNb ResourcesNb
first piece of data is the training dataset with 138, 047 records ResourcesMeanEntropy ResourcesMinEntropy
of sample data. The sample data itself consists of 96, 724 ResourcesMaxEntropy ResourcesMeanSize
records whose legitimate parameter is labeled as “0” and ResourcesMinSize ResourcesMaxSize
LoadConfigurationSize VersionInformationSize
41, 323 records are labeled as “1”. The number of features is legitimate (class Label)
reported as 55. The second part of the data is “unlabeled” test
data with 88, 258 records of sample data and 55 features. Since
the second part of the data is unlabeled, we only conducted
experiments on the first part of the (i.e., the training dataset). C. Experimental Setup
We ignored three features while building our classifiers We used the open-source Anaconda with Python 3.7.4,
namely: ID, Machine, and SizeOfOptionalHeader, TensorFlow 1.14.0, and ran the experiments on a MacBook
since these features appeared to be constant over the data. Pro laptop computer with 2.9 GHz Intel Core i9 processor
Table I lists the features that were used for building the and 32GB of Ram with 2400 MHz DDR4.
classifiers (i.e., 52). As the class label, the legitimate
feature was utilized, where values “0” and “1” represent the V. T HE C HOSEN C LASSIFIERS
benign and malicious files, respectively. A cross-validation The research team chose a diverse set of machine and deep
strategy (i.e., k-fold) is utilized to “validate” the fitness of learning algorithms grouped into three major classes:
the models built.
– Conventional Machine Learners including 1) Gaus-
Figure 1 illustrates the correlation matrix of the features.
sian Naive Bayes, 2) Quadratic Discriminant Analysis
The correlation values vary in the range of (−0.6, +0.3). Most
(QDA), 3) Logistic Regression, 4) AdaBoost, 5) K-
of the correlation values are close to zero indicating that there
Nearest Neighbors, 6) Decision Tree, and 6) Random
is no strong correlation between most of the features.
Forest.
B. Evaluation Metrics – Simple Neural Network with a single Layer, in which
the primary algorithm was the Multi-Layer Perceptron
The model_selection.cross_val_score method
(MLP) with a single layer of training.
was used to evaluate the performance of the classifiers studied.
– Simple Neural Network with Multiple Layers and
The method is offered by the Python sklearn package. Ac-
a Small Value for Epoch, in which an MLP-based
cording to its documentation, the method returns the accuracy
algorithm was utilized with multiple layers and a small
of cross validation where accuracy is defined as the fraction
number of model fitting.
of true (i.e., correct) predictions of the model. The accuracy
– Simple Neural Network with Multiple Layers and a
metric can be formally computed as following:
Larger Value for Epoch, in which an MLP was utilized
#Correct P redictions with multiple layers and a large number of training.
Accuracy =
#T otal P redictions The research team developed several Python scripts and
(1)
TP + TN utilized relevant packages to carry out the experiments. These
=
TP + TN + FP + FN machine/deep learning algorithms take in several parameters
Where TP, TN, FP, and FN represent True Positives, True that were controlled for the purpose of this study. Table II lists
Negatives, False Positive, and False Negatives, respectively. the Python packages used and the parameters sets.
 Fig. 1. Correlation heat map of the features.

– AdaBoostClassifier() is a meta-estimator that even after several hours of executions.

starts off with fitting a classifier on the original dataset. – LogisticRegression() An implementation of reg-
It then fits additional classifier with different weights on ular logistic regression. The default values for the param-
the same dataset to take the classifiers attention towards eters were used.
those instances, which have been classified incorrectly. – The KNeighborsClassifier() classifier is an im-
The n_estimators parameter is the maximum number plementation of the k nearest neighbors clustering. The
of estimators to consider when boosting is performed. parameter n_neighbors was used to determine the
The remaining parameters were left as default. number of clusters. The remaining parameters set as
– The DecisionTreeClassifier() classifier is a de- default.
cision tree classifier whose parameters were set as default. – QuadraticDiscriminantAnalysis() A classi-
– GaussianNB() performs online updates to model pa- fier that employs a quadratic decision boundary to group
rameters. The default parameter values were used. data. The parameters left as default.
– LinearSVC() is an efficient implementation of SVC – RandomForestClassifier() An ensemble-based
with a linear kernel. The parameters were left as default. classifier that fits a number of decision tree classi-
The Python SVC... library, an inefficient implementa- fiers on different subsets of the data. The parameter
tion of the SVM classifier, did not produce any results n_estimators is the number of trees in the forest.
 TABLE II
T HE P YTHON PACKAGES AND PARAMETER SETTINGS EXPLORED IN THIS STUDY.

Classifier Type Python Package Parameter Set & Values

Conventional Machine Learning
Adaptive Boosting Feature Reduction AdaBoostClassifier(...) n_estimators n = 50, 100, 200
Decision Tree Tree-based Decision Support DecisionTreeClassifier(...) Default
Gaussian Naive Bayes Probabilistic GaussianNB(...) Default
Linear SVM (LinearSVC) Supervised Learning LinearSVC(...) Default
Logistic Regression Logistic function LogisticRegression(...) Default
k Nearest Neighbors Unsupervised Learning KNeighborsClassifier(...) n_neighbors n = 3, 5, 7
Quadratic Discriminant Analysis General form of Linear Classifier QuadraticDiscriminantAnalysis(...) Default
Random Decision Forests Ensemble Learners RandomForestClassifier(...) n_estimators n = 10, 50, 100
Neural Networks with Single Layer
Multi-Layer Perceptron: MLP Feed-forward Neural Network MLPClassifier(...) Number of Neurons
Neural Networks with Multiple Layers and Small Epoch
Multi-Layer Perceptron: MLP Feed-forward Neural Network Self-Developed
Neural Networks with Multiple Layers and Larger Epoch
Multi-Layer Perceptron: MLP Feed-forward Neural Network Self-Developed

The remaining parameters left as default. also reports the performance achieved by each classifier when
– MLPClassifier() A feed-forwarding neural network. their data were standardized or left as raw (non-standardized).
The parameter hidden_layer_sizes represents the The classifiers are ordered with respect to accuracy achieved
number of neurons in the hidden layer: by each classifier.
– A call to this library with only one parameter value The classifiers are grouped into five classes with respect to
implies that there is only one layer with the num- their demonstrated performance: I) Poorly performed classi-
ber of neurons specified. MLPClassifier(100) fiers, II) Conventional Machine Learning classifiers with good
implies that there is only one layer with 100 neurons. performance, III) Simple Neural Network with Single Hidden
– A call to this library with two values implies that Layer, IV) Deep Learners with small Epochs, and V) Deep
there are two hidden layers whose number of neurons Learners with larger Epochs.
is specified. For instance, MLPClassifier(100,
1) means that there are two hidden layers each with A. RQ. 1: The Influence of Standardization?
100 and 1 neurons, respectively.
– A call to this library with three parameter values A quick glance at the accuracy achieved by applying each
implies that there are three hidden layers whose classifier on non-standardized and standardized forms of data
number of neurons is specified as inputs. For in- shows that (except a couple of cases (Gaussian, Naive Bayes
stance, MLPClassifier(100, 50, 1) means and Quadratic Discriminant Analysis (QDA))), the classifiers
that there are three hidden layers each with 100, 50, demonstrated an improved accuracy when the given data are
and 1 neurons, respectively. standardized. It was also observed that it takes much smaller
We noticed some differences in the performance when amount of time and effort to train models with standardized
dealing with raw (i.e., non-standardized) versus standardized data; whereas, when non-standardized data are fed into the
data. The standardization of data means changing the values classifiers, it takes considerable amount of time for training
to enforce the distribution of standard deviation from the and fitting the models.
mean to be equal to one. For the comparison purposes and The average mean values for accuracy obtained for different
discovering whether standardization affects the performance, classes of classifiers when non-standardized data are mod-
we performed the experiments on both non-standardized and eled are 61.57%, 90.61%, 94.78%, and 58.70% for Poorly
standardized data. The Python StandardScaler(...) performed classifiers, conventional machine learners, Simple
library was used for standardization. Neural Networks with only one hidden layer, and Deep learn-
We performed a stratified 10-fold cross validation on the ing models, respectively. Whereas, the calculated mean values
dataset. Furthermore, we employed the Python pipeline (i.e., for accuracy when standardized data are used are 47.79%,
pipeline(...)) to sequentially apply a list of transforms 98.88%, 99.21%, and 98.90% for Poorly performed classifiers,
and produce a final estimator. The primary purpose of utilizing conventional machine learners, Simple Neural Networks with
the pipeline is to add several layers of fitting-assessing proce- only one hidden layer, and Deep learning models, respectively.
dure. More specifically, the pipeline helped us with stacking Excluding the poorly performed classes of classifiers, we
two processes: 1) the standardization of the data, and 2) the observe that standardization help in improving accuracy by
application of the specified classifier. 98.88 − 90.61 = 8.27%, 99.21 − 94.78 = 4.43%, and
98.90 − 58.70 = 40.2%, for conventional machine learners,
VI. R ESULTS Simple Neural Networks with only one hidden layer and Deep
Table III reports the performance of each classifiers The learning models, respectively. It was also observed that the
performance is measured in terms of the mean values of accu- standard deviations calculated for accuracy were much higher
racy obtained along with their standard deviations. The table for non-standardized data compared to standardized data. The
 TABLE III
T HE PERFORMANCE OF CLASSIFIERS IN ASCENDING ORDER OF “ACCURACY ” FOR PIPED AND STANDARDIZED DATA (cv = 10).

Accuracy %
No Standardized Piped and Standardized
Classifier Mean SD Mean SD
(Sorted)
I) Poorly Performed Learners
Gaussian Naive Bayes 76.60% 39.49% 46.31% 0.46%
Quadratic Discriminant Analysis (QDA) 46.55% 34.29% 49.28% 0.45%
Average 61.57% 47.79%
II) Conventional Machine Learning
Logistic Regression 30.28% 45.20% 97.51% 0.19%
Linear SVM (LinearSVC) 84.51% 27.68% 97.61% 0.16%
AdaBoost(n = 50) 97.57% 3.65% 98.78% 0.09%
AdaBoost(n = 100) 97.77% 3.35% 98.88% 0.06%
AdaBoost(n = 200) 97.53% 3.87% 98.99% 0.08%
Nearest Neighbors (k = 7) 97.88% 2.01% 98.99% 0.08%
Nearest Neighbors (k = 5) 97.89% 2.12% 99.07% 0.08%
Nearest Neighbors (k = 3) 97.81% 2.45% 99.12% 0.06%
Decision Tree 95.20% 9.60% 99.24% 0.07%
Random Forest (n = 10) 96.85% 6.94% 99.45% 0.06%
Random Forest (n = 100) 97.04% 6.89% 99.51% 0.08%
Random Forest (n = 50) 97.07% 6.66% 99.51% 0.06%
Average 90.61% 98.88%
III) Simple Neural Network (One Layer)
Multi-Layer Perceptron: MLP (52)[BatchSize = Auto] 94.70% 4.36% 99.14% 0.07%
Multi-Layer Perceptron: MLP (100)[BatchSize = Auto] 95.19% 4.55% 99.21% 0.11%
Multi-Layer Perceptron: MLP (200)[BatchSize = Auto] 95.52% 4.12% 99.24% 0.09%
Multi-Layer Perceptron: MLP (400)[BatchSize = Auto] 93.72% 5.05% 99.25% 0.08%
Average 94.78% 99.21%
IV) Deep Learning (Multiple Layers): Epoch = 5
Multi-Layer Perceptron: MLP (30, 1)[BatchSize = 100] 65.01% 11.97% 98.73% 0.09%
Multi-Layer Perceptron: MLP (52, 1)[BatchSize = 100] 57.13% 16.26% 98.79% 0.07%
Multi-Layer Perceptron: MLP (100, 1)[BatchSize = 100] 45.04% 17.57% 98.83% 0.07%
Multi-Layer Perceptron: MLP (30, 1)[BatchSize = 5] 65.88% 11.99% 98.88% 0.09%
Multi-Layer Perceptron: MLP (52, 1)[BatchSize = 5] 58.03% 18.26% 98.90% 0.09%
Multi-Layer Perceptron: MLP (52, 30, 1)[BatchSize = 100] 67.58% 15.08% 98.90% 0.07%
Multi-Layer Perceptron: MLP (100, 1)[BatchSize = 5] 45.04% 17.57% 98.92% 0.10%
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 100] 50.02% 20.05% 98.95% 0.10%
Multi-Layer Perceptron: MLP (52, 30, 1)[BatchSize = 5] 64.54% 18.83% 98.95% 0.09%
Multi-Layer Perceptron: MLP (100, 80, 60, 40, 20, 10, 1)[BatchSize = 5] 66.05% 12.04% 98.97% 0.07%
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 5] 50.02% 20.05% 98.99% 0.10%
Multi-Layer Perceptron: MLP (100, 80, 60, 40, 20, 10, 1)[BatchSize = 100] 70.07% 0.00% 98.99% 0.10%
Average 58.70% 98.90%
V) Deep Learning (Multiple Layers Different Epochs
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 100],[Epoch = 50] – – 99.24% 0.10%
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 100],[Epoch = 100] – – 99.28% 0.08%
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 100],[Epoch = 400] – – 99.29% 0.06%
Multi-Layer Perceptron: MLP (100, 50, 1)[BatchSize = 100],[Epoch = 200] – – 99.33% 0.06%
Average – – 99.28%

primary reason might be due to the fact that larger and wider we employed a 10-fold cross-validation to optimize the models
scale of numerical raw values were used in model fitting. and that might explain why it was infeasible to build a perfect
model with zero false positive and false negative values.
B. RQ. 2: Achieving 100% Accuracy?
The research team tested different classifiers with different C. RQ. 3: The Best Classifier?
tuning parameters with the goal of achieving the mean value As Table III shows, the random forest was the best classifier
of 100% for accuracy. It turned out that achieving such a high with outstanding performance of achieving 99.51% on average
accuracy on model fitting and prediction was infeasible. While for accuracy, followed by decision tree achieving 99.24%.
some of the classifiers demonstrated very high accuracy and The simple neural networks with a single hidden layer also
thus promising results, it seems that building a model that performed very well. On average, they achieved 99.21% on
reduces the false positive and false negative ratios to zero is accuracy. The number of neurons on the singleton hidden
very difficult and hence there are some penalties with missing layer seems to have some light impacts on accuracy. For
such cases for detecting zero-day malware. However, it is also instance, an MLP model with only one hidden layer and
possible that building a perfect model with 100% accuracy 50 neurons achieved 99.14% accuracy; whereas, increasing
may imply that the model is overfitted and may perform poorly the number of neurons to 100, 200, and 400 increased the
for classifying unseen data. To avoid such overfitting problem, accuracy to 99.21%, 99.24%, and 99.25%, respectively. Since
 VII. D ISCUSSION
A. Standardization Is Important for Classification
According to our results, standardization is critical for
classification. The primary reason might be because of compu-
tational expenses involved in dealing with large numbers and
thus with higher standard variations. Some of these classifiers
utilize a distance metric (e.g., Euclidean distance) where the
square roots of the sum of the squared differences between
Fig. 2. Accuracy vs. Epochs. the observations are calculated for clustering the data items.
there are some computational costs associated with the number As a result, to accommodate such expensive computation
of neurons on the layer, and given the slights improvement when larger values are provided as data, the demands for
on the observed accuracy, the question is whether a more computational needs will be increased. Hence, since a larger
complex model is worthy to be built or a simpler model with standard deviation will affect the accuracy of the prediction.
slightly lower accuracy would be sufficient for the prediction.
The choice of this trade-off totally depends on the application B. Feature Reductions and Parameters Tuning
domain. As a special case, detecting zero-day malware is an The authors tuned several parameters of the classifiers.
important and critical task and thus increasing the accuracy as Furthermore, they studied several machine/deep learning al-
much as possible is indeed needed regardless of the cost. gorithms. These learners perform the task of classifications
differently. For instance, one utilizes hyperplanes to create
D. RQ. 4: The Influence of Batch Size in Deep Learning? clusters (e.g., SVM); whereas, some other uses ensemble
learning and take the majority votes to decide (e.g., random
The authors controlled the batch size parameters for deep
forest). Moreover, some of these techniques apply feature
learning classifiers with multiple layers. According to our
reductions and thus tune the parameters and build a model;
observations: the smaller batch size is, the better/fitter the
whereas, some other more advanced algorithms try to take into
model will be. For instance, a neural network with two hidden
account all features and then through deeper analysis adjust
layers each with 10 and 1 neurons but with batch size of
their contributions and weights to the final model. A potential
100 and 5 achieved the accuracy of 98.73% and 98.88%,
drawback of utilizing all features in the computation is the
respectively. The improvement seems to be very small. On the
overfitting problem and thus the model may suffer from being
other hand, training with smaller batch size appeared to take
tightly coupled to the seen data and thus unable to perform
more computation time and thus more expensive than training
well for unseen data. It also may cause adding noisy features
a model with a larger batch size.
into the computations. On the other hand, reducing features
It was also observed that for deep learning-based approaches may cause the problem of missing important relationships
with multiple layers: the larger and deeper the model is, the between parameters. The choice of feature reduction depends
better and fitter the classification will be. For instance, a model on the type of dataset and is an important decision and should
with two layers each having 30 and 1 neurons and batch size be handled with additional care.
of 100 achieved the accuracy of 98.73%. Whereas, a deeper
model with seven layers each having 100, 80, 60, 40, 20, and C. Conventional vs. Deep Learning Classifiers
1 neurons and with the batch size equal to 100 achieved the
accuracy of 98.99%. However, the improvement is very small. The authors expected to observe much better performance
from deep learning-based algorithms. While these deep learn-
ers performed very well, surprisingly, some of the conventional
E. RQ. 5: The Influence of Epochs in Deep Learning?
machine learning classifiers performed comparatively similar
The authors performed a systematic analysis on the influ- or even better. Given the lower cost of training associated with
ence of the number of iterations needed to train the classifiers. the conventional machine learning algorithms and at the same
It was observed that: the greater the number of epochs is, the time a considerably greater cost for training deep classifiers,
more accurate the model will be. For instance, an MLP model the conventional machine learning algorithms might be even a
(100, 50, 1) with Epochs = 50 achieved 99.24% accuracy; better choice compared to the deep learning-based algorithms.
whereas, an increase of Epochs to 200 enhanced the accuracy The deep learning-based classifiers demonstrate a consistent
to 99.33%. This observation might indicate that a smaller improvement achieved by building larger models and addi-
number of epochs might be sufficient to learn the key and tional training. However, a simple random forest algorithm
significant features of the data and thus by adding more rounds still outperforms even larger deep learning-based classifiers
of training stages, the model will not learn anything further with additional training. For instance, the performance demon-
(i.e., all features are already learned). As an example, Figure strated by Random Forest (i.e., 99.51%) and the best per-
2 illustrates the improvement of accuracy over epochs for the formance achieved by deep learning (i.e., 99.33%) after 200
model M LP (100, 50, 1)[BatchSize = 100][Epoch = 200]. epochs with three hidden layers is remarkable.
D. Deep or Deeper Classifiers? [3] Y. Ye, T. Li, D. Adjeroh, and S. S. Iyengar, “A survey on malware
detection using data mining techniques,” ACM Comput. Surv., vol. 50,
According to our results, larger and deeper classifiers tend no. 3, pp. 41:1–41:40, Jun. 2017.
to perform better and they build a more accurate model. [4] K. Rieck, T. Holz, C. Willems, P. Düssel, and P. Laskov, “Learning
and classification of malware behavior,” in Conference on Detection of
However, the improvement does not seem to be significant. Intrusions and Malware, and Vulnerability Assessment (DIMVA), 2008,
A simpler deep learning classifier (e.g., M LP (30, 1) with pp. 108–125.
batch size = 100 and accuracy of 98.73%) might perform [5] D. Gavrilu, M. Cimpoeu, and D. A. L. Ciortuz, “Malware detection
using machine learning,” in International Multi-conference on Computer
comparatively very similar to a deeper and larger classifier Science and Information Technology (IMCSIT), 2009.
(e.g., M LP (100, 80, 60, 40, 20, .10, 1) with batch size = 100 [6] W. Hardy, L. Chen, S. Hou, Y. Ye, and X. Li, “Dl4md: A deep learning
and accuracy of 98.99%. Hence, the choice of the depth of framework for intelligent malwaredetection,” in Int’l Conf. Data Mining
(DMIN’16), 2016.
deep classifiers depends on the desired level of accuracy. [7] S. Siami-Namini, N. Tavakoli, and A. S. Namin, “A comparison of
ARIMA and LSTM in forecasting time series,” in International Con-
VIII. C ONCLUSIONS AND F UTURE W ORK ference on Machine Learning and Applications, ICMLA, Orlando, FL,
USA, 2018, pp. 1394–1401.
This paper empirically explored whether machine and deep [8] N. Tavakoli, “Modeling genome data using bidirectional LSTM,” in
learning classifiers are effective in detecting zero-day malware. Annual Computer Software and Applications Conference, COMPSAC,
Addressing such a question is important from security per- Milwaukee, WI, USA., 2019, pp. 183–188.
[9] M. Chatterjee and A. S. Namin, “Detecting phishing websites through
spective because zero-day malware are unknown applications deep reinforcement learning,” in Annual Computer Software and Appli-
and thus there might not be any malicious signature similar cations Conference, COMPSAC, WI, USA, 2019, pp. 227–232.
to their patterns. We empirically compared a good number of [10] L. Bilge and T. Dumitras, “Before we knew it: An empirical study of
zero-day attacks in the real world,” in ACM Conference on Computer
well-known conventional machine learning and deep learning and Communications Security, 10 2012, pp. 833–844.
classifiers and observed that some of the conventional machine [11] M. G. Miller, “Are we protected yet? developing a machine learning de-
learning algorithms (e.g., random forests) perform very well tection system to combat zero-day malware attacks,” Ph.D. dissertation,
2018.
in comparison with their deep learning-based counterparts. [12] V. Sharma, J. Kim, S. Kwon, I. You, K. Lee, and K. Yim, “A framework
This result implies that some of the conventional and deep for mitigating zero-day attacks in IoT,” CoRR, vol. abs/1804.05549,
learning-based approaches are good classifiers for detecting 2018.
[13] M. Alazab, S. Venkatraman, P. Watters, and M. Alazab, “Zero-day
zero-day malware. However, even though they achieve very malware detection based on supervised learning algorithms of API call
high accuracy (e.g., 99.51%), these algorithms never achieve signatures,” in Ninth Australasian Data Mining Conference - Volume
a 100% accuracy and thus these classifiers might slightly 121, ser. AusDM ’11, 2011, pp. 171–182.
[14] P. M. Comar, L. Liu, S. Saha, P. Tan, and A. Nucci, “Combining
misclassify some of the zero-day malware. supervised and unsupervised learning for zero-day malware detection,”
This paper focused on measuring the accuracy of classifiers in 2013 Proceedings IEEE INFOCOM, April 2013, pp. 2022–2030.
[15] Q. Zhou and D. Pezaros, “Evaluation of machine learning classifiers for
using a 10-fold cross validation. It is important to carry out zero-day intrusion detection - an analysis on CIC-AWS-2018 dataset,”
additional experiments and measure precision, recall, accuracy, CoRR, vol. abs/1905.03685, 2019.
and F1 measures all together along with the ROC measure [16] L. Xiao, X. Wan, X. Lu, Y. Zhang, and D. Wu, “Iot security techniques
based on machine learning: How do IoT devices use AI to enhance
for these classifiers and capture the exact values for false security?” IEEE Signal Processing Magazine, vol. 35, no. 5, pp. 41–49,
positive and false negative. It is also important to replicate Sep. 2018.
the experiments reported in this paper with some other datasets [17] “Malware detection - make your own malware security system, in asso-
ciation with meraz’18 malware security partner max secure software,”
and perform a meta analysis [18] to have a better insights about https://www.kaggle.com/c/malware-detection, Accessed 2019.
the machine leaning algorithms and their classification per- [18] S. Kakarla, S. Momotaz, and A. S. Namin, “An evaluation of mutation
formances. Furthermore, given the outstanding performance and data-flow testing: A meta-analysis,” in International Conference on
Software Testing, Verification and Validation, ICST, Berlin, Germany,
of random forest, it would be interesting to observe whether Workshop Proceedings, 2011, pp. 366–375.
ensemble-based deep learning classifiers perform better than [19] M. Chatterjee, A. S. Namin, and P. Datta, “Evidence fusion for malicious
other classifiers. It is also an interesting question to investigate bot detection in iot,” in International Conference on Big Data, WA, USA,
2018, pp. 4545–4548.
whether evidence theory [19], uncertainty reasoning [20], or [20] S. Sartoli and A. S. Namin, “Adaptive reasoning in the presence of
control-theoretical approaches and decision-based processes imperfect security requirements,” in Annual Computer Software and
[21] can be utilized in accordance with learning algorithms Applications Conference, COMPSAC, GA, USA, 2016, pp. 498–499.
[21] J. Zheng and A. S. Namin, “A markov decision process to determine
to detect zero-day vulnerabilities. optimal policies in moving target,” in ACM SIGSAC Conference on
Computer and Communications Security, Toronto, ON, Canada, 2018,
ACKNOWLEDGMENT pp. 2321–2323.
This work is supported in part by National Science Foun-
dation (NSF) under the grants 1821560 and 1723765.

R EFERENCES
[1] “Yara - the pattern matching swiss knife for malware researchers,”
https://virustotal.github.io/yara/, Accessed 2019.
[2] L. Xie, X. Zhang, J.-P. Seifert, and S. Zhu, “pbmds: A behavior-based
malware detection system for cellphone devices,” in ACM Conference
on Wireless Network Security, ser. WiSec ’10, 2010, pp. 37–48.