Briefings in Bioinformatics, 2023, 24(2), 1–15

[Link]
Advance access publication date 21 January 2023
Problem Solving Protocol

Comprehensive assessment of nine target

prediction web services: which should we choose
for target fishing?
Kai-Yue Ji, Chong Liu, Zhao-Qian Liu, Ya-Feng Deng, Ting-Jun Hou and Dong-Sheng Cao

Downloaded from [Link] by guest on 16 February 2024

Corresponding authors: Dong-Sheng Cao, Xiangya School of Pharmaceutical Sciences, Central South University, Changsha, 410013, Hunan,P. R. China.
Tel.: +86-731-89824761; E-mail: oriental-cds@[Link]; Ting-Jun Hou, College of Pharmaceutical Sciences and State Key Lab of CAD&CG, Zhejiang University,
Hangzhou, Zhejiang310058, P. R. China. Tel.: +86-571-88208412; E-mail: tingjunhou@[Link]

Abstract
Identification of potential targets for known bioactive compounds and novel synthetic analogs is of considerable significance. In silico
target fishing (TF) has become an alternative strategy because of the expensive and laborious wet-lab experiments, explosive growth of
bioactivity data and rapid development of high-throughput technologies. However, these TF methods are based on different algorithms,
molecular representations and training datasets, which may lead to different results when predicting the same query molecules. This
can be confusing for practitioners in practical applications. Therefore, this study systematically evaluated nine popular ligand-based
TF methods based on target and ligand–target pair statistical strategies, which will help practitioners make choices among multiple TF
methods. The evaluation results showed that SwissTargetPrediction was the best method to produce the most reliable predictions while
enriching more targets. High-recall similarity ensemble approach (SEA) was able to find real targets for more compounds compared
with other TF methods. Therefore, SwissTargetPrediction and SEA can be considered as primary selection methods in future studies.
In addition, the results showed that k = 5 was the optimal number of experimental candidate targets. Finally, a novel ensemble TF
method based on consensus voting is proposed to improve the prediction performance. The precision of the ensemble TF method
outperforms the individual TF method, indicating that the ensemble TF method can more effectively identify real targets within a
given top-k threshold. The results of this study can be used as a reference to guide practitioners in selecting the most effective methods
in computational drug discovery.

Keywords: ensemble target prediction method, consensus voting, similarity ensemble approach, SwissTargetPrediction, in silico target
fishing

Introduction for viable alternatives. In silico target fishing (TF), an attractive

Identifying of potential targets for known biologically active com- alternative strategy for target identification, which can signifi-
pounds and novel synthetic analogs is a significant step in early cantly facilitate systematic rational guidance of the process and
drug design, safety evaluation and drug repositioning. It is also a narrows the drug target search space for wet-lab peer studies, has
challenging, expensive and critical step in phenotypic screening become increasingly relevant and popular, because of the rapid
to gain mechanistic understanding of phenotypic activity [1, 2]. growth of chemical and biological data, availability of powerful
Traditional experimental approaches for target identification are high-throughput screening technologies [8], and shift in small-
typically performed at low-throughput scales and are vulnera- molecule drug discovery from the ‘one drug one target’ paradigm
ble to objective conditions [3, 4]. However, the effort and cost to ‘polypharmacology’ [9–12]. In silico TF effectively complements
involved in conducting large-scale experiments and generating expensive and laborious wet-lab experiments, aiming to (i) iden-
statistically significant conclusions are generally prohibitive. In tify potential targets to elucidate the mode of action of com-
addition, severe clinical side effects and cross-reactivity increase pounds [13], (ii) explore the desired polypharmacological effects of
the attrition rate of new drugs [5], the scientific and economic ligands [9, 14], (iii) identify off-targets responsible for side effects
pressure to prospectively design multi-target drugs [6] and the and adverse reactions [15, 16] and (iv) perform drug repositioning
recent boost in drug repurposing [7]. This has prompted the search [17].

Kai-Yue Ji is a graduate student at the Xiangya School of Pharmaceutical Sciences, Central South University, China. Her research focuses on target prediction of
small molecules.
Chong Liu is currently a postdoctoral at the Xiangya School of Pharmaceutical Sciences, Central South University, China. His researches focus on the efficacy and
mechanism of traditional Chinese medicine.
Zhao-Qian Liu is currently a professor at the Xiangya School of Pharmaceutical Sciences, Central South University, China. His researches focus on basic and
clinical applications of pharmacogenomics and individualized therapy for major diseases.
Ya-Feng Deng is a senior researcher at CarbonSilicon AI Technology Co., Ltd, China. His researches focus on artificial intelligence algorithm and product
development.
Ting-Jun Hou is currently a professor at the College of Pharmaceutical Sciences, Zhejiang University, China. His research interests can be found at the website of
his group: [Link]
Dong-Sheng Cao is currently a professor in the Xiangya School of Pharmaceutical Sciences, Central South University, China. His research interests can be found
at the web site of his group: [Link]
Received: November 7, 2022. Revised: December 29, 2022. Accepted: January 3, 2023
© The Author(s) 2023. Published by Oxford University Press. All rights reserved. For Permissions, please email: [Link]@[Link]
2 | Ji et al.

Table 1. Summary of the nine unpaid ligand-based web services that can be utilized for the target prediction of small molecules

Web servers Databases Methods Descriptors URL

SwissTargetPrediction [42] ChEMBL16; 2686 targets; Multiple logistics regression FP2 and 3D fingerprints [Link]
280 381 compounds models based on 2D/3D FP swisstargetprediction.
similarity ch/
SEA [43] ChEMBL16 and MDDR; 246 Cross-target similarity maps Daylight fingerprints [Link]
targets; 65 241 compounds based on an extreme value
distribution
PPB [44] ChEMBL21; 4613 targets; Statistical similarity searching Ten types of fingerprints [Link]
871 673 compounds (fused fingerprints) ch:8080/PPB/
PPB2 [45] ChEMBL22; 1720 targets; DNN (ECfp4); NN (ECfp4); NN ECfp4/Xfp fingerprints [Link]
344 163 compounds; 555 346 (ECfp4) + NB (ECfp4); NN
ligand–target pairs (Xfp) + NB (ECfp4)
ChEMBL [46] ChEMBL22 and other Multi-category NB models ECFP fingerprint [Link]

Downloaded from [Link] by guest on 16 February 2024

databases; 11 000 targets; 1.6 [Link]/ml.
million compounds html
TargetNet [47] BindingDB; 623 targets; Multi-targets QSAR ensemble Seven types of [Link]
109 061 compounds models molecular fingerprints com/

TF can be classified into receptor- and ligand-based methods to validate prediction performance because the iterative modeling
[18, 19]. The receptor-based methods are constructed to rely on process may introduce biases to the test data properties, which
three-dimensional (3D) structural information obtained from var- can lead to overestimation of the TF method performance [48].
ious target receptors [20, 21] or family-specific target [22, 23], but In addition, the use of the receiver operating characteristic (ROC)
these are not available for targets of a certain unknown 3D struc- and area under the curve (AUC) is not suitable for measuring the
ture [24]. This method is not only time-consuming and expensive performance of TF methods [49]. This is because practitioners
but also limited by the reliability of docking simulations, yielding prefer to focus on both the ‘percentage of real targets predicted’
high false-positive rates [25, 26]. Ligand-based methods rely on and the ‘number of real targets predicted’ as criteria for selecting
the similarity principle that similar molecules are likely to share TF methods. The most important question, ‘which TF method
the same target profiles, which can efficiently retrieve any tar- produces the most reliable predictions while covering the largest
get with at least one known ligand without considering the 3D target space’, is usually left unanswered.
structures of target proteins [13, 27]. Compared with receptor- This study was the first to evaluate the performance of nine
based methods, ligand-based methods are more advantageous as popular ligand-based TF methods (Table 1), with the aim of explor-
they provide broader proteomics coverage and consider the multi- ing the best method that would guide practitioners in making a
pharmacological nature of drugs [28, 29]. Furthermore, ligand- choice. In addition, this study analyzed the optimal number of
based methods can be divided into chemical similarity searching experimental candidate targets and the performance of the pro-
methods and machine learning (ML) methods, according to the posed ensemble TF method for TF. Through this study, we aim to
underlying principles [27]. The former are probably the simplest answer the following three questions for practical applications: (i)
approaches for ligand-based methods, and they rely on the prior Which TF method is more effective for enriching real targets? (ii)
knowledge of bioactive ligands and protein targets, molecular fin- Which TF method is more capable of enriching more compound–
gerprints and similarity measures when calculating the similarity target pairs from the prediction list? (iii) Does the ensemble TF
of a query molecule and known bioactive molecules [28, 30, 31]. method improve the accuracy of fishing targets? The process
The latter are applied to predict the targets of small molecules of evaluating the nine TF methods and the construction of the
based on relatively large amounts of bioactivity data and ML ensemble TF method is shown in Figure 1.
algorithms, such as random forests [32], k-nearest neighbor (k-
NN) [33], support vector machines [34, 35], Naïve Bayesian (NB)
[36, 37], deep neural networks (DNNs) [38, 39] and others [40]. Materials and methods
These algorithms map compounds to chemical genome spaces Test dataset
or bioactivity databases using molecular descriptors and finger- In this study, 396 336 ligand–target pairs covering 1410 human
prints and thereafter measure the likelihood between structures protein targets with a total of 271 703 unique compounds from the
and predicted targets [41]. ChEMBL [46] and BindingBD [50] databases were collected to select
In recent years, with the advent of web technologies and the benchmark dataset. Compounds with reported bioactivities
advances in ML algorithms and hardware technologies, unpaid (IC50/Ki/KD/EC50) ≥ 10 μM against human protein targets were
ligand-based TF web services have been widely developed and eliminated to ensure that compounds with reliable targets were
adopted because they do not require us to maintain and manage selected for research. Murcko scaffolds of 271 703 ligands were
the data. However, ligand-based TF methods are based on differ- calculated using MOE software, and a total of 96 817 individual
ent algorithms, molecular representations and training datasets Murcko scaffolds were obtained. The ligands from each target
(Table 1), which may lead to different results when predicting the were clustered according to Murcko scaffolds. Based on the num-
same query molecules. This can be confusing for practitioners in ber of Murcko skeletons, three ligands were randomly selected for
practical applications. Existing research suggests that TF methods each target, targets with less than three ligands were discarded
typically use internal validation as the only means of reporting and not considered further, and only the unique ligand–target
model performance; however, internal validation is not sufficient interaction for each protein target was retained. As shown in
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 3

Downloaded from [Link] by guest on 16 February 2024

Figure 1. Flow chart of performance evaluation of nine TF methods. (1) Test dataset construction. The ligand–target interaction data in the test dataset
were extracted from the ChEMBL and BingdingDB databases. (2) Target prediction. Nine TF methods were used to predict all ligands with known targets.
Each color represents the corresponding TF method, e.g. rose color represents SwissTargetPrecision. Annotation information of the first 10 predicted
targets is converted to Uniport ID for statistical comparison. (3) Performance statistics. The number of all predicted targets and ligand–target pairs
as well as ‘True’ targets for the nine TF methods were counted. Green circles represent true positive and gray circles represent false positive, and the
numbers represent the different targets. (4) Ensemble TF method. The ensemble TF method integrates the number of predicted targets and ligand–target
pairs and ‘True’ targets voted by the two TF methods. The overlapping purple and yellow semicircles represent the predicted targets selected by both
methods, where the green circles represent true positive and the gray circles represent false positive, and the numbers represent the different targets.

Figure 2A, 95% of the active ligands had only one target, fewer SEA is a 2D ligand-based similarity ensemble method that
ligands had more than one target and the maximum number of characterizes each target using its active ligand set of small
targets that a ligand had was 39. Finally, the test dataset con- molecules [43]. The query molecule was compared against all lig-
taining 3348 ligand–target pairs and 1116 targets, covering 2815 and sets of each target set via the Tanimoto similarity of the ECfp4
compounds, was extracted in this study, which included 406 FDA- fingerprints. The Tanimoto similarities were thereafter summed,
approved drug targets. These targets mainly covered seven protein and z-scores calculated. Finally, the z-scores were quantified as
types: enzymes, membrane receptors, ion channels, transporter, an expectation value (E-value) to associate the potential protein
transcription factors, epigenetic regulator and others (Figure 2B). targets.
Enzymes in the current dataset account for 63.5% of biological PPB is a 2D ligand-based similarity method that employs the
targets, indicating the importance of these targets in modern Manhattan similarity index and 10 different types of fingerprints,
pharmacology and the popularity of enzymes as drug targets [51]. including fused fingerprints [44]. For each type of fingerprint,
The physicochemical property of 2815 compounds is shown in target identification was performed by computing the city-block
Figure 2C–F. distance (CBD) between the query and the most similar molecule
in a set of compounds associated with this target. Each distance
distribution was then fitted with a negative binomial distribu-
TF methods tion function to generate the corresponding cumulative density
In this study, we evaluated and compared nine publicly available function, giving the P-value as a function of CBD. The predicted
ligand-based TF methods (Table 1), including SwissTargetPredic- targets of the query molecule are sorted by the sum of their P-
tion, similarity ensemble approach (SEA), TargetNet, ChEMBL, values, which are derived from the nearest neighbors found for
polypharmacology browser (PPB) and the updated version of PPB that target.
(PPB2), with four different TF methods, including NN (ECfp4), NN PPB2 is a public web portal for target prediction that com-
(ECfp4) + NB (ECfp4), NN (Xfp) + NB (ECfp4) and DNN (ECfp4). putes ligand similarities using three type of molecular finger-
SwissTargetPrediction is a combined two-dimensional (2D) and prints (ECfp4, Xfp, MQN) and implements ML approaches (NN,
3D ligand-based similarity method that performs target predic- NB and DNN) as further options [45]. The implementation of
tion by calculating the similarity between query molecule and PPB2 is as follows: (i) NN (ECfp4): compounds are ranked by
ligand sets in a reference database [42]. The quantification of calculating the similarity score between the query molecule and
similarity comprises a 2D approach based on the Tanimoto index each compound in the reference database; thereafter, the targets
between path-based binary fingerprints (FP2) and a 3D approach associated with the top 2000 compounds (nearest neighbors of
based on the Manhattan distance similarity between electroshape the query) in the reference database are extracted. Finally, the
5D (ES5D) vectors [52]. Multiple logistic regression was used to potential targets are ranked according to the similarity score of
combine the different similarity values. The combined scores the most similar nearest neighbor associated with a target. (ii)
were converted into probabilities to identify the likelihood of the Ranking of compounds is performed by calculating the similarity
target–compound association. between the query molecule and each compound in the reference
4 | Ji et al.

Downloaded from [Link] by guest on 16 February 2024

Figure 2. (A) Distribution of the number of targets associated with each ligand. (B) Category distribution map 1116 targets. (C) Distribution of 2815
bioactive compounds according to molecular weight. (D) Distribution of 2815 bioactive compounds according to volume. (E) Distribution of 2815 bioactive
compounds according to rings. (F) Distribution of 2815 bioactive compounds according to density.

database (using ECfp4 or Xfp). The 2000 compounds (nearest reference database extracted from ChEMBL using a DNN model
neighbors of the query) are thereafter used to construct an NB with ECfp4 fingerprints and used to predict the targets of the
ML model using ECfp4 fingerprints to perform target prediction. query molecule.
The top 20 predicted targets of the query molecule were returned. TargetNet, developed by our group, is based on chemical
We compared two NN + NB methods: NN (ECfp4) + NB (ECfp4) and genomics data using the NB model and seven types of molecular
NN (Xfp) + NB (ECfp4). (iii) The DNN (ECfp4) is constructed on a fingerprints to construct 623 SAR models associated with 623
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 5

human proteins to predict the binding of multiple targets for achieves better performance in identifying the real target with
any given molecule, thereby generating drug–target interaction a balance between the combined high-precision and high-recall.
analyses that can be used as feature vectors for wide applications Conversely, a lower Fk score indicates that the TF method achieves
[47]. poor performance at the expense of precision or recall.
ChEMBL web service is a multi-target QSAR model based on
a NB multi-label classifier, which was constructed using open-
source tools such as RDKit and scikit-learn [46]. The associations
of the query molecule–target interactions were predicted using
Results and discussion
the combined QSAR model of each target, which was constructed Which TF method is more effective for enriching
based on the NB algorithm. real targets?
Nine TF methods were used to obtain the potential target In this study, we evaluated the recall of each TF method
information (such as ChEMBL ID and Target Key) of each ligand on 10 independent test datasets (containing 500 drug–target
in the test dataset, and the predicted target information was interactions) that were randomly selected from the test dataset.
converted into a standardized UniProt ID through the UniProt As shown in Figure 3, 10 independent test datasets yielded

Downloaded from [Link] by guest on 16 February 2024

database. Furthermore, we consider real targets of each ligand stable performance for each TF method and quantified the
in the test dataset to have a ‘True’ label. Thus, by comparing the robustness of TF methods in different ranking prediction lists
UniProt ID of the predicted targets and real targets of the ligands (Supplementary Table S1, see Supplementary Data available
in the test dataset, the predicted targets and ligand–target pairs online at [Link] Furthermore, the num-
with the ‘True’ label were considered as true positive. ber of predicted targets for each TF method was calculated using
the test dataset.
Ensemble TF method The results (Figure 4A) show that as k increased from 1 to 10,
The ensemble TF method proposed in this study is based on two the real target recall of the nine TF methods gradually increased.
different TF methods to calculate the most likely protein targets SEA outperforms all other methods in terms of real target recall,
of small molecules. First, the predicted target information of each and the recall order of these methods is identical within a given
ligand in the test dataset was obtained using two different TF top-k (k = 1, 3, 5, 7, 10). SEA achieved a real target recall of 0.73,
methods and converted into a standardized UniProt ID using the which is superior to that of other TF methods for k = 5, as summa-
UniProt database. The UniProt ID of the predicted targets under rized in Supplementary Table S2 (see Supplementary Data avail-
the same ligands was voted within a given ranking (top-k, k = 1, 3, able online at [Link] (SwissTargetPredic-
5, 7, 10). Finally, only the ligand–target pairs voted on by the two tion: 0.69, ChEMBL: 0.63, DNN (ECfp4): 0.59, NN (ECfp4): 0.56,
TF methods in the prediction results were retained. Ligand–target PPB: 0.55, NN (Xfp) + NB (ECfp4): 0.52, NN (ECfp4) + NB (ECfp4):
pairs with the ‘True’ labels were considered as true positive. 0.49, TargetNet: 0.2). Meanwhile, as k increased from 1 to 10,
the real target precision gradually decreased (Figure 4B). However,
Evaluation metrics TagetNet and ChEMBL based on the multi-target QSAR model are
In this study, all possible ligand–target pairs were calculated using affected by the number of targets, which can only return a given
nine TF methods. In addition, the number of predicted targets number of predicted targets for query molecules, resulting in an
and ‘True’ targets for all ligands were counted. To some extent, increase in the real target precision (Figure 4B). QSAR model uses
the calculation of ligand–target pairs is more rigorous because inadequate datasets that may limit the applicability domain and
the former needs to reveal complex interactions between mul- thus fail to predict to the real target [54]. In addition, ChEMBL
tiple molecules and multiple protein targets [53], whereas the outperformed TargetNet, which may be attributed that ChEMBL
latter only needs to correctly predict one of the ligands under is a QSAR model based on multi-task learning, which improves
the target to ref lect the enrichment ability of TF methods [12]. the performance of the model by sharing representation of the
For each test compound, the number of predicted targets (top- data [55]. From Supplementary Table S2 (see Supplementary Data
k) was considered, and that of real targets calculated. Several available online at [Link] in terms of real
evaluation metrics were calculated to evaluate the accuracy of target precision statistics, NN (ECfp4) + NB (ECfp4) has a clear
the TF methods and that of the proposed ensemble TF method, advantage (0.72, k = 1; 0.67, k = 3; 0.65, k = 5; 0.62, k = 7 and 0.59,
including the precision (PRk ) and recall (REk ). k = 10). SwissTaregtPrediction, PPB, NN (ECfp4), DNN (ECfp4) and
NN (Xfp) + NB (ECfp4) yielded similar results in all comparisons,
TPn indicating that their performance was similar for target prediction
PRk = (1)
n using compounds with known targets. The real target precision
of SEA dropped significantly, suggesting that it favors recall over
TPn
REk = (2) precision. Clearly, this result comes at the cost of generating more
m
false positives because the target inference is affected by the
PRk ∗ REk ∗ 2 similarity values of the query molecule and ligand sets in the
Fk = , (3)
PRk + REk training set [13]. We also discuss the Fk curves under different
where PRk (Eq. 1) represents the percentage of predicted results top-k values from the nine TF methods. SwissTargetPredictiom
corresponding to positive predictions, where TPk denotes the num- obtained Fk up to 0.6 (Supplementary Table S2, see Supplemen-
ber of positive predictions of the real targets or ligand–target pairs tary Data available online at [Link] Then,
in the top-k and n denotes the number of predictions of the real among the four methods in PPB2, DNN (ECfp4) performed best. NN
targets or ligand–target pairs in the top-k; REk (Eq. 2) represents (ECfp4) + NB (ECfp4), NN (ECfp4), NN (Xfp) + NB (ECfp4), ChEMBL
the percentage of positive predictions for real targets or ligand– yielded similar results. As shown in Figure 4C, the Fk curve of
target pairs, where m denotes the number of real targets or ligand– SwissTargetPredictiom is always higher than that of the other
target pairs in the test dataset and Fk (Eq. 3) is the harmonic mean methods at arbitrary k values, indicating that it has a stronger
of PRk and REk . A higher Fk score indicates that the TF method ability to enrich real targets.
6 | Ji et al.

Downloaded from [Link] by guest on 16 February 2024

Figure 3. Box plot of real target recall in the top-k (k = 1, 3, 5, 7, 10) for nine TF methods. Diamond labels represent outliers.

In addition, drugs often bind to multiple targets; this is referred k increased from 1 to 10, the performance of various TF methods
to as polypharmacology. The new potential functions of the ‘old’ exhibited highly similar trends in terms of ligand-target pairs, the
targets for seeking new drugs that repurpose existing targets will recall gradually increased and the precision sharply decreased
be a way to gain new treatments for related diseases [56]. In this (Figure 6A and B). SEA and SwissTargetPrediction performed well
study, we also evaluated the performance of nine TF methods in terms of ligand–target pair recall statistics (Figure 6A). When
on FDA-approved targets. As shown in Figure 5A, SEA and k is increased from 1 to 10, the recall increased from 0.30 to
SwissTaregtPrediction outperformed the other methods in the 0.59 for SEA and from 0.33 to 0.53 for SwissTargetPrediction
same ranking, achieving up to 0.8 recall of FDA-approved targets (Supplementary Table S4, see Supplementary Data available
(the percentage of 406 FDA-approved targets in the test set that online at [Link] The recall for the other
are predicted) in the top 5 (Supplementary Table S3, see Supple- methods was essentially similar, indicating that even with a
mentary Data available online at [Link] higher k, the increase in positive prediction would be insignificant.
Meanwhile, as k increased from 1 to 10, the real target precision In terms of the ligand–target pair precision statistics, when k is
gradually decreased and NN (ECfp4) + NB (ECfp4) performed increased from 1 to 10, the precision of the DNN (ECfp4) drops
best (Figure 5B). DNN (ECfp4), NN (Xfp) + NB (ECfp4) and Swis- from 0.56 to 0.08 (Supplementary Table S5, see Supplementary
sTargetPrediction yielded similar results. The Fk curves for Data available online at [Link] As
nine TF methods with different top-k values are discussed shown in Figure 6B, DNN (ECfp4) shows clear advantages and
(Figure 5C). SwissTargetPrediction slightly outperformed the outperforms other methods of PPB2, which may be ascribed to
other methods (Supplementary Table S3, see Supplementary the ability of deep learning methods to learn representations of
Data available online at [Link] This data with multiple levels of abstraction generated by artificial
implies that SwissTargetPrediction have a larger pharmacological neural network with multiple hidden layers and more complex
space within the FDA-approved target coverage and exhibited parameter training procedures [38, 57]. SwissTargetPrediction,
the powerful ability to predict the ‘old’ targets for new drugs. SEA, NN (ECfp4), PPB, NN (Xfp) + NB (ECfp4) and NN (ECfp4) + NB
Although SEA achieved high recall at the expense of precision, (ECfp4) exhibited extremely similar results. In practice, selecting
practitioners prefer to identify as many FDA-approved targets high-recall SwissTargetPrediction and SEA is preferable, as they
related to the compounds of interest as possible in practical are able to identify real targets for a large number of compounds.
applications, so SEA can also be considered as the primary The difference in performance between ChEMBL and TargetNet is
selection method for exploring the ‘old’ targets in future studies. likely ref lected in the size of the dataset. ML methods generally
depend on reliable training datasets, but it is inaccurate to regard
Which TF methods is more capable of enriching all compounds with unknown activity as inactive, which can
more compound–target pairs from the prediction lead to false negatives, thus limiting their application in TF [58,
list? 59]. Therefore, it is not surprising that TargetNet cannot obtain
Figure 6 shows the variation in the performance of the nine TF results to the same extent as other TF methods. Furthermore,
methods in the top-k predicted targets. For a given top-k, the when k was greater than five, various TF methods showed that
performance in terms of ligand–target pair is lower than the the recall slowly increased and the precision slowly decreased.
performance in terms of real target for each TF method because These results imply that the TF method can only identify real
the former is more rigorous and requires correctly predicting targets in a few top-ranked predictions, and further increasing
targets for each ligand and pushing it to the top of the prediction the number of predictions will not produce more real targets.
list, whereas the latter only requires correct identification Therefore, we suggest the top five targets as candidate targets
of one ligand corresponding to the target (Figures 4 and 6). As to assist target identification experiments in the future, and it is
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 7

Downloaded from [Link] by guest on 16 February 2024

Figure 4. Real target results for nine TF methods in the top-k (k = 1, 3, 5, 7, 10). (A) Real target recall for nine TF methods. (B) Real target precision for
nine TF methods. (C) Fk curves for nine TF methods.

necessary to narrow the range of targets to be tested subsequently. studies. In summary, SwissTargetPrediction and SEA provide
From Figure 6C, the results also show that various TF methods better options for identifying and analyzing targets in the context
exhibit a similar trend, where the Fk curve decreases rapidly when of compounds.
k > 1 (Supplementary Table S4, see Supplementary Data available
online at [Link] This can be interpreted Does the ensemble TF method improve the
as obtaining more real targets at the expense of precision, accuracy of TF?
which results in a large number of false positive prediction. Considering the previous discussion, we proposed an ensemble
DNN (ECfp4) and SwissTargetPrediction obtained extremely method to identify the comprehensive and credible targets of a
similar results, and SwissTargetPrediction slightly outperformed query compound by integrating several TF methods. This method
the other methods with higher recall. SwissTargetPrediction estimates the most likely targets of small molecules based on the
can significantly improve the results by combining 2D and 3D a priori hypothesis that the higher the frequency of a target from
similarity measures to push targets with high-similarity ligands different TF methods, the higher the reliability of the target to be
at the top of the list [52, 60]. In conclusion, these results suggest the real target. We integrated the prediction results from the two
that SwissTargetPrediction tends to find more categories of real different TF methods and only considered the targets voted by the
targets in the highest rankings than other TF methods because both TF methods, and used the statistics of the real targets and
of their higher recall and precision. Although SEA achieved high ligand–target pairs to evaluate the accuracy of the ensemble TF
recall at the expense of accuracy, practitioners prefer to identify methods (Supplementary Tables S5 and S6, see Supplementary
as many real targets related to the compounds of interest as Data available online at [Link] For each
possible in practical applications, so SEA can also be considered ensemble TF method, we calculated the recall and precision of
as the primary selection method for target identification in future real targets and ligand-target pairs (Figures 7 and 8). In terms of
8 | Ji et al.

Downloaded from [Link] by guest on 16 February 2024

Figure 5. FDA-approved target results for nine TF methods in the top-k (k = 1, 3, 5, 7, 10). (A) FDA-approved target recall for nine TF methods. (B) FDA-
approved target precision for nine TF methods. (C) Fk curves for nine TF methods.

real target and ligand–target pair precision, SEA achieved 0.14 TF methods in terms of real target and ligand–target pairs.
and 0.26, SwissTargetPrediction achieved 0.14 and 0.61, and the The ensemble of SEA and SwissTargetPrediction method was
ensemble of SEA and SwissTargetPrediction method achieved 0.78 superior to the other methods, obtaining the Fk scores of up
and 0.45 for k = 5 (Figure 9; Supplementary Table S7, see Supple- to 0.6 and 0.4 at k = 5 in terms of real target and ligand–target
mentary Data available online at [Link] pairs (Supplementary Tables S5 and S6, see Supplementary
The results indicate that the most of ensemble TF method Data available online at [Link] This was
yielded excellent results and had a significant improvement in followed by the ensemble of ChEMBL and SwissTargetPrediction
precision compared with the individual TF method. In terms method and the ensemble of SEA and PPB method. This suggests
of real target and ligand–target pair recall, SEA achieved 0.73 that integrating the high-recall individual method is more
and 0.52, SwissTargetPrediction achieved 0.69 and 0.49 and the beneficial for improving the precision of the ensemble method,
ensemble of SEA and SwissTargetPrediction method achieved because it can provide more predictive targets for voting.
0.59 and 0.37 for k = 5 (Supplementary Table S7, see Supple- Therefore, a prerequisite for developing an excellent ensemble
mentary Data available online at [Link] TF method is that the individual TF method performs well in
Supplementary Figure S1, see Supplementary Data available target prediction. In conclusion, the ensemble TF method is
online at [Link] Although the recall of proposed as an excellent scheme for target identification, as
the ensemble TF method is lower than that of the individual TF it has higher precision and makes it easier to find real targets,
method, this can be explained by the fact that the ensemble particularly in the process of exploring the action mechanism of
TF method only considers the same targets from the two TF natural products, which can effectively narrow the target space
methods, whereas some targets that hit by the individual TF of molecules [61].
method are ignored, resulting in a fraction of true targets being
missed. Furthermore, the performance of several ensemble
TF methods is worse than that of the individual TF methods
Discussion
because the ensemble TF methods are mainly based on simple Accurate prediction of bioactive molecule targets is important.
consensus voting. Figure 10 show the Fk curves for ensemble In this study, we systematically evaluated the performance of
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 9

Downloaded from [Link] by guest on 16 February 2024

Figure 6. Ligand–target pair results for nine TF methods in the top-k (k = 1, 3, 5, 7, 10). (A) Ligand–target pair recall for nine TF methods. (B) Ligand–target
pair precision for nine TF methods. (C) Fk curves for nine TF methods.

nine popular ligand-based TF methods based on target and are false positives for a given top-k. This may be attributed to
ligand–target pair statistical strategies that aim to explore the fact that SEA compares drugs to a ligand set based on all
the best methods that produce the most reliable prediction shared chemical patterns and can predict the activity of drugs
results while covering the largest target space. Typically, some with many features of the ligands of a target while ignoring the
performance metrics reported by TF [39], such as ROC and pharmacophores in its predictions [48, 62]. Despite the surprising
AUC, are not suitable for the realistic evaluation of model results of this study, it appears that SEA can be an alternative TF
performance because academic researchers prefer to focus on the method because it has the higher recall and can identify more
two scenarios of ‘what percentage of real targets are predicted, potential targets for more compounds in practical applications.
and how many real targets are predicted’, as the criteria for Overall, our evaluation of nine popular ligand-based TF methods
selecting TF methods. This study recommends the widespread suggests that SwissTargetPrediction is more appropriate if high
use of precision, recall and Fk as critical metrics for the evaluation recall is required while producing high-precision results. If the
and selection of TF methods for practical applications. aim is to explore novel targets for compounds, SEA will achieve
Our results demonstrate that high-Fk SwissTargetPrediction the best performance. Our findings are restricted to a test dataset,
performs better than other methods and is the best method which may be considered to underestimate the effectiveness of TF
for producing the most reliable predictions while enriching for methods, but the fact is that the modeling process of TF methods
more targets. As observed in previous studies, for biologically may introduce biases to the properties of the test data, resulting
active molecules, the greatest advantage of SwissTargetPrediction in an overestimation of model performance [63]. Because the TF
is its combination of 2D fingerprinting and 3D shape-binding method training set is not public, molecules in the test set that are
methods, which can significantly improve performance [42, 60]. similar to compounds in the TF method training set are difficult
Our results also show that SEA can find more compounds of to exclude. Even if similar molecules exist, our evaluation can still
targets in large-scale predictions, but that many of the targets provide scientists with valuable information and help to prioritize
10 | Ji et al.

Downloaded from [Link] by guest on 16 February 2024

Figure 7. Real target results for the ensemble TF method in the top-k (k = 1, 3, 5, 7, 10). (A) Real target recall for the ensemble TF method. (B) Real target
precision for the ensemble TF method.

lead compounds and obtain a true representation of the method ensemble TF method only considers the same targets from the
performance. two TF methods, and this may lead to a small fraction of real
To improve the prediction precision and reduce the wear rate targets being missed, which is difficult to avoid. The bias arising
of wet-lab experiments, we proposed a novel ensemble TF method from the performance statistics of the ensemble method can be
that provides the most reliable prediction targets. However, the reduced by selecting a high-recall TF method and considering the
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 11

Downloaded from [Link] by guest on 16 February 2024

Figure 8. Ligand–target pair results for the ensemble TF method in the top-k (k = 1, 3, 5, 7, 10). (A) Ligand–target pair recall for the ensemble TF method.
(B) Ligand–target pair precision for the ensemble TF method.

number of ranked targets. In addition, the ensemble TF method f lexible. We tried to calculate the performance of the combina-
introduces an additional computational burden, but the signal- tion of three TF methods (Supplementary Table S8, see Supple-
to-noise ratio can be improved using multiple prediction meth- mentary Data available online at [Link]
ods for queries. The ensemble method is not merely applicable The results show that the precision of the combination of the
to the combination of two TF methods. It is extrapolated and three methods is significantly improved. Therefore, by combining
12 | Ji et al.

Downloaded from [Link] by guest on 16 February 2024

Figure 9. Prediction results for the comparison of the combination of SEA and SwissTargetPrediction method, SEA and SwissTargetPrediction in the
top-k (k = 1, 3, 5, 7, 10). (A) Real target precision for three methods. (B) Ligand–target pair precision for three methods.

multiple TF methods with high recall, the performance of iden- different k values and discussing the results in terms of the
tifying real targets may be greatly improved. In practice, practi- statistics of the real target and ligand–target pair, the performance
tioners who prefer the ensemble method with high precision can results for different methods in real application scenarios are
appropriately increase the number of TF methods. Practitioners obtained. The results showed that SwissTargetPrediction with a
who prefer the ensemble method with high recall can appro- high Fk tended to produce the most reliable predictions while
priately increase top-k, although we suggest that combining the enriching more targets. The results also showed that SEA prefers
top 5 targets for each query molecule. In brief, our conclusions recall to precision and can identify real targets for more com-
show that the ensemble TF method is extrapolated, f lexible and pounds. Therefore, SwissTargetPrediction and SEA could be con-
practical to achieve reliable TF, which is useful for reducing the sidered as primary selection methods or target identification in
difficulty for practitioners in selecting among many targets and to future studies. In addition, we suggest k = 5 as the optimal number
improve the hit rate of experimental validation in practical model of experimental candidate targets for the compounds predicted
application scenarios. by the TF method. Finally, this study first proposed a simple,
convenient and general ensemble TF method that implements the
idea of identifying targets by consensus voting and significantly
Conclusion improves target enrichment ability. The ensemble TF method
In this study, we aimed to provide a comprehensive reference can provide valuable information to practitioners and help to
strategy for the evaluation of nine publicly available ligand-based prioritize lead compounds. To the best of our knowledge, this
TF methods. By analyzing the performance of the methods at study is the first extensive performance evaluation of the publicly
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 13

Downloaded from [Link] by guest on 16 February 2024

Figure 10. Fk curves for the ensemble TF method in the top-k (k = 1, 3, 5, 7, 10). (A) Real target Fk curves for the ensemble TF method. (B) Ligand–target
pair Fk curves for the ensemble TF method.

available ligand-based TF method and the first application of TF methods, and the proposed ensemble method helps to solve
consensus voting to TF method. The results reported in this complex scientific puzzles regarding the calculation of bioactive
paper provide initial clues to the applicability of nine ligand-based molecular targets.
14 | Ji et al.

9. Chaudhari R, Tan Z, Huang B, et al. Computational polyphar-

Key Points macology: a new paradigm for drug discovery. Expert Opin Drug
• Nine popular ligand-based target fishing methods are Discovery 2017;12:279–91.
evaluated to determine the best method that would 10. Lavecchia A, Cerchia C. In silico methods to address polyphar-
guide practitioners in making a choice. macology: current status, applications and future perspectives.
• SwissTargetPrediction and SEA can be considered as Drug Discov Today 2016;21:288–98.
primary selection methods in future studies. 11. Liu X, Zhu F, Max H, et al. Predicting targeted polypharmacology
• The top five predicted targets of a molecule are the for drug repositioning and multi-target drug discovery. Curr Med
optimal number of candidate targets to assist target Chem 2013;20:1646–61.
identification experiments in future. 12. Chaudhari R, Fong LW, Tan Z, et al. An up-to-date overview
• A novel ensemble TF method based on consensus voting of computational polypharmacology in modern drug discovery.
is proposed to improve the prediction performance. Expert Opin Drug Discovery 2020;15:1025–44.
• The ensemble TF method outperforms individual TF 13. Agamah FE, Mazandu GK, Hassan R, et al. Computational/in
methods and can identify the real target of the molecule silico methods in drug target and lead prediction. Brief Bioinform

Downloaded from [Link] by guest on 16 February 2024

more effectively. 2020;21:1663–75.
14. Proschak E, Stark H, Merk D. Polypharmacology by design: a
medicinal chemist’s perspective on multitargeting compounds.
J Med Chem 2018;62:420–44.
Supplementary data 15. Lounkine E, Keiser MJ, Whitebread S, et al. Large-scale predic-
Supplementary data are available online at [Link] tion and testing of drug activity on side-effect targets. Nature
org/. 2012;486:361–7.
16. Yera ER, Cleves AE, Jain AN. Chemical structural novelty: on-
targets and off-targets. J Med Chem 2011;54:6771–85.
Acknowledgements 17. Jarada TN, Rokne JG, Alhajj R. A review of computational drug
repositioning: strategies, approaches, opportunities, challenges,
We acknowledge Haikun Xu, and the High-Performance Comput-
and directions. J Chem 2020;12:1–23.
ing Center of Central South University for support. The study was
18. Jenkins JL, Bender A, Davies JW. In silico target fishing: predicting
approved by the university’s review board.
biological targets from chemical structure. Drug Discov Today
Technol 2006;3:413–21.
19. Shaker B, Ahmad S, Lee J, et al. In silico methods and tools for
Funding
drug discovery. Comput Biol Med 2021;137:104851.
National Key Research and Development Program of China 20. Li G-B, Yu Z-J, Liu S, et al. IFPTarget: a customized vir-
(2021YFF1201400); National Natural Science Foundation of tual target identification method based on protein–ligand
China (22173118, 22220102001); Hunan Provincial Science Fund interaction fingerprinting analyses. J Chem Inf Model 2017;57:
for Distinguished Young Scholars (2021JJ10068); Science and 1640–51.
Technology Innovation Program of Hunan Province (2021RC4011); 21. Li H, Gao Z, Kang L, et al. TarFisDock: a web server for iden-
Natural Science Foundation of Hunan Province (2022JJ80104); tifying drug targets with docking approach. Nucleic Acids Res
Changsha Science and Technology Bureau project (kq2001034); 2006;34:W219–24.
The 2020 Guangdong Provincial Science and Technology Innova- 22. Yu J-L, Wu S, Zhou C, et al. MeDBA: the Metalloenzyme data bank
tion Strategy Special Fund (2020B1212030006, Guangdong-Hong and analysis platform. Nucleic Acids Res 2023;51:D593–602.
Kong-Macau Joint Lab). 23. Shen Z, Yan Y-H, Yang S, et al. ProfKin: a comprehensive web
server for structure-based kinase profiling. Eur J Med Chem
2021;225:113772.
References 24. Pinzi L, Rastelli G. Molecular docking: shifting paradigms in drug
1. Lee J, Bogyo M. Target deconvolution techniques in modern discovery. Int J Mol Sci 2019;20:4331.
phenotypic profiling. Curr Opin Chem Biol 2013;17:118–26. 25. Cheng AC, Coleman RG, Smyth KT, et al. Structure-based max-
2. Peón A, Naulaerts S, Ballester PJ. Predicting the reliability of imal affinity model predicts small-molecule druggability. Nat
drug-target interaction predictions with maximum coverage of Biotechnol 2007;25:71–5.
target space. Sci Rep 2017;7:1–11. 26. Koutsoukas A, Simms B, Kirchmair J, et al. From in sil-
3. Drewes G, Knapp S. Chemoproteomics and chemical probes for ico target prediction to multi-target drug design: current
target discovery. Trends Biotechnol 2018;36:1275–86. databases, methods and applications. J Proteomics 2011;74:
4. Bar-Peled L, Kemper EK, Suciu RM, et al. Chemical proteomics 2554–74.
identifies druggable vulnerabilities in a genetically defined can- 27. Cereto-Massagué A, Ojeda MJ, Valls C, et al. Tools for in silico
cer. Cell 2017;171:696, e623–709. target fishing. Methods 2015;71:98–103.
5. Kola I, Landis J. Can the pharmaceutical industry reduce attri- 28. Bender A, Glen RC. Molecular similarity: a key technique in
tion rates? Nat Rev Drug Discov 2004;3:711–6. molecular informatics. Org Biomol Chem 2004;2:3204–18.
6. Morphy R, Kay C, Rankovic Z. From magic bullets to designed 29. Liu X, Xu Y, Li S, et al. In Silicotarget fishing: addressing a “big
multiple ligands. Drug Discov Today 2004;9:641–51. data” problem by ligand-based similarity rankings with data
7. Hopkins AL, Mason JS, Overington JP. Can we rationally design fusion. J Chem 2014;6:1–14.
promiscuous drugs? Curr Opin Struct Biol 2006;16:127–36. 30. Willett P, Barnard JM, Downs GM. Chemical similarity searching.
8. Badrinarayan P, Narahari SG. Virtual high throughput screening J Chem Inf Comput Sci 1998;38:983–96.
in new lead identification. Comb Chem High Throughput Screen 31. Johnson MA, Maggiora GM. Concepts and applications of molecular
2011;14:840–60. similarity. New York: Wiley, 1990.
 Comprehensive assessment of nine target prediction web services: which should we choose for target fishing? | 15

32. Svetnik V, Liaw A, Tong C, et al. Random forest: a classification 48. Mathai N, Chen Y, Kirchmair J. Validation strategies for target
and regression tool for compound classification and QSAR mod- prediction methods. Brief Bioinform 2020;21:791–802.
eling. J Chem Inf Comput Sci 2003;43:1947–58. 49. Peón A, Dang CC, Ballester PJ. How reliable are ligand-centric
33. Shen M, Xiao Y, Golbraikh A, et al. Development and validation of methods for target fishing? Front Chem 2016;4:15.
k-nearest-neighbor QSPR models of metabolic stability of drug 50. Liu T, Lin Y, Wen X, et al. BindingDB: a web-accessible database
candidates. J Med Chem 2003;46:3013–20. of experimentally determined protein–ligand binding affinities.
34. Burbidge R, Trotter M, Buxton B, et al. Drug design by machine Nucleic Acids Res 2007;35:D198–201.
learning: support vector machines for pharmaceutical data 51. Bruder M, Polo G, Trivella DB. Natural allosteric modulators and
analysis. Comput Chem 2001;26:5–14. their biological targets: molecular signatures and mechanisms.
35. Cristianini N, Shawe-Taylor J. An introduction to support vector Nat Prod Rep 2020;37:488–514.
machines and other kernel-based learning methods. Cambridge: Cam- 52. Daina A, Michielin O, Zoete V. SwissTargetPrediction: updated
bridge university press, 2000. data and new features for efficient prediction of protein targets
36. Nidhi Glick M, Davies JW, et al. Prediction of biological targets for of small molecules. Nucleic Acids Res 2019;47:W357–64.
compounds using multiple-category Bayesian models trained 53. Bagherian M, Sabeti E, Wang K, et al. Machine learning

Downloaded from [Link] by guest on 16 February 2024

on chemogenomics databases. J Chem Inf Model 2006;46:1124–33. approaches and databases for prediction of drug–target inter-
37. Özdamar Ö, Eilers RE, Miskiel E, et al. Classification of audio- action: a survey paper. Brief Bioinform 2021;22:247–69.
grams by sequential testing using a dynamic Bayesian proce- 54. Cherkasov A, Muratov EN, Fourches D, et al. QSAR modeling:
dure. J Acoust Soc Am 1990;88:2171–9. where have you been? Where are you going to? J Med Chem 2014;
38. Schmidhuber J. Deep learning in neural networks: an overview. 57:4977–5010.
Neural Netw 2015;61:85–117. 55. Varnek A, Gaudin C, Marcou G, et al. Inductive transfer of
39. Capuzzi SJ, Politi R, Isayev O, et al. QSAR modeling of Tox21 chal- knowledge: application of multi-task learning and feature net
lenge stress response and nuclear receptor signaling toxicity approaches to model tissue-air partition coefficients. J Chem Inf
assays. Front Environ Sci 2016;4:3. Model 2009;49:133–44.
40. Mitchell JB. Machine learning methods in chemoinformatics, 56. Yu H, Chen J, Xu X, et al. A systematic prediction of multiple drug-
Wiley interdisciplinary reviews: computational molecular. Sci- target interactions from chemical, genomic, and pharmacologi-
ence 2014;4:468–81. cal data. PLoS One 2012;7:e37608.
41. Byrne R, Schneider G. In silico target prediction for small 57. Wen M, Zhang Z, Niu S, et al. Deep-learning-based drug–target
molecules. Sys Chem Biol Springer, 2019. pp. 273–309. interaction prediction. J Proteome Res 2017;16:1401–9.
42. Gfeller D, Grosdidier A, Wirth M, et al. SwissTargetPrediction: a 58. Chen R, Liu X, Jin S, et al. Machine learning for drug-target
web server for target prediction of bioactive small molecules. interaction prediction. Molecules 2018;23:2208.
Nucleic Acids Res 2014;42:W32–8. 59. Vamathevan J, Clark D, Czodrowski P, et al. Applications of
43. Keiser MJ, Roth BL, Armbruster BN, et al. Relating protein phar- machine learning in drug discovery and development. Nat Rev
macology by ligand chemistry. Nat Biotechnol 2007;25:197–206. Drug Discov 2019;18:463–77.
44. Awale M, Reymond J-L. The polypharmacology browser: a web- 60. Gfeller D, Michielin O, Zoete V. Shaping the interaction
based multi-fingerprint target prediction tool using ChEMBL landscape of bioactive molecules. Bioinformatics 2013;29:
bioactivity data. J Chem 2017;9:1–10. 3073–9.
45. Awale M, Reymond J-L. Polypharmacology browser PPB2: target 61. Wei H, Guan Y-D, Zhang L-X, et al. A combinatorial target screen-
prediction combining nearest neighbors with machine learning. ing strategy for deorphaning macromolecular targets of natural
J Chem Inf Model 2018;59:10–7. product. Eur J Med Chem 2020;204:112644.
46. Gaulton A, Hersey A, Nowotka M, et al. The ChEMBL database in 62. Keiser MJ, Setola V, Irwin JJ, et al. Predicting new molecular
2017. Nucleic Acids Res 2017;45:D945–54. targets for known drugs. Nature 2009;462:175–81.
47. Yao Z-J, Dong J, Che Y-J, et al. TargetNet: a web service for 63. Baumann D, Baumann K. Reliable estimation of prediction
predicting potential drug–target interaction profiling via multi- errors for QSAR models under model uncertainty using double
target SAR models. J Comput Aided Mol Des 2016;30:413–24. cross-validation. J Chem 2014;6:1–19.